configuration examples in stream markdowns
Project: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/commit/52b5f1ca Tree: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/tree/52b5f1ca Diff: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/diff/52b5f1ca Branch: refs/heads/master Commit: 52b5f1ca010c5f3366e0ab42417883076c9b2307 Parents: 3c1fbde Author: Steve Blackmon @steveblackmon <[email protected]> Authored: Tue Oct 11 17:54:57 2016 -0500 Committer: Steve Blackmon @steveblackmon <[email protected]> Committed: Tue Oct 11 17:54:57 2016 -0500 ---------------------------------------------------------------------- .../markdown/FlinkTwitterFollowingPipeline.md | 61 +++++++++++--------- .../site/markdown/FlinkTwitterPostsPipeline.md | 55 ++++++++++-------- .../markdown/FlinkTwitterSpritzerPipeline.md | 55 ++++++++++-------- .../FlinkTwitterUserInformationPipeline.md | 55 ++++++++++-------- .../src/site/markdown/index.md | 38 ++++-------- .../src/site/markdown/ElasticsearchHdfs.md | 23 +++++++- .../src/site/markdown/HdfsElasticsearch.md | 27 +++++++-- .../src/site/markdown/index.md | 4 +- .../src/site/markdown/ElasticsearchReindex.md | 26 +++++++-- .../src/site/markdown/index.md | 2 +- .../src/site/markdown/MongoElasticsearchSync.md | 22 ++++++- .../src/site/markdown/TwitterFollowNeo4j.md | 18 +++++- .../src/site/markdown/index.md | 15 +---- .../markdown/TwitterHistoryElasticsearch.md | 25 ++++++-- .../src/site/markdown/index.md | 2 +- .../markdown/TwitterUserstreamElasticsearch.md | 13 ++++- .../src/site/markdown/index.md | 2 +- src/site/markdown/credentials/twitter.md | 2 +- 18 files changed, 278 insertions(+), 167 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterFollowingPipeline.md ---------------------------------------------------------------------- diff --git a/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterFollowingPipeline.md b/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterFollowingPipeline.md index 3ad23d3..f9f39e1 100644 --- a/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterFollowingPipeline.md +++ b/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterFollowingPipeline.md @@ -1,42 +1,51 @@ -FlinkTwitterFollowingPipeline -============================= +### FlinkTwitterFollowingPipeline -Description: ------------------ +#### Description: Collects twitter friends or followers with flink. -Specification: ------------------ +#### Configuration: + +[TwitterFollowingPipelineConfiguration.json](TwitterFollowingPipelineConfiguration.json "TwitterFollowingPipelineConfiguration.json" ) + + include "flink.conf" + include "twitter.oauth.conf" + source { + fields = ["ID"] + scheme = file + path = "target/test-classes" + readerPath = "asf.txt" + } + destination { + fields = ["DOC"] + scheme = file + path = "target/test-classes" + writerPath = "FlinkTwitterFollowingPipelineFriendsIT" + } + twitter { + endpoint = friends + ids_only = true + } + +#### Run (Local): -[FlinkTwitterFollowingPipeline.dot](FlinkTwitterFollowingPipeline.dot "FlinkTwitterFollowingPipeline.dot" ) - -Diagram: ------------------ - - - -Example Configuration: ----------------------- + java -cp dist/flink-twitter-collection-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.examples.flink.twitter.collection.FlinkTwitterFollowingPipeline -[FlinkTwitterFollowingPipelineFollowersIT.conf](FlinkTwitterFollowingPipelineFollowersIT.conf "FlinkTwitterFollowingPipelineFollowersIT.conf" ) +#### Run (Flink): -[FlinkTwitterFollowingPipelineFriendsIT.conf](FlinkTwitterFollowingPipelineFriendsIT.conf "FlinkTwitterFollowingPipelineFriendsIT.conf" ) + flink-run.sh dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterFollowingPipeline http://<location_of_config_file> -Run (Local): ------------- +#### Run (YARN): - java -cp dist/flink-twitter-collection-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.examples.flink.twitter.collection.FlinkTwitterFollowingPipeline + flink-run.sh yarn dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterFollowingPipeline http://<location_of_config_file> -Run (Flink): ------------- +#### Specification: - flink-run.sh dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterFollowingPipeline http://<location_of_config_file> +[FlinkTwitterFollowingPipeline.dot](FlinkTwitterFollowingPipeline.dot "FlinkTwitterFollowingPipeline.dot" ) -Run (YARN): ------------ +#### Diagram: - flink-run.sh yarn dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterFollowingPipeline http://<location_of_config_file> + [JavaDocs](apidocs/index.html "JavaDocs") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterPostsPipeline.md ---------------------------------------------------------------------- diff --git a/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterPostsPipeline.md b/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterPostsPipeline.md index fe6b544..0b4c8bd 100644 --- a/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterPostsPipeline.md +++ b/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterPostsPipeline.md @@ -1,40 +1,47 @@ -FlinkTwitterPostsPipeline -========================= +### FlinkTwitterPostsPipeline -Description: ------------------ +#### Description: Collects twitter posts with flink. -Specification: ------------------ +#### Configuration: + +[TwitterPostsPipelineConfiguration.json](TwitterPostsPipelineConfiguration.json "TwitterPostsPipelineConfiguration.json" ) + + include "flink.conf" + include "twitter.oauth.conf" + source { + fields = ["ID"] + scheme = file + path = "target/test-classes" + readerPath = "asf.txt" + } + destination { + fields = ["DOC"] + scheme = file + path = "target/test-classes" + writerPath = "FlinkTwitterPostsPipelineIT" + } + +#### Run (Local): -[FlinkTwitterPostsPipeline.dot](FlinkTwitterPostsPipeline.dot "FlinkTwitterPostsPipeline.dot" ) - -Diagram: ------------------ - - + java -cp dist/flink-twitter-collection-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.examples.flink.twitter.collection.FlinkTwitterPostsPipeline -Example Configuration: ----------------------- +#### Run (Flink): -[FlinkTwitterPostsPipelineIT.conf](FlinkTwitterPostsPipelineIT.conf "FlinkTwitterPostsPipelineIT.conf" ) + flink-run.sh dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterPostsPipeline http://<location_of_config_file> -Run (Local): ------------- +#### Run (YARN): - java -cp dist/flink-twitter-collection-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.examples.flink.twitter.collection.FlinkTwitterPostsPipeline + flink-run.sh yarn dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterPostsPipeline http://<location_of_config_file> -Run (Flink): ------------- +#### Specification: - flink-run.sh dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterPostsPipeline http://<location_of_config_file> +[FlinkTwitterPostsPipeline.dot](FlinkTwitterPostsPipeline.dot "FlinkTwitterPostsPipeline.dot" ) -Run (YARN): ------------ +#### Diagram: - flink-run.sh yarn dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterPostsPipeline http://<location_of_config_file> + [JavaDocs](apidocs/index.html "JavaDocs") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterSpritzerPipeline.md ---------------------------------------------------------------------- diff --git a/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterSpritzerPipeline.md b/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterSpritzerPipeline.md index 1e59039..0a82321 100644 --- a/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterSpritzerPipeline.md +++ b/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterSpritzerPipeline.md @@ -1,40 +1,47 @@ -FlinkTwitterSpritzerPipeline -============================ +### FlinkTwitterSpritzerPipeline -Description: ------------------ +#### Description: Collects twitter posts in real-time from the sample endpoint with flink. -Specification: ------------------ +#### Configuration: + +[TwitterSpritzerPipelineConfiguration.json](TwitterSpritzerPipelineConfiguration.json "TwitterSpritzerPipelineConfiguration.json" ) + + include "flink.conf" + include "twitter.oauth.conf" + destination { + fields = ["DOC"] + scheme = file + path = "target/test-classes" + writerPath = "FlinkTwitterSpritzerPipelineIT" + } + twitter { + endpoint = sample + track = [ + "data" + ] + } + +#### Run (Local): -[FlinkTwitterSpritzerPipeline.dot](FlinkTwitterSpritzerPipeline.dot "FlinkTwitterSpritzerPipeline.dot" ) - -Diagram: ------------------ - - + java -cp dist/flink-twitter-collection-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.examples.flink.twitter.collection.FlinkTwitterSpritzerPipeline -Example Configuration: ----------------------- +#### Run (Flink): -[FlinkTwitterSpritzerPipelineIT.conf](FlinkTwitterSpritzerPipelineIT.conf "FlinkTwitterSpritzerPipelineIT.conf" ) + flink-run.sh dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterSpritzerPipeline http://<location_of_config_file> -Run (Local): ------------- +#### Run (YARN): - java -cp dist/flink-twitter-collection-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.examples.flink.twitter.collection.FlinkTwitterSpritzerPipeline + flink-run.sh yarn dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterSpritzerPipeline http://<location_of_config_file> -Run (Flink): ------------- +#### Specification: - flink-run.sh dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterSpritzerPipeline http://<location_of_config_file> +[FlinkTwitterSpritzerPipeline.dot](FlinkTwitterSpritzerPipeline.dot "FlinkTwitterSpritzerPipeline.dot" ) -Run (YARN): ------------ +#### Diagram: - flink-run.sh yarn dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterSpritzerPipeline http://<location_of_config_file> + [JavaDocs](apidocs/index.html "JavaDocs") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterUserInformationPipeline.md ---------------------------------------------------------------------- diff --git a/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterUserInformationPipeline.md b/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterUserInformationPipeline.md index a465de9..ad90fab 100644 --- a/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterUserInformationPipeline.md +++ b/flink/flink-twitter-collection/src/site/markdown/FlinkTwitterUserInformationPipeline.md @@ -1,40 +1,47 @@ -FlinkTwitterUserInformationPipeline -=================================== +### FlinkTwitterUserInformationPipeline -Description: ------------------ +#### Description: Collects twitter user profiles with flink. -Specification: ------------------ +#### Configuration: + +[TwitterUserInformationPipelineConfiguration.json](TwitterUserInformationPipelineConfiguration.json "TwitterUserInformationPipelineConfiguration.json" ) + + include "flink.conf" + include "twitter.oauth.conf" + source { + fields = ["ID"] + scheme = file + path = "target/test-classes" + readerPath = "1000twitterids.txt" + } + destination { + fields = ["DOC"] + scheme = file + path = "target/test-classes" + writerPath = "FlinkTwitterUserInformationPipelineIT" + } + +#### Run (Local): -[FlinkTwitterUserInformationPipeline.dot](FlinkTwitterUserInformationPipeline.dot "FlinkTwitterUserInformationPipeline.dot" ) - -Diagram: ------------------ - - + java -cp dist/flink-twitter-collection-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.examples.flink.twitter.collection.FlinkTwitterUserInformationPipeline -Example Configuration: ----------------------- +#### Run (Flink): -[FlinkTwitterUserInformationPipelineIT.conf](FlinkTwitterUserInformationPipelineIT.conf "FlinkTwitterUserInformationPipelineIT.conf" ) + flink-run.sh dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterUserInformationPipeline http://<location_of_config_file> -Run (Local): ------------- +#### Run (YARN): - java -cp dist/flink-twitter-collection-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.examples.flink.twitter.collection.FlinkTwitterUserInformationPipeline + flink-run.sh yarn dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterUserInformationPipeline http://<location_of_config_file> -Run (Flink): ------------- +#### Specification: - flink-run.sh dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterUserInformationPipeline http://<location_of_config_file> +[FlinkTwitterUserInformationPipeline.dot](FlinkTwitterUserInformationPipeline.dot "FlinkTwitterUserInformationPipeline.dot" ) -Run (YARN): ------------ +#### Diagram: - flink-run.sh yarn dist/flink-twitter-collection-jar-with-dependencies.jar org.apache.streams.examples.flink.twitter.collection.FlinkTwitterUserInformationPipeline http://<location_of_config_file> + [JavaDocs](apidocs/index.html "JavaDocs") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/flink/flink-twitter-collection/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/flink/flink-twitter-collection/src/site/markdown/index.md b/flink/flink-twitter-collection/src/site/markdown/index.md index 24783be..616bdd7 100644 --- a/flink/flink-twitter-collection/src/site/markdown/index.md +++ b/flink/flink-twitter-collection/src/site/markdown/index.md @@ -1,20 +1,13 @@ -Apache Streams (incubating) -Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 --------------------------------------------------------------------------------- +### flink-twitter-collection -flink-twitter-collection -======================== - -Requirements: -------------- +#### Requirements: - Authorized Twitter API credentials -Description: ------------- +#### Description: + Collects large batches of documents from api.twitter.com from a seed set of ids. -Streams: --------- +#### Streams: <a href="FlinkTwitterFollowingPipeline.html" target="_self">FlinkTwitterFollowingPipeline</a> @@ -24,24 +17,15 @@ Streams: <a href="FlinkTwitterUserInformationPipeline.html" target="_self">FlinkTwitterUserInformationPipeline</a> -Test: ------ +#### Build: + + mvn clean package -Create a local file `application.conf` with valid twitter credentials +#### Test: - twitter { - oauth { - consumerKey = "" - consumerSecret = "" - accessToken = "" - accessTokenSecret = "" - } - } - -Build: ---------- +Build with integration testing enabled, using your credentials - mvn clean test verify -DskipITs=false -DargLine="-Dconfig.file=`pwd`/application.conf" + mvn clean test verify -DskipITs=false -DargLine="-Dconfig.file=./twitter.conf" [JavaDocs](apidocs/index.html "JavaDocs") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md b/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md index ad8ad4a..3294e09 100644 --- a/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md +++ b/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md @@ -6,8 +6,25 @@ Copies documents from elasticsearch to hdfs. #### Configuration: -[ElasticsearchHdfsIT.conf](ElasticsearchHdfsIT.conf "ElasticsearchHdfsIT.conf" ) - +[ElasticsearchHdfs.json](ElasticsearchHdfs.json "ElasticsearchHdfs.json" ) + +##### application.conf + + include "elasticsearch.properties" + include "elasticsearch.conf" + source = ${elasticsearch} + source { + indexes += "elasticsearch_persist_writer_it" + types += "activity" + } + destination { + fields = ["ID","DOC"] + scheme = file + user = hadoop + path = "target/test-classes" + writerPath = "elasticsearch_hdfs_it" + } + #### Run (SBT): sbtx -210 -sbt-create @@ -15,7 +32,7 @@ Copies documents from elasticsearch to hdfs. set libraryDependencies += "org.apache.streams" % "elasticsearch-hdfs" % "0.4-incubating-SNAPSHOT" set fork := true set javaOptions +="-Dconfig.file=application.conf" - run elasticsearch-hdfs org.apache.streams.example.ElasticsearchHdfs + run org.apache.streams.example.ElasticsearchHdfs #### Run (Docker): http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md b/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md index 136b110..ecd8445 100644 --- a/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md +++ b/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md @@ -6,16 +6,35 @@ Copies documents from hdfs to elasticsearch. #### Configuration: -[HdfsElasticsearchIT.conf](HdfsElasticsearchIT.conf "HdfsElasticsearchIT.conf" ) - +[HdfsElasticsearch.json](HdfsElasticsearch.json "HdfsElasticsearch.json" ) + +##### application.conf + + include "elasticsearch.properties" + include "elasticsearch.conf" + source { + fields = ["ID","DOC"] + scheme = file + user = hadoop + path = "target/test-classes" + readerPath = "elasticsearch_hdfs_it" + } + destination = ${elasticsearch} + destination { + index = "hdfs_elasticsearch_it" + type = "activity" + refresh = true + forceUseConfig = true + } + #### Run (SBT): sbtx -210 -sbt-create set resolvers += "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository" set libraryDependencies += "org.apache.streams" % "elasticsearch-hdfs" % "0.4-incubating-SNAPSHOT" set fork := true - set javaOptions +="-Dconfig.file=HdfsElasticsearchIT.conf" - run elasticsearch-hdfs org.apache.streams.example.ElasticsearchHdfs + set javaOptions +="-Dconfig.file=application.conf" + run org.apache.streams.example.ElasticsearchHdfs #### Run (Docker): http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/elasticsearch-hdfs/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-hdfs/src/site/markdown/index.md b/local/elasticsearch-hdfs/src/site/markdown/index.md index d789a2f..4be1820 100644 --- a/local/elasticsearch-hdfs/src/site/markdown/index.md +++ b/local/elasticsearch-hdfs/src/site/markdown/index.md @@ -19,9 +19,9 @@ Start up elasticsearch with docker: mvn -PdockerITs docker:start -Build with integration testing enabled, using your credentials +Build with integration testing enabled: - mvn clean test verify -DskipITs=false -DargLine="-Dconfig.file=`pwd`/application.conf" + mvn clean test verify -DskipITs=false Shutdown elasticsearch when finished: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/elasticsearch-reindex/src/site/markdown/ElasticsearchReindex.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-reindex/src/site/markdown/ElasticsearchReindex.md b/local/elasticsearch-reindex/src/site/markdown/ElasticsearchReindex.md index 2a2a6b2..9bd37d4 100644 --- a/local/elasticsearch-reindex/src/site/markdown/ElasticsearchReindex.md +++ b/local/elasticsearch-reindex/src/site/markdown/ElasticsearchReindex.md @@ -6,20 +6,36 @@ Copies documents into a different index #### Configuration: -[ElasticsearchReindexIT.conf](ElasticsearchReindexIT.conf "ElasticsearchReindexIT.conf" ) - +[ElasticsearchReindex.json](ElasticsearchReindex.json "ElasticsearchReindex.json") + +##### application.conf + + include "elasticsearch.properties" + include "elasticsearch.conf" + source = ${elasticsearch} + source { + indexes += "elasticsearch_persist_writer_it" + types += "activity" + } + destination = ${elasticsearch} + destination { + index: "elasticsearch_reindex_it", + type: "activity", + forceUseConfig": true + } + #### Run (SBT): sbtx -210 -sbt-create set resolvers += "Local Maven Repository" at "file://"+Path.userHome.absolutePath+"/.m2/repository" set libraryDependencies += "org.apache.streams" % "elasticsearch-reindex" % "0.4-incubating-SNAPSHOT" set fork := true - set javaOptions +="-Dconfig.file=ElasticsearchReindexIT.conf" - run elasticsearch-hdfs org.apache.streams.example.ElasticsearchReindex + set javaOptions +="-Dconfig.file=application.conf" + run org.apache.streams.example.ElasticsearchReindex #### Run (Docker): - docker run elasticsearch-reindex java -cp elasticsearch-reindex-jar-with-dependencies.jar -Dconfig.file=`pwd`/HdfsElasticsearchIT.conf org.apache.streams.example.ElasticsearchReindex + docker run elasticsearch-reindex java -cp elasticsearch-reindex-jar-with-dependencies.jar -Dconfig.file=./application.conf org.apache.streams.example.ElasticsearchReindex #### Specification: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/elasticsearch-reindex/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-reindex/src/site/markdown/index.md b/local/elasticsearch-reindex/src/site/markdown/index.md index 87c3e04..66e4a92 100644 --- a/local/elasticsearch-reindex/src/site/markdown/index.md +++ b/local/elasticsearch-reindex/src/site/markdown/index.md @@ -19,7 +19,7 @@ Start up elasticsearch with docker: mvn -PdockerITs docker:start -Build with integration testing enabled, using your credentials +Build with integration testing enabled: mvn clean test verify -DskipITs=false http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/mongo-elasticsearch-sync/src/site/markdown/MongoElasticsearchSync.md ---------------------------------------------------------------------- diff --git a/local/mongo-elasticsearch-sync/src/site/markdown/MongoElasticsearchSync.md b/local/mongo-elasticsearch-sync/src/site/markdown/MongoElasticsearchSync.md index cdbdce1..d1f8f8d 100644 --- a/local/mongo-elasticsearch-sync/src/site/markdown/MongoElasticsearchSync.md +++ b/local/mongo-elasticsearch-sync/src/site/markdown/MongoElasticsearchSync.md @@ -6,7 +6,25 @@ Copies documents from mongodb to elasticsearch #### Configuration: -[MongoElasticsearchSyncIT.conf](MongoElasticsearchSyncIT.conf "MongoElasticsearchSyncIT.conf" ) +[MongoElasticsearchSync.json](MongoElasticsearchSync.json "MongoElasticsearchSync.json") for _ + +##### application.conf + + include "mongo.properties" + include "mongo.conf" + include "elasticsearch.properties" + include "elasticsearch.conf" + source = ${mongo} + source { + db: mongo_persist_it + collection: activity + } + destination = ${elasticsearch} + destination { + index: mongo_elasticsearch_sync_it + type: activity + forceUseConfig": true + } #### Run (SBT): @@ -15,7 +33,7 @@ Copies documents from mongodb to elasticsearch set libraryDependencies += "org.apache.streams" % "mongo-elasticsearch-sync" % "0.4-incubating-SNAPSHOT" set fork := true set javaOptions +="-Dconfig.file=application.conf" - run mongo-elasticsearch-sync org.apache.streams.example.MongoElasticsearchSync + run org.apache.streams.example.MongoElasticsearchSync #### Run (Docker): http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/twitter-follow-neo4j/src/site/markdown/TwitterFollowNeo4j.md ---------------------------------------------------------------------- diff --git a/local/twitter-follow-neo4j/src/site/markdown/TwitterFollowNeo4j.md b/local/twitter-follow-neo4j/src/site/markdown/TwitterFollowNeo4j.md index 936efb4..c241b60 100644 --- a/local/twitter-follow-neo4j/src/site/markdown/TwitterFollowNeo4j.md +++ b/local/twitter-follow-neo4j/src/site/markdown/TwitterFollowNeo4j.md @@ -6,7 +6,20 @@ Collects friend or follower connections for a set of twitter users to build a gr #### Configuration: -[TwitterFollowNeo4jIT.conf](TwitterFollowNeo4jIT.conf "TwitterFollowNeo4jIT.conf" ) +[TwitterFollowNeo4j.json](TwitterFollowNeo4j.json "TwitterFollowNeo4j.json") for _ + +##### application.conf + + include "neo4j.properties" + include "neo4j.conf" + include "twitter.oauth.conf" + twitter { + endpoint = "friends" + info = [ + 18055613 + ] + twitter.max_items = 1000 + } #### Run (SBT): @@ -15,7 +28,7 @@ Collects friend or follower connections for a set of twitter users to build a gr set libraryDependencies += "org.apache.streams" % "twitter-follow-neo4j" % "0.4-incubating-SNAPSHOT" set fork := true set javaOptions +="-Dconfig.file=application.conf" - run org.apache.streams.example.graph.TwitterFollowNeo4j + run org.apache.streams.example.TwitterFollowNeo4j #### Run (Docker): @@ -29,5 +42,4 @@ Collects friend or follower connections for a set of twitter users to build a gr  - ###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/twitter-follow-neo4j/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/twitter-follow-neo4j/src/site/markdown/index.md b/local/twitter-follow-neo4j/src/site/markdown/index.md index 3efdc5b..50a7456 100644 --- a/local/twitter-follow-neo4j/src/site/markdown/index.md +++ b/local/twitter-follow-neo4j/src/site/markdown/index.md @@ -10,28 +10,17 @@ #### Build: - mvn clean package verify + mvn clean package #### Test: -Create a local file `application.conf` with valid twitter credentials - - twitter { - oauth { - consumerKey = "" - consumerSecret = "" - accessToken = "" - accessTokenSecret = "" - } - } - Start up neo4j with docker: mvn -PdockerITs docker:start Build with integration testing enabled, using your credentials - mvn clean test verify -DskipITs=false -DargLine="-Dconfig.file=`pwd`/application.conf" + mvn clean test verify -DskipITs=false -DargLine="-Dconfig.file=./twitter.conf" Shutdown neo4j when finished: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/twitter-history-elasticsearch/src/site/markdown/TwitterHistoryElasticsearch.md ---------------------------------------------------------------------- diff --git a/local/twitter-history-elasticsearch/src/site/markdown/TwitterHistoryElasticsearch.md b/local/twitter-history-elasticsearch/src/site/markdown/TwitterHistoryElasticsearch.md index 9b696c2..09d7f5a 100644 --- a/local/twitter-history-elasticsearch/src/site/markdown/TwitterHistoryElasticsearch.md +++ b/local/twitter-history-elasticsearch/src/site/markdown/TwitterHistoryElasticsearch.md @@ -8,11 +8,26 @@ Converts them to activities, and writes them in activity format to Elasticsearch #### Configuration: -[TwitterHistoryElasticsearchIT.conf](TwitterHistoryElasticsearchIT.conf "TwitterHistoryElasticsearchIT.conf" ) - -In the Twitter section you should place all of your relevant authentication keys and whichever Twitter IDs you want to pull history for. - -Twitter IDs can be converted from screennames at http://www.gettwitterid.com +[TwitterHistoryElasticsearch.json](TwitterHistoryElasticsearch.json "TwitterHistoryElasticsearch.json") for _ + +##### application.conf + + include "elasticsearch.properties" + include "elasticsearch.conf" + include "twitter.oauth.conf" + twitter { + info = [ + 18055613 + ] + twitter.max_items = 1000 + } + elasticsearch { + index = twitter_history + type = activity + forceUseConfig = true + } + +[TwitterHistoryElasticsearchIT.conf](TwitterHistoryElasticsearchIT.conf "TwitterHistoryElasticsearchIT.conf") #### Run (SBT): http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/twitter-history-elasticsearch/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/twitter-history-elasticsearch/src/site/markdown/index.md b/local/twitter-history-elasticsearch/src/site/markdown/index.md index e737a12..28154cb 100644 --- a/local/twitter-history-elasticsearch/src/site/markdown/index.md +++ b/local/twitter-history-elasticsearch/src/site/markdown/index.md @@ -31,7 +31,7 @@ Start up elasticsearch with docker: Build with integration testing enabled, using your credentials - mvn clean test verify -DskipITs=false -DargLine="-Dconfig.file=`pwd`/application.conf" + mvn clean test verify -DskipITs=false -DargLine="-Dconfig.file=./twitter.conf" Shutdown elasticsearch when finished: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/twitter-userstream-elasticsearch/src/site/markdown/TwitterUserstreamElasticsearch.md ---------------------------------------------------------------------- diff --git a/local/twitter-userstream-elasticsearch/src/site/markdown/TwitterUserstreamElasticsearch.md b/local/twitter-userstream-elasticsearch/src/site/markdown/TwitterUserstreamElasticsearch.md index 36f4244..c812749 100644 --- a/local/twitter-userstream-elasticsearch/src/site/markdown/TwitterUserstreamElasticsearch.md +++ b/local/twitter-userstream-elasticsearch/src/site/markdown/TwitterUserstreamElasticsearch.md @@ -6,7 +6,18 @@ This example connects to an active twitter account and stores the userstream as #### Configuration: -[TwitterUserstreamElasticsearchIT.conf](TwitterUserstreamElasticsearchIT.conf "TwitterUserstreamElasticsearchIT.conf" ) +[TwitterUserstreamElasticsearch.json](TwitterUserstreamElasticsearch.json "TwitterUserstreamElasticsearch.json") for _ + +##### application.conf + + include "elasticsearch.properties" + include "elasticsearch.conf" + include "twitter.oauth.conf" + elasticsearch { + index = twitter_userstream + type = activity + forceUseConfig = true + } #### Run (SBT): http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/local/twitter-userstream-elasticsearch/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/twitter-userstream-elasticsearch/src/site/markdown/index.md b/local/twitter-userstream-elasticsearch/src/site/markdown/index.md index 833efde..10575d3 100644 --- a/local/twitter-userstream-elasticsearch/src/site/markdown/index.md +++ b/local/twitter-userstream-elasticsearch/src/site/markdown/index.md @@ -31,7 +31,7 @@ Start up elasticsearch with docker: Build with integration testing enabled, using your credentials - mvn clean test verify -DskipITs=false -DargLine="-Dconfig.file=`pwd`/application.conf" + mvn clean test verify -DskipITs=false -DargLine="-Dconfig.file=./twitter.conf" Shutdown elasticsearch when finished: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/52b5f1ca/src/site/markdown/credentials/twitter.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/credentials/twitter.md b/src/site/markdown/credentials/twitter.md index 098dabd..b716115 100644 --- a/src/site/markdown/credentials/twitter.md +++ b/src/site/markdown/credentials/twitter.md @@ -1,6 +1,6 @@ ## Twitter Credentials -Create a local file `twitter.conf` with valid twitter credentials +Create a local file `twitter.oauth.conf` with valid twitter credentials twitter { oauth {
