Repository: incubator-streams Updated Branches: refs/heads/0.1-rc1 [created] ce0b7f7e7
first pass, sensible README in all modules Project: http://git-wip-us.apache.org/repos/asf/incubator-streams/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-streams/commit/c2a482b2 Tree: http://git-wip-us.apache.org/repos/asf/incubator-streams/tree/c2a482b2 Diff: http://git-wip-us.apache.org/repos/asf/incubator-streams/diff/c2a482b2 Branch: refs/heads/0.1-rc1 Commit: c2a482b254708e310786aa8d6d776b216787c428 Parents: 7ba66e0 Author: sblackmon <sblack...@w2odigital.com> Authored: Fri Dec 5 19:45:40 2014 -0600 Committer: sblackmon <sblack...@w2odigital.com> Committed: Fri Dec 5 19:45:40 2014 -0600 ---------------------------------------------------------------------- README.md | 52 ++++++++++++++++++++ README.txt | 52 -------------------- streams-components/README.md | 10 ++++ streams-config/README.md | 4 ++ streams-contrib/README.md | 7 +++ .../streams-persist-s3/README.md | 18 +++++++ .../streams-persist-console/README.md | 16 ++++++ streams-contrib/streams-persist-hbase/README.md | 20 ++++++++ streams-contrib/streams-persist-hdfs/README.md | 29 +++++++++++ streams-contrib/streams-persist-kafka/README.md | 14 ++++++ streams-contrib/streams-persist-mongo/README.md | 13 +++-- .../streams-processor-jackson/README.md | 5 ++ .../streams-processor-json/README.md | 5 ++ .../streams-processor-peoplepattern/README.md | 13 +++++ .../streams-processor-regex/README.md | 5 ++ .../streams-processor-urls/README.md | 5 ++ .../streams-provider-datasift/README.md | 18 +++++++ .../streams-provider-facebook/README.md | 17 +++++++ .../google-gmail/README.md | 14 ++++++ .../google-gplus/README.md | 22 +++++++++ .../streams-provider-instagram/README.md | 1 + .../streams-provider-moreover/README.md | 17 +++++++ .../streams-provider-rss/README.markdown | 1 + .../streams-provider-sysomos/README.md | 19 +++++++ streams-core/README.md | 4 ++ streams-monitoring/README.md | 4 ++ streams-pojo-extensions/README.md | 4 ++ streams-pojo/README.md | 4 ++ streams-runtimes/README.md | 7 +++ streams-util/README.md | 4 ++ 30 files changed, 345 insertions(+), 59 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md new file mode 100644 index 0000000..43aef2a --- /dev/null +++ b/README.md @@ -0,0 +1,52 @@ +Apache Streams (incubating) +Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 +-------------------------------------------------------------------------------- + +About +===== +Apache Streams is a lightweight (yet scalable) server for ActivityStreams. The role of Apache Streams is to provide a central point of aggregation, filtering and querying for Activities that have been submitted by disparate systems. Apache Streams also intends to include a mechanism for intelligent filtering and recommendation to reduce the noise to end users. + + + +Release Notes +============= + - Initial project structure and master POM release + + +Getting Started +=============== +Please visit the project website for the latest information: + http://streams.incubator.apache.org/ + +Along with the developer mailing list archive: + http://mail-archives.apache.org/mod_mbox/streams-dev/ + + +System Requirements +=================== +You need a platform that supports Java SE 6 or later. + +Building and running +==================== +To build from source code: + + - Requirements: + Sources compilation require Java SE 6 or higher. + The project is built with Apache Maven 3+ (suggested is 3.0.3). + You need to download and install Maven 3 from: http://maven.apache.org/ + + - The Streams project itself (this one) depends on the separate Streams Master project + which defines general and global settings for the whole of the Streams project, + independent of a specific release. + As its streams-master-pom is already published to the Apache Snapshots repository, + there is no need to check it out manually and build it locally yourself, + unless changes are needed on general and global level. + + If so needed, the Streams Master project can be checked out from: + http://svn.apache.org/repos/asf/incubator/streams/streams-master-pom/trunk streams-master-pom + + After check out, cd into streams-master-pom and invoke maven to install it using: + $mvn install + + - To build the Streams project invoke maven in the root directory: + $mvn install http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/README.txt ---------------------------------------------------------------------- diff --git a/README.txt b/README.txt deleted file mode 100644 index 6d23022..0000000 --- a/README.txt +++ /dev/null @@ -1,52 +0,0 @@ -Apache Streams (incubating) - README.txt -Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 --------------------------------------------------------------------------------- - -About -===== -Apache Streams is a lightweight (yet scalable) server for ActivityStreams. The role of Apache Streams is to provide a central point of aggregation, filtering and querying for Activities that have been submitted by disparate systems. Apache Streams also intends to include a mechanism for intelligent filtering and recommendation to reduce the noise to end users. - - - -Release Notes -============= - - Initial project structure and master POM release - - -Getting Started -=============== -Please visit the project website for the latest information: - http://streams.incubator.apache.org/ - -Along with the developer mailing list archive: - http://mail-archives.apache.org/mod_mbox/streams-dev/ - - -System Requirements -=================== -You need a platform that supports Java SE 6 or later. - -Building and running -==================== -To build from source code: - - - Requirements: - Sources compilation require Java SE 6 or higher. - The project is built with Apache Maven 3+ (suggested is 3.0.3). - You need to download and install Maven 3 from: http://maven.apache.org/ - - - The Streams project itself (this one) depends on the separate Streams Master project - which defines general and global settings for the whole of the Streams project, - independent of a specific release. - As its streams-master-pom is already published to the Apache Snapshots repository, - there is no need to check it out manually and build it locally yourself, - unless changes are needed on general and global level. - - If so needed, the Streams Master project can be checked out from: - http://svn.apache.org/repos/asf/incubator/streams/streams-master-pom/trunk streams-master-pom - - After check out, cd into streams-master-pom and invoke maven to install it using: - $mvn install - - - To build the Streams project invoke maven in the root directory: - $mvn install http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-components/README.md ---------------------------------------------------------------------- diff --git a/streams-components/README.md b/streams-components/README.md new file mode 100644 index 0000000..eb7dd43 --- /dev/null +++ b/streams-components/README.md @@ -0,0 +1,10 @@ +streams-components +============== + +These modules contain generic providers, processors, and writers which +can fulfill a variety of purposes based on the configuration they receive. + +They have been designed to be extended in cases where a specific integration +needs to do something a bit different. + +Please help improve these packages, or create new ones! \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-config/README.md ---------------------------------------------------------------------- diff --git a/streams-config/README.md b/streams-config/README.md new file mode 100644 index 0000000..7456311 --- /dev/null +++ b/streams-config/README.md @@ -0,0 +1,4 @@ +streams-config +============== + +streams-config provides a centralized configuration system for streams. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/README.md b/streams-contrib/README.md new file mode 100644 index 0000000..6d5427b --- /dev/null +++ b/streams-contrib/README.md @@ -0,0 +1,7 @@ +streams-contrib +============== + +These modules contain data schemas, API synchronizations, and ActivityStreams +conversions unique to specific sites and software packages. + +Please help improve these packages, or create new ones! \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-amazon-aws/streams-persist-s3/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-amazon-aws/streams-persist-s3/README.md b/streams-contrib/streams-amazon-aws/streams-persist-s3/README.md new file mode 100644 index 0000000..5e5aa2b --- /dev/null +++ b/streams-contrib/streams-amazon-aws/streams-persist-s3/README.md @@ -0,0 +1,18 @@ +streams-persist-s3 +============== + +Write documents to an S3 bucket. + +Example writer configuration: + + "s3": { + "key": "", + "secretKey": "", + "bucket": "", + "protocol": "HTTP", + "writerPath": "path/", + "writerFilePrefix": "prefix", + "chunk": false, + "region": "us-east-1" + } + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-persist-console/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-persist-console/README.md b/streams-contrib/streams-persist-console/README.md new file mode 100644 index 0000000..237c025 --- /dev/null +++ b/streams-contrib/streams-persist-console/README.md @@ -0,0 +1,16 @@ +streams-persist-console +===================== + +Read and write documents from console + +This can also be used to read and write files. + +For example, to read a stream from a file of json documents: + + cat inputfile.txt | java -jar stream.jar + +For example, to write a stream to a file of json documents: + + java -jar stream.jar > outputfile.txt + +NOTE: Be sure to disable the console logger in the latter case. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-persist-hbase/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-persist-hbase/README.md b/streams-contrib/streams-persist-hbase/README.md new file mode 100644 index 0000000..5cca090 --- /dev/null +++ b/streams-contrib/streams-persist-hbase/README.md @@ -0,0 +1,20 @@ +streams-persist-hbase +===================== + +Read and write to HBase + +Example reader or writer configuration: + + hbase { + rootdir = "hdfs://cloudera-vm-local:8020/hbase" + zookeeper { + quorum = "cloudera-vm-local" + property { + clientPort = 2181 + } + } + table = "test_table" + family = "test_family" + qualifier = "test_column" + } + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-persist-hdfs/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-persist-hdfs/README.md b/streams-contrib/streams-persist-hdfs/README.md new file mode 100644 index 0000000..1b038b3 --- /dev/null +++ b/streams-contrib/streams-persist-hdfs/README.md @@ -0,0 +1,29 @@ +streams-persist-hdfs +===================== + +Read and write to HDFS + +Example reader configuration: + + hdfs { + host = "localhost" + port = "50070" + path = "/user/cloudera" + user = "cloudera" + password = "cloudera" + } + +Example writer configuration: + + hdfs { + host = "localhost" + port = "50070" + path = "/user/cloudera" + user = "cloudera" + password = "cloudera" + writerPath = "/history/twitter/example" + writerFilePrefix = "streams-" + } + + + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-persist-kafka/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-persist-kafka/README.md b/streams-contrib/streams-persist-kafka/README.md new file mode 100644 index 0000000..e110492 --- /dev/null +++ b/streams-contrib/streams-persist-kafka/README.md @@ -0,0 +1,14 @@ +streams-persist-kafka +===================== + +Read and write to Kafka + +Example reader / writer configuration: + + kafka { + brokerlist = "localhost:9092" + zkconnect = "localhost:2181" + topic = "topic" + groupId = "group" + } + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-persist-mongo/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-persist-mongo/README.md b/streams-contrib/streams-persist-mongo/README.md index 57d18c7..1dfee2f 100644 --- a/streams-contrib/streams-persist-mongo/README.md +++ b/streams-contrib/streams-persist-mongo/README.md @@ -5,12 +5,11 @@ Read and write to MongoDB Example configuration: - { - "mongo": { - "host": "localhost", - "port": 27017, - "db": "local", - "collection": "startup_log" - } + "mongo": { + "host": "localhost", + "port": 27017, + "db": "local", + "collection": "startup_log" } + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-processor-jackson/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-processor-jackson/README.md b/streams-contrib/streams-processor-jackson/README.md new file mode 100644 index 0000000..b0dc3c7 --- /dev/null +++ b/streams-contrib/streams-processor-jackson/README.md @@ -0,0 +1,5 @@ +streams-processor-jackson +===================== + +Jackson Utility Processors + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-processor-json/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-processor-json/README.md b/streams-contrib/streams-processor-json/README.md new file mode 100644 index 0000000..3c88021 --- /dev/null +++ b/streams-contrib/streams-processor-json/README.md @@ -0,0 +1,5 @@ +streams-processor-json +===================== + +JsonPath Utility Processors + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-processor-peoplepattern/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-processor-peoplepattern/README.md b/streams-contrib/streams-processor-peoplepattern/README.md new file mode 100644 index 0000000..aa8dae0 --- /dev/null +++ b/streams-contrib/streams-processor-peoplepattern/README.md @@ -0,0 +1,13 @@ +streams-processor-peoplepattern +===================== + +PeoplePattern API Processors + +Example configuration: + + "peoplepattern": { + "username": "", + "password": "" + } + + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-processor-regex/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-processor-regex/README.md b/streams-contrib/streams-processor-regex/README.md new file mode 100644 index 0000000..8e45714 --- /dev/null +++ b/streams-contrib/streams-processor-regex/README.md @@ -0,0 +1,5 @@ +streams-processor-regex +===================== + +Regex Utility Processors + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-processor-urls/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-processor-urls/README.md b/streams-contrib/streams-processor-urls/README.md new file mode 100644 index 0000000..ea858b8 --- /dev/null +++ b/streams-contrib/streams-processor-urls/README.md @@ -0,0 +1,5 @@ +streams-processor-urls +===================== + +URL Utility Processors + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-provider-datasift/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-provider-datasift/README.md b/streams-contrib/streams-provider-datasift/README.md new file mode 100644 index 0000000..a29fb16 --- /dev/null +++ b/streams-contrib/streams-provider-datasift/README.md @@ -0,0 +1,18 @@ +streams-provider-datasift +===================== + +Datasift Provider + +Example configuration: + + datasift { + apiKey = "" + userName = "" + hashes = [ + "b8aaf7cec5faa2fadbd55d651933a31e", + "f41f054e2a2ba8d2e7b0d74f56e727d6" + ] + } + + + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-provider-facebook/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-provider-facebook/README.md b/streams-contrib/streams-provider-facebook/README.md new file mode 100644 index 0000000..ad58d68 --- /dev/null +++ b/streams-contrib/streams-provider-facebook/README.md @@ -0,0 +1,17 @@ +streams-provider-facebook +===================== + +Facebook Provider + +Example configuration: + + facebook { + â â oauth { + â â â â appId = "" + â â â â appSecret = "" + â â â â accessToken = "" + â â } + } + + + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-provider-google/google-gmail/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-provider-google/google-gmail/README.md b/streams-contrib/streams-provider-google/google-gmail/README.md new file mode 100644 index 0000000..fe2d138 --- /dev/null +++ b/streams-contrib/streams-provider-google/google-gmail/README.md @@ -0,0 +1,14 @@ +google-gmail +===================== + +GMail Provider + +Example configuration: + + gmail { + username = "" + password = "" + } + + + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-provider-google/google-gplus/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-provider-google/google-gplus/README.md b/streams-contrib/streams-provider-google/google-gplus/README.md new file mode 100644 index 0000000..b2175a0 --- /dev/null +++ b/streams-contrib/streams-provider-google/google-gplus/README.md @@ -0,0 +1,22 @@ +google-gplus +===================== + +Google+ Provider + +Example configuration: + + gplus { + â â googlePlusUsers = [ + { + userId = "" + } + ] + oauth { + â â â â appName = "" + â â â â pathToP12KeyFile = "Absolute Path to key file" + â â â â serviceAccountEmailAddress = "" + â â } + } + + + http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-provider-instagram/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-provider-instagram/README.md b/streams-contrib/streams-provider-instagram/README.md index 3bca23b..95de3f3 100644 --- a/streams-contrib/streams-provider-instagram/README.md +++ b/streams-contrib/streams-provider-instagram/README.md @@ -1,4 +1,5 @@ streams-provider-instagram +========================== Purpose http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-provider-moreover/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-provider-moreover/README.md b/streams-contrib/streams-provider-moreover/README.md new file mode 100644 index 0000000..d7fcbfb --- /dev/null +++ b/streams-contrib/streams-provider-moreover/README.md @@ -0,0 +1,17 @@ +streams-provider-moreover +========================== + +Purpose + + Module connects to moreover API, collects events, converts to activity, and passes each activity downstream. + +Example configuration + + moreover { + â â apiKeys { + key { + key = "" + startingSequence = "" + } + } + } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-provider-rss/README.markdown ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-provider-rss/README.markdown b/streams-contrib/streams-provider-rss/README.markdown index e31b5ce..7199fe9 100644 --- a/streams-contrib/streams-provider-rss/README.markdown +++ b/streams-contrib/streams-provider-rss/README.markdown @@ -1,4 +1,5 @@ streams-provider-rss +==================== Purpose http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-contrib/streams-provider-sysomos/README.md ---------------------------------------------------------------------- diff --git a/streams-contrib/streams-provider-sysomos/README.md b/streams-contrib/streams-provider-sysomos/README.md new file mode 100644 index 0000000..9b25f55 --- /dev/null +++ b/streams-contrib/streams-provider-sysomos/README.md @@ -0,0 +1,19 @@ +streams-provider-sysomos +========================== + +Purpose + + Module connects to sysomos API, collects events, converts to activity, and passes each activity downstream. + +Example configuration + + sysomos { + heartbeatIds = [ + HBID + ] + apiBatchSize = 500 + apiKey = KEY + minDelayMs = 10000 + scheduledDelayMs = 120000 + maxBatchSize = 10000 + } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-core/README.md ---------------------------------------------------------------------- diff --git a/streams-core/README.md b/streams-core/README.md new file mode 100644 index 0000000..3201490 --- /dev/null +++ b/streams-core/README.md @@ -0,0 +1,4 @@ +streams-core +============== + +streams-core contains core classes and interfaces. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-monitoring/README.md ---------------------------------------------------------------------- diff --git a/streams-monitoring/README.md b/streams-monitoring/README.md new file mode 100644 index 0000000..153b0d4 --- /dev/null +++ b/streams-monitoring/README.md @@ -0,0 +1,4 @@ +streams-monitoring +============== + +streams-monitoring contains classes and interfaces for monitoring executing streams. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-pojo-extensions/README.md ---------------------------------------------------------------------- diff --git a/streams-pojo-extensions/README.md b/streams-pojo-extensions/README.md new file mode 100644 index 0000000..dd46309 --- /dev/null +++ b/streams-pojo-extensions/README.md @@ -0,0 +1,4 @@ +streams-pojo-extensions +============== + +Support for adding and modifying extension properties. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-pojo/README.md ---------------------------------------------------------------------- diff --git a/streams-pojo/README.md b/streams-pojo/README.md new file mode 100644 index 0000000..2642093 --- /dev/null +++ b/streams-pojo/README.md @@ -0,0 +1,4 @@ +streams-pojo +============== + +streams-pojo contains ActivityStream json schema definitions and utility classes. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-runtimes/README.md ---------------------------------------------------------------------- diff --git a/streams-runtimes/README.md b/streams-runtimes/README.md new file mode 100644 index 0000000..97e412a --- /dev/null +++ b/streams-runtimes/README.md @@ -0,0 +1,7 @@ +streams-runtimes +============== + +These modules contain builders and helpers that support instantiation, +setup, execution, and cleanup of streams in a specific runtime context. + +Please help improve these packages, or create new ones! \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/c2a482b2/streams-util/README.md ---------------------------------------------------------------------- diff --git a/streams-util/README.md b/streams-util/README.md new file mode 100644 index 0000000..6ec1d51 --- /dev/null +++ b/streams-util/README.md @@ -0,0 +1,4 @@ +streams-util +============== + +streams-util contains utility classes and testing support classes. \ No newline at end of file
