http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/faq.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/faq.md b/src/site/markdown/faq.md new file mode 100644 index 0000000..49f5feb --- /dev/null +++ b/src/site/markdown/faq.md @@ -0,0 +1,138 @@ +## Frequently Asked Questions + +### Why should I adopt activity streams for my project? + +Odds are the dataset you are working with is some combination of timestamped events and observations of entities and their relationships at various points in time. Activity Streams provides a simple yet powerful standard format for these types of data, regardless of their origin, publisher, or specific details. Activity Streams is a community-driven specification designed for interoperability and flexibility. By supporting activity streams you maximize the chance that a new data-source of interest to you will be compatible with your existing data, and that your data will be compatible with that of other communities working on similar projects. + +### What organizations exchange data in activity streams formats? + +A short list of organizations and products that support activity streams format is compiled [here](who.html "here"). + +If your organization supports activity streams, please let us know on the project mailing list. + +### Why should I consider using Apache Streams for my project? + +If you are working with structured event and or entity data that fits the Activity Streams model, and working with a JVM language, Apache Streams can simplify many of the challenging aspects involved with these types of projects. For example: + +* Keeping track of the original source of each piece of information +* Harmonizing a multitude of date-time formats +* Moving between JSON, XML, YAML, and binary serializations +* Writing processing logic that can run in both batch and real-time workflows +* Defining constraints and validation rules for up-stream (third-party) and in-stream (your sphere of control) data +* Supplying run-time configuration globally and per-stream-component in a sensible manner + +### What does Apache Streams actually do? + +Apache Streams is + +* an SDK for data-centric JVM software +* a set of modules that connect data-providing APIs and data-persisting analytical systems +* a community working to make web and enterprise datasets interoperable by default + +Apache Streams is not + +* one-size-fits-all +* prescriptive or opinionated about how it should be used +* only useful for projects fully dedicated to activity streams datasets + +The primary Streams git repository incubator-streams (org.apache.streams:streams-project) contains + +* core interfaces and utilities +* plugins for transforming schemas into source code and other artifacts +* a library of modules for acquiring, transforming, and enriching data streams. + +Similar modules can also be hosted externally - so long as they publish maven artifacts compatible with your version of streams, you can import and use them in your streams easily. + +The streams community also supports a separate repository incubator-streams-examples (org.apache.streams:streams-examples) which contains a library of simple streams that are 'ready-to-run'. Look here to see what Streams user code look like. + +### Why bother with any data framework at all? + +Why use Postgres, Elasticsearch, Cassandra, Hadoop, Linux, or Java? + +Frameworks make important but boring parts of systems and code just work so your team can focus on features important to your users. + +If you are sure you can write code that is some combination of faster, more readable, better tested, easier to learn, easier to build with, or more maintainable than any existing framework (including Streams), maybe you should. + +But you are probably under-estimating how difficult it will be to optimize across all of these considerations, stay current with upgrades to underlying libraries, and fix whatever bugs are discovered. + +Or maybe you are capable of doing it all flawlessly, but your time is just more valuable focused on your product rather than on plumbing. + +By joining forces with others who care about clean running water, everyone can run better, faster, stronger code assembled with more diverse expertise, tested and tuned under more use cases. + +### How is streams different than "*processing framework*"? + +You don't have to look hard to find great data processing frameworks for batch or for real-time. Pig, Hive, Storm, Spark, Samza, Flink, and Google Cloud Dataflow (soon-to-be Apache Beam) are all great. Apex and NiFi are interesting newer options. This list only includes Apache Foundation JVM projects! + +At the core these platforms help you connect inputs and outputs to a directed graph of computation, and run your code at scale. + +Streams use this computational model as well, but is more focused on intelligently and correctly modeling the data that will flow through the stream than on stream execution. In this sense Streams is an alternative to avro or protocol buffers - one which prioritizes flexibility, expressivity, interoperability, and tooling ahead of speed or compute efficiency. + +Streams seeks to make it easy to design and evolve streams, and to configure complex streams sensibly. Where many processing frameworks leave all business logic and configuration issues to the developer, streams modules are designed to mix-and-match. Streams modules expect to be embedded with other frameworks and are organized to make that process painless. + +Streams also contains a library of plug-and-play data providers to collect and normalize data from a variety of popular sources. + +### How do I deploy Streams? + +Currently you cannot deploy Streams (uppercase). Streams has no shrink-wrapped ready-to-run server process. You can however deploy streams (lowercase). The right method for packaging, deploying, and running streams depends on what runtime you are going to use. + +Streams includes a local runtime that uses multi-threaded execution and blocking queues within a single process. In this scenario you build an uberjar with few exclusions and ship it to a target environment however you want - maven, scp, docker, etc... You launch the stream process with an appropriate configuration and watch the magic / catastrophic fail. + +Alternatively, components written to streams interfaces can be bound within other platforms such as pig or spark. In this scenario, you build an uberjar that excludes the platform parts of the classpath and launch your stream using the launch style of that platform. + +### Can't I just dump source data directly into files or databases? + +Absolutely - and that will work great right up until the point where the requirements, the tools, or the way you want to index your data need to change. + +### What if I need data from "*specific API*"? + +No problem - anyone can write a Streams provider. The project contains providers that use a variety of strategies to generate near-real-time data streams, including: + +* sockets +* webhooks +* polling +* scraping + +Providers can run continuously and pass-through new data, or they can work sequentially through a backlog of items. If you need to collect so many items that you can't fit all of their ids in the memory available to your stream, it's pretty simple to sub-divide your backlog into small batches and launch a series of providers for collection using frameworks such as Flink or Spark Streaming. + +### What if I want to keep data in "*unsupported database*"? + +No problem - anyone can write a Streams persist reader or persist writer. The project contains persist writers that: + +* write documents efficiently with batch-style binary indexing +* write documents one-by-one to services with REST api endpoints +* write data to local or distributed buffers. + +If you just want to use streams providers to collect and feed incoming data into a queueing system to work with outside of streams that's just fine. + +### Can't I just use "*third-party SDK*" to do the same thing? + +Describe any specific data collection, processing, or storage function and there are probably several if not tens of basic implementations on GitHub. There may even be language-specific libraries published by a vendor with a commercial interest in a related technology. + +However, in general there are a set of tradeoffs involved when relying on these packages. + +* They often have transitive dependencies. +* They may not use performant HTTP and JSON libraries. +* The object representations and lifecycle mechanisms they provide may not be consistent with the rest of your code. +* They may source configuration properties in a problematic or cumbersome fashion. +* Their licenses may be restrictive or undocumented. + +Streams goes to great lengths to regularize many of these issues so that they are uniform across project modules, and easy to reuse within new and external modules. + +Where quality java libraries exist, their most useful parts may be included within a streams module, with unnecessary or difficult parts of their dependency tree excluded. + +### Where do I start? + +Work your way through the 'Tutorial' menu to get up and running with streams. + +Then browse the 'Other Resources' menu to learn more about how streams works and why. + +### How can I help? + +* Join our mailing list. +* Ask questions and suggest features. +* Contribute to the documentation in one of the streams repositories. +* Write a new provider using an existing provider as a template. +* Add new features (and / or tests) to an existing module you intend to use. +* Build and contributing a new example. + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0
http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/index.md b/src/site/markdown/index.md new file mode 100644 index 0000000..8754944 --- /dev/null +++ b/src/site/markdown/index.md @@ -0,0 +1,17 @@ +## Overview + +Apache Streams (incubating) unifies a diverse world of digital profiles and online activities into common formats and vocabularies, and makes these datasets accessible across a variety of databases, devices, and platforms for streaming, browsing, search, sharing, and analytics use-cases. + +### What is Streams? +Apache Streams contains JRE-based modules that developers can use to easily integrate with online data sources and build polyglot indexes of activities, entities, and relationships - all based on public standards such as [Activity Streams](http://activitystrea.ms/), or other published organizational standards. + +### Why use Streams? +Streams contains libraries and patterns for specifying, publishing, and inter-linking schemas, and assists with conversion of activities (posts, shares, likes, follows, etc.) and objects (profiles, pages, photos, videos, etc.) between the representation, format, and encoding preferred by supported data providers (Twitter, Instagram, etc.), and storage services (Cassandra, Elasticsearch, HBase, HDFS, Neo4J, etc.) + +### Why is Streams important? +The project aims to provide simple two-way data interchange with all popular REST APIs in activity streams formats using a universal protocol. No other active open-source project has this ambitious goal, as well as production-worthy implementations for >10 services. Streams compatibility with multiple storage back-ends and ability to be embedded within any java-based real-time or batch data processing platform ensures that its interoperability features come with little technical baggage. + +### Disclaimer +Apache Streams is an effort undergoing incubation at [The Apache Software Foundation (ASF)](http://apache.org) sponsored by the [Apache Incubator PMC](http://incubator.apache.org). Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/install/docker.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/install/docker.md b/src/site/markdown/install/docker.md new file mode 100644 index 0000000..996bd4e --- /dev/null +++ b/src/site/markdown/install/docker.md @@ -0,0 +1,24 @@ +## Docker + +Run from your command line: + + docker version + +| Possible result | Explanation | +|-----------------|-------------| +| bash: docker: No such file or directory | You need to install docker | +| Client: Version: < 1.0.0 | You need a newer version of docker | +| Server: Version: < 1.0.0 | You need a newer version of docker | +| Client: Version: > 1.0.0\nServer: Version: > 1.0.0 | You are all good | + +If you need to install docker, start here: + +[https://docs.docker.com/engine/installation/](https://docs.docker.com/engine/installation/) + +Run from your command line: + + docker ps + +If you see a (possibly empty) list of running containers, you are good. + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/install/git.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/install/git.md b/src/site/markdown/install/git.md new file mode 100644 index 0000000..c1aad85 --- /dev/null +++ b/src/site/markdown/install/git.md @@ -0,0 +1,15 @@ +#### Git + + git -version + +| Possible result | Explanation | +|-----------------|-------------| +| bash: git: No such file or directory | You need to install git | +| git version < 2.7 | You should upgrade git for security reasons | +| git version > 2.7 | You are all good | + +If you need to install docker, start here: + +[https://git-scm.com/book/en/v2/Getting-Started-Installing-Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/install/java.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/install/java.md b/src/site/markdown/install/java.md new file mode 100644 index 0000000..24b527f --- /dev/null +++ b/src/site/markdown/install/java.md @@ -0,0 +1,18 @@ +#### Java SDK + +Run from your command line: + + java -version + +| Possible result | Explanation | +|-----------------|-------------| +| Java Version >= 1.7.0u72) | You're all good | +| Java Version >= 1.8.0u25) | You're all good | +| Java Version < 1.7.0u72 | You need a newer JDK | +| Java Version < 1.8.0u25 | You need a newer JDK | + +If you need to install or upgrade Java, start here: + +[http://www.oracle.com/technetwork/java/javase/downloads/index.html](http://www.oracle.com/technetwork/java/javase/downloads/index.html) + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/install/maven.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/install/maven.md b/src/site/markdown/install/maven.md new file mode 100644 index 0000000..17d8d3a --- /dev/null +++ b/src/site/markdown/install/maven.md @@ -0,0 +1,21 @@ +#### Maven + +Run from your command line: + + mvn -version + +| Possible result | Explanation | +|-----------------|-------------| +| -bash: mvn: command not found | You need to install maven | +| Error: JAVA_HOME is not defined correctly. | You need to install JDK | +| Apache Maven >= 3.2.5+\nJava Version >= 1.7.0u72) | You're all good | +| Apache Maven >= 3.2.5+\nJava Version >= 1.8.0u25) | You're all good | +| Apache Maven < 3.2.5 | You need a newer version of maven | +| Java Version < 1.7.0u72 | You need a newer version of maven | +| Java Version < 1.8.0u25 | You need a newer JDK | + +If you need to install maven, start here: + +[http://maven.apache.org/install.html](http://maven.apache.org/install.html) + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/install/sbt.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/install/sbt.md b/src/site/markdown/install/sbt.md new file mode 100644 index 0000000..b8bc684 --- /dev/null +++ b/src/site/markdown/install/sbt.md @@ -0,0 +1,17 @@ +## SBT + +Run from your command line: + + sbt + +| Possible result | Explanation | +|-----------------|-------------| +| bash: sbt: command not found | You need to install sbt | + +You should really install sbt-extras, like this: + + curl -s https://raw.githubusercontent.com/paulp/sbt-extras/master/sbt > /usr/bin/sbtx && chmod 0755 /usr/bin/sbtx + +Now you can easily run the streams examples using SBT, like a boss. + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/integration.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/integration.md b/src/site/markdown/integration.md new file mode 100644 index 0000000..05e0dc0 --- /dev/null +++ b/src/site/markdown/integration.md @@ -0,0 +1,7 @@ +## Continuous Integration + +This project uses [Jenkins](http://jenkins-ci.org/ "Jenkins") for continuous integration. + + + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/release-setup.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/release-setup.md b/src/site/markdown/release-setup.md new file mode 100644 index 0000000..d9dd778 --- /dev/null +++ b/src/site/markdown/release-setup.md @@ -0,0 +1,76 @@ +###Release Setup + +These setup steps only need to be performed on a particular machine once. + +Developers using Linux workstations can skip over the references to Cygwin. If using Windows, install cygwin, including Utils/gnupg and Net/openssh packages. + +####Create and install a SSH key + +1. Open a shell window. If using Windows, open a cygwin window. +2. Use ssh-keygen to create an SSH key. + + ssh-keygen -t rsa -b 4096 + + - Follow the latest steps and guides on the [ASF website](http://www.apache.org/dev/openpgp.html#generate-key) as you should **NOT** be using SHA1 and new keys **MUST** be at least 4096 bits. + - Program defaults should be fine. No passphrase is required for the ssh key generation. The keys will be saved in ~/.ssh/id_dsa (private) and ~/.ssh/id_dsa.pub (public). + - See [Authenticating By Public Key (OpenSSH)](http://www.networknewz.com/networknewz-10-20030707AuthenticatingbyPublicKeyOpenSSH.html) for a good description on why and how to perform this task. +3. SCP your SSH public key ~/.ssh/id_dsa.pub created in last step to ~/id_dsa.pub on people.apache.org. + + cd ~/.ssh + scp id_dsa.pub <your userid>@people.apache.org:id_dsa.pub + + - You will be prompted for your password. +4. Use ssh to login to people.apache.org + + cd ~ + ssh <your userid>@people.apache.org + + - At this point, you will still be prompted for your password. +5. Create a ~/.ssh folder in your home directory on people.apache.org and change its file mode to 700. + + mkdir ~/.ssh + chmod 700 ~/.ssh + +6. Move or append ~/id_dsa.pub to ~/.ssh/authorized_keys and change its file mode to 600. + + mv ~/id_dsa.pub ~/.ssh/authorized_keys + chmod 600 ~/.ssh/authorized_keys + + - Each public key in the authorized_keys spans only one line. For example: "ssh-dss AAAAB3NzaC1kc3MAAA ..... agBmmfZ9uAbSqA== dsa-key-20071107" + - *'#' in the first column is a comment line.* + +7. Exit out of this ssh session. + +8. Start a new ssh session. No login should be required this time due to the private ssh key on your local box matching up with the public ssh key in your home directory (~/.ssh). + + ssh <userid>@people.apache.org + + - If you are still prompted for a password, then you have not set up the ssh keys properly. Review the steps above and ensure that all of the steps were followed properly. Or, maybe the instructions are still not quite right and they still need some adjusting. In that case, please update the instructions accordingly.* + +####Create a GPG key + +1. Open a shell window. If using Windows, open a cygwin window. +2. Generate a key-pair with gpg, using default key kind ("RSA and RSA") and keys size (4096). + + gpg --gen-key + + - The program's default values should be fine. For the "Real Name" enter your full name (ie. Stan Programmer). For the "e-mail address" enter your apache address (ie. sprogram...@apache.org). You will also be required to enter a "passphrase" for the GPG key generation. Keep track of this as you will need this for the Release processing. + - *The generated keys are stored in $HOME/.gnupg or %HOME%\Application Data\gnupg subdirectory.* + - *Save the content in this subdirectory to a safe media. This contains your private key used to sign all the Streams release materials.* + +3. Backup your home directory to another media || +4. Add your public key to the [SVN repository](https://svn.apache.org/repos/asf/incubator/streams/KEYS). See the commands describe at the beginning of this KEYS file to perform this task. The gpg key-pair is used to sign the published artifacts for the Streams releases. + + gpg --list-sigs <Real Name> && gpg --armor -- export <Real Name> + + *The [KEYS](https://svn.apache.org/repos/asf/incubator/streams/KEYS) file is updated via normal svn commit procedures. The one under w.a.o/dist/ has to be manually updated from svn.* + +5. Submit your public key to a key server. E.g. [SURFNET](http://pgp.surfnet.nl:11371/) or [MIT](http://pgp.mit.edu/) + +6. Following the instructions in http://people.apache.org/~henkp/trust/ and ask multiple (at least 3) current Apache committers to sign your public key. + +####Configure Maven + +1. Update your ~/.m2/settings.xml with the properties from [Publishing Maven Artifacts](http://www.apache.org/dev/publishing-maven-artifacts.html#dev-env) + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/release.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/release.md b/src/site/markdown/release.md new file mode 100644 index 0000000..7998b4f --- /dev/null +++ b/src/site/markdown/release.md @@ -0,0 +1,251 @@ +##Release Process + +There are two distinct sets of artifacts that are released on independent schedules: streams-master & streams-project. The streams-master is the project metadata and only needs to be released when there is a change in the structure of the project itself. The streams-project artifacts are comprised of all streams source code, binaries and a standalone demo. For release setup information, refer to [Release Setup Information](/release-setup.html). + +All of the steps below apply to all streams repository releases, unless otherwise specified. As an alternative to releasing separately, the projects MAY be released together as one combined release. The steps for this can be found below. ([Combined Release Steps](#combined)) + +NOTE: + +Releases should always be built and published in the following order: + +* streams-master +* streams-project +* streams-examples + +In the instructions below, ${project.name} should be one of these top-level repository aggregator pom project names. + +As an alternative to releasing separately, the projects MAY be released together as one combined release. The steps for this can be found below. (Combined Release Steps) + +###Common Release Steps + +1. Environment setup for releasing artifacts (same for SNAPSHOTs and releases) + + 1. Increase the default Java heap available to Maven (required for Java SE 6) + + export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m" + + 2. Use the latest Sun 1.7.0 JDK + 3. Use Maven 3.2.1 or later + 4. Make sure the [Release Setup](release-setup.html) steps have been performed. + +2. Prepare the source for release: + + 1. Cleanup JIRA so the Fix Version in issues resolved since the last release includes this release version correctly. + 2. Update the text files in a working copy of the project root - + 1. Update the CHANGELOG based on the Text release reports from JIRA. + 2. Review and update README.md if needed. + 3. Commit any changes back to git + 3. Stage any Roadmap or Release landing pages on the site. + +3. Create a release candidate branch from master. + X should start at 1 and increment if early release candidates fail to complete the release cycle. + + git checkout master + git branch ${project.name}-${project.version}-rcX + +4. Verify the source has the required license headers before trying to release: + + mvn -Papache-release clean apache-rat:check -e -DskipTests + +5. Do a dry run of the release:prepare step: + + mvn -Papache-release release:prepare -DautoVersionSubmodules=true -DdryRun=true + + - The dry run will not commit any changes back to SCM and gives you the opportunity to verify that the release process will complete as expected. You will be prompted for the following information : + + * Release version - take the default (should be ${project.version}-incubating) + * SCM release tag - *DO NOT TAKE THE DEFAULT* - ${project.artifactId}-${project.version}-rcX + * New development version - take the default (should be ${project.version}-incubating-SNAPSHOT) + * GPG Passphrase + + - *If you cancel a release:prepare before it updates the pom.xml versions, then use the release:clean goal to just remove the extra files that were created.* + + - The Maven release plugin checks for SNAPSHOT dependencies in pom's. It will not complete the prepare goal until all SNAPSHOT dependencies are resolved. + +6. Verify that the release process completed as expected + 1. The release plugin will create pom.xml.tag files which contain the changes that would have been committed to SVN. The only differences between pom.xml.tag and it's corresponding pom.xml file should be the version number. + 2. Check release.properties and make sure that the scm properties have the right version. Sometimes the scm location can be the previous version not the next version. + 3. Verify signatures ([Verifying release signatures](#verify_signatures)) + +7. Cleanup the release prepare files again: + + mvn -Papache-release release:clean + +8. Prepare the release + 1. Run the "release:prepare" step for real this time. You'll be prompted for the same version information. + + mvn -Papache-release -U clean release:prepare -DautoVersionSubmodules=true + + 2. Backup (zip or tar) your local release candidate directory in case you need to rollback the release after the next step is performed. +9. Perform the release + * This step will create a maven staging repository and site for use in testing and voting. + + mvn -Papache-release -Darguments='-Dmaven.test.skip.exec=true' release:perform -Dgoals=deploy -DlocalRepoDirectory=. -DlocalCheckout=true + + * If your local OS userid doesn't match your Apache userid, then you'll have to also override the value provided by the OS to Maven for the site-deploy step to work. This is known to work for Linux, but not for Mac and unknown for Windows.* + + -Duser.name=[your_apache_uid] + +10. Verify the Nexus release artifacts + 1. Verify the staged artifacts in the nexus repo + * https://repository.apache.org/index.html + * Staging repositories (under Build Promotion) --> Name column --> org.apache.streams + * Navigate through the artifact tree and make sure that all javadoc, sources, tests, jars, ... have .asc (GPG signature) and .md5 files. See http://people.apache.org/~henkp/repo/faq.html and http://www.apache.org/dev/release-signing.html#openpgp-ascii-detach-sig + 2. Close the nexus staging repo + * https://repository.apache.org/index.html + * Staging repositories (under Build Promotion) --> Name column --> org.apache.streams + * Click checkbox for the open staging repo (org.apache.streams-XXX) and press Close in the menu bar. +11. Put the release candidate up for a vote + 1. Create a VOTE email thread on dev@ to record votes as replies + 2. Create a DISCUSS email thread on dev@ for any vote questions + 3. Perform a review of the release and cast your vote. See the following for more details on Apache releases + [http://www.apache.org/dev/release.html](http://www.apache.org/dev/release.html) + 4. A -1 vote does not necessarily mean that the vote must be redone, however it is usually a good idea to rollback the release if a -1 vote is received. See - Recovering from a vetoed release + 5. After the vote has been open for at least 72 hours, has at least three +1 PMC votes and no -1 votes, then post the results to the vote thread by - + * reply to the initial email and prepend to the original subject "[RESULT]" + * Include a list of everyone who voted +1, 0 or -1. + 6. Promote the staged nexus artifacts + * https://repository.apache.org/index.html + * Staging repositories (under Build Promotion) --> Name column --> org.apache.streams + * Click checkbox of the closed staging repo (org.apache.streams-XXX) and select Release. +13. Put the release candidate up for an IPMC vote if there are fewer than 3 +1 (binding) votes from IPMC members + 1. Copy the source artifacts over to the release dev area + + svn co https://dist.apache.org/repos/dist/dev/incubator/streams ./streams-dev (KEEP this directory until after the release process has been completed) + cd ./streams-dev + mkdir ${project.version} + cd ./${project.version} + wget https://repository.apache.org/content/repositories/releases/org/apache/streams/${project.name}/${project.version}/${project.name}-${project.version}-source-release.zip + wget https://repository.apache.org/content/repositories/releases/org/apache/streams/${project.name}/${project.version}/${project.name}-${project.version}-source-release.zip.asc + wget https://repository.apache.org/content/repositories/releases/org/apache/streams/${project.name}/${project.version}/${project.name}-${project.version}-source-release.zip.md5 + svn add ${project.name}-* + svn commit -m "Committing Source Release for ${project.name}-${project.version} + + 2. Submit a vote to gene...@incubator.apache.org requesting additional IPMC member votes. + +14. Complete the release + 1. Copy the source artifacts over to the distribution area + + svn co https://dist.apache.org/repos/dist/release/incubator/streams/releases ./streams-releases (KEEP this directory until after the release process has been completed) + cd ./streams-releases + mkdir ${project.version} + cd ./${project.version} + wget https://repository.apache.org/content/repositories/releases/org/apache/streams/${project.name}/${project.version}/${project.name}-${project.version}-source-release.zip + wget https://repository.apache.org/content/repositories/releases/org/apache/streams/${project.name}/${project.version}/${project.name}-${project.version}-source-release.zip.asc + wget https://repository.apache.org/content/repositories/releases/org/apache/streams/${project.name}/${project.version}/${project.name}-${project.version}-source-release.zip.md5 + svn add ${project.name}-* + svn commit -m "Committing Source Release for ${project.name}-${project.version} + + 3. Create an official release tag from the successful release candidate tag. + + git checkout ${project.name}-${project.version}-rcX + git tag -a ${project.name}-${project.version} -m 'release tag ${project.name}-${project.version}' + git push origin :refs/tags/streams-project-${project.version} + + 4. Update the staged website + * Update the downloads page (downloads.md) to add new version using the mirrored URLs + * Modify the URL for the prior release to the archived URL for the release + 5. Publish the website (see [website](website.html "Website Management")) + * WAIT 24hrs after committing releases for mirrors to replicate + * Publish updates to the download page + 6. Delete the prior versions + * Navigate to the release directories checked out in the prior steps + * Delete the prior release artifacts using the svn delete command + * Commit the deletion +14. Update the JIRA versions page to close all issues, mark the version as "released", and set the date to the date that the release was approved. You may also need to make a new release entry for the next release. +15. Announcing the release + * Make a news announcement on the streams homepage. + * Make an announcement about the release on the d...@streams.incubator.apache.org, and annou...@incubator.apache.org list as per the Apache Announcement Mailing Lists page) + +####Recovering from a vetoed release + +1. Reply to the initial vote email and prepend to the original subject - + [CANCELED] +2. Clean the release prepare files and hard reset the release candidate branch. + + mvn -P apache-release release:clean + +3. Delete the git tag created by the release:perform step - + + git tag -d streams-project-${project.version}-rcX + git push origin :refs/tags/streams-project-${project.version}-rcX + +4. Delete the build artifacts on people & www + + rm -rfv /www/people.apache.org/builds/streams/${project.version} + rm -rfv /www/www.apache.org/dist/streams/${project.version} + +5. Drop the nexus staging repo + 1. https://repository.apache.org/index.html + 2. Enterprise --> Staging + 3. Staging tab --> Name column --> org.apache.streams + 4. Right click on the closed staging repo (org.apache.streams-XXX) and select Drop. +5. Remove the staged site +6. Make the required updates that caused the vote to be canceled during the next release cycle + +<a name="verify_signatures" ></a> +####Verifying release signatures +On unix platforms the following command can be executed - + + for file in `find . -type f -iname '*.asc'` + do + gpg --verify ${file} + done + +You'll need to look at the output to ensure it contains only good signatures - + +gpg: Good signature from ... +gpg: Signature made ... + + +<a name="combined" ></a> +####Combined Release +In order to perform a combined release of the streams-master and streams-project trunks, do the following: + + * Perform Steps 1-9 of the [release](#release-steps) for streams-master, streams-project, and streams-examples + * Do NOT perform step 10 until steps 1-9 have been completed for BOTH projects + * Build the streams-master FIRST + * When prompted to change dependencies on SNAPSHOTs, do so to the corresponding releases that you just built + * Execute the remaining steps using the following e-mail template + + to: streams-...@incubator.apache.org + subject: [VOTE] Apache Streams ${release.version} Release Candidate + + I've created a combined ${release.version} release candidate, with the + following artifacts up for a vote: + + incubator-streams-master source tag (r${release.version}): + https://git-wip-us.apache.org/repos/asf?p=incubator-streams-master.git;a=commit;h=... + + incubator-streams source tag (r${release.version}): + https://git-wip-us.apache.org/repos/asf?p=incubator-streams.git;a=commit;h=... + + incubator-streams-examples source tag (r${release.version}): + https://git-wip-us.apache.org/repos/asf?p=incubator-streams-examples.git;a=commit;h=... + + Maven staging repo: + https://repository.apache.org/content/repositories/${release.master.repository} + https://repository.apache.org/content/repositories/${release.project.repository} + https://repository.apache.org/content/repositories/${release.examples.repository} + + Source releases: + https://repository.apache.org/content/repositories/${release.master.repository}/org/apache/streams/streams-master/${release.version}/streams-master-${release.version}-source-release.zip + https://repository.apache.org/content/repositories/${release.project.repository}/org/apache/rave/rave-project/${release.version}/streams-project-${release.version}-incubating-source-release.zip + https://repository.apache.org/content/repositories/${release.project.repository}/org/apache/rave/rave-project/${release.version}/streams-project-${release.version}-incubating-source-release.zip + + Checksums of streams-master-${release.version}-source-release.zip: + MD5: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX + SHA1: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX + + Release artifacts are signed with the following key: + https://people.apache.org/keys/committer/... + + Please take the time to verify the artifacts before casting your vote. + + Vote will be open for 72 hours. + + [ ] +1 approve + [ ] +0 no opinion + [ ] -1 disapprove (and reason why) + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/services/elasticsearch.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/services/elasticsearch.md b/src/site/markdown/services/elasticsearch.md new file mode 100644 index 0000000..58d096b --- /dev/null +++ b/src/site/markdown/services/elasticsearch.md @@ -0,0 +1,35 @@ +## Elasticsearch + +Start elasticsearch via docker with the docker maven plugin: + + docker -PdockerITs docker:start + +Confirm that elasticsearch is running: + + docker ps + +Confirm that host and post(s) are in property file: + + cat elasticsearch.properties + +Create a local file `elasticsearch.conf` with cluster details: + + elasticsearch { + hosts += ${es.tcp.host} + port = ${es.tcp.port} + clusterName = "elasticsearch" + } + +When configuring a stream, include these files: + + include "elasticsearch.properties" + include "elasticsearch.conf" + +Supply application-specific configuration as well: + + elasticsearch { + index: "" + type: "" + } + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/services/mongo.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/services/mongo.md b/src/site/markdown/services/mongo.md new file mode 100644 index 0000000..c2b182d --- /dev/null +++ b/src/site/markdown/services/mongo.md @@ -0,0 +1,34 @@ +## Mongo + +Start mongo via docker with the docker maven plugin: + + docker -PdockerITs docker:start + +Confirm that elasticsearch is running: + + docker ps + +Confirm that host and post(s) are in property file: + + cat mongo.properties + +Create a local file `elasticsearch.conf` with cluster details: + + mongo { + host = ${mongo.tcp.host} + port = ${mongo.tcp.port} + } + +When configuring a stream, include these files: + + include "mongo.properties" + include "mongo.conf" + +Supply application-specific configuration as well: + + mongo { + db: "", + collection: "" + } + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/services/neo4j.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/services/neo4j.md b/src/site/markdown/services/neo4j.md new file mode 100644 index 0000000..1d88ed0 --- /dev/null +++ b/src/site/markdown/services/neo4j.md @@ -0,0 +1,29 @@ +## Neo4j + +Start mongo via docker with the docker maven plugin: + + docker -PdockerITs docker:start + +Confirm that elasticsearch is running: + + docker ps + +Confirm that host and post(s) are in property file: + + cat neo4j.properties + +Create a local file `neo4j.conf` with cluster details: + + neo4j { + hostname = ${neo4j.tcp.host} + port = ${neo4j.tcp.port} + type = "neo4j" + graph = "data" + } + +When configuring a stream, include these files: + + include "neo4j.properties" + include "neo4j.conf" + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/source-repository.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/source-repository.md b/src/site/markdown/source-repository.md new file mode 100644 index 0000000..f94e7fb --- /dev/null +++ b/src/site/markdown/source-repository.md @@ -0,0 +1,27 @@ +## Source Repositories + +This project uses [Git](http://git.org/ "Git") for source control. + +Only project committers are permitted to push to git-wip-us.apache.org. + +However, anyone may open pull requests via github.org. + +###streams-master + +[streams-master](index.html "streams-master") source is officially hosted at [git-wip-us.apache.org/repos/asf/incubator-streams-master.git](https://git-wip-us.apache.org/repos/asf/incubator-streams-master.git "https://git-wip-us.apache.org/repos/asf/incubator-streams-master.git";) + +[streams-master](index.html "streams-master") source is also hosted at [github.com/apache/incubator-streams-master.git](https://github.com/apache/incubator-streams-master.git "https://github.com/apache/incubator-streams-master.git";) + +###streams-project + +[streams-project](http://streams.incubator.apache.org/site/0.2-incubating/streams-project "streams-project") source is officially hosted at [git-wip-us.apache.org/repos/asf/incubator-streams.git](https://git-wip-us.apache.org/repos/asf/incubator-streams.git "https://git-wip-us.apache.org/repos/asf/incubator-streams.git";) + +[streams-project](index.html "streams-project") source is also hosted at [github.com/apache/incubator-streams.git](https://github.com/apache/incubator-streams.git "https://github.com/apache/incubator-streams.git";) + +###streams-examples + +[streams-examples](http://streams.incubator.apache.org/site/0.2-incubating/streams-project "streams-examples") source is officially hosted at [git-wip-us.apache.org/repos/asf/incubator-streams-examples.git](https://git-wip-us.apache.org/repos/asf/incubator-streams-examples.git "https://git-wip-us.apache.org/repos/asf/incubator-streams-examples.git";) + +[streams-examples](http://streams.incubator.apache.org/site/0.2-incubating/streams-project "streams-examples") source is also hosted at [github.com/apache/incubator-streams-examples.git](https://github.com/apache/incubator-streams-examples.git "https://github.com/apache/incubator-streams-examples.git";) + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/website.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/website.md b/src/site/markdown/website.md new file mode 100644 index 0000000..43dee43 --- /dev/null +++ b/src/site/markdown/website.md @@ -0,0 +1,194 @@ +## Website Management + +http://streams.incubator.apache.org is a set of interconnected websites generated +by the maven site plugin. The front page, this page, and most pages linked in the +top nav of the site are part of incubator-streams-master. incubator-streams and +incubator-streams-examples also contain documentation of specific modules and examples that +are part of the project webpage. + +### Website Content + +Pages, diagrams, and other hard-linkable resources are stored throughout the source tree. + +#### Markdown + +* src/site/markdown + +Most HTML pages are generated from markdown. The maven-site-plugin does this during the site build process. + +#### Schemas + +* src/main/jsonschema +* src/main/xmlschema + +JSON and XML Schemas through-out the project are copied to the web page of their respective modules so they can be +linked to in other projects. + +This allows users to extend the canonical streams schemas incrementally and/or re-use specific fields as they see fit. + +#### Resources + +* src/main/resources +* src/test/resources + +Other resources including .conf and .properties files are copied to the web page of their respective modules so +they can be linked to across projects and in external projects. + +This allows users to import HOCON from modules outside their sphere of control and adapt to changes upstream. + +### Website Updates + +The project website(s) are hosted by the Apache foundation and updated via SVN. + +Currently pushing website changes is a manual process with several steps, performed by whomever is making the change. + +This typically requires checking out the current website from SVN. + + svn co https://svn.apache.org/repos/asf/incubator/streams/site/trunk/content + cd content + +NOTE: + +Repositories should always be built and published in the following order: + +* streams-master +* streams-project +* streams-examples + +#### Preparing to publishing a new website version + +The instructions below presume: + +* you have a shell open in the SVN content directory +* you know the artifactId and version of the repository you want to publish. + +If this is a brand new snapshot or release version, you first need to create a directory corresponding to the new version. + + mkdir site/${project.version} + svn add site/${project.version} + svn commit -m "svn add site/${project.version}" + +The first time a specific site is being published for this version, you must create the directory where it will be published. + + mkdir site/${project.version}/${project.artifactId} + svn add site/${project.version}/${project.artifactId} + svn commit -m "svn add site/${project.version}/${project.artifactId}" + +If you are published over an existing snapshot, you may need to first remove the existing version and recreate an empty directory. + + rm -rf site/${project.version}/${project.artifactId} + svn delete site/${project.version}/${project.artifactId} + svn commit -m "svn delete site/${project.version}/${project.artifactId}" + mkdir site/${project.version}/${project.artifactId} + svn add site/${project.version}/${project.artifactId} + svn commit -m "svn add site/${project.version}/${project.artifactId}" + +In general however the scm plugin is smart enough to just add new resources and commit changes. + +If you are publishing a release, it's appropriate to delete the site snapshots related to the prior releases. + +For example when 0.3-incubating is published, 0.2-incubating-SNAPSHOT should be deleted. + +This policy of removing old snapshots keeps external projects from linking to snapshot artifacts indefinitely. + +Release artifacts should be retained indefinitely. + +#### Generating and committing a new website version + +The instructions below presume: + +* you have a shell open in the root of a project repository +* you know the artifactId and version of the repository you want to publish. + +First, ensure that you have local credentials capable of publishing the site. + + <server> + <id>site.streams.{master|project|examples}</id> + <username>{your apache ID}</username> + <privateKey>{absolute path to your private key</privateKey> + <passphrase>{your private key passphrase}</passphrase> + <filePermissions>664</filePermissions> + <directoryPermissions>775</directoryPermissions> + <configuration></configuration> + </server> + +Next, generate SVG resources for all DOT diagrams in the source tree + + for dot in $(find . -name *.dot); do dot -Tsvg $dot -o $dot.svg; done + +Then, generate the site that will be published + + mvn clean generate-sources package -Dmaven.test.skip.exec=true site:site site:stage + +Double-check the logs and determine where exactly the staged site is located on your local drive. + +At this point you can open target/staging/index.html (or wherever) and do a basic sanity check on the site you intend to publish. + +Finally, publish the site. + + mvn scm-publish:publish-scm + +You may need to provide -Dscmpublish.content=<> depending exactly where the staging site directory winds up. + +Note the revision number checked in at the bottom of the maven logs. + +#### Updating the staging site + +Next step is to update the staging site and check it out. + +Log into https://cms.apache.org with your apache credentials. + +Use https://cms.apache.org/streams to access the streams website. + +Typically you can use Get streams Working Copy, although you might need to Force if you run into conflicts in SVN. + +You'll probably need to 'Update this directory' if you want to inspect the changes you committed above. + +'View Staging Builds' should show a build around the time of the previous commit. This means the change has been staged. + +You should now be able to access and review the published site(s) via the staging URL: + +* http://streams.staging.apache.org/ + +At this point use explicit versions to access and review the new documentation, i.e. + +* http://streams.staging.apache.org/site/0.3-incubating-SNAPSHOT/streams-master/index.html +* http://streams.staging.apache.org/site/0.3-incubating-SNAPSHOT/streams-project/index.html +* http://streams.staging.apache.org/site/0.3-incubating-SNAPSHOT/streams-examples/index.html + +#### Managing version pointers + +When new versions of these sites are built for the first time, an additional set to alter Apache rules may be appropriate. + +The convention we use exposes the latest specific site version(s) using redirects maintained in the .htaccess file of project website SVN. + +These rules are located in the '.htaccess' file in the root of the SVN content directory and looks something like this: + + Options +FollowSymLinks + RewriteEngine on + RedirectMatch "^/$" "/site/0.3-incubating-SNAPSHOT/streams-master" + Redirect /site/latest /site/0.3-incubating-SNAPSHOT + Redirect /site/latest/streams-master /site/0.3-incubating-SNAPSHOT/streams-master + Redirect /site/latest/streams-project /site/0.3-incubating-SNAPSHOT/streams-project + Redirect /site/latest/streams-examples /site/0.3-incubating-SNAPSHOT/streams-examples + +Adjust the Redirect rules as appropriate for the project and version you are deploying. You can do this directly from the CMS. + +Commit your changes, wait a few seconds, click Follow Staging Build, and you should see a new build with a 'Build Successful' message. + +You should now be able to use 'latest' as an alias for the docs you are deploying, i.e. + +* http://streams.staging.apache.org/site/latest/streams-master/index.html +* http://streams.staging.apache.org/site/latest/streams-project/index.html +* http://streams.staging.apache.org/site/latest/streams-examples/index.html + +#### Promoting a new website version + +All that's left at this point is to promote from staging to production. + +If a release is happening, this should happen just prior to the release announcement. + +If you are just updating content associated with a snapshot, use good judgement to determine whether the list should have a chance to review +and/or vote on the changes in staging prior to promotion. + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/markdown/who.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/who.md b/src/site/markdown/who.md new file mode 100644 index 0000000..4add70f --- /dev/null +++ b/src/site/markdown/who.md @@ -0,0 +1,12 @@ +### Who uses activity streams? + +| Organization (Product) | Reference | +|------------------------|-----------| +| Atlassian (JIRA) | [Consuming an Activity Streams Feed](https://developer.atlassian.com/docs/atlassian-platform-common-components/activity-streams/consuming-an-activity-streams-feed "Consuming an Activity Streams Feed") | +| Google (Google Plus) | [Google+ Platform for Web](https://developers.google.com/+/web/api/rest/latest/activities#resource-representations "Google+ Platform for Web") | +| IBM | [Activity Streams](https://developer.ibm.com/open/activity-streams/ "Activity Streams") | +| Twitter (Gnip) | [Activity Streams Intro](http://support.gnip.com/articles/activity-streams-intro.html "Activity Streams Intro") | + +If your organization supports activity streams, please let us know on the project mailing list. + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams/blob/367d45ff/src/site/site.xml ---------------------------------------------------------------------- diff --git a/src/site/site.xml b/src/site/site.xml index 3805a13..b4e6792 100644 --- a/src/site/site.xml +++ b/src/site/site.xml @@ -17,10 +17,45 @@ ~ under the License. --> <project> + <publishDate position="bottom"/> + <version position="bottom"/> + <skin> + <groupId>org.apache.maven.skins</groupId> + <artifactId>maven-fluido-skin</artifactId> + <version>1.5</version> + </skin> + <bannerLeft> + <name>Apache Streams</name> + <src>http://streams.incubator.apache.org/images/streams_logo.jpg</src> + <href>http://streams.incubator.apache.org</href> + <height>100</height> + <width>150</width> + </bannerLeft> + <bannerRight> + <name>Apache Incubator</name> + <src>http://incubator.apache.org/images/egg-logo.png</src> + <href>http://incubator.apache.org</href> + </bannerRight> + <custom> + <fluidoSkin> + <topBarEnabled>true</topBarEnabled> + <navBarStyle>navbar-inverse</navBarStyle> + <sideBarEnabled>false</sideBarEnabled> + <gitHub> + <projectId>apache/incubator-streams</projectId> + <ribbonOrientation>right</ribbonOrientation> + <ribbonColor>black</ribbonColor> + </gitHub> + <twitter> + <user>ApacheStreams</user> + <showUser>true</showUser> + <showFollowers>true</showFollowers> + </twitter> + </fluidoSkin> + </custom> <body> <breadcrumbs> <item name="Incubator" href="http://incubator.apache.org/"/> - <item name="Streams" href="http://streams.incubator.apache.org/"/> </breadcrumbs> <head> <
