Repository: bahir-website Updated Branches: refs/heads/master d09e63b3b -> f8d356989
[BAHIR-50] Publish extensions documentation to website Provide a script to gather extension documentation based on its README.md and publish it as part of the current documentation on the website. When a release is performed, the "current" folder should be copied to the release name (e.g. 2.0.0) as a documentation snapshot. Project: http://git-wip-us.apache.org/repos/asf/bahir-website/repo Commit: http://git-wip-us.apache.org/repos/asf/bahir-website/commit/f8d35698 Tree: http://git-wip-us.apache.org/repos/asf/bahir-website/tree/f8d35698 Diff: http://git-wip-us.apache.org/repos/asf/bahir-website/diff/f8d35698 Branch: refs/heads/master Commit: f8d356989c150427f4d85c1ff12e2f06864f4d63 Parents: d09e63b Author: Luciano Resende <[email protected]> Authored: Sun Aug 7 19:53:11 2016 +0300 Committer: Luciano Resende <[email protected]> Committed: Wed Aug 10 14:13:21 2016 -0700 ---------------------------------------------------------------------- README.md | 21 ++++++- site/_data/navigation.yml | 5 ++ site/docs/spark/current/.gitignore | 1 + site/docs/spark/current/documentation.md | 46 +++++++++++++++ .../current/spark-sql-streaming-mqtt.template | 27 +++++++++ .../spark/current/spark-streaming-akka.template | 26 +++++++++ .../spark/current/spark-streaming-mqtt.template | 27 +++++++++ .../current/spark-streaming-twitter.template | 26 +++++++++ .../current/spark-streaming-zeromq.template | 26 +++++++++ update-doc.sh | 61 ++++++++++++++++++++ 10 files changed, 264 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 418049f..c72acee 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,22 @@ running from within the directory: ``` +## Update extensions documentation + +To update the current documentation contents, use the update-doc script. +This script will update the documentation contents based on the README.md +files from the source of each extension. + +``` +1. update-doc.sh +2. git commit -a -m"My updates" +3. git push +4. ./publish.sh +5. git push origin asf-site +``` + + + ## Publishing to live site Bahir is using [gitpubsub](http://www.apache.org/dev/gitpubsub.html) for publishing the website, @@ -48,8 +64,9 @@ shell script. ``` 1. Make your changes 2. git commit -a -m"My updates" -3. ./publish.sh -4. git push origin asf-site +3. git push +4. ./publish.sh +5. git push origin asf-site ``` Within a few minutes, gitpubsub should kick in and you'll be able to http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/site/_data/navigation.yml ---------------------------------------------------------------------- diff --git a/site/_data/navigation.yml b/site/_data/navigation.yml index 8aa4588..16ccd13 100644 --- a/site/_data/navigation.yml +++ b/site/_data/navigation.yml @@ -45,6 +45,11 @@ topnav: - title: Project Committers url: /community-members +- title: Documentation + subcategories: + - title: Current + url: /docs/spark/current/documentation + - title: GitHub url: https://github.com/apache/bahir http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/site/docs/spark/current/.gitignore ---------------------------------------------------------------------- diff --git a/site/docs/spark/current/.gitignore b/site/docs/spark/current/.gitignore new file mode 100644 index 0000000..6b7415a --- /dev/null +++ b/site/docs/spark/current/.gitignore @@ -0,0 +1 @@ +spark*.md http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/site/docs/spark/current/documentation.md ---------------------------------------------------------------------- diff --git a/site/docs/spark/current/documentation.md b/site/docs/spark/current/documentation.md new file mode 100644 index 0000000..16b6527 --- /dev/null +++ b/site/docs/spark/current/documentation.md @@ -0,0 +1,46 @@ +--- +layout: page +title: Extensions for Apache Spark +description: Extensions for Apache Spark +group: nav-right +--- +<!-- +{% comment %} +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to you under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +{% endcomment %} +--> + +{% include JB/setup %} + +### Apache Bahir Extensions for Apache Spark + +<br/> + +#### Structured Streaming Data Sources + +[MQTT data source](/docs/spark/current/spark-sql-streaming-mqtt) + +<br/> + +#### Discretized Streams (DStreams) Connectors + +[Akka connector](/docs/spark/current/spark-streaming-akka) + +[MQTT connector](/docs/spark/current/spark-streaming-mqtt) + +[Twitter connector](/docs/spark/current/spark-streaming-twitter) + +[ZeroMQ connector](/docs/spark/current/spark-streaming-zeromq) \ No newline at end of file http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/site/docs/spark/current/spark-sql-streaming-mqtt.template ---------------------------------------------------------------------- diff --git a/site/docs/spark/current/spark-sql-streaming-mqtt.template b/site/docs/spark/current/spark-sql-streaming-mqtt.template new file mode 100644 index 0000000..fd08baa --- /dev/null +++ b/site/docs/spark/current/spark-sql-streaming-mqtt.template @@ -0,0 +1,27 @@ +--- +layout: page +title: Spark Structured Streaming MQTT +description: Spark Structured Streaming MQTT +group: nav-right +--- +<!-- +{% comment %} +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to you under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +{% endcomment %} +--> + +{% include JB/setup %} + http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/site/docs/spark/current/spark-streaming-akka.template ---------------------------------------------------------------------- diff --git a/site/docs/spark/current/spark-streaming-akka.template b/site/docs/spark/current/spark-streaming-akka.template new file mode 100644 index 0000000..78b7d14 --- /dev/null +++ b/site/docs/spark/current/spark-streaming-akka.template @@ -0,0 +1,26 @@ +--- +layout: page +title: Spark Streaming Akka +description: Spark Streaming Akka +group: nav-right +--- +<!-- +{% comment %} +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to you under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +{% endcomment %} +--> + +{% include JB/setup %} http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/site/docs/spark/current/spark-streaming-mqtt.template ---------------------------------------------------------------------- diff --git a/site/docs/spark/current/spark-streaming-mqtt.template b/site/docs/spark/current/spark-streaming-mqtt.template new file mode 100644 index 0000000..fd08baa --- /dev/null +++ b/site/docs/spark/current/spark-streaming-mqtt.template @@ -0,0 +1,27 @@ +--- +layout: page +title: Spark Structured Streaming MQTT +description: Spark Structured Streaming MQTT +group: nav-right +--- +<!-- +{% comment %} +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to you under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +{% endcomment %} +--> + +{% include JB/setup %} + http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/site/docs/spark/current/spark-streaming-twitter.template ---------------------------------------------------------------------- diff --git a/site/docs/spark/current/spark-streaming-twitter.template b/site/docs/spark/current/spark-streaming-twitter.template new file mode 100644 index 0000000..9a97ced --- /dev/null +++ b/site/docs/spark/current/spark-streaming-twitter.template @@ -0,0 +1,26 @@ +--- +layout: page +title: Spark Streaming Twitter +description: Spark Streaming Twitter +group: nav-right +--- +<!-- +{% comment %} +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to you under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +{% endcomment %} +--> + +{% include JB/setup %} http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/site/docs/spark/current/spark-streaming-zeromq.template ---------------------------------------------------------------------- diff --git a/site/docs/spark/current/spark-streaming-zeromq.template b/site/docs/spark/current/spark-streaming-zeromq.template new file mode 100644 index 0000000..6a6a02a --- /dev/null +++ b/site/docs/spark/current/spark-streaming-zeromq.template @@ -0,0 +1,26 @@ +--- +layout: page +title: Spark Streaming ZeroMQ +description: Spark Streaming ZeroMQ +group: nav-right +--- +<!-- +{% comment %} +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to you under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + +http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +{% endcomment %} +--> + +{% include JB/setup %} http://git-wip-us.apache.org/repos/asf/bahir-website/blob/f8d35698/update-doc.sh ---------------------------------------------------------------------- diff --git a/update-doc.sh b/update-doc.sh new file mode 100755 index 0000000..2373215 --- /dev/null +++ b/update-doc.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -e + +BASE_DIR=$(pwd) +WEBSITE_DOC_DIR=$BASE_DIR/site/docs/spark/current +BAHIR_SOURCE_DIR=$BASE_DIR/target/bahir + +function checkout_code { + # Checkout code + rm -rf target + mkdir target + cd target + rm -rf bahir + git clone https://git-wip-us.apache.org/repos/asf/bahir.git + cd bahir + git checkout $GIT_REF + git_hash=`git rev-parse --short HEAD` + echo "Checked out Bahir git hash $git_hash" + + git clean -d -f -x + + cd "$BASE_DIR" #return to base dir +} + +checkout_code + +rm $WEBSITE_DOC_DIR/spark*.md + +cp $WEBSITE_DOC_DIR/spark-sql-streaming-mqtt.template $WEBSITE_DOC_DIR/spark-sql-streaming-mqtt.md +cat $BAHIR_SOURCE_DIR/sql-streaming-mqtt/README.md >> $WEBSITE_DOC_DIR/spark-sql-streaming-mqtt.md + +cp $WEBSITE_DOC_DIR/spark-streaming-akka.template $WEBSITE_DOC_DIR/spark-streaming-akka.md +cat $BAHIR_SOURCE_DIR/streaming-akka/README.md >> $WEBSITE_DOC_DIR/spark-streaming-akka.md + +cp $WEBSITE_DOC_DIR/spark-streaming-mqtt.template $WEBSITE_DOC_DIR/spark-streaming-mqtt.md +cat $BAHIR_SOURCE_DIR/streaming-mqtt/README.md >> $WEBSITE_DOC_DIR/spark-streaming-mqtt.md + +cp $WEBSITE_DOC_DIR/spark-streaming-twitter.template $WEBSITE_DOC_DIR/spark-streaming-twitter.md +cat $BAHIR_SOURCE_DIR/streaming-twitter/README.md >> $WEBSITE_DOC_DIR/spark-streaming-twitter.md + +cp $WEBSITE_DOC_DIR/spark-streaming-zeromq.template $WEBSITE_DOC_DIR/spark-streaming-zeromq.md +cat $BAHIR_SOURCE_DIR/streaming-zeromq/README.md >> $WEBSITE_DOC_DIR/spark-streaming-zeromq.md + +set +e
