This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 83f943c [SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and
help messages for Kinesis integration
83f943c is described below
commit 83f943c4b922413159cdcc09dfee7ef67d586215
Author: Kengo Seki <[email protected]>
AuthorDate: Sun Mar 29 14:27:19 2020 -0700
[SPARK-31293][DSTREAMS][KINESIS][DOC] Fix wrong examples and help messages
for Kinesis integration
### What changes were proposed in this pull request?
This PR (SPARK-31293) fixes wrong command examples, parameter descriptions
and help message format for Amazon Kinesis integration with Spark Streaming.
### Why are the changes needed?
To improve usability of those commands.
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
I ran the fixed commands manually and confirmed they worked as expected.
Closes #28063 from sekikn/SPARK-31293.
Authored-by: Kengo Seki <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 60dd1a690fed62b1d6442cdc8cf3f89ef4304d5a)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/streaming-kinesis-integration.md | 3 +--
.../main/python/examples/streaming/kinesis_wordcount_asl.py | 7 ++++---
.../apache/spark/examples/streaming/KinesisWordCountASL.scala | 10 +++++-----
3 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/docs/streaming-kinesis-integration.md
b/docs/streaming-kinesis-integration.md
index e68d513..db813c4 100644
--- a/docs/streaming-kinesis-integration.md
+++ b/docs/streaming-kinesis-integration.md
@@ -246,8 +246,7 @@ To run the example,
</div>
<div data-lang="python" markdown="1">
- ./bin/spark-submit --jars external/kinesis-asl/target/scala-*/\
- spark-streaming-kinesis-asl-assembly_*.jar \
+ ./bin/spark-submit --jars
'external/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar'
\
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
\
[Kinesis app name] [Kinesis stream name] [endpoint URL] [region
name]
diff --git
a/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
b/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
index 49794fa..777a332 100644
---
a/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
+++
b/external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
@@ -26,6 +26,7 @@
<stream-name> name of the Kinesis stream (ie. mySparkStream)
<endpoint-url> endpoint of the Kinesis service
(e.g. https://kinesis.us-east-1.amazonaws.com)
+ <region-name> region name of the Kinesis endpoint (e.g. us-east-1)
Example:
@@ -34,10 +35,10 @@
$ export AWS_SECRET_KEY=<your-secret-key>
# run the example
- $ bin/spark-submit -jars external/kinesis-asl/target/scala-*/\
- spark-streaming-kinesis-asl-assembly_*.jar \
+ $ bin/spark-submit --jars \
+
'external/kinesis-asl-assembly/target/spark-streaming-kinesis-asl-assembly_*.jar'
\
external/kinesis-asl/src/main/python/examples/streaming/kinesis_wordcount_asl.py
\
- myAppName mySparkStream https://kinesis.us-east-1.amazonaws.com
+ myAppName mySparkStream https://kinesis.us-east-1.amazonaws.com
us-east-1
There is a companion helper class called KinesisWordProducerASL which puts
dummy data
onto the Kinesis stream.
diff --git
a/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
b/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
index a5d5ac7..32f4a67 100644
---
a/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
+++
b/external/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala
@@ -78,7 +78,7 @@ object KinesisWordCountASL extends Logging {
if (args.length != 3) {
System.err.println(
"""
- |Usage: KinesisWordCountASL <app-name> <stream-name> <endpoint-url>
<region-name>
+ |Usage: KinesisWordCountASL <app-name> <stream-name> <endpoint-url>
|
| <app-name> is the name of the consumer app, used to track the
read data in DynamoDB
| <stream-name> is the name of the Kinesis stream
@@ -171,11 +171,11 @@ object KinesisWordCountASL extends Logging {
* <endpoint-url> is the endpoint of the Kinesis service
* (ie. https://kinesis.us-east-1.amazonaws.com)
* <records-per-sec> is the rate of records per second to put onto the stream
- * <words-per-record> is the rate of records per second to put onto the
stream
+ * <words-per-record> is the number of words per record
*
* Example:
* $ SPARK_HOME/bin/run-example streaming.KinesisWordProducerASL
mySparkStream \
- * https://kinesis.us-east-1.amazonaws.com us-east-1 10 5
+ * https://kinesis.us-east-1.amazonaws.com 10 5
*/
object KinesisWordProducerASL {
def main(args: Array[String]): Unit = {
@@ -183,13 +183,13 @@ object KinesisWordProducerASL {
System.err.println(
"""
|Usage: KinesisWordProducerASL <stream-name> <endpoint-url>
<records-per-sec>
- <words-per-record>
+ | <words-per-record>
|
| <stream-name> is the name of the Kinesis stream
| <endpoint-url> is the endpoint of the Kinesis service
| (e.g. https://kinesis.us-east-1.amazonaws.com)
| <records-per-sec> is the rate of records per second to put onto
the stream
- | <words-per-record> is the rate of records per second to put
onto the stream
+ | <words-per-record> is the number of words per record
|
""".stripMargin)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
