[
https://issues.apache.org/jira/browse/SPARK-3639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14193285#comment-14193285
]
Chris Fregly commented on SPARK-3639:
-------------------------------------
great catch, aniket!
for some background context, i was trying to make the sample easier to run out
of the box. i overlooked the spark-submit scenario, unfortunately. thanks for
fixing this.
few things:
1) does the Streaming Kinesis Guide (docs/streaming-kinesis-integration.md)
need updating with your change? specifically, the Running the Example section?
i don't think so, but something to double-check.
2) i noticed you put a comment in the scaladoc about needing +1 workers/threads
than receivers, perhaps we should reword this to say
(number of kinesis shards+1) workers/threads are needed
because the number of receivers is determined by the number of shards in the
kinesis stream. might tighten up the message a bit.
3) should we throw an error if the number of workers/threads is not
sufficient? nobody likes an error message, but might be helpful here. this is
the basis of https://issues.apache.org/jira/browse/SPARK-2475, btw. might want
to keep an eye on that jira.
thanks again, man. great catch.
-chris
> Kinesis examples set master as local
> ------------------------------------
>
> Key: SPARK-3639
> URL: https://issues.apache.org/jira/browse/SPARK-3639
> Project: Spark
> Issue Type: Bug
> Components: Examples, Streaming
> Affects Versions: 1.0.2, 1.1.0
> Reporter: Aniket Bhatnagar
> Assignee: Aniket Bhatnagar
> Priority: Minor
> Labels: examples
> Fix For: 1.1.1, 1.2.0
>
>
> Kinesis examples set master as local thus not allowing the example to be
> tested on a cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
