[
https://issues.apache.org/jira/browse/SPARK-2475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114204#comment-14114204
]
Chris Fregly commented on SPARK-2475:
-------------------------------------
another option for the examples, specifically, is to default the number of
local threads similar to to how the Kinesis example does it:
https://github.com/apache/spark/blob/ae58aea2d1435b5bb011e68127e1bcddc2edf5b2/extras/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala#L104
i get the number of shards in the given Kinesis stream and add 1. the goal was
to make this example work out of the box with little friction - even an error
message can be discouraging.
for the other examples, we could just default to 2. the advanced user can
override if they want. though i don't think i support an override in my
kinesis example. whoops! :)
> Check whether #cores > #receivers in local mode
> -----------------------------------------------
>
> Key: SPARK-2475
> URL: https://issues.apache.org/jira/browse/SPARK-2475
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: Tathagata Das
>
> When the number of slots in local mode is not more than the number of
> receivers, then the system should throw an error. Otherwise the system just
> keeps waiting for resources to process the received data.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
