[jira] [Commented] (SPARK-7788) Streaming | Kinesis | KinesisReceiver blocks in onStart
[ https://issues.apache.org/jira/browse/SPARK-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555711#comment-14555711 ] Apache Spark commented on SPARK-7788: - User 'tdas' has created a pull request for this issue: https://github.com/apache/spark/pull/6348 > Streaming | Kinesis | KinesisReceiver blocks in onStart > --- > > Key: SPARK-7788 > URL: https://issues.apache.org/jira/browse/SPARK-7788 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.3.0, 1.3.1 >Reporter: Aniket Bhatnagar >Assignee: Tathagata Das >Priority: Blocker > Labels: kinesis > > KinesisReceiver calls worker.run() which is a blocking call (while loop) as > per source code of kinesis-client library - > https://github.com/awslabs/amazon-kinesis-client/blob/v1.2.1/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java. > This results in infinite loop while calling > sparkStreamingContext.stop(stopSparkContext = false, stopGracefully = true) > perhaps because ReceiverTracker is never able to register the receiver (it's > receiverInfo field is a empty map) causing it to be stuck in infinite loop > while waiting for running flag to be set to false. > Also, we should investigate a way to have receiver restart in case of > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7788) Streaming | Kinesis | KinesisReceiver blocks in onStart
[ https://issues.apache.org/jira/browse/SPARK-7788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14553893#comment-14553893 ] Tathagata Das commented on SPARK-7788: -- This may be a good catch. I will try to look into it very soon and try to solve it for 1.4 > Streaming | Kinesis | KinesisReceiver blocks in onStart > --- > > Key: SPARK-7788 > URL: https://issues.apache.org/jira/browse/SPARK-7788 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.3.0, 1.3.1 >Reporter: Aniket Bhatnagar >Assignee: Tathagata Das >Priority: Blocker > Labels: kinesis > > KinesisReceiver calls worker.run() which is a blocking call (while loop) as > per source code of kinesis-client library - > https://github.com/awslabs/amazon-kinesis-client/blob/v1.2.1/src/main/java/com/amazonaws/services/kinesis/clientlibrary/lib/worker/Worker.java. > This results in infinite loop while calling > sparkStreamingContext.stop(stopSparkContext = false, stopGracefully = true) > perhaps because ReceiverTracker is never able to register the receiver (it's > receiverInfo field is a empty map) causing it to be stuck in infinite loop > while waiting for running flag to be set to false. > Also, we should investigate a way to have receiver restart in case of > failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
