[
https://issues.apache.org/jira/browse/BEAM-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366369#comment-16366369
]
Paweł Kaczmarczyk commented on BEAM-2582:
-----------------------------------------
This one should not happen anymore. Closed shareds are already handled in
BEAM-2469
> KinesisIO incorrectly handles closed shards
> -------------------------------------------
>
> Key: BEAM-2582
> URL: https://issues.apache.org/jira/browse/BEAM-2582
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
> Reporter: adam gray
> Assignee: Chamikara Jayalath
> Priority: Major
>
> The KinesisIO throws an exception when consuming closed Kinesis shards, which
> return null from `GetShardIterator`, as it tries to call `GetRecords` with
> the null `shardIterator` value instead of abandoning the closed shard.
> This means KinesisIO fails after re-sharding a stream with an exception like
> the following:
> {noformat}
> Exception in thread "main" java.lang.RuntimeException: Kinesis client side
> failure
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:151)
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:115)
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:102)
> at
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readMoreIfNecessary(ShardRecordsIterator.java:79)
> at
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.next(ShardRecordsIterator.java:64)
> at
> org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.java:86)
> at
> org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.startReader(UnboundedReadEvaluatorFactory.java:190)
> at
> org.apache.beam.runners.direct.UnboundedReadEvaluatorFactory$UnboundedReadEvaluator.processElement(UnboundedReadEvaluatorFactory.java:128)
> at
> org.apache.beam.runners.direct.TransformExecutor.processElements(TransformExecutor.java:139)
> at
> org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:107)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.amazonaws.AmazonServiceException: 1 validation error detected:
> Value null at 'shardIterator' failed to satisfy constraint: Member must not
> be null (Service: AmazonKinesis; Status Code: 400; Error Code:
> ValidationException; Request ID: d764e747-9616-5db3-86ba-08a0bc44cb39)
> at
> com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1378)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:924)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:702)
> at
> com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:454)
> at
> com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:416)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:365)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2016)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1986)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:985)
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient$3.call(SimplifiedKinesisClient.java:118)
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient$3.call(SimplifiedKinesisClient.java:115)
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:140)
> ... 14 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
