[
https://issues.apache.org/jira/browse/BEAM-6262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Romanenko updated BEAM-6262:
-----------------------------------
Description:
While reading a stream with limited number of records or time it throws an
exception in the end. Though, it seems that there is no data loss, so it's a
minor issue.
{noformat}
INFO: Creating new reader using [Checkpoint AT_TIMESTAMP for stream
apache-beam, shard shardId-000000000002: null, Checkpoint AT_TIMESTAMP for
stream apache-beam, shard shardId-000000000001: null]
Dec 18, 2018 6:41:44 PM org.apache.beam.sdk.io.kinesis.KinesisReader start
INFO: Starting reader using [Checkpoint AT_TIMESTAMP for stream apache-beam,
shard shardId-000000000002: null, Checkpoint AT_TIMESTAMP for stream
apache-beam, shard shardId-000000000001: null]
Dec 18, 2018 6:41:46 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
INFO: Closing shard iterators pool
Dec 18, 2018 6:41:46 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool readLoop
SEVERE: Unexpected exception occurred
java.lang.RuntimeException: Not retryable client failure
at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:225)
at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:134)
at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:119)
at
org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.fetchRecords(ShardRecordsIterator.java:86)
at
org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:76)
at
org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:123)
at
org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:114)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.AbortedException:
at
com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)
at
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)
at
com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
at
com.fasterxml.jackson.dataformat.cbor.CBORParserBootstrapper.ensureLoaded(CBORParserBootstrapper.java:196)
at
com.fasterxml.jackson.dataformat.cbor.CBORParserBootstrapper.constructParser(CBORParserBootstrapper.java:90)
at
com.fasterxml.jackson.dataformat.cbor.CBORFactory._createParser(CBORFactory.java:350)
at
com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:287)
at
com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:26)
at
com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:109)
at
com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:43)
at
com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2276)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2252)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.executeGetRecords(AmazonKinesisClient.java:1062)
at
com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:1038)
at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.lambda$getRecords$2(SimplifiedKinesisClient.java:137)
at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:210)
... 11 more
{noformat}
was:
While reading a stream with limited number of records or time it throws an
exception in the end. Though, it seems that there is no data loss, so it's a
minor issue.
{{INFO: Creating new reader using [Checkpoint AT_TIMESTAMP for stream
apache-beam, shard shardId-000000000002: null, Checkpoint AT_TIMESTAMP for
stream apache-beam, shard shardId-000000000001: null]}}
{{ Dec 18, 2018 6:41:44 PM org.apache.beam.sdk.io.kinesis.KinesisReader start}}
{{ INFO: Starting reader using [Checkpoint AT_TIMESTAMP for stream apache-beam,
shard shardId-000000000002: null, Checkpoint AT_TIMESTAMP for stream
apache-beam, shard shardId-000000000001: null]}}
{{ Dec 18, 2018 6:41:46 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool
stop}}
{{ INFO: Closing shard iterators pool}}
{{ Dec 18, 2018 6:41:46 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool
readLoop}}
{{ SEVERE: Unexpected exception occurred}}
{{ java.lang.RuntimeException: Not retryable client failure}}
{{ at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:225)}}
{{ at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:134)}}
{{ at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:119)}}
{{ at
org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.fetchRecords(ShardRecordsIterator.java:86)}}
{{ at
org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:76)}}
{{ at
org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:123)}}
{{ at
org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:114)}}
{{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
{{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
{{ at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
{{ at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
{{ at java.lang.Thread.run(Thread.java:748)}}
{{ Caused by: com.amazonaws.AbortedException: }}
{{ at
com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)}}
{{ at
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)}}
{{ at
com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)}}
{{ at
com.fasterxml.jackson.dataformat.cbor.CBORParserBootstrapper.ensureLoaded(CBORParserBootstrapper.java:196)}}
{{ at
com.fasterxml.jackson.dataformat.cbor.CBORParserBootstrapper.constructParser(CBORParserBootstrapper.java:90)}}
{{ at
com.fasterxml.jackson.dataformat.cbor.CBORFactory._createParser(CBORFactory.java:350)}}
{{ at
com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:287)}}
{{ at
com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:26)}}
{{ at
com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:109)}}
{{ at
com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:43)}}
{{ at
com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)}}
{{ at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554)}}
{{ at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)}}
{{ at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)}}
{{ at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)}}
{{ at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)}}
{{ at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)}}
{{ at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)}}
{{ at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)}}
{{ at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)}}
{{ at
com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2276)}}
{{ at
com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2252)}}
{{ at
com.amazonaws.services.kinesis.AmazonKinesisClient.executeGetRecords(AmazonKinesisClient.java:1062)}}
{{ at
com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:1038)}}
{{ at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.lambda$getRecords$2(SimplifiedKinesisClient.java:137)}}
{{ at
org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:210)}}
{{ ... 11 more}}
> KinesisIO - ShardReadersPool - Unexpected exception occurred
> ------------------------------------------------------------
>
> Key: BEAM-6262
> URL: https://issues.apache.org/jira/browse/BEAM-6262
> Project: Beam
> Issue Type: Bug
> Components: io-java-kinesis
> Affects Versions: 2.7.0, 2.8.0, 2.9.0
> Reporter: Alexey Romanenko
> Assignee: Alexey Romanenko
> Priority: Minor
>
> While reading a stream with limited number of records or time it throws an
> exception in the end. Though, it seems that there is no data loss, so it's a
> minor issue.
> {noformat}
> INFO: Creating new reader using [Checkpoint AT_TIMESTAMP for stream
> apache-beam, shard shardId-000000000002: null, Checkpoint AT_TIMESTAMP for
> stream apache-beam, shard shardId-000000000001: null]
> Dec 18, 2018 6:41:44 PM org.apache.beam.sdk.io.kinesis.KinesisReader start
> INFO: Starting reader using [Checkpoint AT_TIMESTAMP for stream apache-beam,
> shard shardId-000000000002: null, Checkpoint AT_TIMESTAMP for stream
> apache-beam, shard shardId-000000000001: null]
> Dec 18, 2018 6:41:46 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool stop
> INFO: Closing shard iterators pool
> Dec 18, 2018 6:41:46 PM org.apache.beam.sdk.io.kinesis.ShardReadersPool
> readLoop
> SEVERE: Unexpected exception occurred
> java.lang.RuntimeException: Not retryable client failure
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:225)
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:134)
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.getRecords(SimplifiedKinesisClient.java:119)
> at
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.fetchRecords(ShardRecordsIterator.java:86)
> at
> org.apache.beam.sdk.io.kinesis.ShardRecordsIterator.readNextBatch(ShardRecordsIterator.java:76)
> at
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.readLoop(ShardReadersPool.java:123)
> at
> org.apache.beam.sdk.io.kinesis.ShardReadersPool.lambda$startReadingShards$0(ShardReadersPool.java:114)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.amazonaws.AbortedException:
> at
> com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)
> at
> com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)
> at
> com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
> at
> com.fasterxml.jackson.dataformat.cbor.CBORParserBootstrapper.ensureLoaded(CBORParserBootstrapper.java:196)
> at
> com.fasterxml.jackson.dataformat.cbor.CBORParserBootstrapper.constructParser(CBORParserBootstrapper.java:90)
> at
> com.fasterxml.jackson.dataformat.cbor.CBORFactory._createParser(CBORFactory.java:350)
> at
> com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:287)
> at
> com.fasterxml.jackson.dataformat.cbor.CBORFactory.createParser(CBORFactory.java:26)
> at
> com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:109)
> at
> com.amazonaws.http.JsonResponseHandler.handle(JsonResponseHandler.java:43)
> at
> com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2276)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2252)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.executeGetRecords(AmazonKinesisClient.java:1062)
> at
> com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:1038)
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.lambda$getRecords$2(SimplifiedKinesisClient.java:137)
> at
> org.apache.beam.sdk.io.kinesis.SimplifiedKinesisClient.wrapExceptions(SimplifiedKinesisClient.java:210)
> ... 11 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
