[jira] [Commented] (FLINK-10154) Make sure we always read at least one record in KinesisConnector
[ https://issues.apache.org/jira/browse/FLINK-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582718#comment-16582718 ] ASF GitHub Bot commented on FLINK-10154: asfgit closed pull request #6564: [FLINK-10154] [kinesis connector] Make sure we always read at least one record in Kinesis Connector. URL: https://github.com/apache/flink/pull/6564 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java b/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java index d698ecf2e9d..6de72783f1f 100644 --- a/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java +++ b/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java @@ -278,8 +278,8 @@ private int adaptRecordsToRead(long runLoopTimeNanos, int numRecords, long recor double loopFrequencyHz = 10.0d / runLoopTimeNanos; double bytesPerRead = KINESIS_SHARD_BYTES_PER_SECOND_LIMIT / loopFrequencyHz; maxNumberOfRecordsPerFetch = (int) (bytesPerRead / averageRecordSizeBytes); - // Ensure the value is not more than 1L - maxNumberOfRecordsPerFetch = Math.min(maxNumberOfRecordsPerFetch, ConsumerConfigConstants.DEFAULT_SHARD_GETRECORDS_MAX); + // Ensure the value is greater than 0 and not more than 1L + maxNumberOfRecordsPerFetch = Math.max(1, Math.min(maxNumberOfRecordsPerFetch, ConsumerConfigConstants.DEFAULT_SHARD_GETRECORDS_MAX)); } return maxNumberOfRecordsPerFetch; } This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make sure we always read at least one record in KinesisConnector > > > Key: FLINK-10154 > URL: https://issues.apache.org/jira/browse/FLINK-10154 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Affects Versions: 1.6.0 >Reporter: Jamie Grier >Assignee: Jamie Grier >Priority: Minor > Labels: pull-request-available > > It's possible in some cases to request zero records from Kinesis in the > Kinesis connector. This can happen when the "adpative reads" feature is > enabled. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10154) Make sure we always read at least one record in KinesisConnector
[ https://issues.apache.org/jira/browse/FLINK-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582704#comment-16582704 ] ASF GitHub Bot commented on FLINK-10154: StefanRRichter commented on issue #6564: [FLINK-10154] [kinesis connector] Make sure we always read at least one record in Kinesis Connector. URL: https://github.com/apache/flink/pull/6564#issuecomment-413587065 LGTM 👍 Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make sure we always read at least one record in KinesisConnector > > > Key: FLINK-10154 > URL: https://issues.apache.org/jira/browse/FLINK-10154 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Affects Versions: 1.6.0 >Reporter: Jamie Grier >Assignee: Jamie Grier >Priority: Minor > Labels: pull-request-available > > It's possible in some cases to request zero records from Kinesis in the > Kinesis connector. This can happen when the "adpative reads" feature is > enabled. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10154) Make sure we always read at least one record in KinesisConnector
[ https://issues.apache.org/jira/browse/FLINK-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581706#comment-16581706 ] ASF GitHub Bot commented on FLINK-10154: bowenli86 commented on issue #6564: [FLINK-10154] [kinesis connector] Make sure we always read at least one record in Kinesis Connector. URL: https://github.com/apache/flink/pull/6564#issuecomment-413363019 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make sure we always read at least one record in KinesisConnector > > > Key: FLINK-10154 > URL: https://issues.apache.org/jira/browse/FLINK-10154 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Affects Versions: 1.6.0 >Reporter: Jamie Grier >Assignee: Jamie Grier >Priority: Minor > Labels: pull-request-available > > It's possible in some cases to request zero records from Kinesis in the > Kinesis connector. This can happen when the "adpative reads" feature is > enabled. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10154) Make sure we always read at least one record in KinesisConnector
[ https://issues.apache.org/jira/browse/FLINK-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581414#comment-16581414 ] Jamie Grier commented on FLINK-10154: - PR: [https://github.com/apache/flink/pull/6564] > Make sure we always read at least one record in KinesisConnector > > > Key: FLINK-10154 > URL: https://issues.apache.org/jira/browse/FLINK-10154 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Affects Versions: 1.6.0 >Reporter: Jamie Grier >Assignee: Jamie Grier >Priority: Minor > Labels: pull-request-available > > It's possible in some cases to request zero records from Kinesis in the > Kinesis connector. This can happen when the "adpative reads" feature is > enabled. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10154) Make sure we always read at least one record in KinesisConnector
[ https://issues.apache.org/jira/browse/FLINK-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16581412#comment-16581412 ] ASF GitHub Bot commented on FLINK-10154: jgrier opened a new pull request #6564: [FLINK-10154] [kinesis connector] Make sure we always read at least one record in Kinesis Connector. URL: https://github.com/apache/flink/pull/6564 ## What is the purpose of the change Fixing a bug that can cause the Kinesis Connector to request zero records and therefore throw an exception. ## Brief change log One line fix to ensure a minimum Kinesis request size of 1 record. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Make sure we always read at least one record in KinesisConnector > > > Key: FLINK-10154 > URL: https://issues.apache.org/jira/browse/FLINK-10154 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Affects Versions: 1.6.0 >Reporter: Jamie Grier >Assignee: Jamie Grier >Priority: Minor > Labels: pull-request-available > > It's possible in some cases to request zero records from Kinesis in the > Kinesis connector. This can happen when the "adpative reads" feature is > enabled. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
