[jira] [Created] (FLINK-35174) Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink RabbitMQ connector
Danny Cranmer created FLINK-35174: - Summary: Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink RabbitMQ connector Key: FLINK-35174 URL: https://issues.apache.org/jira/browse/FLINK-35174 Project: Flink Issue Type: Technical Debt Components: Connectors / AWS Reporter: Danny Cranmer Assignee: Danny Cranmer Fix For: aws-connector-4.3.0 Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink AWS connectors -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35141) Release flink-connector-pulsar vX.X.X for Flink 1.19
Danny Cranmer created FLINK-35141: - Summary: Release flink-connector-pulsar vX.X.X for Flink 1.19 Key: FLINK-35141 URL: https://issues.apache.org/jira/browse/FLINK-35141 Project: Flink Issue Type: Sub-task Components: Connectors / Pulsar Reporter: Danny Cranmer https://github.com/apache/flink-connector-pulsar -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35134) Release flink-connector-elasticsearch vX.X.X for Flink 1.18/1.19
Danny Cranmer created FLINK-35134: - Summary: Release flink-connector-elasticsearch vX.X.X for Flink 1.18/1.19 Key: FLINK-35134 URL: https://issues.apache.org/jira/browse/FLINK-35134 Project: Flink Issue Type: Sub-task Components: Connectors / ElasticSearch Reporter: Danny Cranmer https://github.com/apache/flink-connector-elasticsearch -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35136) Release flink-connector-hbase vX.X.X for Flink 1.19
Danny Cranmer created FLINK-35136: - Summary: Release flink-connector-hbase vX.X.X for Flink 1.19 Key: FLINK-35136 URL: https://issues.apache.org/jira/browse/FLINK-35136 Project: Flink Issue Type: Sub-task Components: Connectors / HBase Reporter: Danny Cranmer -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35142) Release flink-connector-rabbitmq vX.X.X for Flink 1.19
Danny Cranmer created FLINK-35142: - Summary: Release flink-connector-rabbitmq vX.X.X for Flink 1.19 Key: FLINK-35142 URL: https://issues.apache.org/jira/browse/FLINK-35142 Project: Flink Issue Type: Sub-task Components: Connectors/ RabbitMQ Reporter: Danny Cranmer https://github.com/apache/flink-connector-rabbitmq -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35140) Release flink-connector-opensearch vX.X.X for Flink 1.19
Danny Cranmer created FLINK-35140: - Summary: Release flink-connector-opensearch vX.X.X for Flink 1.19 Key: FLINK-35140 URL: https://issues.apache.org/jira/browse/FLINK-35140 Project: Flink Issue Type: Sub-task Components: Connectors / Opensearch Reporter: Danny Cranmer -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35133) Release flink-connector-cassandra v4.3.0 for Flink 1.18/1.19
Danny Cranmer created FLINK-35133: - Summary: Release flink-connector-cassandra v4.3.0 for Flink 1.18/1.19 Key: FLINK-35133 URL: https://issues.apache.org/jira/browse/FLINK-35133 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer https://github.com/apache/flink-connector-cassandra -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35139) Release flink-connector-mongodb vX.X.X for Flink 1.19
Danny Cranmer created FLINK-35139: - Summary: Release flink-connector-mongodb vX.X.X for Flink 1.19 Key: FLINK-35139 URL: https://issues.apache.org/jira/browse/FLINK-35139 Project: Flink Issue Type: Sub-task Components: Connectors / MongoDB Reporter: Danny Cranmer https://github.com/apache/flink-connector-mongodb -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35138) Release flink-connector-kafka vX.X.X for Flink 1.19
Danny Cranmer created FLINK-35138: - Summary: Release flink-connector-kafka vX.X.X for Flink 1.19 Key: FLINK-35138 URL: https://issues.apache.org/jira/browse/FLINK-35138 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer https://github.com/apache/flink-connector-kafka -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35137) Release flink-connector-jdbc vX.X.X for Flink 1.19
Danny Cranmer created FLINK-35137: - Summary: Release flink-connector-jdbc vX.X.X for Flink 1.19 Key: FLINK-35137 URL: https://issues.apache.org/jira/browse/FLINK-35137 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer https://github.com/apache/flink-connector-jdbc -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35135) Release flink-connector-gcp-pubsub vX.X.X for Flink 1.18/1.19
Danny Cranmer created FLINK-35135: - Summary: Release flink-connector-gcp-pubsub vX.X.X for Flink 1.18/1.19 Key: FLINK-35135 URL: https://issues.apache.org/jira/browse/FLINK-35135 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer https://github.com/apache/flink-connector-gcp-pubsub -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35132) Release flink-connector-aws v4.3.0 for Flink 1.18/1.19
Danny Cranmer created FLINK-35132: - Summary: Release flink-connector-aws v4.3.0 for Flink 1.18/1.19 Key: FLINK-35132 URL: https://issues.apache.org/jira/browse/FLINK-35132 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer Fix For: aws-connector-4.3.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35131) Support and Release Connectors for Flink 1.19
Danny Cranmer created FLINK-35131: - Summary: Support and Release Connectors for Flink 1.19 Key: FLINK-35131 URL: https://issues.apache.org/jira/browse/FLINK-35131 Project: Flink Issue Type: Improvement Reporter: Danny Cranmer This is the parent task to contain connector support and releases for Flink 1.19. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35124) Connector Release Fails to run Checkstyle
Danny Cranmer created FLINK-35124:
-
Summary: Connector Release Fails to run Checkstyle
Key: FLINK-35124
URL: https://issues.apache.org/jira/browse/FLINK-35124
Project: Flink
Issue Type: Bug
Components: Build System
Reporter: Danny Cranmer
During a release of the AWS connectors the build was failing at the
\{{./tools/releasing/shared/stage_jars.sh}} step due to a checkstyle error.
{code:java}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-checkstyle-plugin:3.1.2:check (validate) on
project flink-connector-aws: Failed during checkstyle execution: Unable to find
suppressions file at location: /tools/maven/suppressions.xml: Could not find
resource '/tools/maven/suppressions.xml'. -> [Help 1] {code}
Looks like it is caused by this
[https://github.com/apache/flink-connector-shared-utils/commit/a75b89ee3f8c9a03e97ead2d0bd9d5b7bb02b51a]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35113) Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink AWS connectors
Danny Cranmer created FLINK-35113: - Summary: Bump org.apache.commons:commons-compress from 1.25.0 to 1.26.1 for Flink AWS connectors Key: FLINK-35113 URL: https://issues.apache.org/jira/browse/FLINK-35113 Project: Flink Issue Type: Technical Debt Components: Connectors / Kafka Reporter: Martijn Visser Assignee: Martijn Visser Fix For: kafka-4.0.0, kafka-3.1.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34342) Address ListShards Consistency for DDB Streams Source
Danny Cranmer created FLINK-34342: - Summary: Address ListShards Consistency for DDB Streams Source Key: FLINK-34342 URL: https://issues.apache.org/jira/browse/FLINK-34342 Project: Flink Issue Type: Sub-task Components: Connectors / DynamoDB Reporter: Danny Cranmer -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34341) Implement DDB Streams Table API support
Danny Cranmer created FLINK-34341: - Summary: Implement DDB Streams Table API support Key: FLINK-34341 URL: https://issues.apache.org/jira/browse/FLINK-34341 Project: Flink Issue Type: Sub-task Components: Connectors / Kinesis Reporter: Hong Liang Teoh Implement Table API support for KDS Source. Consider: * Configurations to support. Should have customisation parity with DataStream API * Testing should include both SQL client + Table API via Java -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34340) Add support for DDB Streams
Danny Cranmer created FLINK-34340:
-
Summary: Add support for DDB Streams
Key: FLINK-34340
URL: https://issues.apache.org/jira/browse/FLINK-34340
Project: Flink
Issue Type: Sub-task
Components: Connectors / DynamoDB
Reporter: Danny Cranmer
In the legacy KDS source we support Amazon DynamoDB streams via an adapter
shim. Both KDS and DDB streams have a similar API.
This task builds upon https://issues.apache.org/jira/browse/FLINK-34339 and
will add a {{DynamoDBStreamsSource}} which will setup a DDB SDK client shim.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34339) Add connector abstraction layer to remove reliance on AWWS SDK classes
Danny Cranmer created FLINK-34339:
-
Summary: Add connector abstraction layer to remove reliance on
AWWS SDK classes
Key: FLINK-34339
URL: https://issues.apache.org/jira/browse/FLINK-34339
Project: Flink
Issue Type: Sub-task
Reporter: Danny Cranmer
In order to shim DDB streams we need to be able to support the
Stream/Shard/Record etc concepts without tying to a specific implementation.
This will allow us to mimic the KDS/DDB streams support in the old connector by
providing a shim at the AWS SDK client.
# Model {{software.amazon.awssdk.services.kinesis}} classes as native concepts
# Push down any usage of {{software.amazon.awssdk.services.kinesis}} to a KDS
specific class
# Ensure that the bulk of the connector logic is reusable, the top level class
would be implementation specific and shim in the write client factories and
configuration
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34024) Update connector release process for Python connectors
Danny Cranmer created FLINK-34024: - Summary: Update connector release process for Python connectors Key: FLINK-34024 URL: https://issues.apache.org/jira/browse/FLINK-34024 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer Work out how to release the Python libs for Flink connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33413) Bump Avro in AWS Connectors to address CVE-2023-39410
Danny Cranmer created FLINK-33413: - Summary: Bump Avro in AWS Connectors to address CVE-2023-39410 Key: FLINK-33413 URL: https://issues.apache.org/jira/browse/FLINK-33413 Project: Flink Issue Type: Bug Components: Connectors / AWS Affects Versions: aws-connector-4.1.0 Reporter: Danny Cranmer Fix For: aws-connector-4.2.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33409) Bump Guava to address CVE-2020-8908/CVE-2023-2976
Danny Cranmer created FLINK-33409:
-
Summary: Bump Guava to address CVE-2020-8908/CVE-2023-2976
Key: FLINK-33409
URL: https://issues.apache.org/jira/browse/FLINK-33409
Project: Flink
Issue Type: Bug
Components: Connectors / AWS
Affects Versions: aws-connector-4.1.0, aws-connector-3.0.0
Reporter: Danny Cranmer
Fix For: aws-connector-4.2.0
Bump Guava from {{32.0.0-jre}} to {{32.1.3-jre}} to mitigate
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33279) KinesisStream e2e test failing on PR during cleanup
Danny Cranmer created FLINK-33279: - Summary: KinesisStream e2e test failing on PR during cleanup Key: FLINK-33279 URL: https://issues.apache.org/jira/browse/FLINK-33279 Project: Flink Issue Type: Bug Components: Connectors / Kinesis Reporter: Danny Cranmer We recently added e2e tests to hit the real KDS (AWS) endpoints on CI. We added a separate sanity test to cleanup streams that were not gracefully deleted. This tests is running on PR and it should not be, resulting in failing actions runs: https://github.com/apache/flink-connector-aws/actions/runs/6524289454/job/17731091860?pr=105 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33260) Custom Error Handling for Kinesis Consumer
Danny Cranmer created FLINK-33260: - Summary: Custom Error Handling for Kinesis Consumer Key: FLINK-33260 URL: https://issues.apache.org/jira/browse/FLINK-33260 Project: Flink Issue Type: Improvement Components: Connectors / Kinesis Reporter: Danny Cranmer Background The Kinesis Consumer exposes various configuration that allows the user to define retry and backoff strategies when dealing with errors. However, the configuration does not allow the user to configure which errors are retryable, or different strategies for different errors. The error handling logic is hard coded within the connector. Over time we discover errors that should be retryable that are not, for example KDS throwing 500 on SubscribeToShare or transient DNS issues. h3. Scope Add the ability for the user to define retry/backoff strategy per error. This could be achieved using flexible configuration keys, or allowing the user to register their own retry strategies on the connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33151) Prometheus Sink Connector - Create Github Repo
Danny Cranmer created FLINK-33151:
-
Summary: Prometheus Sink Connector - Create Github Repo
Key: FLINK-33151
URL: https://issues.apache.org/jira/browse/FLINK-33151
Project: Flink
Issue Type: Sub-task
Reporter: Danny Cranmer
Create the \{{flink-connector-prometheus}} repo
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33048) Project website refresh Light Mode
Danny Cranmer created FLINK-33048: - Summary: Project website refresh Light Mode Key: FLINK-33048 URL: https://issues.apache.org/jira/browse/FLINK-33048 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33049) Project website refresh Code Samples Widget
Danny Cranmer created FLINK-33049: - Summary: Project website refresh Code Samples Widget Key: FLINK-33049 URL: https://issues.apache.org/jira/browse/FLINK-33049 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33047) Project website refresh Dark Mode
Danny Cranmer created FLINK-33047: - Summary: Project website refresh Dark Mode Key: FLINK-33047 URL: https://issues.apache.org/jira/browse/FLINK-33047 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33046) [FLIP-333] Redesign Apache Flink website
Danny Cranmer created FLINK-33046: - Summary: [FLIP-333] Redesign Apache Flink website Key: FLINK-33046 URL: https://issues.apache.org/jira/browse/FLINK-33046 Project: Flink Issue Type: New Feature Components: Project Website Reporter: Danny Cranmer Apply styling as per https://cwiki.apache.org/confluence/display/FLINK/FLIP-333%3A+Redesign+Apache+Flink+website -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33043) Search is broken on the Flink website
Danny Cranmer created FLINK-33043: - Summary: Search is broken on the Flink website Key: FLINK-33043 URL: https://issues.apache.org/jira/browse/FLINK-33043 Project: Flink Issue Type: Bug Components: Project Website Reporter: Danny Cranmer The Search box on [https://flink.apache.org/] does not work. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31918) Pulsar Source does not failing build against Flink 1.18 on nightly CI
Danny Cranmer created FLINK-31918:
-
Summary: Pulsar Source does not failing build against Flink 1.18
on nightly CI
Key: FLINK-31918
URL: https://issues.apache.org/jira/browse/FLINK-31918
Project: Flink
Issue Type: Bug
Components: Connectors / Pulsar
Reporter: Danny Cranmer
[https://github.com/apache/flink-connector-pulsar/actions/runs/4783897408/jobs/8504710249]
{{Error: Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile)
on project flink-connector-pulsar: Compilation failure }}
{{[150|https://github.com/apache/flink-connector-pulsar/actions/runs/4783897408/jobs/8504710249#step:13:151]Error:
/home/runner/work/flink-connector-pulsar/flink-connector-pulsar/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/reader/PulsarSourceFetcherManager.java:[52,8]
org.apache.flink.connector.pulsar.source.reader.PulsarSourceFetcherManager is
not abstract and does not override abstract method
removeSplits(java.util.List)
in org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31847) Flink Connector JDBC CI Failing
Danny Cranmer created FLINK-31847: - Summary: Flink Connector JDBC CI Failing Key: FLINK-31847 URL: https://issues.apache.org/jira/browse/FLINK-31847 Project: Flink Issue Type: Technical Debt Components: Connectors / JDBC Reporter: Danny Cranmer The CI is failing, or Flaky: * [https://github.com/apache/flink-connector-jdbc/actions/runs/4712658186|https://github.com/apache/flink-connector-jdbc/pull/41] * https://github.com/apache/flink-connector-jdbc/actions/runs/4710470564 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31793) Remove dependency on flink-shaded for flink-connector-jdbc
Danny Cranmer created FLINK-31793: - Summary: Remove dependency on flink-shaded for flink-connector-jdbc Key: FLINK-31793 URL: https://issues.apache.org/jira/browse/FLINK-31793 Project: Flink Issue Type: Technical Debt Components: Connectors / Cassandra Reporter: Martijn Visser Assignee: Etienne Chauchot Fix For: cassandra-4.0.0, cassandra-3.0.1 The Cassandra connector relies on flink-shaded and uses Flinks' shaded Guava. With the externalization of connector, connectors shouldn't rely on Flink-Shaded but instead shade dependencies such as this one themselves -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31704) Pulsar docs should be pulled from dedicated branch
Danny Cranmer created FLINK-31704: - Summary: Pulsar docs should be pulled from dedicated branch Key: FLINK-31704 URL: https://issues.apache.org/jira/browse/FLINK-31704 Project: Flink Issue Type: Technical Debt Components: Connectors / Pulsar, Documentation Reporter: Danny Cranmer -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31703) Update Flink docs for AWS v4.1.0
Danny Cranmer created FLINK-31703: - Summary: Update Flink docs for AWS v4.1.0 Key: FLINK-31703 URL: https://issues.apache.org/jira/browse/FLINK-31703 Project: Flink Issue Type: Improvement Components: Connectors / Opensearch, Documentation Reporter: Danny Cranmer Assignee: Danny Cranmer Fix For: 1.18.0, 1.17.1 Update Flink docs for 1.16 to pull in the opensearch docs from flink-connector-opensearch repo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31702) Integrate Opensearch connector docs into Flink docs v1.17/master
Danny Cranmer created FLINK-31702: - Summary: Integrate Opensearch connector docs into Flink docs v1.17/master Key: FLINK-31702 URL: https://issues.apache.org/jira/browse/FLINK-31702 Project: Flink Issue Type: Improvement Components: Connectors / Opensearch, Documentation Reporter: Danny Cranmer Assignee: Danny Cranmer Fix For: 1.16.1 Update Flink docs for 1.16 to pull in the opensearch docs from flink-connector-opensearch repo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31701) RabbitMQ nightly CI failure
Danny Cranmer created FLINK-31701:
-
Summary: RabbitMQ nightly CI failure
Key: FLINK-31701
URL: https://issues.apache.org/jira/browse/FLINK-31701
Project: Flink
Issue Type: Technical Debt
Components: Connectors / MongoDB
Reporter: Danny Cranmer
Investigate and fix the nightly CI failure. Example
https://github.com/apache/flink-connector-mongodb/actions/runs/4585933750
{code:java}
Error:
/home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/sink/writer/context/DefaultMongoSinkContext.java:[33,8]
org.apache.flink.connector.mongodb.sink.writer.context.DefaultMongoSinkContext
is not abstract and does not override abstract method getAttemptNumber() in
org.apache.flink.api.connector.sink2.Sink.InitContext{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31700) MongoDB nightly CI failure
Danny Cranmer created FLINK-31700:
-
Summary: MongoDB nightly CI failure
Key: FLINK-31700
URL: https://issues.apache.org/jira/browse/FLINK-31700
Project: Flink
Issue Type: Technical Debt
Components: Connectors / JDBC
Reporter: Danny Cranmer
Investigate and fix the nightly CI failure. Example
[https://github.com/apache/flink-connector-jdbc/actions/runs/4585903259]
{code:java}
Error: Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on
project flink-connector-jdbc: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test failed:
org.junit.platform.commons.JUnitException: TestEngine with ID 'archunit' failed
to discover tests:
com.tngtech.archunit.lang.syntax.elements.MethodsThat.areAnnotatedWith(Ljava/lang/Class;)Ljava/lang/Object;
-> [Help 1]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31699) JDBC nightly CI failure
Danny Cranmer created FLINK-31699:
-
Summary: JDBC nightly CI failure
Key: FLINK-31699
URL: https://issues.apache.org/jira/browse/FLINK-31699
Project: Flink
Issue Type: Technical Debt
Components: Connectors / Cassandra
Reporter: Danny Cranmer
Investigate and fix the nightly CI failure. Example
[https://github.com/apache/flink-connector-cassandra/actions/runs/4585936901]
{code:java}
Error: Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on
project flink-connector-cassandra_2.12: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test failed:
org.junit.platform.commons.JUnitException: TestEngine with ID 'junit-jupiter'
failed to discover tests: org/junit/jupiter/api/io/CleanupMode:
org.junit.jupiter.api.io.CleanupMode -> [Help 1]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31698) Cassandra nightly CI failure
Danny Cranmer created FLINK-31698:
-
Summary: Cassandra nightly CI failure
Key: FLINK-31698
URL: https://issues.apache.org/jira/browse/FLINK-31698
Project: Flink
Issue Type: Technical Debt
Components: Connectors / Opensearch
Reporter: Danny Cranmer
Investigate and fix the nightly CI failure. Example
[https://github.com/apache/flink-connector-opensearch/actions/runs/4585851921]
{code:java}
Error: Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on
project flink-connector-opensearch: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test failed:
org.junit.platform.commons.JUnitException: TestEngine with ID 'archunit' failed
to discover tests: 'java.lang.Object
com.tngtech.archunit.lang.syntax.elements.MethodsThat.areAnnotatedWith(java.lang.Class)'
-> [Help 1]{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31697) OpenSearch nightly CI failure
Danny Cranmer created FLINK-31697:
-
Summary: OpenSearch nightly CI failure
Key: FLINK-31697
URL: https://issues.apache.org/jira/browse/FLINK-31697
Project: Flink
Issue Type: Technical Debt
Components: Connectors / ElasticSearch
Reporter: Danny Cranmer
Investigate and fix the nightly CI failure. Example
[https://github.com/apache/flink-connector-elasticsearch/actions/runs/4585918498/jobs/8098357503]
{code:java}
Error: Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on
project flink-connector-elasticsearch-base: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test failed:
org.junit.platform.commons.JUnitException: TestEngine with ID 'archunit' failed
to discover tests: 'java.lang.Object
com.tngtech.archunit.lang.syntax.elements.MethodsThat.areAnnotatedWith(java.lang.Class)'
-> [Help 1] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31696) ElasticSearch nightly CI failure
Danny Cranmer created FLINK-31696:
-
Summary: ElasticSearch nightly CI failure
Key: FLINK-31696
URL: https://issues.apache.org/jira/browse/FLINK-31696
Project: Flink
Issue Type: Technical Debt
Components: Connectors / ElasticSearch
Reporter: Danny Cranmer
Investigate and fix the nightly CI failure. Example
[https://github.com/apache/flink-connector-elasticsearch/actions/runs/4585918498/jobs/8098357503]
{code:java}
Error: Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on
project flink-connector-elasticsearch-base: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test failed:
org.junit.platform.commons.JUnitException: TestEngine with ID 'archunit' failed
to discover tests: 'java.lang.Object
com.tngtech.archunit.lang.syntax.elements.MethodsThat.areAnnotatedWith(java.lang.Class)'
-> [Help 1] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31692) Integrate MongoDB connector docs into Flink website
Danny Cranmer created FLINK-31692: - Summary: Integrate MongoDB connector docs into Flink website Key: FLINK-31692 URL: https://issues.apache.org/jira/browse/FLINK-31692 Project: Flink Issue Type: Improvement Components: Connectors / MongoDB Reporter: Danny Cranmer Fix For: 1.18.0 Update Flink docs build to pull the MongoDB connector docs [1] [1] https://github.com/apache/flink-connector-mongodb -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31183) Flink Kinesis EFO Consumer can fail to stop gracefully
Danny Cranmer created FLINK-31183:
-
Summary: Flink Kinesis EFO Consumer can fail to stop gracefully
Key: FLINK-31183
URL: https://issues.apache.org/jira/browse/FLINK-31183
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: aws-connector-4.0.0, 1.16.1, aws-connector-3.0.0, 1.15.3
Reporter: Danny Cranmer
Fix For: 1.17.0, 1.15.4, aws-connector-4.1.0, 1.16.2
*Background*
When stopping a Flink job using the stop-with-savepoint API the EFO Kinesis
source can fail to close gracefully.
Sample stack trace
{code:java}
2023-02-16 20:45:40
org.apache.flink.runtime.checkpoint.CheckpointException: Task has failed.
at
org.apache.flink.runtime.messages.checkpoint.SerializedCheckpointException.unwrap(SerializedCheckpointException.java:51)
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:1013)
at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$declineCheckpoint$2(ExecutionGraphHandler.java:103)
at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.flink.runtime.checkpoint.CheckpointException: Task name
with subtask : Source: vas_source_stream (38/48)#0 Failure reason: Task has
failed.
at
org.apache.flink.runtime.taskmanager.Task.declineCheckpoint(Task.java:1395)
at
org.apache.flink.runtime.taskmanager.Task.lambda$triggerCheckpointBarrier$3(Task.java:1338)
at
java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
at
java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:343)
Caused by: java.util.concurrent.CompletionException:
java.util.concurrent.RejectedExecutionException: event executor terminated
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
... 3 more
Caused by: java.util.concurrent.RejectedExecutionException: event executor
terminated
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:923)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:350)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:343)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:825)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:815)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription.cancel(HandlerPublisher.java:502)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.async.DelegatingSubscription.cancel(DelegatingSubscription.java:37)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.http2.Http2ResetSendingSubscription.cancel(Http2ResetSendingSubscription.java:41)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.async.DelegatingSubscription.cancel(DelegatingSubscription.java:37)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$OnCancelSubscription.cancel(ResponseHandler.java:409)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.async.FlatteningSubscriber$1.cancel(FlatteningSubscriber.java:98)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.async.FlatteningSubscriber.handleStateUpdate(FlatteningSubscriber.java:170)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.utils.async.FlatteningSubscriber.access$100(FlatteningSubscriber.java:29)
at
org.apache.flink.kinesis.shaded
[jira] [Created] (FLINK-31041) Race condition in DefaultScheduler results in memory leak and busy loop
Danny Cranmer created FLINK-31041:
-
Summary: Race condition in DefaultScheduler results in memory leak
and busy loop
Key: FLINK-31041
URL: https://issues.apache.org/jira/browse/FLINK-31041
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.16.1, 1.15.3
Reporter: Danny Cranmer
Fix For: 1.17.0, 1.15.4, 1.16.2
h4. Context
When a job creates multiple sources that use the {{SourceCoordinator}}
(FLIP-27), there is a failure race condition that results in:
* Memory leak of \{{ExecutionVertexVersion}}
* Busy loop constantly trying to restart job
* Restart strategy is not respected
This results in the Job Manager becoming unresponsive.
h4. Reproduction Steps
This can be reproduced by a job that creates multiple sources that fail in the
{{{}SplitEnumerator{}}}. We observed this with multiple {{KafkaSource's}}
trying to load a non-existent cert from the file system and throwing FNFE.
Thus, here is a simple job to reproduce (BE WARNED: running this locally will
lock up your IDE):
{code:java}
StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
env.setRestartStrategy(new
RestartStrategies.FailureRateRestartStrategyConfiguration(1, Time.of(10,
TimeUnit.SECONDS), Time.of(10, TimeUnit.SECONDS)));
KafkaSource source = KafkaSource.builder()
.setProperty("security.protocol", "SASL_SSL")
// SSL configurations
// Configure the path of truststore (CA) provided by the server
.setProperty("ssl.truststore.location",
"/path/to/kafka.client.truststore.jks")
.setProperty("ssl.truststore.password", "test1234")
// Configure the path of keystore (private key) if client
authentication is required
.setProperty("ssl.keystore.location",
"/path/to/kafka.client.keystore.jks")
.setProperty("ssl.keystore.password", "test1234")
// SASL configurations
// Set SASL mechanism as SCRAM-SHA-256
.setProperty("sasl.mechanism", "SCRAM-SHA-256")
// Set JAAS configurations
.setProperty("sasl.jaas.config",
"org.apache.kafka.common.security.scram.ScramLoginModule required
username=\"username\" password=\"password\";")
.setBootstrapServers("http://localhost:3456";)
.setTopics("input-topic")
.setGroupId("my-group")
.setStartingOffsets(OffsetsInitializer.earliest())
.setValueOnlyDeserializer(new SimpleStringSchema())
.build();
List> sources = IntStream.range(0, 32)
.mapToObj(i -> env
.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka
Source " + i).uid("source-" + i)
.keyBy(s -> s.charAt(0))
.map(s -> s))
.collect(Collectors.toList());
env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka
Source").uid("source")
.keyBy(s -> s.charAt(0))
.union(sources.toArray(new SingleOutputStreamOperator[] {}))
.print();
env.execute("test job"); {code}
h4. Root Cause
We can see that the {{OperatorCoordinatorHolder}} already has a [debounce
mechanism|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/operators/coordination/OperatorCoordinatorHolder.java#L609],
however the default scheduler does not, and processes many
{{OperatorCoordinatorHolder}}.
h4. Fix
I have managed to fix this, I will open a PR, but would need feedback from
people who understand this code better than me!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30633) Update AWS SDKv2 to v2.19.14
Danny Cranmer created FLINK-30633: - Summary: Update AWS SDKv2 to v2.19.14 Key: FLINK-30633 URL: https://issues.apache.org/jira/browse/FLINK-30633 Project: Flink Issue Type: Improvement Components: Connectors / AWS, Connectors / DynamoDB, Connectors / Firehose, Connectors / Kinesis Reporter: Danny Cranmer Fix For: 1.17.0, 1.16.1, 1.15.4, aws-connector-4.1.0, aws-connector-3.1.0 Update AWS SDK to latest version since it fixes a bug: - https://issues.apache.org/jira/browse/FLINK-30304 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30557) Remove flink-connector-aws-kinesis-streams from Flink master branch
Danny Cranmer created FLINK-30557:
-
Summary: Remove flink-connector-aws-kinesis-streams from Flink
master branch
Key: FLINK-30557
URL: https://issues.apache.org/jira/browse/FLINK-30557
Project: Flink
Issue Type: Sub-task
Reporter: Danny Cranmer
Assignee: Danny Cranmer
Remove:
* {{flink-connector-aws-kinesis-firehose}}
* {{flink-sql-connector-aws-kinesis-firehose}}
{{Also remove documentation }}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30550) Flink master to integrate KDS/KDF docs from flink-connector-aws
Danny Cranmer created FLINK-30550: - Summary: Flink master to integrate KDS/KDF docs from flink-connector-aws Key: FLINK-30550 URL: https://issues.apache.org/jira/browse/FLINK-30550 Project: Flink Issue Type: Technical Debt Components: Connectors / AWS Affects Versions: 1.16.0 Reporter: Danny Cranmer Assignee: Danny Cranmer Fix For: 1.16.1 KDS/KDF connectors have been externalized to the flink-connector-aws repo. Update the docs to reflect this -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30549) Remove flink-connector-firehose from Flink master branch
Danny Cranmer created FLINK-30549: - Summary: Remove flink-connector-firehose from Flink master branch Key: FLINK-30549 URL: https://issues.apache.org/jira/browse/FLINK-30549 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30436) Integrate Opensearch connector docs into Flink docs v1.16
Danny Cranmer created FLINK-30436: - Summary: Integrate Opensearch connector docs into Flink docs v1.16 Key: FLINK-30436 URL: https://issues.apache.org/jira/browse/FLINK-30436 Project: Flink Issue Type: Improvement Components: Connectors / Opensearch, Documentation Reporter: Danny Cranmer Fix For: 1.16.1 Update Flink docs for 1.16 to pull in the opensearch docs from flink-connector-opensearch repo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30388) Add support for ElementConverted open() method for KDS/KDF/DDB
Danny Cranmer created FLINK-30388:
-
Summary: Add support for ElementConverted open() method for
KDS/KDF/DDB
Key: FLINK-30388
URL: https://issues.apache.org/jira/browse/FLINK-30388
Project: Flink
Issue Type: Technical Debt
Components: Connectors / DynamoDB, Connectors / Firehose, Connectors
/ Kinesis
Reporter: Danny Cranmer
FLINK-29938 added support for an {{open()}} method in Async Sink
ElementConverter. Once flink-connector-aws upgrades to Flink 1.17 we should
implement this method. It was originally implemented
[here|http://example.com|https://github.com/apache/flink/pull/21265] but was
yanked during the [sync|FLINK-30384]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30385) Remove KDS/KDF/AWS Base connectors from master branch
Danny Cranmer created FLINK-30385: - Summary: Remove KDS/KDF/AWS Base connectors from master branch Key: FLINK-30385 URL: https://issues.apache.org/jira/browse/FLINK-30385 Project: Flink Issue Type: Technical Debt Reporter: Danny Cranmer KDS/KDF/AWS Base connectors have been externalised and can now be removed from the master branch -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30384) Sync master KDS/KDF connector changes to flink-connector-aws
Danny Cranmer created FLINK-30384: - Summary: Sync master KDS/KDF connector changes to flink-connector-aws Key: FLINK-30384 URL: https://issues.apache.org/jira/browse/FLINK-30384 Project: Flink Issue Type: Technical Debt Components: Connectors / Firehose, Connectors / Kinesis Reporter: Danny Cranmer Fix For: aws-connector-4.1.0 Update flink-connector-aws to take the KDS/KDF/AWS base and docs changes from the Flink codebase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30382) Flink 1.16 to integrate KDS/KDF docs from flink-connector-aws
Danny Cranmer created FLINK-30382: - Summary: Flink 1.16 to integrate KDS/KDF docs from flink-connector-aws Key: FLINK-30382 URL: https://issues.apache.org/jira/browse/FLINK-30382 Project: Flink Issue Type: Technical Debt Components: Connectors / AWS Affects Versions: 1.16.0 Reporter: Danny Cranmer Fix For: 1.16.1 KDS/KDF connectors have been externalized to the flink-connector-aws repo. Update the docs to reflect this -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30356) Update NOTICE files to say "Apache Flink AWS connectors"
Danny Cranmer created FLINK-30356: - Summary: Update NOTICE files to say "Apache Flink AWS connectors" Key: FLINK-30356 URL: https://issues.apache.org/jira/browse/FLINK-30356 Project: Flink Issue Type: Technical Debt Components: Connectors / AWS Affects Versions: aws-connector-3.0.0, aws-connector-4.0.0 Reporter: Danny Cranmer Fix For: aws-connector-4.1.0, aws-connector-3.1.0 [https://lists.apache.org/thread/8bb8kh3w5ohztj50k4cgsqt97466t9fj] Update all NOTICE files as per: - "Not a blocking issue, but the source NOTICE currently says "Apache Flink", where it should say "Apache Flink AWS connector" or something." -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30304) Possible Deadlock in Kinesis/Firehose/DynamoDB Connector
Danny Cranmer created FLINK-30304:
-
Summary: Possible Deadlock in Kinesis/Firehose/DynamoDB Connector
Key: FLINK-30304
URL: https://issues.apache.org/jira/browse/FLINK-30304
Project: Flink
Issue Type: Technical Debt
Reporter: Danny Cranmer
Fix For: aws-connector-4.0.0
AWS Sinks based on Async Sink can enter a deadlock situation if the AWS async
client throws error outside of the future. We observed this with a local
application:
{code:java}
java.lang.NullPointerException
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.closedChannelMessage(NettyUtils.java:135)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.decorateException(NettyUtils.java:71)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:310)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:189)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.CancellableAcquireChannelPool.lambda$acquire$1(CancellableAcquireChannelPool.java:58)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.ensureAcquiredChannelIsHealthy(HealthCheckedChannelPool.java:114)
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.lambda$tryAcquire$1(HealthCheckedChannelPool.java:97)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:502)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at
org.apache.flink.kinesis.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at
org.apache.flink.kinesis.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at
org.apache.flink.kinesis.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Thread.java:829){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30291) Integrate flink-connector-aws into Flink docs
Danny Cranmer created FLINK-30291:
-
Summary: Integrate flink-connector-aws into Flink docs
Key: FLINK-30291
URL: https://issues.apache.org/jira/browse/FLINK-30291
Project: Flink
Issue Type: Technical Debt
Components: Connectors / AWS, Documentation
Reporter: Danny Cranmer
Fix For: 1.17.0, 1.16.1
Update the docs render to integrate {{{}flink-connector-aws{}}}.
Add a new shortcode to handle rendering the SQL connector correctly
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30230) Drop Support for Flink 1.15
Danny Cranmer created FLINK-30230: - Summary: Drop Support for Flink 1.15 Key: FLINK-30230 URL: https://issues.apache.org/jira/browse/FLINK-30230 Project: Flink Issue Type: Improvement Components: Connectors / AWS Reporter: Danny Cranmer Fix For: aws-connector-4.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30229) SQL IT case for DynamoDB Sink
Danny Cranmer created FLINK-30229: - Summary: SQL IT case for DynamoDB Sink Key: FLINK-30229 URL: https://issues.apache.org/jira/browse/FLINK-30229 Project: Flink Issue Type: Improvement Components: Connectors / DynamoDB Reporter: Danny Cranmer Fix For: aws-connector-2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30008) Add Flink 1.16.0 Support
Danny Cranmer created FLINK-30008: - Summary: Add Flink 1.16.0 Support Key: FLINK-30008 URL: https://issues.apache.org/jira/browse/FLINK-30008 Project: Flink Issue Type: Sub-task Components: Connectors / AWS Reporter: Danny Cranmer Fix For: aws-connector-2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29960) Update README to Reflect AWS Connectors Single Repo
Danny Cranmer created FLINK-29960:
-
Summary: Update README to Reflect AWS Connectors Single Repo
Key: FLINK-29960
URL: https://issues.apache.org/jira/browse/FLINK-29960
Project: Flink
Issue Type: Sub-task
Components: Connectors / AWS
Reporter: Danny Cranmer
Fix For: aws-connector-2.0.0
We recently renamed and pivoted the {{flink-connectors-dynamodb}} repository to
be a more general {{{}flink-connectors-aws{}}}. Update the README to reflect
these changes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29954) Improve DynamoDB Connector Sample Code
Danny Cranmer created FLINK-29954:
-
Summary: Improve DynamoDB Connector Sample Code
Key: FLINK-29954
URL: https://issues.apache.org/jira/browse/FLINK-29954
Project: Flink
Issue Type: Improvement
Components: Connectors / DynamoDB
Reporter: Danny Cranmer
Fix For: aws-connector-2.0.0
As per https://lists.apache.org/thread/lhvhsyrl3o88lsb6so8158nz6qogsqvn.
The {{SinkIntoDynamoDb}} sample app should not use AWS SDK types between
operators of the Flink job graph
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29938) Add open() Method to AsyncSink ElementConverter
Danny Cranmer created FLINK-29938:
-
Summary: Add open() Method to AsyncSink ElementConverter
Key: FLINK-29938
URL: https://issues.apache.org/jira/browse/FLINK-29938
Project: Flink
Issue Type: Improvement
Components: Connectors / Common
Reporter: Danny Cranmer
The {{ElementConverter}} is used to convert records to objects that a Sink can
deliver to a destination. In some sink implementations, the
{{ElementConverter}} needs to be serialized and sent to TM, DynamoDB is a good
example. For DynamoDB we need to lazily instantiate some objects, and an
`open()` method would provide a clean hook for this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29937) Enhanced DynamoDB Element Converter
Danny Cranmer created FLINK-29937:
-
Summary: Enhanced DynamoDB Element Converter
Key: FLINK-29937
URL: https://issues.apache.org/jira/browse/FLINK-29937
Project: Flink
Issue Type: Improvement
Components: Connectors / DynamoDB
Reporter: Danny Cranmer
Fix For: aws-connector-2.0.0
The new DynamoDB sink requires the user to supply an ElementConverter to
convert their POJO into a {{DynamoDbWriteRequest}}. DynamoDB provide an
"enhanced client" that supports annotations on your model class.
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/examples-dynamodb-enhanced.html
Add a new element converter, that can be optionally used to convert a
{{@DynamoDbBean}} to a {{DynamoDbWriteRequest}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29908) Externalize and configure E2E tests
Danny Cranmer created FLINK-29908: - Summary: Externalize and configure E2E tests Key: FLINK-29908 URL: https://issues.apache.org/jira/browse/FLINK-29908 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer Migrate Amazon Kinesis and Firehose E2E test modules from Flink core to flink-connector-aws -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29907) Externalize AWS connectors from Flink core
Danny Cranmer created FLINK-29907:
-
Summary: Externalize AWS connectors from Flink core
Key: FLINK-29907
URL: https://issues.apache.org/jira/browse/FLINK-29907
Project: Flink
Issue Type: Sub-task
Components: Connectors / AWS
Reporter: Danny Cranmer
Fix For: aws-connector-2.0.0
Externlize the following modules from Flink core to the connectors repo:
- {{flink-connector-aws-base}}
- {{flink-connector-kinesis}}
- {{flink-connector-sql-kinesis}}
- {{flink-connector-aws-kinesis-streams}}
- {{flink-connector-sql-aws-kinesis-streams}}
- {{flink-connector-aws-kinesis-firehose}}
- {{flink-connector-sql-aws-kinesis-firehose}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29906) Address tabs/spaces in checkstyle/spotless
Danny Cranmer created FLINK-29906: - Summary: Address tabs/spaces in checkstyle/spotless Key: FLINK-29906 URL: https://issues.apache.org/jira/browse/FLINK-29906 Project: Flink Issue Type: Sub-task Components: Connectors / AWS Reporter: Danny Cranmer The DynamoDB connector has a mix of tabs and spaces, the quality config does not seem to be enforcing anything. The code style guide says we should [use spaces|https://flink.apache.org/contributing/code-style-and-quality-formatting.html#whitespaces]. - Update code to use spaces - Fix quality plugin to reject tabs -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29905) Migrate flink-connector-dynamodb-parent to flink-connector-aws-parent
Danny Cranmer created FLINK-29905:
-
Summary: Migrate flink-connector-dynamodb-parent to
flink-connector-aws-parent
Key: FLINK-29905
URL: https://issues.apache.org/jira/browse/FLINK-29905
Project: Flink
Issue Type: Sub-task
Reporter: Danny Cranmer
Update {{flink-connector-dynamodb-parent}} pom to be more general for the AWS
connector repo
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29896) DynamoDB CI Failing to run checks
Danny Cranmer created FLINK-29896: - Summary: DynamoDB CI Failing to run checks Key: FLINK-29896 URL: https://issues.apache.org/jira/browse/FLINK-29896 Project: Flink Issue Type: Bug Components: Connectors / DynamoDB Reporter: Danny Cranmer Fix For: aws-connector-2.0.0 The checks are failing to actually run the build and test step: - https://github.com/apache/flink-connector-aws/actions/runs/3396739520/jobs/5648279513 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29894) Rename flink-connector-dynamodb to flink-connector-aws
Danny Cranmer created FLINK-29894:
-
Summary: Rename flink-connector-dynamodb to flink-connector-aws
Key: FLINK-29894
URL: https://issues.apache.org/jira/browse/FLINK-29894
Project: Flink
Issue Type: Sub-task
Reporter: Danny Cranmer
The existing
{{[flink-connector-dynamodb|[http://example.com|https://github.com/apache/flink-connector-dynamodb]]}}
repository should be renamed to {{flink-connector-aws}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29854) Make Record Size Flush Strategy Optional for Async Sink
Danny Cranmer created FLINK-29854:
-
Summary: Make Record Size Flush Strategy Optional for Async Sink
Key: FLINK-29854
URL: https://issues.apache.org/jira/browse/FLINK-29854
Project: Flink
Issue Type: Improvement
Components: Connectors / Common
Reporter: Danny Cranmer
h3. Background
Currently AsyncSinkWriter supports three mechanisms that trigger a flush to the
destination:
* TIme based
* Batch size in bytes
* Number of records in the batch
For "batch size in bytes" one must implement
[getSizeInBytes|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L202]
in order for the base to calculate the total batch size. In some cases
computing the batch size within the AsyncSinkWriter is an expensive operation,
or not possible. For example, the DynamoDB connector needs to determined the
serialized size of {{DynamoDbWriteRequest}}.
h3. Scope
Add a feature to make "size in bytes" support optional, this includes:
- Connectors will not be required to implement {{getSizeInBytes}}
- Batches will not be validated for max size
- Records will not be validated size
The sink implementer can decide if it is appropriate to enable this feature.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29688) Build time compatibility check for DynamoDB SDK
Danny Cranmer created FLINK-29688:
-
Summary: Build time compatibility check for DynamoDB SDK
Key: FLINK-29688
URL: https://issues.apache.org/jira/browse/FLINK-29688
Project: Flink
Issue Type: Improvement
Components: Connectors / DynamoDB
Reporter: Danny Cranmer
Fix For: dynamodb-1.0.0
The DynamoDB connector exposes SDK classes to the end user code, and also is
responsible for de/serialization of these classes. Add a build time check to
ensure the client model is binary equivalent of a known good version. This will
prevent us updating the SDK and unexpectedly breaking the de/serialization.
We use {{japicmp-maven-plugin}} to do something similar for Flink, we can
potentially reuse this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29444) Setup release scripts
Danny Cranmer created FLINK-29444: - Summary: Setup release scripts Key: FLINK-29444 URL: https://issues.apache.org/jira/browse/FLINK-29444 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer See https://issues.apache.org/jira/browse/FLINK-29320 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29443) Replicate packaging tests
Danny Cranmer created FLINK-29443: - Summary: Replicate packaging tests Key: FLINK-29443 URL: https://issues.apache.org/jira/browse/FLINK-29443 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer See https://issues.apache.org/jira/browse/FLINK-29316 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29442) Setup license checks
Danny Cranmer created FLINK-29442: - Summary: Setup license checks Key: FLINK-29442 URL: https://issues.apache.org/jira/browse/FLINK-29442 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer See https://issues.apache.org/jira/browse/FLINK-29310 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29441) Setup dependency convergence check
Danny Cranmer created FLINK-29441:
-
Summary: Setup dependency convergence check
Key: FLINK-29441
URL: https://issues.apache.org/jira/browse/FLINK-29441
Project: Flink
Issue Type: Sub-task
Reporter: Danny Cranmer
Following the Flink repo:
* Convergence should be disabled by default
* Enabled via {{-Pcheck-convergence}}
* Enabled on CI
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29440) Setup CI Logging
Danny Cranmer created FLINK-29440: - Summary: Setup CI Logging Key: FLINK-29440 URL: https://issues.apache.org/jira/browse/FLINK-29440 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer Configure Log4J for CI builds -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29439) Add DynamoDB project readme
Danny Cranmer created FLINK-29439: - Summary: Add DynamoDB project readme Key: FLINK-29439 URL: https://issues.apache.org/jira/browse/FLINK-29439 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer Copy and update from the ElasticSearch connector repo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29438) Setup DynamoDB Connector Project Structure
Danny Cranmer created FLINK-29438: - Summary: Setup DynamoDB Connector Project Structure Key: FLINK-29438 URL: https://issues.apache.org/jira/browse/FLINK-29438 Project: Flink Issue Type: Improvement Components: Connectors / DynamoDB Reporter: Danny Cranmer Fix For: dynamodb-1.0.0 Umbrella task for DynamoDB Connector setup. Please put all DynamoDB connector setup related tasks under this task. We will review the umbrella task to develop the migration guide and best practices for further connector creation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29246) Setup DynamoDB Connector Project Structure
Danny Cranmer created FLINK-29246: - Summary: Setup DynamoDB Connector Project Structure Key: FLINK-29246 URL: https://issues.apache.org/jira/browse/FLINK-29246 Project: Flink Issue Type: Improvement Reporter: Danny Cranmer Setup initial project structure for [flink-connector-dynamodb|https://github.com/apache/flink-connector-dynamodb] including: - Parent and module pom files - Basic build configuration - Quality plugin configuration (checkstyle) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29205) FlinkKinesisConsumer not respecting Credential Provider configuration for EFO
Danny Cranmer created FLINK-29205: - Summary: FlinkKinesisConsumer not respecting Credential Provider configuration for EFO Key: FLINK-29205 URL: https://issues.apache.org/jira/browse/FLINK-29205 Project: Flink Issue Type: Bug Components: Connectors / Kinesis Affects Versions: 1.15.2 Reporter: Danny Cranmer Fix For: 1.16.0, 1.15.3 Reported in [https://lists.apache.org/thread/xgpk0n59z5wq7kg6j8m8pgy5mcjzvvw5] FlinkKinesisConsumer is not respecting the credential provider configuration. It appears as though the legacy property transform is discarding valid config, this results in the AUTO credential provider being used - https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxyV2Factory.java#L65 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28094) [aws][Kinesis][Firehose] Upgrade AWS SDK to support ap-southeast-3
Danny Cranmer created FLINK-28094:
-
Summary: [aws][Kinesis][Firehose] Upgrade AWS SDK to support
ap-southeast-3
Key: FLINK-28094
URL: https://issues.apache.org/jira/browse/FLINK-28094
Project: Flink
Issue Type: Improvement
Components: Connectors / Kinesis
Reporter: Danny Cranmer
Fix For: 1.12.8, 1.16.0, 1.13.7, 1.15.2, 1.14.6
The AWS base module pulls AWS SDK v2.17.52 which does not support
{{ap-southeast-3}}. Update to the latest version.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (FLINK-26890) [Kinesis][Consumer] DynamoDB consumer error consuming partitions close to retention
Danny Cranmer created FLINK-26890:
-
Summary: [Kinesis][Consumer] DynamoDB consumer error consuming
partitions close to retention
Key: FLINK-26890
URL: https://issues.apache.org/jira/browse/FLINK-26890
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: 1.15.0
Reporter: Danny Cranmer
Fix For: 1.15.0, 1.16.0, 1.13.7, 1.14.5
*Background*
The Amazon Kinesis Data Streams consumer supports consuming from Amazon
DynamoDB via the [DynamoDB Streams Kinesis
Adapter|https://github.com/awslabs/dynamodb-streams-kinesis-adapter].
*Problem*
We have seen instances of consumer throwing {{ResouceNotFoundException}} when
attempting to invoke {{GetShardIterator}}.
{code}
com.amazonaws.services.kinesis.model.ResourceNotFoundException: Requested
resource not found: Shard does not exist
{code}
According to the DynamoDB team, the {{DescribeStream}} call may return shard
IDs that are no longer valid, and this exception needs to be handled by the
client.
*Solution*
Modify the DynamoDB consumer to treat {{ResourceNotFoundException}} as a shard
closed signal.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26417) Async can Sink violate At Least Once during checkpointing
Danny Cranmer created FLINK-26417:
-
Summary: Async can Sink violate At Least Once during checkpointing
Key: FLINK-26417
URL: https://issues.apache.org/jira/browse/FLINK-26417
Project: Flink
Issue Type: Bug
Components: Connectors / Common
Reporter: Danny Cranmer
Fix For: 1.15.0
Async Sink holds data to send in 2 places:
- Buffered in queue
- "In-flight", being submitted by the async writer
When checkpointing we only capture the buffered requests. It seems like the
code to [wait for in-flight requests to
complete](https://github.com/apache/flink/blob/aab253a9141f1cdff81658eb47c7bb6ee9d2105c/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L402-L420)
has been removed/
Reinstate this mechanism to wait for in-flight requests to flush within
{{prepareCommit}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26256) AWS SDK Async Event Loop Group Classloader Issue
Danny Cranmer created FLINK-26256:
-
Summary: AWS SDK Async Event Loop Group Classloader Issue
Key: FLINK-26256
URL: https://issues.apache.org/jira/browse/FLINK-26256
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Reporter: Danny Cranmer
Assignee: Zichen Liu
Fix For: 1.15.0
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31044&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d
(shortened stack trace, as full is too large)
{noformat}
Feb 09 20:05:04 java.util.concurrent.ExecutionException:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: Trying to access closed classloader. Please check if you store
classloaders directly or indirectly in static fields. If the stacktrace
suggests that the leak occurs in a third party library and cannot be fixed
immediately, you can disable this check with the configuration
'classloader.check-leaked-classloader'.
Feb 09 20:05:04 at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
Feb 09 20:05:04 at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
(...)
Feb 09 20:05:04 Caused by:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute
HTTP request: Trying to access closed classloader. Please check if you store
classloaders directly or indirectly in static fields. If the stacktrace
suggests that the leak occurs in a third party library and cannot be fixed
immediately, you can disable this check with the configuration
'classloader.check-leaked-classloader'.
Feb 09 20:05:04 at
software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
Feb 09 20:05:04 at
software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
Feb 09 20:05:04 at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204)
Feb 09 20:05:04 at
software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:200)
Feb 09 20:05:04 at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:179)
Feb 09 20:05:04 at
software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
(...)
Feb 09 20:05:04 Caused by: java.lang.IllegalStateException: Trying to access
closed classloader. Please check if you store classloaders directly or
indirectly in static fields. If the stacktrace suggests that the leak occurs in
a third party library and cannot be fixed immediately, you can disable this
check with the configuration 'classloader.check-leaked-classloader'.
Feb 09 20:05:04 at
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:164)
Feb 09 20:05:04 at
org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.getResources(FlinkUserCodeClassLoaders.java:188)
Feb 09 20:05:04 at
java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:348)
Feb 09 20:05:04 at
java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
Feb 09 20:05:04 at
java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
Feb 09 20:05:04 at
javax.xml.stream.FactoryFinder$1.run(FactoryFinder.java:352)
Feb 09 20:05:04 at java.security.AccessController.doPrivileged(Native
Method)
Feb 09 20:05:04 at
javax.xml.stream.FactoryFinder.findServiceProvider(FactoryFinder.java:341)
Feb 09 20:05:04 at
javax.xml.stream.FactoryFinder.find(FactoryFinder.java:313)
Feb 09 20:05:04 at
javax.xml.stream.FactoryFinder.find(FactoryFinder.java:227)
Feb 09 20:05:04 at
javax.xml.stream.XMLInputFactory.newInstance(XMLInputFactory.java:154)
Feb 09 20:05:04 at
software.amazon.awssdk.protocols.query.unmarshall.XmlDomParser.createXmlInputFactory(XmlDomParser.java:124)
Feb 09 20:05:04 at
java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:284)
Feb 09 20:05:04 at
java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
Feb 09 20:05:04 at java.lang.ThreadLocal.get(ThreadLocal.java:170)
(...)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26008) [FLIP-171] Update Kinesalite docker container reference
Danny Cranmer created FLINK-26008:
-
Summary: [FLIP-171] Update Kinesalite docker container reference
Key: FLINK-26008
URL: https://issues.apache.org/jira/browse/FLINK-26008
Project: Flink
Issue Type: Sub-task
Components: Connectors / Kinesis
Reporter: Danny Cranmer
We are currently referencing {{:latest}} Kinesalite image for tests. Update
this to the more recent version:
- https://github.com/apache/flink/pull/18661/files#r801531468
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25848) [FLIP-171] KDS Sink does not fast fail when invalid configuration supplied
Danny Cranmer created FLINK-25848:
-
Summary: [FLIP-171] KDS Sink does not fast fail when invalid
configuration supplied
Key: FLINK-25848
URL: https://issues.apache.org/jira/browse/FLINK-25848
Project: Flink
Issue Type: Sub-task
Components: Connectors / Kinesis
Reporter: Danny Cranmer
Fix For: 1.15.0
h4. Description
KDS sink does not fail job when invalid configuration provided.
h4. Reproduction Steps
- Start a job using an Async Sink implementation, for example KDS
- Specify an invalid credential provider configuration, for example
{code}
CREATE TABLE orders (
`code` STRING,
`quantity` BIGINT
) WITH (
'connector' = 'kinesis',
'stream' = 'source',
'aws.credentials.provider' = 'ASSUME_ROLE',
'aws.region' = 'us-east-1',
'format' = 'json'
);
{code}
h4. Actual Results
- Sink operator transitions to running, consistently retrying
{code}
2022-01-27 08:29:31,582 WARN
org.apache.flink.connector.kinesis.sink.KinesisDataStreamsSinkWriter [] - KDS
Sink failed to persist 5 entries to KDS
java.util.concurrent.CompletionException:
org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.sts.model.StsException:
2 validation errors detected: Value null at 'roleArn' failed to satisfy
constraint: Member must not be null; Value null at 'roleSessionName' failed to
satisfy constraint: Member must not be null (Service: Sts, Status Code: 400,
Request ID: af8f2176-aafa-4230-805b-72d90e418810, Extended Request ID: null)
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
~[?:?]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)
~[?:?]
at
java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
~[?:?]
at
java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
~[?:?]
at
org.apache.flink.kinesis.shaded.software.amazon.awssdk.services.kinesis.DefaultKinesisAsyncClient.putRecords(DefaultKinesisAsyncClient.java:2112)
~[blob_p-a167cef2d4088e526c4e209bb2b4e5a83f8ab359-fc0375a6ef7545bc316b770d110d4564:1.15-SNAPSHOT]
at
org.apache.flink.connector.kinesis.sink.KinesisDataStreamsSinkWriter.submitRequestEntries(KinesisDataStreamsSinkWriter.java:122)
~[blob_p-a167cef2d4088e526c4e209bb2b4e5a83f8ab359-fc0375a6ef7545bc316b770d110d4564:1.15-SNAPSHOT]
at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.flush(AsyncSinkWriter.java:311)
~[flink-connector-files-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.connector.base.sink.writer.AsyncSinkWriter.prepareCommit(AsyncSinkWriter.java:391)
~[flink-connector-files-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.streaming.runtime.operators.sink.SinkOperator.endInput(SinkOperator.java:192)
~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.endOperatorInput(StreamOperatorWrapper.java:96)
~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.endInput(RegularOperatorChain.java:97)
~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:68)
~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:517)
~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:802)
~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:751)
~[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
[flink-dist-1.15-SNAPSHOT.jar:1.15-SNAPSHOT]
at java.lang.Thread.run(Thread.java:829) [?:?]
{code}
h4. Expected Results
- Job fails fast
h4. Suggested Resolution
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25846) Async Sink does not gracefully shutdown on Cancel
Danny Cranmer created FLINK-25846: - Summary: Async Sink does not gracefully shutdown on Cancel Key: FLINK-25846 URL: https://issues.apache.org/jira/browse/FLINK-25846 Project: Flink Issue Type: Sub-task Reporter: Danny Cranmer h4. Description Async Sink does not react gracefully to cancellation signal h4. Reproduction Steps - Start a job using an Async Sink implementation, for example KDS - Navigate to Flink Dashboard - Click Job > Cancel h4. Actual Results - Sink operator stuck in Cancelling, retrying h4. Expected Results - Sink operator closes h4. Suggested Resolution - Async Sink should treat `InterruptedException` as stop signal -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24431) [Kinesis][EFO] EAGER registration strategy does not work when job fails over
Danny Cranmer created FLINK-24431:
-
Summary: [Kinesis][EFO] EAGER registration strategy does not work
when job fails over
Key: FLINK-24431
URL: https://issues.apache.org/jira/browse/FLINK-24431
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: 1.13.2, 1.12.5, 1.14.0
Reporter: Danny Cranmer
Fix For: 1.12.6, 1.13.3, 1.15.0, 1.14.1
*Background*
The EFO Kinesis connector will register and de-register stream consumers based
on the configured [registration
strategy|https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/connectors/datastream/kinesis/#efo-stream-consumer-registrationderegistration].
When {{EAGER}} is used, the client (usually job manager) will register the
consumer and then the task managers will de-register the consumer when job
stops/fails. If the job is configured to restart on fail, then the consumer
will not exist and the job will continuously fail over.
*Solution*
The proposal is to not deregister the stream consumer when {{EAGER}} is used.
The documentation should be updated to reflect this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23996) Make Scala 2.12 default build target
Danny Cranmer created FLINK-23996:
-
Summary: Make Scala 2.12 default build target
Key: FLINK-23996
URL: https://issues.apache.org/jira/browse/FLINK-23996
Project: Flink
Issue Type: Improvement
Components: Build System
Reporter: Danny Cranmer
Fix For: 1.15.0
*Background*
Flink currently supports Scala 2.11 and 2.12. There is a plan to [drop Scala
2.11 support|https://issues.apache.org/jira/browse/FLINK-20845], however in the
meantime, 2.11 is the default Scala version.
*What*
Make Scala 2.12 the default version.
*How*
- Update {{pom.xml}} files to default to Scala 2.12
- Update documentation to reflect this and to instruct how to build Scala 2.11
- Update CI scripts
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-23802) [kinesis][efo] Reduce ReadTimeoutExceptions for Kinesis Consumer
Danny Cranmer created FLINK-23802:
-
Summary: [kinesis][efo] Reduce ReadTimeoutExceptions for Kinesis
Consumer
Key: FLINK-23802
URL: https://issues.apache.org/jira/browse/FLINK-23802
Project: Flink
Issue Type: Improvement
Components: Connectors / Kinesis
Affects Versions: 1.13.2, 1.12.5, 1.12.4, 1.13.1, 1.12.3, 1.13.0, 1.12.2,
1.12.1, 1.12.0
Reporter: Danny Cranmer
Assignee: Danny Cranmer
Fix For: 1.14.0, 1.12.6, 1.13.3
h3. Background
The Kinesis EFO consumer uses an async AWS SDK Netty client to read records
from Kinesis. When the client is inactive for 30 seconds a
{{ReadTimeoutException}} is thrown by Netty. The consumer will terminate the
subscription, backoff and retry. Jobs with high backpressure can result in
frequent {{ReadTImeoutException}} and the frequent backoff and retry can cause
unnecessary overhead.
h3. What?
Reduce/eliminate {{ReadTimeoutException}} from the EFO consumer
h3. How?
There are 2 improvements to be made:
1. Request next record from the Flink source thread rather than the AWS SDK
response thread. This means that there will always be space in the input buffer
queue. The AWS SDK async response thread is no longer blocking on this queue.
Backpressure is now applied by the Flink source thread rather than the AWS SDK
thread.
2. Increase the Read Timeout (30s) to be higher than the maximum Shard
subscription duration (5m) and enable TCP keep alive
h3. References
This has already been implemented and tested in
[amazon-kinesis-connector-flink|https://github.com/awslabs/amazon-kinesis-connector-flink]:
- [Prevent SDK threads
blocking|https://github.com/awslabs/amazon-kinesis-connector-flink/pull/40]
- [Increase read timeout and enable TCP
keepalive|https://github.com/awslabs/amazon-kinesis-connector-flink/pull/42]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22479) [Kinesis][Consumer] Potential lock-up under error condition
Danny Cranmer created FLINK-22479: - Summary: [Kinesis][Consumer] Potential lock-up under error condition Key: FLINK-22479 URL: https://issues.apache.org/jira/browse/FLINK-22479 Project: Flink Issue Type: Bug Components: Connectors / Kinesis Affects Versions: 1.12.2, 1.12.1, 1.12.0 Reporter: Danny Cranmer Assignee: Danny Cranmer Fix For: 1.13.0, 1.12.3 *Background* This connector has been [forked](https://github.com/awslabs/amazon-kinesis-connector-flink) by AWS for use on KDA with Flink 1.11. Bugs have been encountered: - Under high backpressure scenarios - When an error is thrown during tear down *Scope* Pull in the following fixes from AWS fork: - // TODO -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21933) [kinesis][efo] EFO consumer treats interrupts as retryable exceptions
Danny Cranmer created FLINK-21933: - Summary: [kinesis][efo] EFO consumer treats interrupts as retryable exceptions Key: FLINK-21933 URL: https://issues.apache.org/jira/browse/FLINK-21933 Project: Flink Issue Type: Bug Components: Connectors / Kinesis Affects Versions: 1.12.2, 1.12.1, 1.12.0 Reporter: Danny Cranmer Assignee: Danny Cranmer Fix For: 1.13.0, 1.12.3 *Background* When an Flink job is cancelled or failed, the source threads get interrupted. The EFO consumer is treating this exception as retryable and will retry until MAX_RETRIES is exceeded. *Scope* Update error handling to treat Interrupted exceptions as non-recoverable. The source should gracefully shutdown. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21830) [kinesis][auth] Add support for Session Token
Danny Cranmer created FLINK-21830:
-
Summary: [kinesis][auth] Add support for Session Token
Key: FLINK-21830
URL: https://issues.apache.org/jira/browse/FLINK-21830
Project: Flink
Issue Type: Improvement
Components: Connectors / Kinesis
Reporter: Danny Cranmer
*Background*
The {{FlinkKinesisConsumer}} and {{FlinkKinesisProducer}} support a variety of
AWS authentication mechanisms, including {{BASIC}}. When using {{BASIC}},
{{SESSION_KEY}} is not supported.
*Scope*
Add support for {{SESSION_KEY}} when constructing the AWS credential provider.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21391) Migrate AWS Glue e2e test from sh to Java
Danny Cranmer created FLINK-21391:
-
Summary: Migrate AWS Glue e2e test from sh to Java
Key: FLINK-21391
URL: https://issues.apache.org/jira/browse/FLINK-21391
Project: Flink
Issue Type: Improvement
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: Danny Cranmer
We added an e2e test using {{sh}} script with AWS Glue Schema Registry support.
We are pushing to migrate tests to the new Java framework.
Migrate {{ flink-end-to-end-tests/test-scripts/test_glue_schema_registry.sh}}
to a Java e2e test:
-
https://lists.apache.org/thread.html/rdc2894c67c6da3fa92f85ec2cde1d5a0c551748050431d36c13bf7a3%40%3Cdev.flink.apache.org%3E
This change will require supporting Kinesis in the Java e2e framework and allow
us to more easily migrate the Kinesis e2e tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21228) [Kinesis][Producer] Deadlock in KinesisProducer
Danny Cranmer created FLINK-21228:
-
Summary: [Kinesis][Producer] Deadlock in KinesisProducer
Key: FLINK-21228
URL: https://issues.apache.org/jira/browse/FLINK-21228
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: 1.12.1
Reporter: Danny Cranmer
*Background*
Application sink failed and resulted in:
- Indefinite backpressure being applied
- Exception never thrown causing job to fail
Application running with:
{code:java}
flinkKinesisProducer.setQueueLimit(1);
flinkKinesisProducer.setFailOnError(true);
{code}
- {{KinesisProducer}} is waiting for queue to empty before sending the next
record
([code|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisProducer.java#L303])
- KPL ran out of memory, which raised an error, however this is processed async
([code|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisProducer.java#L275])
- {{KinesisProducer}} would have rethrown the error and restarted the job,
however operator stuck in an infinite loop enforcing the queue limit (which
never clears)
([code|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisProducer.java#L306])
*Proposal*
- {{checkAndPropagateAsyncError()}} while enforcing queue limit in
{{enforceQueueLimit()}} to break deadlock
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20630) [Kinesis][DynamoDB] DynamoDB Consumer fails to consume from Latest
Danny Cranmer created FLINK-20630:
-
Summary: [Kinesis][DynamoDB] DynamoDB Consumer fails to consume
from Latest
Key: FLINK-20630
URL: https://issues.apache.org/jira/browse/FLINK-20630
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: 1.12.0
Reporter: Danny Cranmer
Fix For: 1.12.1
*Background*
When consuming from `LATEST`, the `KinesisDataFetcher` converts the shard
iterator type into an `AT_TIMESTAMP` to ensure all shards start from the same
position. When `LATEST` is used each shared would effectively start from a
different point in the time.
DynamoDB streams do not support `AT_TIMESTAMP`.
*Scope*
Remove shard iterator type transform for DynamoDB streams consumer.
*Reproduction Steps*
Create a simple application that consumer from `LATEST` using
`FlinkDynamoDBStreamsConsumer`
*Expected Results*
Consumer starts reading records from the head of the stream
*Actual Results*
An exception is thrown:
{code}
Caused by:
org.apache.flink.kinesis.shaded.com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:
1 validation error detected: Value 'AT_TIMESTAMP' at 'shardIteratorType'
failed to satisfy constraint: Member must satisfy enum value set:
[AFTER_SEQUENCE_NUMBER, LATEST, AT_SEQUENCE_NUMBER, TRIM_HORIZON] (Service:
AmazonDynamoDBStreams; Status Code: 400; Error Code: ValidationException;
Request ID: AFQ8KCJAP74IN5MR5KD2FP0CTBVV4KQNSO5AEMVJF66Q9ASUAAJG)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1799)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1383)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1359)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
at
org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
at
org.apache.flink.kinesis.shaded.com.amazonaws.services.dynamodbv2.AmazonDynamoDBStreamsClient.doInvoke(AmazonDynamoDBStreamsClient.java:686)
at
org.apache.flink.kinesis.shaded.com.amazonaws.services.dynamodbv2.AmazonDynamoDBStreamsClient.invoke(AmazonDynamoDBStreamsClient.java:653)
at
org.apache.flink.kinesis.shaded.com.amazonaws.services.dynamodbv2.AmazonDynamoDBStreamsClient.invoke(AmazonDynamoDBStreamsClient.java:642)
at
org.apache.flink.kinesis.shaded.com.amazonaws.services.dynamodbv2.AmazonDynamoDBStreamsClient.executeGetShardIterator(AmazonDynamoDBStreamsClient.java:544)
at
org.apache.flink.kinesis.shaded.com.amazonaws.services.dynamodbv2.AmazonDynamoDBStreamsClient.getShardIterator(AmazonDynamoDBStreamsClient.java:515)
at
org.apache.flink.kinesis.shaded.com.amazonaws.services.dynamodbv2.streamsadapter.AmazonDynamoDBStreamsAdapterClient.getShardIterator(AmazonDynamoDBStreamsAdapterClient.java:355)
at
org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardIterator(KinesisProxy.java:311)
at
org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getShardIterator(KinesisProxy.java:302)
at
org.apache.flink.streaming.connectors.kinesis.internals.publisher.polling.PollingRecordPublisher.getShardIterator(PollingRecordPublisher.java:173)
at
org.apache.flink.streaming.connectors.kinesis.internals.publisher.polling.PollingRecordPublisher.(PollingRecordPublisher.java:93)
at
org.apache.flink.streaming.connectors.kinesis.internals.publisher.polling.PollingRecordPublisherFactory.create(PollingRecordPublisherFactory.java:85)
at
org.apache.flink.streaming.connectors.kinesis.internals.publisher.polling.PollingRecordPublisherFactory.create(PollingRecordPublisherFactory.java:36)
at
org.apache.flink.streaming.connectors.kinesis.internals.KinesisDataFetcher.createRecordPubl
[jira] [Created] (FLINK-20629) [Kinesis][EFO] Migrate from DescribeStream to DescribeStreamSummary
Danny Cranmer created FLINK-20629:
-
Summary: [Kinesis][EFO] Migrate from DescribeStream to
DescribeStreamSummary
Key: FLINK-20629
URL: https://issues.apache.org/jira/browse/FLINK-20629
Project: Flink
Issue Type: Improvement
Components: Connectors / Kinesis
Affects Versions: 1.12.0
Reporter: Danny Cranmer
Fix For: 1.12.1
*Background*
The Kinesis EFO connector invokes {{DescribeStream}} during startup to acquire
the stream ARN. This call also includes the shard information and has a TPS of
10. A similar service exists, {{DescribeStreamSummary}} that has a TPS of 20
and a lighter response payload size.
During startup sources with high parallelism compete to call this service (in
{{LAZY}} mode), resulting in backoff and retry. Essentially the startup time
can grow by 1s for every 10 parallelism, due to the 10 TPS. Migrating to
{{DescribeStreamSummary}} will improve startup time.
*Scope*
Migrate call to {{DescribeStream}} to use {{DescribeStreamSummary}} instead.
*Note*
I have targeted {{1.12.1}}, let me know if we should instead target {{1.13}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20350) [Kinesis][GCP PubSub] Incompatible Connectors due to Guava conflict
Danny Cranmer created FLINK-20350:
-
Summary: [Kinesis][GCP PubSub] Incompatible Connectors due to
Guava conflict
Key: FLINK-20350
URL: https://issues.apache.org/jira/browse/FLINK-20350
Project: Flink
Issue Type: Bug
Components: Connectors / Google Cloud PubSub, Connectors / Kinesis
Affects Versions: 1.11.2, 1.11.1
Reporter: Danny Cranmer
*Problem*
Kinesis and GCP PubSub connector do not work together. The following error is
thrown.
{code}
java.lang.NoClassDefFoundError: Could not initialize class
io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder
at
org.apache.flink.streaming.connectors.gcp.pubsub.DefaultPubSubSubscriberFactory.getSubscriber(DefaultPubSubSubscriberFactory.java:52)
~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.createAndSetPubSubSubscriber(PubSubSource.java:213)
~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.open(PubSubSource.java:102)
~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
~[flink-core-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
~[flink-runtime_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-runtime_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_252]
{code}
*Cause*
This is caused by a Guava dependency conflict:
- Kinesis Consumer > {{18.0}}
- GCP PubSub > {{26.0-android}}
{{NettyChannelBuilder}} fails to initialise due to missing method in guava:
-
{{com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V}}
*Possible Fixes*
- Align Guava versions
- Shade Guava in either connector
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20088) [Kinesis][Polling] Issue using Polling consumer at timestamp with empty shard
Danny Cranmer created FLINK-20088:
-
Summary: [Kinesis][Polling] Issue using Polling consumer at
timestamp with empty shard
Key: FLINK-20088
URL: https://issues.apache.org/jira/browse/FLINK-20088
Project: Flink
Issue Type: Improvement
Components: Connectors / Kinesis
Reporter: Danny Cranmer
Fix For: 1.12.0
*Background*
The consumer fails when a Polling record publisher uses a timestamp sentinel
starting position and the first record batch is empty. This is because the
consumer tries to recalculate the start position from the timestamp sentinel,
this operation is not supported.
*Reproduction Steps*
Setup an application consuming from Kinesis with following properties and
consume from an empty shard:
{code:java}
String format = "-MM-dd'T'HH:mm:ss";
String date = new SimpleDateFormat(format).format(new Date());
consumerConfig.setProperty(ConsumerConfigConstants.STREAM_INITIAL_TIMESTAMP,
date);
consumerConfig.setProperty(ConsumerConfigConstants.STREAM_TIMESTAMP_DATE_FORMAT,
format);
consumerConfig.setProperty(ConsumerConfigConstants.STREAM_INITIAL_POSITION,
"AT_TIMESTAMP"); {code}
*Error*
{code:java}
Exception in thread "main"
org.apache.flink.runtime.client.JobExecutionException: Job execution
failed.Exception in thread "main"
org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
at
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:119)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:229)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:996)
at akka.dispatch.OnComplete.internal(Future.scala:264) at
akka.dispatch.OnComplete.internal(Future.scala:261) at
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191) at
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188) at
scala.concurrent.impl.CallbackRunnable.run$$$capture(Promise.scala:36) at
scala.concurrent.impl.CallbackRunnable.run(Promise.scala) at
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572) at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
at
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436) at
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435) at
scala.concurrent.impl.CallbackRunnable.run$$$capture(Promise.scala:36) at
scala.concurrent.impl.CallbackRunnable.run(Promise.scala) at
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90) at
akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)Caused
by: org.apache.flink.runtime.JobException: Recovery is suppressed by
NoRestartBackoffTimeStrategy at
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(Executio
