[jira] [Resolved] (FLINK-35182) Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink Pulsar connector
[ https://issues.apache.org/jira/browse/FLINK-35182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen resolved FLINK-35182. --- Fix Version/s: pulsar-4.2.0 Assignee: Zhongqiang Gong Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/commit/b37a8b32f30683664ff25888d403c4de414043e1 > Bump org.apache.commons:commons-compress from 1.24.0 to 1.26.1 for Flink > Pulsar connector > - > > Key: FLINK-35182 > URL: https://issues.apache.org/jira/browse/FLINK-35182 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / Pulsar >Reporter: Zhongqiang Gong >Assignee: Zhongqiang Gong >Priority: Minor > Labels: pull-request-available > Fix For: pulsar-4.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-32645) Flink pulsar sink is having poor performance
[ https://issues.apache.org/jira/browse/FLINK-32645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen resolved FLINK-32645. --- Resolution: Fixed > Flink pulsar sink is having poor performance > > > Key: FLINK-32645 > URL: https://issues.apache.org/jira/browse/FLINK-32645 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.2 > Environment: !Screenshot 2023-07-22 at 1.59.42 PM.png!!Screenshot > 2023-07-22 at 2.03.53 PM.png! > >Reporter: Vijaya Bhaskar V >Assignee: Zili Chen >Priority: Major > Fix For: pulsar-3.0.2 > > Attachments: Screenshot 2023-07-22 at 2.03.53 PM.png, Screenshot > 2023-07-22 at 2.56.55 PM.png, Screenshot 2023-07-22 at 3.45.21 PM-1.png, > Screenshot 2023-07-22 at 3.45.21 PM.png, pom.xml > > > Found following issue with flink pulsar sink: > > Flink pulsar sink is always waiting while enqueueing the message and making > the task slot busy no matter how many free slots we provide. Attached the > screen shot of the same > Just sending messages of less rate 8k msg/sec and stand alone flink job with > discarding sink is able to receive full rate if 8K msg/sec > Where as pulsar sink was consuming only upto 2K msg/sec and the sink is > always busy waiting. Snapshot of thread dump attached. > Also snap shot of flink stream graph attached > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34962) flink-connector-pulsa starts failed due to incorrect use of Pulsar API: LookupService. getPartitionedTopicMetadata
[ https://issues.apache.org/jira/browse/FLINK-34962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen updated FLINK-34962: -- Affects Version/s: (was: pulsar-4.2.0) > flink-connector-pulsa starts failed due to incorrect use of Pulsar API: > LookupService. getPartitionedTopicMetadata > -- > > Key: FLINK-34962 > URL: https://issues.apache.org/jira/browse/FLINK-34962 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: pulsar-4.1.1 > Environment: * flink 1.17 > * pulsar client 3.0.0 > * org.apache.flink:flink-connector-pulsar:4.1.0-1.17 (connector) >Reporter: Yubiao Feng >Priority: Major > Labels: easyfix, pull-request-available > Fix For: pulsar-4.2.0 > > > - The unnecessary codes calls > `pulsarClient.getLookup().getPartitionedTopicMetadata()` to create the > partitioned topic metadata(in fact, this behavior of is not correct) > - Why it is unnecessary: the [following > code]([https://github.com/apache/flink-connector-pulsar/blob/main/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/ProducerRegister.java#L245]) > that is creating a producer will also trigger partitioned topic metadata to > create. > - The method `pulsarClient.getLookup().getPartitionedTopicMetadata()` will > not retry if the connection is closed so that users will get an error. The > following code creates a producer that will retry if the connection is > closed, reducing the probability of an error occurring. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34962) flink-connector-pulsa starts failed due to incorrect use of Pulsar API: LookupService. getPartitionedTopicMetadata
[ https://issues.apache.org/jira/browse/FLINK-34962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen updated FLINK-34962: -- Affects Version/s: (was: pulsar-4.1.1) > flink-connector-pulsa starts failed due to incorrect use of Pulsar API: > LookupService. getPartitionedTopicMetadata > -- > > Key: FLINK-34962 > URL: https://issues.apache.org/jira/browse/FLINK-34962 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar > Environment: * flink 1.17 > * pulsar client 3.0.0 > * org.apache.flink:flink-connector-pulsar:4.1.0-1.17 (connector) >Reporter: Yubiao Feng >Priority: Major > Labels: easyfix, pull-request-available > Fix For: pulsar-4.2.0 > > > - The unnecessary codes calls > `pulsarClient.getLookup().getPartitionedTopicMetadata()` to create the > partitioned topic metadata(in fact, this behavior of is not correct) > - Why it is unnecessary: the [following > code]([https://github.com/apache/flink-connector-pulsar/blob/main/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/ProducerRegister.java#L245]) > that is creating a producer will also trigger partitioned topic metadata to > create. > - The method `pulsarClient.getLookup().getPartitionedTopicMetadata()` will > not retry if the connection is closed so that users will get an error. The > following code creates a producer that will retry if the connection is > closed, reducing the probability of an error occurring. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-34962) flink-connector-pulsa starts failed due to incorrect use of Pulsar API: LookupService. getPartitionedTopicMetadata
[ https://issues.apache.org/jira/browse/FLINK-34962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen resolved FLINK-34962. --- Fix Version/s: pulsar-4.2.0 Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/commit/7340f713422b1734e84ec0602f154441b8da7fab > flink-connector-pulsa starts failed due to incorrect use of Pulsar API: > LookupService. getPartitionedTopicMetadata > -- > > Key: FLINK-34962 > URL: https://issues.apache.org/jira/browse/FLINK-34962 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: pulsar-4.2.0, pulsar-4.1.1 > Environment: * flink 1.17 > * pulsar client 3.0.0 > * org.apache.flink:flink-connector-pulsar:4.1.0-1.17 (connector) >Reporter: Yubiao Feng >Priority: Major > Labels: easyfix, pull-request-available > Fix For: pulsar-4.2.0 > > > - The unnecessary codes calls > `pulsarClient.getLookup().getPartitionedTopicMetadata()` to create the > partitioned topic metadata(in fact, this behavior of is not correct) > - Why it is unnecessary: the [following > code]([https://github.com/apache/flink-connector-pulsar/blob/main/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/ProducerRegister.java#L245]) > that is creating a producer will also trigger partitioned topic metadata to > create. > - The method `pulsarClient.getLookup().getPartitionedTopicMetadata()` will > not retry if the connection is closed so that users will get an error. The > following code creates a producer that will retry if the connection is > closed, reducing the probability of an error occurring. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33884) Update Pulsar dependency to 3.0.2 in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-33884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen resolved FLINK-33884. --- Fix Version/s: pulsar-4.2.0 Resolution: Fixed Master via https://github.com/apache/flink-connector-pulsar/commit/9f4b902c2a478d0105eec1e32bac3ea40f318d00 > Update Pulsar dependency to 3.0.2 in Pulsar Connector > - > > Key: FLINK-33884 > URL: https://issues.apache.org/jira/browse/FLINK-33884 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.1 >Reporter: David Christle >Assignee: David Christle >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.2.0 > > > The [3.0.2 > patch|https://pulsar.apache.org/release-notes/versioned/pulsar-3.0.2/] > includes various bug fixes, including a few for the Pulsar client (e.g. > [link]([https://github.com/apache/pulsar/pull/21144)). Upgrading the > dependency in the connector will pick up these fixes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-34629) Pulsar source lost topic subscribe
[ https://issues.apache.org/jira/browse/FLINK-34629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen resolved FLINK-34629. --- Fix Version/s: pulsar-4.2.0 Resolution: Fixed Master via https://github.com/apache/flink-connector-pulsar/commit/7a5eef268cb3f598589ad9cc32648ac92fbbee1d > Pulsar source lost topic subscribe > -- > > Key: FLINK-34629 > URL: https://issues.apache.org/jira/browse/FLINK-34629 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: pulsar-3.0.1 >Reporter: WangMinChao >Assignee: WangMinChao >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.2.0 > > > The non-partition pulsar topic partition id is `-1`, using multiples of the > non-partition topics > in Pulsar source maybe lose topic subscribe. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33970) Add necessary checks for connector document
[ https://issues.apache.org/jira/browse/FLINK-33970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-33970. - Resolution: Fixed master via 5b70da8f88e21057a5c590d139eab558f87e5dca Thanks a lot [~gongzhongqiang]! > Add necessary checks for connector document > --- > > Key: FLINK-33970 > URL: https://issues.apache.org/jira/browse/FLINK-33970 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Leonard Xu >Assignee: Zhongqiang Gong >Priority: Major > Labels: pull-request-available > > In FLINK-33964, we found the documentation files in independent connector > repos lacks basic checks like broken url, this ticket aims to add necessary > checks and avoid similar issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33970) Add necessary checks for connector document
[ https://issues.apache.org/jira/browse/FLINK-33970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen updated FLINK-33970: -- Fix Version/s: pulsar-4.2.0 pulsar-4.1.1 > Add necessary checks for connector document > --- > > Key: FLINK-33970 > URL: https://issues.apache.org/jira/browse/FLINK-33970 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Leonard Xu >Assignee: Zhongqiang Gong >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.2.0, pulsar-4.1.1 > > > In FLINK-33964, we found the documentation files in independent connector > repos lacks basic checks like broken url, this ticket aims to add necessary > checks and avoid similar issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33400) Pulsar connector doesn't compile for Flink 1.18 due to Archunit update
[ https://issues.apache.org/jira/browse/FLINK-33400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-33400. - Fix Version/s: pulsar-4.1.0 Assignee: Zili Chen (was: Martijn Visser) Resolution: Fixed master via 707e49472d557bafa58013c17e3194b64fb4b3ef > Pulsar connector doesn't compile for Flink 1.18 due to Archunit update > -- > > Key: FLINK-33400 > URL: https://issues.apache.org/jira/browse/FLINK-33400 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.1 >Reporter: Martijn Visser >Assignee: Zili Chen >Priority: Blocker > Labels: pull-request-available > Fix For: pulsar-4.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33602) Pulsar connector should be compatible with Flink 1.18
Zili Chen created FLINK-33602: - Summary: Pulsar connector should be compatible with Flink 1.18 Key: FLINK-33602 URL: https://issues.apache.org/jira/browse/FLINK-33602 Project: Flink Issue Type: Bug Components: Connectors / Pulsar Affects Versions: 1.18.0, pulsar-4.1.0 Reporter: Zili Chen Assignee: Zili Chen Currently, the build and test job always fails - https://github.com/apache/flink-connector-pulsar/actions/runs/6937440214 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33357) add Apache Software License 2
[ https://issues.apache.org/jira/browse/FLINK-33357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-33357: - Assignee: 蔡灿材 > add Apache Software License 2 > - > > Key: FLINK-33357 > URL: https://issues.apache.org/jira/browse/FLINK-33357 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.6.0 >Reporter: 蔡灿材 >Assignee: 蔡灿材 >Priority: Minor > Labels: pull-request-available > Fix For: kubernetes-operator-1.5.0 > > Attachments: 2023-10-25 12-08-58屏幕截图.png > > > Flinkdeployments.flink.apache.org - v1. Currently yml and > flinksessionjobs.flink.apache.org - v1. Yml don't > add add Apache Software License 2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-32938) flink-connector-pulsar should remove all `PulsarAdmin` calls
[
https://issues.apache.org/jira/browse/FLINK-32938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen resolved FLINK-32938.
---
Fix Version/s: pulsar-4.1.0
Resolution: Fixed
master via 78d00ea9e3e278d4ce2fbb0c8a8d380abef7b858
> flink-connector-pulsar should remove all `PulsarAdmin` calls
>
>
> Key: FLINK-32938
> URL: https://issues.apache.org/jira/browse/FLINK-32938
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
>Reporter: Neng Lu
>Assignee: Neng Lu
>Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-4.1.0
>
>
> The flink-connector-pulsar should not access and interact with the admin
> endpoint. This could introduce potential security issues.
> In a production environment, a Pulsar cluster admin will not grant the
> permissions for the flink application to conduct any admin operations.
> Currently, the connector does various admin calls:
> ```{{{}{}}}{{{}{}}}
> PulsarAdmin.topics().getPartitionedTopicMetadata(topic)
> PulsarAdmin.namespaces().getTopics(namespace)
> PulsarAdmin.topics().getLastMessageId(topic)
> PulsarAdmin.topics().getMessageIdByTimestamp(topic, timestamp)
> PulsarAdmin.topics().getSubscriptions(topic)
> PulsarAdmin.topics().createSubscription(topic, subscription,
> MessageId.earliest)
> PulsarAdmin.topics().resetCursor(topic, subscription, initial, !include)
> ```
> We need to replace these calls with consumer or client calls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-32938) flink-connector-pulsar should remove all `PulsarAdmin` calls
[
https://issues.apache.org/jira/browse/FLINK-32938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen reassigned FLINK-32938:
-
Assignee: Neng Lu
> flink-connector-pulsar should remove all `PulsarAdmin` calls
>
>
> Key: FLINK-32938
> URL: https://issues.apache.org/jira/browse/FLINK-32938
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
>Reporter: Neng Lu
>Assignee: Neng Lu
>Priority: Major
> Labels: pull-request-available
>
> The flink-connector-pulsar should not access and interact with the admin
> endpoint. This could introduce potential security issues.
> In a production environment, a Pulsar cluster admin will not grant the
> permissions for the flink application to conduct any admin operations.
> Currently, the connector does various admin calls:
> ```{{{}{}}}{{{}{}}}
> PulsarAdmin.topics().getPartitionedTopicMetadata(topic)
> PulsarAdmin.namespaces().getTopics(namespace)
> PulsarAdmin.topics().getLastMessageId(topic)
> PulsarAdmin.topics().getMessageIdByTimestamp(topic, timestamp)
> PulsarAdmin.topics().getSubscriptions(topic)
> PulsarAdmin.topics().createSubscription(topic, subscription,
> MessageId.earliest)
> PulsarAdmin.topics().resetCursor(topic, subscription, initial, !include)
> ```
> We need to replace these calls with consumer or client calls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33111) Flink Pulsar Connector to Pulsar Client Version Mismatch
[ https://issues.apache.org/jira/browse/FLINK-33111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767131#comment-17767131 ] Zili Chen commented on FLINK-33111: --- We upgrade the Pulsar client version in https://github.com/apache/flink-connector-pulsar/pull/25 which brings benefits for the new version. Perhaps we should update the document and people who use Pulsar 2.10.x can use 3.x connector. > Flink Pulsar Connector to Pulsar Client Version Mismatch > > > Key: FLINK-33111 > URL: https://issues.apache.org/jira/browse/FLINK-33111 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.17.1 >Reporter: Jason Kania >Priority: Major > > In the documentation for the Flink Pulsar Connector, > ([https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/pulsar/]) > it indicates that 2.10.0 and above versions of the pulsar client are > supported "You can use the connector with the Pulsar 2.10.0 or higher" and > the pom file entry references the 4.0.0-1.17 version of the connector which > points to the 2.11.0 version of the Pulsar client. However, when using Pulsar > Client 2.10.4 or 2.10.5, the following error is generated: > > java.lang.NoSuchMethodError: 'org.apache.pulsar.client.api.ClientBuilder > org.apache.pulsar.client.api.ClientBuilder.connectionMaxIdleSeconds(int)' > at > org.apache.flink.connector.pulsar.common.config.PulsarClientFactory.createClient(PulsarClientFactory.java:127) > at > org.apache.flink.connector.pulsar.source.reader.PulsarSourceReader.create(PulsarSourceReader.java:266) > at > org.apache.flink.connector.pulsar.source.PulsarSource.createReader(PulsarSource.java:137) > at > org.apache.flink.streaming.api.operators.SourceOperator.initReader(SourceOperator.java:312) > at > org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask.init(SourceOperatorStreamTask.java:93) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:699) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675) > at > org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952) > at > org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) > at java.base/java.lang.Thread.run(Thread.java:829) > > The referenced method 'connectionMaxIdleSeconds' is only available in the > Pulsar 2.11 client when looking at the source code. I am not sure whether the > documentation is wrong and the Flink Pulsar Connector 2.11 is the intended > Pulsar version. However, my understanding is that Pulsar 2.11 is targeted > toward java 17. This would create the need for mixed Java 11 and Java 17 > deployment unless the Pulsar client code is compiled for 2.11. > > Documentation cleanup and a reference to the appropriate Java versions is > needed. A fix to the 1.17.1 Flink pulsar connector may alternatively be > required. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode
[
https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764950#comment-17764950
]
Zili Chen commented on FLINK-33053:
---
But it's possible to add an option to explicitly identify the ownership. You
can open an issue on the Curator JIRA project and let me with the other
maintainers to figure it out.
> Watcher leak in Zookeeper HA mode
> -
>
> Key: FLINK-33053
> URL: https://issues.apache.org/jira/browse/FLINK-33053
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0, 1.17.1
>Reporter: Yangze Guo
>Assignee: Yangze Guo
>Priority: Blocker
> Attachments: 26.dump.zip, 26.log,
> taskmanager_flink-native-test-117-taskmanager-1-9_thread_dump (1).json
>
>
> We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA
> mode. TM's watches on the leader of JobMaster has not been stopped after job
> finished.
> Here is how we re-produce this issue:
> - Start a session cluster and enable Zookeeper HA mode.
> - Continuously and concurrently submit short queries, e.g. WordCount to the
> cluster.
> - echo -n wchp | nc \{zk host} \{zk port} to get current watches.
> We can see a lot of watches on
> /flink/\{cluster_name}/leader/\{job_id}/connection_info.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode
[
https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764948#comment-17764948
]
Zili Chen commented on FLINK-33053:
---
No. Both {{CuratorCache}} and {{TreeCache}} doesn't "own" the path so it's
unclear if other recipes share the same client (connection) set up watches
also. This is different from {{LeaderLatch}} which owns the path so it can
ensure that no one else (should) access the related nodes.
> Watcher leak in Zookeeper HA mode
> -
>
> Key: FLINK-33053
> URL: https://issues.apache.org/jira/browse/FLINK-33053
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0, 1.17.1
>Reporter: Yangze Guo
>Assignee: Yangze Guo
>Priority: Blocker
> Attachments: 26.dump.zip, 26.log,
> taskmanager_flink-native-test-117-taskmanager-1-9_thread_dump (1).json
>
>
> We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA
> mode. TM's watches on the leader of JobMaster has not been stopped after job
> finished.
> Here is how we re-produce this issue:
> - Start a session cluster and enable Zookeeper HA mode.
> - Continuously and concurrently submit short queries, e.g. WordCount to the
> cluster.
> - echo -n wchp | nc \{zk host} \{zk port} to get current watches.
> We can see a lot of watches on
> /flink/\{cluster_name}/leader/\{job_id}/connection_info.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode
[
https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764723#comment-17764723
]
Zili Chen commented on FLINK-33053:
---
But we don't have other shared watchers so we can force remove watches as above.
> Watcher leak in Zookeeper HA mode
> -
>
> Key: FLINK-33053
> URL: https://issues.apache.org/jira/browse/FLINK-33053
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0, 1.17.1
>Reporter: Yangze Guo
>Priority: Blocker
> Attachments: 26.dump.zip, 26.log,
> taskmanager_flink-native-test-117-taskmanager-1-9_thread_dump (1).json
>
>
> We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA
> mode. TM's watches on the leader of JobMaster has not been stopped after job
> finished.
> Here is how we re-produce this issue:
> - Start a session cluster and enable Zookeeper HA mode.
> - Continuously and concurrently submit short queries, e.g. WordCount to the
> cluster.
> - echo -n wchp | nc \{zk host} \{zk port} to get current watches.
> We can see a lot of watches on
> /flink/\{cluster_name}/leader/\{job_id}/connection_info.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode
[
https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764721#comment-17764721
]
Zili Chen commented on FLINK-33053:
---
See https://lists.apache.org/thread/3b9hn9j4c05yfztlr2zcctbg7sqwdh58.
This seems to be a ZK issue that I met one year ago..
> Watcher leak in Zookeeper HA mode
> -
>
> Key: FLINK-33053
> URL: https://issues.apache.org/jira/browse/FLINK-33053
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0, 1.17.1
>Reporter: Yangze Guo
>Priority: Blocker
> Attachments: 26.dump.zip, 26.log,
> taskmanager_flink-native-test-117-taskmanager-1-9_thread_dump (1).json
>
>
> We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA
> mode. TM's watches on the leader of JobMaster has not been stopped after job
> finished.
> Here is how we re-produce this issue:
> - Start a session cluster and enable Zookeeper HA mode.
> - Continuously and concurrently submit short queries, e.g. WordCount to the
> cluster.
> - echo -n wchp | nc \{zk host} \{zk port} to get current watches.
> We can see a lot of watches on
> /flink/\{cluster_name}/leader/\{job_id}/connection_info.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode
[
https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17764702#comment-17764702
]
Zili Chen commented on FLINK-33053:
---
I noticed that the {{TreeCache}}'s close call {{removeWatches}} instead of
{{removeAllWatches}} called by your scripts above.
{{removeWatches}} only remove the watcher in client side so remain the server
side watcher as is.
> Watcher leak in Zookeeper HA mode
> -
>
> Key: FLINK-33053
> URL: https://issues.apache.org/jira/browse/FLINK-33053
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0, 1.17.1
>Reporter: Yangze Guo
>Priority: Blocker
> Attachments: 26.dump.zip, 26.log,
> taskmanager_flink-native-test-117-taskmanager-1-9_thread_dump (1).json
>
>
> We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA
> mode. TM's watches on the leader of JobMaster has not been stopped after job
> finished.
> Here is how we re-produce this issue:
> - Start a session cluster and enable Zookeeper HA mode.
> - Continuously and concurrently submit short queries, e.g. WordCount to the
> cluster.
> - echo -n wchp | nc \{zk host} \{zk port} to get current watches.
> We can see a lot of watches on
> /flink/\{cluster_name}/leader/\{job_id}/connection_info.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33019) Pulsar tests hangs during nightly builds
[ https://issues.apache.org/jira/browse/FLINK-33019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763587#comment-17763587 ] Zili Chen edited comment on FLINK-33019 at 9/11/23 7:57 AM: It seems we can sometimes pass the test https://github.com/apache/flink-connector-pulsar/actions/runs/6133935359 So perhaps it's because we add the new SQL connector whose tests takes more time to complete and then cause a trivial timeout? was (Author: tison): It seems we can sometimes pass the test https://github.com/apache/flink-connector-pulsar/actions/runs/6133935359 So perhaps it's because we add the new SQL connector tests whose takes more time to complete and then cause a trivial timeout? > Pulsar tests hangs during nightly builds > > > Key: FLINK-33019 > URL: https://issues.apache.org/jira/browse/FLINK-33019 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Reporter: Martijn Visser >Priority: Blocker > > https://github.com/apache/flink-connector-pulsar/actions/runs/6067569890/job/16459404675#step:13:25195 > The thread dump shows multiple parked/sleeping threads. No clear indicator of > what's wrong -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33019) Pulsar tests hangs during nightly builds
[ https://issues.apache.org/jira/browse/FLINK-33019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763587#comment-17763587 ] Zili Chen edited comment on FLINK-33019 at 9/11/23 7:57 AM: It seems we can sometimes pass the test https://github.com/apache/flink-connector-pulsar/actions/runs/6133935359 So perhaps it's because we add the new SQL connector tests whose takes more time to complete and then cause a trivial timeout? was (Author: tison): It seems we can sometimes pass the test https://github.com/apache/flink-connector-pulsar/actions/runs/6133935359 So perhaps it's because we add the new SQL connector tests and it takes more time to complete and then cause a trivial timeout? > Pulsar tests hangs during nightly builds > > > Key: FLINK-33019 > URL: https://issues.apache.org/jira/browse/FLINK-33019 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Reporter: Martijn Visser >Priority: Blocker > > https://github.com/apache/flink-connector-pulsar/actions/runs/6067569890/job/16459404675#step:13:25195 > The thread dump shows multiple parked/sleeping threads. No clear indicator of > what's wrong -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33019) Pulsar tests hangs during nightly builds
[ https://issues.apache.org/jira/browse/FLINK-33019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17763587#comment-17763587 ] Zili Chen commented on FLINK-33019: --- It seems we can sometimes pass the test https://github.com/apache/flink-connector-pulsar/actions/runs/6133935359 So perhaps it's because we add the new SQL connector tests and it takes more time to complete and then cause a trivial timeout? > Pulsar tests hangs during nightly builds > > > Key: FLINK-33019 > URL: https://issues.apache.org/jira/browse/FLINK-33019 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Reporter: Martijn Visser >Priority: Blocker > > https://github.com/apache/flink-connector-pulsar/actions/runs/6067569890/job/16459404675#step:13:25195 > The thread dump shows multiple parked/sleeping threads. No clear indicator of > what's wrong -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode
[
https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762965#comment-17762965
]
Zili Chen commented on FLINK-33053:
---
The log seems trimed. I saw:
2023-09-08 11:09:03,738 DEBUG
org.apache.flink.shaded.curator5.org.apache.curator.framework.imps.WatcherRemovalManager
[] - Removing watcher for path:
/flink/flink-native-test-117/leader/7db5c7316828f598234677e2169e7b0f/connection_info
So the TM has issued watcher removal request.
> Watcher leak in Zookeeper HA mode
> -
>
> Key: FLINK-33053
> URL: https://issues.apache.org/jira/browse/FLINK-33053
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.17.1
>Reporter: Yangze Guo
>Priority: Critical
> Attachments: 26.dump.zip, 26.log
>
>
> We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA
> mode. TM's watches on the leader of JobMaster has not been stopped after job
> finished.
> Here is how we re-produce this issue:
> - Start a session cluster and enable Zookeeper HA mode.
> - Continuously and concurrently submit short queries, e.g. WordCount to the
> cluster.
> - echo -n wchp | nc \{zk host} \{zk port} to get current watches.
> We can see a lot of watches on
> /flink/\{cluster_name}/leader/\{job_id}/connection_info.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode
[
https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762657#comment-17762657
]
Zili Chen commented on FLINK-33053:
---
Perhaps you can enable debug logs and check "Removing watcher for path: " from
Curator to see if the related watchers are issued removing.
> Watcher leak in Zookeeper HA mode
> -
>
> Key: FLINK-33053
> URL: https://issues.apache.org/jira/browse/FLINK-33053
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.17.1
>Reporter: Yangze Guo
>Priority: Critical
>
> We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA
> mode. TM's watches on the leader of JobMaster has not been stopped after job
> finished.
> Here is how we re-produce this issue:
> - Start a session cluster and enable Zookeeper HA mode.
> - Continuously and concurrently submit short queries, e.g. WordCount to the
> cluster.
> - echo -n wchp | nc \{zk host} \{zk port} to get current watches.
> We can see a lot of watches on
> /flink/\{cluster_name}/leader/\{job_id}/connection_info.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33053) Watcher leak in Zookeeper HA mode
[
https://issues.apache.org/jira/browse/FLINK-33053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762599#comment-17762599
]
Zili Chen commented on FLINK-33053:
---
The recipe in use is {{TreeCache}}, which doesn't change from 5.0.0. And it
also closes watches on {{close}}.
Do you have a bisect which version introduced this regression?
> Watcher leak in Zookeeper HA mode
> -
>
> Key: FLINK-33053
> URL: https://issues.apache.org/jira/browse/FLINK-33053
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0, 1.17.1
>Reporter: Yangze Guo
>Priority: Critical
>
> We observe a watcher leak in our OLAP stress test when enabling Zookeeper HA
> mode. TM's watches on the leader of JobMaster has not been stopped after job
> finished.
> Here is how we re-produce this issue:
> - Start a session cluster and enable Zookeeper HA mode.
> - Continuously and concurrently submit short queries, e.g. WordCount to the
> cluster.
> - echo -n wchp | nc \{zk host} \{zk port} to get current watches.
> We can see a lot of watches on
> /flink/\{cluster_name}/leader/\{job_id}/connection_info.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33019) Pulsar tests hangs during nightly builds
[ https://issues.apache.org/jira/browse/FLINK-33019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17762412#comment-17762412 ] Zili Chen commented on FLINK-33019: --- [~martijnvisser] It seems all for the SNAPSHOT version and with JDK 11. Is there anything that can be a (internal) breaking change with this property? > Pulsar tests hangs during nightly builds > > > Key: FLINK-33019 > URL: https://issues.apache.org/jira/browse/FLINK-33019 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Reporter: Martijn Visser >Priority: Blocker > > https://github.com/apache/flink-connector-pulsar/actions/runs/6067569890/job/16459404675#step:13:25195 > The thread dump shows multiple parked/sleeping threads. No clear indicator of > what's wrong -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29390) Pulsar SQL Connector: SQLClient E2E testing
[ https://issues.apache.org/jira/browse/FLINK-29390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-29390: - Assignee: Zili Chen > Pulsar SQL Connector: SQLClient E2E testing > --- > > Key: FLINK-29390 > URL: https://issues.apache.org/jira/browse/FLINK-29390 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Yufei Zhang >Assignee: Zili Chen >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31427) Pulsar Catalog support with Schema translation
[ https://issues.apache.org/jira/browse/FLINK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-31427. - Fix Version/s: (was: pulsar-4.0.1) Resolution: Later It's rare that users would use Pulsar as Catalog source. Postpone for later. > Pulsar Catalog support with Schema translation > -- > > Key: FLINK-31427 > URL: https://issues.apache.org/jira/browse/FLINK-31427 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available, stale-assigned > > This task will make the Pulsar serve as the Flink catalog. It will expose the > Pulsar's namespace as the Flink's database, the topic as the Flink's table. > You can easily create a table and database on Pulsar. The table can be > consumed by other clients with a valid schema check. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29360) Pulsar Table Connector Documentation
[ https://issues.apache.org/jira/browse/FLINK-29360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-29360: - Assignee: Zili Chen > Pulsar Table Connector Documentation > > > Key: FLINK-29360 > URL: https://issues.apache.org/jira/browse/FLINK-29360 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Yufei Zhang >Assignee: Zili Chen >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29359) Pulsar Table Connector pom config and packaging
[ https://issues.apache.org/jira/browse/FLINK-29359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760687#comment-17760687 ] Zili Chen commented on FLINK-29359: --- [~affe] [~syhily] [~leonard] I'm unsure if this ticket means to support SQL jar packaging. I can see that we already have the module flink-sql-connector-pulsar. > Pulsar Table Connector pom config and packaging > --- > > Key: FLINK-29359 > URL: https://issues.apache.org/jira/browse/FLINK-29359 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Yufei Zhang >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-29357) Pulsar Table Sink code: implementation
[ https://issues.apache.org/jira/browse/FLINK-29357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen resolved FLINK-29357. --- Fix Version/s: pulsar-4.1.0 Resolution: Fixed master via c71fc862e0d4a782c19f361d3bf581da836cca79 > Pulsar Table Sink code: implementation > -- > > Key: FLINK-29357 > URL: https://issues.apache.org/jira/browse/FLINK-29357 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Yufei Zhang >Assignee: Zili Chen >Priority: Minor > Fix For: pulsar-4.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29357) Pulsar Table Sink code: implementation
[ https://issues.apache.org/jira/browse/FLINK-29357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-29357: - Assignee: Zili Chen > Pulsar Table Sink code: implementation > -- > > Key: FLINK-29357 > URL: https://issues.apache.org/jira/browse/FLINK-29357 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Yufei Zhang >Assignee: Zili Chen >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-29358) Pulsar Table Connector testing
[ https://issues.apache.org/jira/browse/FLINK-29358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen resolved FLINK-29358. --- Fix Version/s: pulsar-4.1.0 Assignee: Zili Chen Resolution: Fixed master via c71fc862e0d4a782c19f361d3bf581da836cca79 > Pulsar Table Connector testing > -- > > Key: FLINK-29358 > URL: https://issues.apache.org/jira/browse/FLINK-29358 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Yufei Zhang >Assignee: Zili Chen >Priority: Minor > Fix For: pulsar-4.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-29356) Pulsar Table Source code :implementation
[ https://issues.apache.org/jira/browse/FLINK-29356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen resolved FLINK-29356. --- Fix Version/s: pulsar-4.1.0 Resolution: Fixed master via c71fc862e0d4a782c19f361d3bf581da836cca79 > Pulsar Table Source code :implementation > > > Key: FLINK-29356 > URL: https://issues.apache.org/jira/browse/FLINK-29356 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Yufei Zhang >Assignee: Zili Chen >Priority: Minor > Fix For: pulsar-4.1.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29356) Pulsar Table Source code :implementation
[ https://issues.apache.org/jira/browse/FLINK-29356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-29356: - Assignee: Zili Chen > Pulsar Table Source code :implementation > > > Key: FLINK-29356 > URL: https://issues.apache.org/jira/browse/FLINK-29356 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Yufei Zhang >Assignee: Zili Chen >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-26203) Support Table API in Pulsar Connector
[ https://issues.apache.org/jira/browse/FLINK-26203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760680#comment-17760680 ] Zili Chen commented on FLINK-26203: --- Yeah. I'm updating the tickets here now. > Support Table API in Pulsar Connector > - > > Key: FLINK-26203 > URL: https://issues.apache.org/jira/browse/FLINK-26203 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufan Sheng >Priority: Minor > Labels: Pulsar, auto-deprioritized-major, pull-request-available > > Currently Pulsar connector only supports DataStream API. We plan to support > Table API as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-29527) Make unknownFieldsIndices work for single ParquetReader
[ https://issues.apache.org/jira/browse/FLINK-29527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen resolved FLINK-29527. --- Resolution: Fixed master via 50622df1f01cd9cade78bfe2add5a7faff678e3e Shall we pick to other version? It seems somehow a feature catch up or fix. > Make unknownFieldsIndices work for single ParquetReader > --- > > Key: FLINK-29527 > URL: https://issues.apache.org/jira/browse/FLINK-29527 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Sun Shun >Assignee: Sun Shun >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Currently, from the improvement FLINK-23715, Flink use a collection named > `unknownFieldsIndices` to track the nonexistent fields, and it is kept inside > the `ParquetVectorizedInputFormat`, and applied to all parquet files under > given path. > However, some fields may only be nonexistent in some of the historical > parquet files, while exist in latest ones. And based on > `unknownFieldsIndices`, flink will always skip these fields, even thought > they are existing in the later parquets. > As a result, the value of these fields will become empty when they are > nonexistent in some historical parquet files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29527) Make unknownFieldsIndices work for single ParquetReader
[ https://issues.apache.org/jira/browse/FLINK-29527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen updated FLINK-29527: -- Fix Version/s: 1.19.0 > Make unknownFieldsIndices work for single ParquetReader > --- > > Key: FLINK-29527 > URL: https://issues.apache.org/jira/browse/FLINK-29527 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Sun Shun >Assignee: Sun Shun >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Currently, from the improvement FLINK-23715, Flink use a collection named > `unknownFieldsIndices` to track the nonexistent fields, and it is kept inside > the `ParquetVectorizedInputFormat`, and applied to all parquet files under > given path. > However, some fields may only be nonexistent in some of the historical > parquet files, while exist in latest ones. And based on > `unknownFieldsIndices`, flink will always skip these fields, even thought > they are existing in the later parquets. > As a result, the value of these fields will become empty when they are > nonexistent in some historical parquet files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32645) Flink pulsar sink is having poor performance
[ https://issues.apache.org/jira/browse/FLINK-32645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen updated FLINK-32645: -- Fix Version/s: pulsar-3.0.2 > Flink pulsar sink is having poor performance > > > Key: FLINK-32645 > URL: https://issues.apache.org/jira/browse/FLINK-32645 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.2 > Environment: !Screenshot 2023-07-22 at 1.59.42 PM.png!!Screenshot > 2023-07-22 at 2.03.53 PM.png! > >Reporter: Vijaya Bhaskar V >Assignee: Zili Chen >Priority: Major > Fix For: pulsar-3.0.2 > > Attachments: Screenshot 2023-07-22 at 2.03.53 PM.png, Screenshot > 2023-07-22 at 2.56.55 PM.png, Screenshot 2023-07-22 at 3.45.21 PM-1.png, > Screenshot 2023-07-22 at 3.45.21 PM.png, pom.xml > > > Found following issue with flink pulsar sink: > > Flink pulsar sink is always waiting while enqueueing the message and making > the task slot busy no matter how many free slots we provide. Attached the > screen shot of the same > Just sending messages of less rate 8k msg/sec and stand alone flink job with > discarding sink is able to receive full rate if 8K msg/sec > Where as pulsar sink was consuming only upto 2K msg/sec and the sink is > always busy waiting. Snapshot of thread dump attached. > Also snap shot of flink stream graph attached > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-32645) Flink pulsar sink is having poor performance
[ https://issues.apache.org/jira/browse/FLINK-32645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-32645: - Assignee: Zili Chen > Flink pulsar sink is having poor performance > > > Key: FLINK-32645 > URL: https://issues.apache.org/jira/browse/FLINK-32645 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.2 > Environment: !Screenshot 2023-07-22 at 1.59.42 PM.png!!Screenshot > 2023-07-22 at 2.03.53 PM.png! > >Reporter: Vijaya Bhaskar V >Assignee: Zili Chen >Priority: Major > Attachments: Screenshot 2023-07-22 at 2.03.53 PM.png, Screenshot > 2023-07-22 at 2.56.55 PM.png, Screenshot 2023-07-22 at 3.45.21 PM-1.png, > Screenshot 2023-07-22 at 3.45.21 PM.png, pom.xml > > > Found following issue with flink pulsar sink: > > Flink pulsar sink is always waiting while enqueueing the message and making > the task slot busy no matter how many free slots we provide. Attached the > screen shot of the same > Just sending messages of less rate 8k msg/sec and stand alone flink job with > discarding sink is able to receive full rate if 8K msg/sec > Where as pulsar sink was consuming only upto 2K msg/sec and the sink is > always busy waiting. Snapshot of thread dump attached. > Also snap shot of flink stream graph attached > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32645) Flink pulsar sink is having poor performance
[ https://issues.apache.org/jira/browse/FLINK-32645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746193#comment-17746193 ] Zili Chen commented on FLINK-32645: --- The new version series use a differnt GAV. You can find them at https://mvnrepository.com/artifact/org.apache.flink/flink-connector-pulsar. 3.0.1 should be compatible with Flink 1.16. > Flink pulsar sink is having poor performance > > > Key: FLINK-32645 > URL: https://issues.apache.org/jira/browse/FLINK-32645 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.2 > Environment: !Screenshot 2023-07-22 at 1.59.42 PM.png!!Screenshot > 2023-07-22 at 2.03.53 PM.png! > >Reporter: Vijaya Bhaskar V >Priority: Major > Attachments: Screenshot 2023-07-22 at 2.03.53 PM.png, Screenshot > 2023-07-22 at 2.56.55 PM.png, Screenshot 2023-07-22 at 3.45.21 PM-1.png, > Screenshot 2023-07-22 at 3.45.21 PM.png > > > Found following issue with flink pulsar sink: > > Flink pulsar sink is always waiting while enqueueing the message and making > the task slot busy no matter how many free slots we provide. Attached the > screen shot of the same > Just sending messages of less rate 8k msg/sec and stand alone flink job with > discarding sink is able to receive full rate if 8K msg/sec > Where as pulsar sink was consuming only upto 2K msg/sec and the sink is > always busy waiting. Snapshot of thread dump attached. > Also snap shot of flink stream graph attached > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32645) Flink pulsar sink is having poor performance
[ https://issues.apache.org/jira/browse/FLINK-32645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17746177#comment-17746177 ] Zili Chen commented on FLINK-32645: --- What version of Flink Pulsar connector did you use? 1.16.2 is Flink version and now Pulsar Flink connector takes it own version series https://github.com/apache/flink-connector-pulsar. Generally, 3.0.1 or 4.0.0. > Flink pulsar sink is having poor performance > > > Key: FLINK-32645 > URL: https://issues.apache.org/jira/browse/FLINK-32645 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.2 > Environment: !Screenshot 2023-07-22 at 1.59.42 PM.png!!Screenshot > 2023-07-22 at 2.03.53 PM.png! > >Reporter: Vijaya Bhaskar V >Priority: Major > Attachments: Screenshot 2023-07-22 at 2.03.53 PM.png, Screenshot > 2023-07-22 at 2.56.55 PM.png, Screenshot 2023-07-22 at 3.45.21 PM-1.png, > Screenshot 2023-07-22 at 3.45.21 PM.png > > > Found following issue with flink pulsar sink: > > Flink pulsar sink is always waiting while enqueueing the message and making > the task slot busy no matter how many free slots we provide. Attached the > screen shot of the same > Just sending messages of less rate 8k msg/sec and stand alone flink job with > discarding sink is able to receive full rate if 8K msg/sec > Where as pulsar sink was consuming only upto 2K msg/sec and the sink is > always busy waiting. Snapshot of thread dump attached. > Also snap shot of flink stream graph attached > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-24302) Direct buffer memory leak on Pulsar connector with Java 11
[
https://issues.apache.org/jira/browse/FLINK-24302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17745015#comment-17745015
]
Zili Chen edited comment on FLINK-24302 at 7/20/23 10:24 AM:
-
Temporarily work around with
https://github.com/apache/flink-connector-pulsar/commit/3b6c3aff8aeca3cc17673bbc84b90b70c1c680a9
for test coverage.
If there is good news from the Pulsar side, we can follow up for tuning.
was (Author: tison):
Temporarily work around with
https://github.com/apache/flink-connector-pulsar/commit/3b6c3aff8aeca3cc17673bbc84b90b70c1c680a9
for test coverage.
If there is good news from the Pulsar side, we can follow up for tunning.
> Direct buffer memory leak on Pulsar connector with Java 11
> --
>
> Key: FLINK-24302
> URL: https://issues.apache.org/jira/browse/FLINK-24302
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Yufan Sheng
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: pulsar-4.1.0
>
>
> Running the Pulsar connector with multiple split readers on Java 11 could
> throw {{a java.lang.OutOfMemoryError exception}}.
> {code:java}
> Caused by: java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503)
> ... 42 more
> Caused by:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:249)
> ... 39 more
> Caused by: org.apache.pulsar.shade.io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104)
> at
> org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:303)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.NettyRequestSender.writeRequest(NettyRequestSender.java:420)
> ... 23 more
> {code}
> The reason is that under Java 11, the Netty will allocate memory from the
> pool of Java Direct Memory and is affected by the MaxDirectMemory limit.
> Under Java 8, it allocates native memory and is not affected by that setting.
> We have to reduce the direct memory usage by using a newer Pulsar client
> which has a memory-limits configuration.
> This issue is addressed on Pulsar, and
>
[jira] [Closed] (FLINK-24302) Direct buffer memory leak on Pulsar connector with Java 11
[
https://issues.apache.org/jira/browse/FLINK-24302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-24302.
-
Resolution: Workaround
Temporarily work around with
https://github.com/apache/flink-connector-pulsar/commit/3b6c3aff8aeca3cc17673bbc84b90b70c1c680a9
for test coverage.
If there is good news from the Pulsar side, we can follow up for tunning.
> Direct buffer memory leak on Pulsar connector with Java 11
> --
>
> Key: FLINK-24302
> URL: https://issues.apache.org/jira/browse/FLINK-24302
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Yufan Sheng
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: pulsar-4.1.0
>
>
> Running the Pulsar connector with multiple split readers on Java 11 could
> throw {{a java.lang.OutOfMemoryError exception}}.
> {code:java}
> Caused by: java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503)
> ... 42 more
> Caused by:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:249)
> ... 39 more
> Caused by: org.apache.pulsar.shade.io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104)
> at
> org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:303)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.NettyRequestSender.writeRequest(NettyRequestSender.java:420)
> ... 23 more
> {code}
> The reason is that under Java 11, the Netty will allocate memory from the
> pool of Java Direct Memory and is affected by the MaxDirectMemory limit.
> Under Java 8, it allocates native memory and is not affected by that setting.
> We have to reduce the direct memory usage by using a newer Pulsar client
> which has a memory-limits configuration.
> This issue is addressed on Pulsar, and
> [PIP-74|https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits]
> has been created for resolving this issue.
> We should keep this issue open with no resolved versions until Pulsar
> provides a new client with memory limits.
> h2. Update: 2022/08/04
> The memory limit on consumer API has been released
>
[jira] [Updated] (FLINK-24302) Direct buffer memory leak on Pulsar connector with Java 11
[
https://issues.apache.org/jira/browse/FLINK-24302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen updated FLINK-24302:
--
Fix Version/s: pulsar-4.1.0
(was: pulsar-4.0.1)
> Direct buffer memory leak on Pulsar connector with Java 11
> --
>
> Key: FLINK-24302
> URL: https://issues.apache.org/jira/browse/FLINK-24302
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Yufan Sheng
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: pulsar-4.1.0
>
>
> Running the Pulsar connector with multiple split readers on Java 11 could
> throw {{a java.lang.OutOfMemoryError exception}}.
> {code:java}
> Caused by: java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503)
> ... 42 more
> Caused by:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:249)
> ... 39 more
> Caused by: org.apache.pulsar.shade.io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104)
> at
> org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:303)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.NettyRequestSender.writeRequest(NettyRequestSender.java:420)
> ... 23 more
> {code}
> The reason is that under Java 11, the Netty will allocate memory from the
> pool of Java Direct Memory and is affected by the MaxDirectMemory limit.
> Under Java 8, it allocates native memory and is not affected by that setting.
> We have to reduce the direct memory usage by using a newer Pulsar client
> which has a memory-limits configuration.
> This issue is addressed on Pulsar, and
> [PIP-74|https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits]
> has been created for resolving this issue.
> We should keep this issue open with no resolved versions until Pulsar
> provides a new client with memory limits.
> h2. Update: 2022/08/04
> The memory limit on consumer API has been released
> https://github.com/apache/pulsar/pull/15216, we need to add
> autoScaledReceiverQueueSizeEnabled option to enable this feature. This memory
> limit will get released on Pulsar
[jira] [Assigned] (FLINK-24302) Direct buffer memory leak on Pulsar connector with Java 11
[
https://issues.apache.org/jira/browse/FLINK-24302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen reassigned FLINK-24302:
-
Assignee: Zili Chen
> Direct buffer memory leak on Pulsar connector with Java 11
> --
>
> Key: FLINK-24302
> URL: https://issues.apache.org/jira/browse/FLINK-24302
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Yufan Sheng
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: pulsar-4.0.1
>
>
> Running the Pulsar connector with multiple split readers on Java 11 could
> throw {{a java.lang.OutOfMemoryError exception}}.
> {code:java}
> Caused by: java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503)
> ... 42 more
> Caused by:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:249)
> ... 39 more
> Caused by: org.apache.pulsar.shade.io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104)
> at
> org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:303)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.NettyRequestSender.writeRequest(NettyRequestSender.java:420)
> ... 23 more
> {code}
> The reason is that under Java 11, the Netty will allocate memory from the
> pool of Java Direct Memory and is affected by the MaxDirectMemory limit.
> Under Java 8, it allocates native memory and is not affected by that setting.
> We have to reduce the direct memory usage by using a newer Pulsar client
> which has a memory-limits configuration.
> This issue is addressed on Pulsar, and
> [PIP-74|https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits]
> has been created for resolving this issue.
> We should keep this issue open with no resolved versions until Pulsar
> provides a new client with memory limits.
> h2. Update: 2022/08/04
> The memory limit on consumer API has been released
> https://github.com/apache/pulsar/pull/15216, we need to add
> autoScaledReceiverQueueSizeEnabled option to enable this feature. This memory
> limit will get released on Pulsar 2.11.0. We will get this fixed after that.
>
[jira] [Assigned] (FLINK-32630) The log level of job failed info should change from INFO to WARN/ERROR if job failed
[
https://issues.apache.org/jira/browse/FLINK-32630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen reassigned FLINK-32630:
-
Assignee: Matt Wang
> The log level of job failed info should change from INFO to WARN/ERROR if job
> failed
>
>
> Key: FLINK-32630
> URL: https://issues.apache.org/jira/browse/FLINK-32630
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission, Runtime / Coordination
>Affects Versions: 1.17.1
>Reporter: Matt Wang
>Assignee: Matt Wang
>Priority: Minor
>
> When a job fails to submit or run, the following log level should be changed
> to WARN or ERROR, INFO will confuse users
> {code:java}
> 2023-07-14 20:05:26,863 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> flink_test_job (08eefd50) switched from state FAILING
> to FAILED.
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> FailureRateRestartBackoffTimeStrategy(FailureRateRestartBackoffTimeStrategy(failuresIntervalMS=240,backoffTimeMS=2,maxFailuresPerInterval=100)
>
> 2023-07-14 20:05:26,889 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
> 08eefd50 reached terminal state FAILED.
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> FailureRateRestartBackoffTimeStrategy(FailureRateRestartBackoffTimeStrategy(failuresIntervalMS=240,backoffTimeMS=2,maxFailuresPerInterval=100)
> 2023-07-14 20:05:26,956 INFO
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap
> [] - Application FAILED:
> java.util.concurrent.CompletionException:
> org.apache.flink.client.deployment.application.ApplicationExecutionException:
> Could not execute application.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32630) The log level of job failed info should change from INFO to WARN/ERROR if job failed
[
https://issues.apache.org/jira/browse/FLINK-32630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744605#comment-17744605
]
Zili Chen commented on FLINK-32630:
---
You can ping me on a patch ready.
> The log level of job failed info should change from INFO to WARN/ERROR if job
> failed
>
>
> Key: FLINK-32630
> URL: https://issues.apache.org/jira/browse/FLINK-32630
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission, Runtime / Coordination
>Affects Versions: 1.17.1
>Reporter: Matt Wang
>Assignee: Matt Wang
>Priority: Minor
>
> When a job fails to submit or run, the following log level should be changed
> to WARN or ERROR, INFO will confuse users
> {code:java}
> 2023-07-14 20:05:26,863 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
> flink_test_job (08eefd50) switched from state FAILING
> to FAILED.
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> FailureRateRestartBackoffTimeStrategy(FailureRateRestartBackoffTimeStrategy(failuresIntervalMS=240,backoffTimeMS=2,maxFailuresPerInterval=100)
>
> 2023-07-14 20:05:26,889 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job
> 08eefd50 reached terminal state FAILED.
> org.apache.flink.runtime.JobException: Recovery is suppressed by
> FailureRateRestartBackoffTimeStrategy(FailureRateRestartBackoffTimeStrategy(failuresIntervalMS=240,backoffTimeMS=2,maxFailuresPerInterval=100)
> 2023-07-14 20:05:26,956 INFO
> org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap
> [] - Application FAILED:
> java.util.concurrent.CompletionException:
> org.apache.flink.client.deployment.application.ApplicationExecutionException:
> Could not execute application.{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-24302) Direct buffer memory leak on Pulsar connector with Java 11
[
https://issues.apache.org/jira/browse/FLINK-24302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744604#comment-17744604
]
Zili Chen commented on FLINK-24302:
---
Pending to close. This should be an issue on the Pulsar side. Little thing we
can do only from the connector side.
> Direct buffer memory leak on Pulsar connector with Java 11
> --
>
> Key: FLINK-24302
> URL: https://issues.apache.org/jira/browse/FLINK-24302
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Yufan Sheng
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: pulsar-4.0.1
>
>
> Running the Pulsar connector with multiple split readers on Java 11 could
> throw {{a java.lang.OutOfMemoryError exception}}.
> {code:java}
> Caused by: java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503)
> ... 42 more
> Caused by:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:249)
> ... 39 more
> Caused by: org.apache.pulsar.shade.io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104)
> at
> org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:303)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.NettyRequestSender.writeRequest(NettyRequestSender.java:420)
> ... 23 more
> {code}
> The reason is that under Java 11, the Netty will allocate memory from the
> pool of Java Direct Memory and is affected by the MaxDirectMemory limit.
> Under Java 8, it allocates native memory and is not affected by that setting.
> We have to reduce the direct memory usage by using a newer Pulsar client
> which has a memory-limits configuration.
> This issue is addressed on Pulsar, and
> [PIP-74|https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits]
> has been created for resolving this issue.
> We should keep this issue open with no resolved versions until Pulsar
> provides a new client with memory limits.
> h2. Update: 2022/08/04
> The memory limit on consumer API has been released
> https://github.com/apache/pulsar/pull/15216, we need to add
> autoScaledReceiverQueueSizeEnabled option to enable this feature.
[jira] [Commented] (FLINK-24302) Direct buffer memory leak on Pulsar connector with Java 11
[
https://issues.apache.org/jira/browse/FLINK-24302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17744603#comment-17744603
]
Zili Chen commented on FLINK-24302:
---
Workaround to turn on tests for JDK11 in
https://github.com/apache/flink-connector-pulsar/pull/55.
> Direct buffer memory leak on Pulsar connector with Java 11
> --
>
> Key: FLINK-24302
> URL: https://issues.apache.org/jira/browse/FLINK-24302
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Yufan Sheng
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: pulsar-4.0.1
>
>
> Running the Pulsar connector with multiple split readers on Java 11 could
> throw {{a java.lang.OutOfMemoryError exception}}.
> {code:java}
> Caused by: java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
> at
> java.base/java.util.concurrent.CompletableFuture$OrApply.tryFire(CompletableFuture.java:1503)
> ... 42 more
> Caused by:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason: java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:249)
> ... 39 more
> Caused by: org.apache.pulsar.shade.io.netty.handler.codec.EncoderException:
> java.lang.OutOfMemoryError: Direct buffer memory
> at
> org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:104)
> at
> org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:346)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:303)
> at
> org.apache.pulsar.shade.io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
> at
> org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> at
> org.apache.pulsar.shade.io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.request.NettyRequestSender.writeRequest(NettyRequestSender.java:420)
> ... 23 more
> {code}
> The reason is that under Java 11, the Netty will allocate memory from the
> pool of Java Direct Memory and is affected by the MaxDirectMemory limit.
> Under Java 8, it allocates native memory and is not affected by that setting.
> We have to reduce the direct memory usage by using a newer Pulsar client
> which has a memory-limits configuration.
> This issue is addressed on Pulsar, and
> [PIP-74|https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits]
> has been created for resolving this issue.
> We should keep this issue open with no resolved versions until Pulsar
> provides a new client with memory limits.
> h2. Update: 2022/08/04
> The memory limit on consumer API has been released
> https://github.com/apache/pulsar/pull/15216, we need to add
> autoScaledReceiverQueueSizeEnabled option to enable this feature. This memory
>
[jira] [Created] (FLINK-32625) MiniClusterTestEnvironment supports customized MiniClusterResourceConfiguration
Zili Chen created FLINK-32625: - Summary: MiniClusterTestEnvironment supports customized MiniClusterResourceConfiguration Key: FLINK-32625 URL: https://issues.apache.org/jira/browse/FLINK-32625 Project: Flink Issue Type: Improvement Components: Test Infrastructure Reporter: Zili Chen Assignee: Zili Chen -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32612) Upgrade Pulsar version to 3.0.0 for the connector
[ https://issues.apache.org/jira/browse/FLINK-32612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-32612. - Fix Version/s: pulsar-4.0.1 Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/commit/279a678dd4f39e1e268d911dc4096d23ab15bc73 > Upgrade Pulsar version to 3.0.0 for the connector > - > > Key: FLINK-32612 > URL: https://issues.apache.org/jira/browse/FLINK-32612 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Reporter: Zili Chen >Assignee: Zili Chen >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.0.1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32612) Upgrade Pulsar version to 3.0.0 for the connector
Zili Chen created FLINK-32612: - Summary: Upgrade Pulsar version to 3.0.0 for the connector Key: FLINK-32612 URL: https://issues.apache.org/jira/browse/FLINK-32612 Project: Flink Issue Type: Improvement Components: Connectors / Pulsar Reporter: Zili Chen Assignee: Zili Chen -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32585) Filter javax.xml.bind:jaxb-api false positive for Pulsar connector
[ https://issues.apache.org/jira/browse/FLINK-32585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-32585. - Resolution: Fixed master via d7a3b3847dc5c680b32d1997d448b7dac44e529c > Filter javax.xml.bind:jaxb-api false positive for Pulsar connector > -- > > Key: FLINK-32585 > URL: https://issues.apache.org/jira/browse/FLINK-32585 > Project: Flink > Issue Type: Improvement > Components: Build System / CI >Reporter: Zili Chen >Assignee: Zili Chen >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32585) Filter javax.xml.bind:jaxb-api false positive for Pulsar connector
Zili Chen created FLINK-32585: - Summary: Filter javax.xml.bind:jaxb-api false positive for Pulsar connector Key: FLINK-32585 URL: https://issues.apache.org/jira/browse/FLINK-32585 Project: Flink Issue Type: Improvement Components: Build System / CI Reporter: Zili Chen Assignee: Zili Chen -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32003) Release 3.0.0-1.16 and 1.16.1 doesn't work with OAuth2
[
https://issues.apache.org/jira/browse/FLINK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17725683#comment-17725683
]
Zili Chen commented on FLINK-32003:
---
The main branch already has more later version. It's not affected by this issue.
> Release 3.0.0-1.16 and 1.16.1 doesn't work with OAuth2
> --
>
> Key: FLINK-32003
> URL: https://issues.apache.org/jira/browse/FLINK-32003
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: pulsar-3.0.0
>Reporter: Neng Lu
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-3.0.1
>
>
> The release for 3.0.0-1.16 and 1.16.1 depends on Pulsar client version 2.10.1.
> There is an issue using OAuth2 with this client version which results in the
> following error.
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.pulsar.client.admin.PulsarAdminException:
> java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason:
> https://func-test-31a67160-533f-4a5f-81a8-30b6221f34a9.gcp-shared-gcp-usce1-martin.streamnative.g.snio.cloud:443
> at me.nlu.pulsar.PulsarAdminTester.main(PulsarAdminTester.java:56)
> Caused by: org.apache.pulsar.client.admin.PulsarAdminException:
> java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason:
> https://func-test-31a67160-533f-4a5f-81a8-30b6221f34a9.gcp-shared-gcp-usce1-martin.streamnative.g.snio.cloud:443
> at
> org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251)
> at
> org.apache.pulsar.client.admin.internal.TopicsImpl$1.failed(TopicsImpl.java:187)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:292)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:274)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:244)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178)
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:218)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:277)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:181)
> at
>
[jira] [Commented] (FLINK-32003) Release 3.0.0-1.16 and 1.16.1 doesn't work with OAuth2
[
https://issues.apache.org/jira/browse/FLINK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720101#comment-17720101
]
Zili Chen commented on FLINK-32003:
---
Thanks for reporting this issue [~nlu90]! It's reasonable to upgrade patch
versions as long as it holds the semantic.
> Release 3.0.0-1.16 and 1.16.1 doesn't work with OAuth2
> --
>
> Key: FLINK-32003
> URL: https://issues.apache.org/jira/browse/FLINK-32003
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Reporter: Neng Lu
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
>
> The release for 3.0.0-1.16 and 1.16.1 depends on Pulsar client version 2.10.1.
> There is an issue using OAuth2 with this client version which results in the
> following error.
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.pulsar.client.admin.PulsarAdminException:
> java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason:
> https://func-test-31a67160-533f-4a5f-81a8-30b6221f34a9.gcp-shared-gcp-usce1-martin.streamnative.g.snio.cloud:443
> at me.nlu.pulsar.PulsarAdminTester.main(PulsarAdminTester.java:56)
> Caused by: org.apache.pulsar.client.admin.PulsarAdminException:
> java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason:
> https://func-test-31a67160-533f-4a5f-81a8-30b6221f34a9.gcp-shared-gcp-usce1-martin.streamnative.g.snio.cloud:443
> at
> org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251)
> at
> org.apache.pulsar.client.admin.internal.TopicsImpl$1.failed(TopicsImpl.java:187)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:292)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:274)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:244)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178)
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:218)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:277)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:181)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.channel.NettyConnectListener$1.onFailure(NettyConnectListener.java:151)
>
[jira] [Assigned] (FLINK-32003) Release 3.0.0-1.16 and 1.16.1 doesn't work with OAuth2
[
https://issues.apache.org/jira/browse/FLINK-32003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen reassigned FLINK-32003:
-
Assignee: Zili Chen
> Release 3.0.0-1.16 and 1.16.1 doesn't work with OAuth2
> --
>
> Key: FLINK-32003
> URL: https://issues.apache.org/jira/browse/FLINK-32003
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Reporter: Neng Lu
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
>
> The release for 3.0.0-1.16 and 1.16.1 depends on Pulsar client version 2.10.1.
> There is an issue using OAuth2 with this client version which results in the
> following error.
> {code:java}
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.pulsar.client.admin.PulsarAdminException:
> java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason:
> https://func-test-31a67160-533f-4a5f-81a8-30b6221f34a9.gcp-shared-gcp-usce1-martin.streamnative.g.snio.cloud:443
> at me.nlu.pulsar.PulsarAdminTester.main(PulsarAdminTester.java:56)
> Caused by: org.apache.pulsar.client.admin.PulsarAdminException:
> java.util.concurrent.CompletionException:
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
> Could not complete the operation. Number of retries has been exhausted.
> Failed reason:
> https://func-test-31a67160-533f-4a5f-81a8-30b6221f34a9.gcp-shared-gcp-usce1-martin.streamnative.g.snio.cloud:443
> at
> org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251)
> at
> org.apache.pulsar.client.admin.internal.TopicsImpl$1.failed(TopicsImpl.java:187)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:247)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processFailure(ClientRuntime.java:242)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.access$100(ClientRuntime.java:62)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.lambda$failure$1(ClientRuntime.java:178)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:292)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:274)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:244)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.failure(ClientRuntime.java:178)
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:218)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:277)
> at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.abort(NettyResponseFuture.java:273)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:181)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.channel.NettyConnectListener$1.onFailure(NettyConnectListener.java:151)
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:26)
>
[jira] [Created] (FLINK-31748) Adapt SplitFetcherManager#removeSplit for flink-connector-pulsar
Zili Chen created FLINK-31748: - Summary: Adapt SplitFetcherManager#removeSplit for flink-connector-pulsar Key: FLINK-31748 URL: https://issues.apache.org/jira/browse/FLINK-31748 Project: Flink Issue Type: Sub-task Reporter: Zili Chen Assignee: Yufan Sheng -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-25509) FLIP-208: Add RecordEvaluator to dynamically stop source based on de-serialized records
[ https://issues.apache.org/jira/browse/FLINK-25509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17708825#comment-17708825 ] Zili Chen commented on FLINK-25509: --- For the Pulsar side, I have a conversation with [~syhily] and he's willing to follow up this ticket. Shall we have a ticket for the subtask? Also, I wonder what versions will be affected. Since the external connectors compiled with release versions, shall we do the subtask after a specific Flink release? > FLIP-208: Add RecordEvaluator to dynamically stop source based on > de-serialized records > --- > > Key: FLINK-25509 > URL: https://issues.apache.org/jira/browse/FLINK-25509 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common, Connectors / Kafka >Reporter: Dong Lin >Assignee: Hang Ruan >Priority: Major > Labels: pull-request-available > > This feature is needed to migrate applications which uses > KafkaDeserializationSchema::isEndOfStream() from using FlinkKafkaConsumer to > using KafkaSource. > Please checkout > https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Add+RecordEvaluator+to+dynamically+stop+source+based+on+de-serialized+records > for the motivation and the proposed changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31334) E2e ci fail with unsupported file exception
[
https://issues.apache.org/jira/browse/FLINK-31334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696803#comment-17696803
]
Zili Chen commented on FLINK-31334:
---
cc [~chesnay] I guess you're familiar with this part?
> E2e ci fail with unsupported file exception
>
>
> Key: FLINK-31334
> URL: https://issues.apache.org/jira/browse/FLINK-31334
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Reporter: luoyuxia
>Priority: Critical
>
> The e2e ci throw
> "E: Unsupported file ./libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb given on
> commandline".
> The full exception message is
> {code:java}
> Installing required software
> Reading package lists...
> Building dependency tree...
> Reading state information...
> bc is already the newest version (1.07.1-2build1).
> bc set to manually installed.
> libapr1 is already the newest version (1.6.5-1ubuntu1).
> libapr1 set to manually installed.
> 0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
> --2023-03-06 07:22:17--
> http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb
> Resolving security.ubuntu.com (security.ubuntu.com)... 185.125.190.39,
> 185.125.190.36, 91.189.91.39, ...
> Connecting to security.ubuntu.com (security.ubuntu.com)|185.125.190.39|:80...
> connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2023-03-06 07:22:17 ERROR 404: Not Found.
> WARNING: apt does not have a stable CLI interface. Use with caution in
> scripts.
> Reading package lists...
> E: Unsupported file ./libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb given on
> commandline
> ##[error]Bash exited with code '100'.
> Finishing: Prepare E2E run
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46628=logs=81be5d54-0dc6-5130-d390-233dd2956037=d72874ca-f446-5272-2efd-0705f108dbf6
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31206) Broken links on flink.apache.org
[ https://issues.apache.org/jira/browse/FLINK-31206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693137#comment-17693137 ] Zili Chen commented on FLINK-31206: --- [~martijnvisser] Thanks for your follow-ups! See https://github.com/apache/flink-web/commit/9dc7f0cc1954ff4f845a34f9aeaa2723b345ba74#r101925551. > Broken links on flink.apache.org > > > Key: FLINK-31206 > URL: https://issues.apache.org/jira/browse/FLINK-31206 > Project: Flink > Issue Type: Bug > Components: Project Website >Reporter: Zili Chen >Assignee: Martijn Visser >Priority: Minor > > Previously page link > https://flink.apache.org/contribute/code-style-and-quality/preamble/ is > broken, new link is > https://flink.apache.org/how-to-contribute/code-style-and-quality-preamble/. > Shall we set up a redirection or just let those broken links wait for > maintainers fixing? > cc [~martijnvisser] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31206) Broken links on flink.apache.org
Zili Chen created FLINK-31206: - Summary: Broken links on flink.apache.org Key: FLINK-31206 URL: https://issues.apache.org/jira/browse/FLINK-31206 Project: Flink Issue Type: Bug Reporter: Zili Chen Previously page link https://flink.apache.org/contribute/code-style-and-quality/preamble/ is broken, new link is https://flink.apache.org/how-to-contribute/code-style-and-quality-preamble/. Shall we set up a redirection or just let those broken links wait for maintainers fixing? cc [~martijnvisser] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-22922) Migrate flink project website to Hugo
[ https://issues.apache.org/jira/browse/FLINK-22922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692982#comment-17692982 ] Zili Chen commented on FLINK-22922: --- Hi [~martijnvisser]! I see now we use `asf-site` for both dev and deploy. How can we ensure the website is always rebuilt, or now we do it manually? > Migrate flink project website to Hugo > - > > Key: FLINK-22922 > URL: https://issues.apache.org/jira/browse/FLINK-22922 > Project: Flink > Issue Type: Improvement > Components: Project Website >Reporter: Chesnay Schepler >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > > Hugo is working like a charm for the Flink documentation. To reduce the > number of software stacks, and massively reduce friction when building the > current Flink website, we should migrate the Flink website to hugo as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31086) [DOC]update connector lable for blackhole and kafka
[ https://issues.apache.org/jira/browse/FLINK-31086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-31086. - Resolution: Fixed master via e292a105eb9f7ddccfba2b0d53430dcdb06c0280 > [DOC]update connector lable for blackhole and kafka > --- > > Key: FLINK-31086 > URL: https://issues.apache.org/jira/browse/FLINK-31086 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: hehuiyuan >Assignee: hehuiyuan >Priority: Minor > Labels: pull-request-available > > pdate connector label for kafka and blackhole. > Blackhole: sink:bounded unbounded > Kafka: source bounded > !https://user-images.githubusercontent.com/18002496/216600374-5c9d16db-66ac-42a4-8b21-16245a71f9ef.png|width=468,height=168! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31086) [DOC]update connector lable for blackhole and kafka
[ https://issues.apache.org/jira/browse/FLINK-31086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen updated FLINK-31086: -- Issue Type: Improvement (was: Bug) > [DOC]update connector lable for blackhole and kafka > --- > > Key: FLINK-31086 > URL: https://issues.apache.org/jira/browse/FLINK-31086 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: hehuiyuan >Assignee: hehuiyuan >Priority: Minor > Labels: pull-request-available > > pdate connector label for kafka and blackhole. > Blackhole: sink:bounded unbounded > Kafka: source bounded > !https://user-images.githubusercontent.com/18002496/216600374-5c9d16db-66ac-42a4-8b21-16245a71f9ef.png|width=468,height=168! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-31086) [DOC]update connector lable for blackhole and kafka
[ https://issues.apache.org/jira/browse/FLINK-31086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-31086: - Assignee: hehuiyuan > [DOC]update connector lable for blackhole and kafka > --- > > Key: FLINK-31086 > URL: https://issues.apache.org/jira/browse/FLINK-31086 > Project: Flink > Issue Type: Bug > Components: Documentation >Reporter: hehuiyuan >Assignee: hehuiyuan >Priority: Minor > Labels: pull-request-available > > pdate connector label for kafka and blackhole. > Blackhole: sink:bounded unbounded > Kafka: source bounded > !https://user-images.githubusercontent.com/18002496/216600374-5c9d16db-66ac-42a4-8b21-16245a71f9ef.png|width=468,height=168! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30606) Bump Pulsar to 2.11.0
[ https://issues.apache.org/jira/browse/FLINK-30606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-30606. - Resolution: Fixed master via 8b58e7f81058ee50f7d194319ca122cc134033dc > Bump Pulsar to 2.11.0 > - > > Key: FLINK-30606 > URL: https://issues.apache.org/jira/browse/FLINK-30606 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.0 >Reporter: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > > The fifth release candidate for Apache Pulsar 2.11.0 is voting and be > approved. So the final 2.11.0 will be released soon. This is a huge release > which fixes a lot of bugs. > We will bump the Pulsar client to this version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28505) Pulsar sink doesn't support topic auto creation
[ https://issues.apache.org/jira/browse/FLINK-28505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-28505. - Resolution: Duplicate > Pulsar sink doesn't support topic auto creation > --- > > Key: FLINK-28505 > URL: https://issues.apache.org/jira/browse/FLINK-28505 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30109) Checked exceptions are sneakingly transformed into unchecked exceptions in the Pulsar
[
https://issues.apache.org/jira/browse/FLINK-30109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17687944#comment-17687944
]
Zili Chen commented on FLINK-30109:
---
master via 2278653d67a8ddf171c88d538a288e503221625a
I'm not sure whether other versions should be picked.
> Checked exceptions are sneakingly transformed into unchecked exceptions in
> the Pulsar
> -
>
> Key: FLINK-30109
> URL: https://issues.apache.org/jira/browse/FLINK-30109
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Pulsar, Documentation
>Affects Versions: 1.16.0, 1.17.0, 1.15.2
>Reporter: Matthias Pohl
>Assignee: Yufan Sheng
>Priority: Major
> Labels: pull-request-available
>
> [PulsarExceptionUtils|https://github.com/apache/flink/blob/c675f786c51038801161e861826d1c54654f0dde/flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarExceptionUtils.java#L33]
> provides {{sneaky*}} utility methods for hiding checked exceptions. This is
> rather unusual coding. Based on what's provided in the code I would have
> concerns as a reader that we're not handling errors properly in calling code.
> Either, we remove these methods and add proper exception handling or we add
> proper documentation on why this workaround is necessary.
> [~syhily] already hinted in his [FLINK-29830 PR
> comment|https://github.com/apache/flink/pull/21252#discussion_r1019822514]
> that this is related to flaws of the Pulsar API.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30489) flink-sql-connector-pulsar doesn't shade all dependencies
[ https://issues.apache.org/jira/browse/FLINK-30489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-30489. - Fix Version/s: pulsar-4.0.0 Resolution: Fixed master via 5f0bb2db9f1f357a11cce965eb5832bb908523a1 > flink-sql-connector-pulsar doesn't shade all dependencies > - > > Key: FLINK-30489 > URL: https://issues.apache.org/jira/browse/FLINK-30489 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.0 >Reporter: Henri Yandell >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > > Looking at > [https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-pulsar/1.16.0/flink-sql-connector-pulsar-1.16.0.jar] > I'm seeing that some dependencies are shaded (com.fasterxml, com.yahoo etc), > but others are not (org.sfl4j, org.bouncycastel, com.scurrilous, ...) and > will presumably clash with other jar files. > Additionally, this bundling is going on in the '.jar' file rather than in a > more clearly indicated separate -bundle or -shaded jar file. > As a jar file this seems confusing and potentially bug inducing; though I > note I'm just a review of the jar and not Flink experienced. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-30489) flink-sql-connector-pulsar doesn't shade all dependencies
[ https://issues.apache.org/jira/browse/FLINK-30489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-30489: - Assignee: Yufan Sheng > flink-sql-connector-pulsar doesn't shade all dependencies > - > > Key: FLINK-30489 > URL: https://issues.apache.org/jira/browse/FLINK-30489 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.0 >Reporter: Henri Yandell >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > > Looking at > [https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-pulsar/1.16.0/flink-sql-connector-pulsar-1.16.0.jar] > I'm seeing that some dependencies are shaded (com.fasterxml, com.yahoo etc), > but others are not (org.sfl4j, org.bouncycastel, com.scurrilous, ...) and > will presumably clash with other jar files. > Additionally, this bundling is going on in the '.jar' file rather than in a > more clearly indicated separate -bundle or -shaded jar file. > As a jar file this seems confusing and potentially bug inducing; though I > note I'm just a review of the jar and not Flink experienced. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30654) Add option to force consumption from StartCursor every time the application starts
[ https://issues.apache.org/jira/browse/FLINK-30654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-30654. - Fix Version/s: pulsar-4.0.0 Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/pull/23 > Add option to force consumption from StartCursor every time the application > starts > -- > > Key: FLINK-30654 > URL: https://issues.apache.org/jira/browse/FLINK-30654 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.0 >Reporter: likang >Assignee: Yufan Sheng >Priority: Minor > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > Attachments: WechatIMG250.png > > > Pulsar does not take effect when the subscription is set to start position > consumption. It is recommended to add an option to be determined by the user > or to add a lastAck strategy to adapt to the scenario of whether to start > consumption from the last consumption submission position -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30622) Support auto consume schema in Pulsar source
[ https://issues.apache.org/jira/browse/FLINK-30622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-30622. - Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/pull/19 > Support auto consume schema in Pulsar source > > > Key: FLINK-30622 > URL: https://issues.apache.org/jira/browse/FLINK-30622 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > > Pulsar client supports [auto > schema|https://pulsar.apache.org/docs/2.10.x/schema-understand/#auto-schema] > if you don't know the schema type of a Pulsar topic in advance. > This can make the message consuming from Pulsar topic which contains multiple > schema extremely easy. We plan to add auto schema support in Pulsar source. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28351) Pulsar Sink should support dynamic generated topic from record
[ https://issues.apache.org/jira/browse/FLINK-28351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17685937#comment-17685937 ] Zili Chen commented on FLINK-28351: --- [~syhily] I've written a release note here. Please check and comment (or if you have the permission, update directly) if changes are needed. > Pulsar Sink should support dynamic generated topic from record > -- > > Key: FLINK-28351 > URL: https://issues.apache.org/jira/browse/FLINK-28351 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > > Some people would like to use dynamically-generated topics from messages and > use the key hash range policy. This is not supported by the Pulsar sink > currently. We would introduce a new interface named TopicExacter and add a > new setTopics(TopicExacter) in PulsarSinkBuilder. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28351) Pulsar Sink should support dynamic generated topic from record
[ https://issues.apache.org/jira/browse/FLINK-28351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-28351. - Fix Version/s: pulsar-4.0.0 Release Note: The default value of `pulsar.source.partitionDiscoveryIntervalMs` is changed to 30 (5 minutes) for less burden. Resolution: Fixed Master via https://github.com/apache/flink-connector-pulsar/pull/14 > Pulsar Sink should support dynamic generated topic from record > -- > > Key: FLINK-28351 > URL: https://issues.apache.org/jira/browse/FLINK-28351 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Affects Versions: 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > > Some people would like to use dynamically-generated topics from messages and > use the key hash range policy. This is not supported by the Pulsar sink > currently. We would introduce a new interface named TopicExacter and add a > new setTopics(TopicExacter) in PulsarSinkBuilder. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30689) Use auto produce bytes schema in Pulsar sink
[
https://issues.apache.org/jira/browse/FLINK-30689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-30689.
-
Resolution: Fixed
master via https://github.com/apache/flink-connector-pulsar/pull/20
> Use auto produce bytes schema in Pulsar sink
>
>
> Key: FLINK-30689
> URL: https://issues.apache.org/jira/browse/FLINK-30689
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
>Affects Versions: pulsar-4.0.0
>Reporter: Yufan Sheng
>Assignee: Yufan Sheng
>Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-4.0.0
>
>
> Pulsar has a {{Schema.AUTO_PRODUCE_BYTES()}} for sending the byte array
> messages to broker with extra schema validation. This would be better than
> directly using {{Schema.BYTES}} which bypass the validation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30657) Remove Shared and Key_Shared related tests in Pulsar connector
[ https://issues.apache.org/jira/browse/FLINK-30657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682356#comment-17682356 ] Zili Chen commented on FLINK-30657: --- flink-connector-pulsar 3.0 via https://github.com/apache/flink-connector-pulsar/pull/17 I'm not sure whether we should backport to 1.15 and 1.16. > Remove Shared and Key_Shared related tests in Pulsar connector > -- > > Key: FLINK-30657 > URL: https://issues.apache.org/jira/browse/FLINK-30657 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.0, 1.15.3 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.15.4, 1.16.2 > > > As the [FLINK-30413|https://issues.apache.org/jira/browse/FLINK-30413] issue > talked, we have dropped the Shared and Key_Shared supported in upcoming > flink-connector-pulsar 4.0 release. The flaky tests of Shared and Key_Shared > still matters the old Flink build. > Cause these tests are useless now, we can just disable them without any fix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-30622) Support auto consume schema in Pulsar source
[ https://issues.apache.org/jira/browse/FLINK-30622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-30622: - Assignee: Yufan Sheng > Support auto consume schema in Pulsar source > > > Key: FLINK-30622 > URL: https://issues.apache.org/jira/browse/FLINK-30622 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > > Pulsar client supports [auto > schema|https://pulsar.apache.org/docs/2.10.x/schema-understand/#auto-schema] > if you don't know the schema type of a Pulsar topic in advance. > This can make the message consuming from Pulsar topic which contains multiple > schema extremely easy. We plan to add auto schema support in Pulsar source. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30681) Pulsar-Flink connector corrupts its output topic
[ https://issues.apache.org/jira/browse/FLINK-30681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-30681. - Assignee: Yufan Sheng Resolution: Not A Bug Explained in https://github.com/apache/flink-connector-pulsar/pull/18. > Pulsar-Flink connector corrupts its output topic > > > Key: FLINK-30681 > URL: https://issues.apache.org/jira/browse/FLINK-30681 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.3 >Reporter: Jacek Wislicki >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > > When PulsarSink writes a message to its output topic, the topic gets > permanently corrupted and cannot be used anymore (even with newly created > subscriptions). > We have isolated this behaviour to a minimal project demonstrating the > problem available on [GitHub|https://github.com/JacekWislicki/vp-test5]: > # There are 2 topics: IN and OUT > # IN is subscribed by a Flink's InToOutJob (with PulsarSource) and writes to > OUT (with PulsarSink) > # OUT is subscribed by a Pulsar's OutReadFunction > # When we write directly to OUT (e.g., with OutTopicProducer), > OutReadFunction gets each message from its backlog and processes it with no > issue (the ledger position updates) > # When we write to IN (e.g., with InTopicProducer), InToOutJob reads the > message, processes it and writes to OUT > # OutReadFunction reads the message, the ledger position updates, but nothing > happens > ## Further messages written to OUT are not read as OUT is blocked on the last > message from Flink > ## Truncating OUT does not help, neither does unsubscribing or creating a new > subscription > Reproduced with Pulsar 2.9.1, 2.9.2 and 2.10.2. > The issue does not occur when we use our custom temporary old SinkFunction > implementation based on a Pulsar producer writing to OUT. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-30413) Drop Shared and Key_Shared subscription support in Pulsar connector
[
https://issues.apache.org/jira/browse/FLINK-30413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen reopened FLINK-30413:
---
> Drop Shared and Key_Shared subscription support in Pulsar connector
> ---
>
> Key: FLINK-30413
> URL: https://issues.apache.org/jira/browse/FLINK-30413
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
>Affects Versions: 1.16.0, 1.17.0, 1.15.3
>Reporter: Yufan Sheng
>Assignee: Yufan Sheng
>Priority: Critical
> Labels: pull-request-available
>
> A lot of Pulsar connector test unstable issues are related to {{Shared}} and
> {{Key_Shared}} subscription. Because this two subscription is designed to
> consume the records in an unordered way. And we can support multiple
> consumers in same topic partition. But this feature lead to some drawbacks in
> connector.
> 1. Performance
> Flink is a true stream processor with high correctness support. But support
> multiple consumer will require higher correctness which depends on Pulsar
> transaction. But the internal implementation of Pulsar transaction on source
> is record the message one by one and stores all the pending ack status in
> client side. Which is slow and memory inefficient.
> This means that we can only use {{Shared}} and {{Key_Shared}} on Flink with
> low throughput. This against our intention to support these two subscription.
> Because adding multiple consumer to same partition can increase the consuming
> speed.
> 2. Unstable
> Pulsar transaction acknowledge the messages one by one in an internal
> Pulsar's topic. But it's not stable enough to get it works. A lot of pending
> issues in Flink JIRA are related to Pulsar transaction and we don't have any
> workaround.
> 3. Complex
> Support {{Shared}} and {{Key_Shared}} subscription make the connector's code
> more complex than we expect. We have to make every part of code into ordered
> and unordered way. Which is hard to understand for the maintainer.
> 4. Necessary
> The current implementation on {{Shared}} and {{Key_Shared}} is completely
> unusable to use in Production environment. For the user, this function is not
> necessary. Because there is no bottleneck in consuming data from Pulsar, the
> bottleneck is in processing the data, which we can achieve by increasing the
> parallelism of the processing operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30413) Drop Shared and Key_Shared subscription support in Pulsar connector
[
https://issues.apache.org/jira/browse/FLINK-30413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-30413.
-
Fix Version/s: pulsar-4.0.0
Resolution: Fixed
> Drop Shared and Key_Shared subscription support in Pulsar connector
> ---
>
> Key: FLINK-30413
> URL: https://issues.apache.org/jira/browse/FLINK-30413
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
>Affects Versions: 1.16.0, 1.17.0, 1.15.3
>Reporter: Yufan Sheng
>Assignee: Yufan Sheng
>Priority: Critical
> Labels: pull-request-available
> Fix For: pulsar-4.0.0
>
>
> A lot of Pulsar connector test unstable issues are related to {{Shared}} and
> {{Key_Shared}} subscription. Because this two subscription is designed to
> consume the records in an unordered way. And we can support multiple
> consumers in same topic partition. But this feature lead to some drawbacks in
> connector.
> 1. Performance
> Flink is a true stream processor with high correctness support. But support
> multiple consumer will require higher correctness which depends on Pulsar
> transaction. But the internal implementation of Pulsar transaction on source
> is record the message one by one and stores all the pending ack status in
> client side. Which is slow and memory inefficient.
> This means that we can only use {{Shared}} and {{Key_Shared}} on Flink with
> low throughput. This against our intention to support these two subscription.
> Because adding multiple consumer to same partition can increase the consuming
> speed.
> 2. Unstable
> Pulsar transaction acknowledge the messages one by one in an internal
> Pulsar's topic. But it's not stable enough to get it works. A lot of pending
> issues in Flink JIRA are related to Pulsar transaction and we don't have any
> workaround.
> 3. Complex
> Support {{Shared}} and {{Key_Shared}} subscription make the connector's code
> more complex than we expect. We have to make every part of code into ordered
> and unordered way. Which is hard to understand for the maintainer.
> 4. Necessary
> The current implementation on {{Shared}} and {{Key_Shared}} is completely
> unusable to use in Production environment. For the user, this function is not
> necessary. Because there is no bottleneck in consuming data from Pulsar, the
> bottleneck is in processing the data, which we can achieve by increasing the
> parallelism of the processing operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-30657) Disable Shared and Key_Shared related tests in Pulsar connector
[ https://issues.apache.org/jira/browse/FLINK-30657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-30657: - Assignee: Yufan Sheng > Disable Shared and Key_Shared related tests in Pulsar connector > --- > > Key: FLINK-30657 > URL: https://issues.apache.org/jira/browse/FLINK-30657 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.16.0, 1.15.3 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Blocker > Fix For: 1.16.1, 1.15.4 > > > As the [FLINK-30413|https://issues.apache.org/jira/browse/FLINK-30413] issue > talked, we have dropped the Shared and Key_Shared supported in upcoming > flink-connector-pulsar 4.0 release. The flaky tests of Shared and Key_Shared > still matters the old Flink build. > Cause these tests are useless now, we can just disable them without any fix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30413) Drop Shared and Key_Shared subscription support in Pulsar connector
[
https://issues.apache.org/jira/browse/FLINK-30413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-30413.
-
Fix Version/s: (was: pulsar-4.0.0)
Resolution: Won't Do
> Drop Shared and Key_Shared subscription support in Pulsar connector
> ---
>
> Key: FLINK-30413
> URL: https://issues.apache.org/jira/browse/FLINK-30413
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
>Affects Versions: 1.16.0, 1.17.0, 1.15.3
>Reporter: Yufan Sheng
>Assignee: Yufan Sheng
>Priority: Critical
> Labels: pull-request-available
>
> A lot of Pulsar connector test unstable issues are related to {{Shared}} and
> {{Key_Shared}} subscription. Because this two subscription is designed to
> consume the records in an unordered way. And we can support multiple
> consumers in same topic partition. But this feature lead to some drawbacks in
> connector.
> 1. Performance
> Flink is a true stream processor with high correctness support. But support
> multiple consumer will require higher correctness which depends on Pulsar
> transaction. But the internal implementation of Pulsar transaction on source
> is record the message one by one and stores all the pending ack status in
> client side. Which is slow and memory inefficient.
> This means that we can only use {{Shared}} and {{Key_Shared}} on Flink with
> low throughput. This against our intention to support these two subscription.
> Because adding multiple consumer to same partition can increase the consuming
> speed.
> 2. Unstable
> Pulsar transaction acknowledge the messages one by one in an internal
> Pulsar's topic. But it's not stable enough to get it works. A lot of pending
> issues in Flink JIRA are related to Pulsar transaction and we don't have any
> workaround.
> 3. Complex
> Support {{Shared}} and {{Key_Shared}} subscription make the connector's code
> more complex than we expect. We have to make every part of code into ordered
> and unordered way. Which is hard to understand for the maintainer.
> 4. Necessary
> The current implementation on {{Shared}} and {{Key_Shared}} is completely
> unusable to use in Production environment. For the user, this function is not
> necessary. Because there is no bottleneck in consuming data from Pulsar, the
> bottleneck is in processing the data, which we can achieve by increasing the
> parallelism of the processing operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30413) Drop Shared and Key_Shared subscription support in Pulsar connector
[
https://issues.apache.org/jira/browse/FLINK-30413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676411#comment-17676411
]
Zili Chen commented on FLINK-30413:
---
I second [~syhily]'s comment that it can be a break change and we'd either fix
the case or directly remove the tests. Since we don't maintain those functions
in 1.15 or 1.16, I'll close this issue in favor of FLINK-30657.
> Drop Shared and Key_Shared subscription support in Pulsar connector
> ---
>
> Key: FLINK-30413
> URL: https://issues.apache.org/jira/browse/FLINK-30413
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
>Affects Versions: 1.16.0, 1.17.0, 1.15.3
>Reporter: Yufan Sheng
>Assignee: Yufan Sheng
>Priority: Critical
> Labels: pull-request-available
> Fix For: pulsar-4.0.0
>
>
> A lot of Pulsar connector test unstable issues are related to {{Shared}} and
> {{Key_Shared}} subscription. Because this two subscription is designed to
> consume the records in an unordered way. And we can support multiple
> consumers in same topic partition. But this feature lead to some drawbacks in
> connector.
> 1. Performance
> Flink is a true stream processor with high correctness support. But support
> multiple consumer will require higher correctness which depends on Pulsar
> transaction. But the internal implementation of Pulsar transaction on source
> is record the message one by one and stores all the pending ack status in
> client side. Which is slow and memory inefficient.
> This means that we can only use {{Shared}} and {{Key_Shared}} on Flink with
> low throughput. This against our intention to support these two subscription.
> Because adding multiple consumer to same partition can increase the consuming
> speed.
> 2. Unstable
> Pulsar transaction acknowledge the messages one by one in an internal
> Pulsar's topic. But it's not stable enough to get it works. A lot of pending
> issues in Flink JIRA are related to Pulsar transaction and we don't have any
> workaround.
> 3. Complex
> Support {{Shared}} and {{Key_Shared}} subscription make the connector's code
> more complex than we expect. We have to make every part of code into ordered
> and unordered way. Which is hard to understand for the maintainer.
> 4. Necessary
> The current implementation on {{Shared}} and {{Key_Shared}} is completely
> unusable to use in Production environment. For the user, this function is not
> necessary. Because there is no bottleneck in consuming data from Pulsar, the
> bottleneck is in processing the data, which we can achieve by increasing the
> parallelism of the processing operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-30654) start cursor issue
[ https://issues.apache.org/jira/browse/FLINK-30654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen reassigned FLINK-30654: - Assignee: Yufan Sheng > start cursor issue > -- > > Key: FLINK-30654 > URL: https://issues.apache.org/jira/browse/FLINK-30654 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.0 >Reporter: likang >Assignee: Yufan Sheng >Priority: Minor > > Pulsar does not take effect when the subscription is set to start position > consumption. It is recommended to add an option to be determined by the user > or to add a lastAck strategy to adapt to the scenario of whether to start > consumption from the last consumption submission position > !image-2023-01-12-23-13-48-411.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30552) Pulsar connector shouldn't assert the BatchMessageId size.
[
https://issues.apache.org/jira/browse/FLINK-30552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen updated FLINK-30552:
--
Fix Version/s: pulsar-3.0.1
> Pulsar connector shouldn't assert the BatchMessageId size.
> --
>
> Key: FLINK-30552
> URL: https://issues.apache.org/jira/browse/FLINK-30552
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: pulsar-4.0.0
>Reporter: Yufan Sheng
>Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-3.0.1, pulsar-4.0.0
>
>
> Pulsar will try to assert the batch message id size in {{MessageIdUtils}},
> but the batch size is determined by the producer in batch mode. So we can
> promise the size could be 1.
> And the next message id calculation should be calculated by using
> BatchMessageId.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-30552) Pulsar connector shouldn't assert the BatchMessageId size.
[
https://issues.apache.org/jira/browse/FLINK-30552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen reassigned FLINK-30552:
-
Assignee: Yufan Sheng
> Pulsar connector shouldn't assert the BatchMessageId size.
> --
>
> Key: FLINK-30552
> URL: https://issues.apache.org/jira/browse/FLINK-30552
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: pulsar-4.0.0
>Reporter: Yufan Sheng
>Assignee: Yufan Sheng
>Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-3.0.1, pulsar-4.0.0
>
>
> Pulsar will try to assert the batch message id size in {{MessageIdUtils}},
> but the batch size is determined by the producer in batch mode. So we can
> promise the size could be 1.
> And the next message id calculation should be calculated by using
> BatchMessageId.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30552) Pulsar connector shouldn't assert the BatchMessageId size.
[
https://issues.apache.org/jira/browse/FLINK-30552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17665060#comment-17665060
]
Zili Chen commented on FLINK-30552:
---
3.0.1 via https://github.com/apache/flink-connector-pulsar/pull/16
> Pulsar connector shouldn't assert the BatchMessageId size.
> --
>
> Key: FLINK-30552
> URL: https://issues.apache.org/jira/browse/FLINK-30552
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: pulsar-4.0.0
>Reporter: Yufan Sheng
>Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-3.0.1, pulsar-4.0.0
>
>
> Pulsar will try to assert the batch message id size in {{MessageIdUtils}},
> but the batch size is determined by the producer in batch mode. So we can
> promise the size could be 1.
> And the next message id calculation should be calculated by using
> BatchMessageId.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-28870) Pulsar Source hangs on the small incoming message rates
[ https://issues.apache.org/jira/browse/FLINK-28870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-28870. - Fix Version/s: pulsar-4.0.0 Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/pull/15 > Pulsar Source hangs on the small incoming message rates > --- > > Key: FLINK-28870 > URL: https://issues.apache.org/jira/browse/FLINK-28870 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Reporter: xie >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > > When using Pulsar Source to consume data, if the data rate is small, e.g. 2 > msg/s, there will be long periods of time when no messages are consumed. This > is caused by the default PulsarSourceOptions.PULSAR_MAX_FETCH_TIME and > PulsarSourceOptions.PULSAR_MAX_FETCH_RECORDS. Pulsar Source will try to pull > messages until any one of the condition exceed. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30616) Don't support batchMessageId when restore from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-30616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17656508#comment-17656508
]
Zili Chen commented on FLINK-30616:
---
[~songv] According to [~syhily]'s comment above, you may try out the master
branch code on https://github.com/apache/flink-connector-pulsar to see if it
solves your case. Also, I'd like to know whether the next feature release is
also acceptable for you.
> Don't support batchMessageId when restore from checkpoint
> -
>
> Key: FLINK-30616
> URL: https://issues.apache.org/jira/browse/FLINK-30616
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.16.0, pulsar-3.0.0
> Environment: flink version: 1.16.0
> flink-connector-pulsar version: 1.16.0
>Reporter: songv
>Priority: Major
>
> I have a non-partition topic:
> * the producer for the topic sends batch messages to the topic(to improve
> the speed of producers)
> * the flink job consumes this topic by Exclusive subscription type
> When the flink task manager is restarted for some reason, an exception is
> thrown when restored from the checkpoint:
> {code:java}
> java.lang.RuntimeException: One or more fetchers have encountered the
> exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
> ~[flink-connector-files-1.16.0.jar:1.16.0]
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
> ~[flink-connector-files-1.16.0.jar:1.16.0]
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
> ~[flink-connector-files-1.16.0.jar:1.16.0]
> at
> org.apache.flink.connector.pulsar.source.reader.source.PulsarOrderedSourceReader.pollNext(PulsarOrderedSourceReader.java:106)
> ~[?:?]
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNextNotReading(SourceOperator.java:403)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:387)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:542)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at java.lang.Thread.run(Unknown Source) ~[?:?]
> Caused by: java.lang.RuntimeException: SplitFetcher thread 0 received
> unexpected exception while polling the records
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:150)
> ~[flink-connector-files-1.16.0.jar:1.16.0]
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105)
> ~[flink-connector-files-1.16.0.jar:1.16.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
> at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
> ... 1 more
> Caused by: java.lang.IllegalArgumentException: We only support normal message
> id currently. This batch size is %d [83]
> at org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:160)
> ~[flink-dist-1.16.0.jar:1.16.0]
> at
> org.apache.flink.connector.pulsar.source.enumerator.cursor.MessageIdUtils.unwrapMessageId(MessageIdUtils.java:65)
> ~[?:?]
> at
> org.apache.flink.connector.pulsar.source.enumerator.cursor.MessageIdUtils.nextMessageId(MessageIdUtils.java:44)
> ~[?:?]
> at
>
[jira] [Closed] (FLINK-28082) Support end to end encryption on Pulsar connector.
[ https://issues.apache.org/jira/browse/FLINK-28082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-28082. - Fix Version/s: pulsar-4.0.0 (was: 1.17.0) Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/pull/13 > Support end to end encryption on Pulsar connector. > -- > > Key: FLINK-28082 > URL: https://issues.apache.org/jira/browse/FLINK-28082 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Affects Versions: 1.16.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: pulsar-4.0.0 > > > Add this Pulsar encryption support: > https://pulsar.apache.org/docs/security-encryption/ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-25686) Support Pulsar Schema evolution in Pulsar Source Connector
[ https://issues.apache.org/jira/browse/FLINK-25686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-25686. - Fix Version/s: pulsar-4.0.0 Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/pull/12 > Support Pulsar Schema evolution in Pulsar Source Connector > -- > > Key: FLINK-25686 > URL: https://issues.apache.org/jira/browse/FLINK-25686 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufan Sheng >Priority: Minor > Labels: Pulsar, pull-request-available, stale-assigned > Fix For: pulsar-4.0.0 > > > Currently the source connector reads all messages as Message, and > implements a deserialization abstraction to deserialize the byte array > message. This deserialization abstraction supports both flink schema and > pulsar schema. However because of this extra layer of abstraction, even when > using pulsar schema, the schema info is not provided to the consumer > (client), so it is not using any pulsar’s schema validation mechanism. (If > pulsar client has a schema, it will send the schema to broker on connect and > broker will validate if it is a valid schema according to a preset > compatibility rule :[ > https://pulsar.apache.org/docs/en/schema-evolution-compatibility/|https://pulsar.apache.org/docs/en/schema-evolution-compatibility/]). > we need to support this schema evolution -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30552) Pulsar connector shouldn't assert the BatchMessageId size.
[
https://issues.apache.org/jira/browse/FLINK-30552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-30552.
-
Resolution: Fixed
master via https://github.com/apache/flink-connector-pulsar/pull/11
> Pulsar connector shouldn't assert the BatchMessageId size.
> --
>
> Key: FLINK-30552
> URL: https://issues.apache.org/jira/browse/FLINK-30552
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: pulsar-4.0.0
>Reporter: Yufan Sheng
>Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-4.0.0
>
>
> Pulsar will try to assert the batch message id size in {{MessageIdUtils}},
> but the batch size is determined by the producer in batch mode. So we can
> promise the size could be 1.
> And the next message id calculation should be calculated by using
> BatchMessageId.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-27917) PulsarUnorderedPartitionSplitReaderTest.consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition failed with AssertionError
[
https://issues.apache.org/jira/browse/FLINK-27917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-27917.
-
Fix Version/s: (was: 1.14.7)
(was: 1.16.1)
(was: 1.15.4)
Resolution: Invalid
After FLINK-30413 dropped the related support, this test was also dropped.
Invalid now.
> PulsarUnorderedPartitionSplitReaderTest.consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition
> failed with AssertionError
> ---
>
> Key: FLINK-27917
> URL: https://issues.apache.org/jira/browse/FLINK-27917
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Pulsar
>Affects Versions: 1.14.5, 1.15.1
>Reporter: Huang Xingbo
>Assignee: Yufan Sheng
>Priority: Major
> Labels: pull-request-available, test-stability
>
> {code:java}
> 2022-06-06T06:34:46.7906026Z Jun 06 06:34:46 [ERROR]
> org.apache.flink.connector.pulsar.source.reader.split.PulsarUnorderedPartitionSplitReaderTest.consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition(PulsarPartitionSplitReaderBase)[1]
> Time elapsed: 9.774 s <<< FAILURE!
> 2022-06-06T06:34:46.7919217Z Jun 06 06:34:46 java.lang.AssertionError:
> 2022-06-06T06:34:46.7920918Z Jun 06 06:34:46 [We should fetch the expected
> size]
> 2022-06-06T06:34:46.7921479Z Jun 06 06:34:46 Expected size: 20 but was: 3 in:
> 2022-06-06T06:34:46.7922019Z Jun 06 06:34:46 [PulsarMessage{id=58:0:0:0,
> value=ElpTDLGvKz, eventTime=0},
> 2022-06-06T06:34:46.7922757Z Jun 06 06:34:46 PulsarMessage{id=58:1:0:0,
> value=cDGEGcCZnP, eventTime=0},
> 2022-06-06T06:34:46.7924900Z Jun 06 06:34:46 PulsarMessage{id=58:2:0:0,
> value=rZmaCxrhZF, eventTime=0}]
> 2022-06-06T06:34:46.7926359Z Jun 06 06:34:46 at
> org.apache.flink.connector.pulsar.source.reader.split.PulsarPartitionSplitReaderTestBase.fetchedMessages(PulsarPartitionSplitReaderTestBase.java:186)
> 2022-06-06T06:34:46.7928019Z Jun 06 06:34:46 at
> org.apache.flink.connector.pulsar.source.reader.split.PulsarPartitionSplitReaderTestBase.fetchedMessages(PulsarPartitionSplitReaderTestBase.java:156)
> 2022-06-06T06:34:46.7930207Z Jun 06 06:34:46 at
> org.apache.flink.connector.pulsar.source.reader.split.PulsarPartitionSplitReaderTestBase.consumeMessageCreatedBeforeHandleSplitsChangesAndResetToEarliestPosition(PulsarPartitionSplitReaderTestBase.java:247)
> 2022-06-06T06:34:46.7931943Z Jun 06 06:34:46 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-06-06T06:34:46.7933282Z Jun 06 06:34:46 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-06-06T06:34:46.7934885Z Jun 06 06:34:46 at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-06-06T06:34:46.7936182Z Jun 06 06:34:46 at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 2022-06-06T06:34:46.7937301Z Jun 06 06:34:46 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> 2022-06-06T06:34:46.7938744Z Jun 06 06:34:46 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2022-06-06T06:34:46.7939650Z Jun 06 06:34:46 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-06-06T06:34:46.7940516Z Jun 06 06:34:46 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2022-06-06T06:34:46.7941737Z Jun 06 06:34:46 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> 2022-06-06T06:34:46.7942588Z Jun 06 06:34:46 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:92)
> 2022-06-06T06:34:46.7943874Z Jun 06 06:34:46 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2022-06-06T06:34:46.7945291Z Jun 06 06:34:46 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-06-06T06:34:46.7946812Z Jun 06 06:34:46 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-06-06T06:34:46.7948852Z Jun 06 06:34:46 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-06-06T06:34:46.7950462Z Jun 06 06:34:46 at
>
[jira] [Closed] (FLINK-29835) NoClassDefFoundError in PulsarSourceUnorderedE2ECase
[
https://issues.apache.org/jira/browse/FLINK-29835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-29835.
-
Resolution: Invalid
After FLINK-30413 dropped the related support, this test was dropped also.
Invalid now.
> NoClassDefFoundError in PulsarSourceUnorderedE2ECase
>
>
> Key: FLINK-29835
> URL: https://issues.apache.org/jira/browse/FLINK-29835
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Pulsar, Runtime / Coordination
>Affects Versions: 1.16.0, 1.15.3, pulsar-3.0.0, pulsar-4.0.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
> Attachments:
> PulsarSourceUnorderedE2ECase.testSavepoint.FileNotFoundException.log
>
>
> [This
> build|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42680=logs=87489130-75dc-54e4-1f45-80c30aa367a3=73da6d75-f30d-5d5a-acbe-487a9dcff678=16001]
> failed in {{PulsarSourceUnorderedE2ECase.testSavepoint}} due to some job
> timeout.
> The logs reveal {{{}NoClassDefFoundErrors{}}}:
> {code:java}
> 2022-11-14 15:36:59,696 WARN
> org.apache.pulsar.shade.org.asynchttpclient.DefaultAsyncHttpClient [] -
> Unexpected error on ChannelManager close
> java.lang.NoClassDefFoundError:
> org/apache/pulsar/shade/io/netty/util/concurrent/DefaultPromise$1
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499)
> ~[?:?]
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
> ~[?:?]
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
> ~[?:?]
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
> ~[?:?]
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.ensureThreadStarted(SingleThreadEventExecutor.java:970)
> ~[?:?]
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.shutdownGracefully(SingleThreadEventExecutor.java:661)
> ~[?:?]
> at
> org.apache.pulsar.shade.io.netty.util.concurrent.MultithreadEventExecutorGroup.shutdownGracefully(MultithreadEventExecutorGroup.java:163)
> ~[?:?]
> at
> org.apache.pulsar.shade.org.asynchttpclient.netty.channel.ChannelManager.close(ChannelManager.java:307)
> ~[?:?]
> at
> org.apache.pulsar.shade.org.asynchttpclient.DefaultAsyncHttpClient.close(DefaultAsyncHttpClient.java:120)
> ~[?:?]
> at
> org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.close(AsyncHttpConnector.java:346)
> ~[?:?]
> at
> org.apache.pulsar.client.admin.internal.PulsarAdminImpl.close(PulsarAdminImpl.java:490)
> ~[?:?]
> at
> org.apache.flink.connector.pulsar.source.enumerator.PulsarSourceEnumerator.close(PulsarSourceEnumerator.java:172)
> ~[?:?]
> at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:255)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.runtime.source.coordinator.SourceCoordinator.close(SourceCoordinator.java:265)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.coordination.ComponentClosingUtils.lambda$closeAsyncWithTimeout$0(ComponentClosingUtils.java:76)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
> Caused by: java.lang.ClassNotFoundException:
> org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise$1
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> ~[?:1.8.0_342]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_342]
> at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_342]
> ... 16 more {code}
> Outdated:
> -The issue seems to be related to the BlobServer failing to provide some
> artifacts ({{{}java.io.FileNotFoundException{}}}) that consequently causes
> classes not being found.-
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29755) PulsarSourceUnorderedE2ECase.testSavepoint failed because of missing TaskManagers
[
https://issues.apache.org/jira/browse/FLINK-29755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-29755.
-
Resolution: Invalid
After FLINK-30413 dropped the related support, this test was dropped also.
Invalid now.
> PulsarSourceUnorderedE2ECase.testSavepoint failed because of missing
> TaskManagers
> -
>
> Key: FLINK-29755
> URL: https://issues.apache.org/jira/browse/FLINK-29755
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Pulsar
>Affects Versions: 1.16.0, pulsar-3.0.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
> Attachments: PulsarSourceUnorderedE2ECase.testSavepoint.log
>
>
> [This
> build|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42325=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=13932]
> failed (not exclusively) due to a problem with
> {{PulsarSourceUnorderedE2ECase.testSavepoint}}. It seems like there were no
> TaskManagers spun up which resulted in the test job failing with a
> {{NoResourceAvailableException}}.
> {code}
> org.apache.flink.runtime.jobmaster.slotpool.DeclarativeSlotPoolBridge [] -
> Could not acquire the minimum required resources, failing slot requests.
> Acquired: []. Current slot pool status: Registered TMs: 0, registered slots:
> 0 free slots: 0
> {code}
> I didn't raise this one to critical because it looks like a missing
> TaskManager test environment issue. I attached the e2e test-specific logs to
> the Jira issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30393) PulsarSourceUnorderedE2ECase.testScaleDown failed with TimeoutException
[ https://issues.apache.org/jira/browse/FLINK-30393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-30393. - Resolution: Invalid After FLINK-30413 dropped the related support, this test was dropped also. Invalid now. > PulsarSourceUnorderedE2ECase.testScaleDown failed with TimeoutException > --- > > Key: FLINK-30393 > URL: https://issues.apache.org/jira/browse/FLINK-30393 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: Qingsheng Ren >Priority: Major > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43904=logs=f8e16326-dc75-5ba0-3e95-6178dd55bf6c=15c1d318-5ca8-529f-77a2-d113a700ec34=16386 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30340) PulsarUnorderedSourceITCase fails with IllegalStateException
[
https://issues.apache.org/jira/browse/FLINK-30340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-30340.
-
Resolution: Invalid
After FLINK-30413 dropped the related support, this test was dropped also.
Invalid now.
> PulsarUnorderedSourceITCase fails with IllegalStateException
>
>
> Key: FLINK-30340
> URL: https://issues.apache.org/jira/browse/FLINK-30340
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Pulsar
>Affects Versions: pulsar-4.0.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: test-stability
>
> We have test failures in \{{PulsarUnorderedSourceITCase}} in the following
> build due to an {{IllegalStateException}} being caused by the job reaching
> {{CANCELLED}} state:
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43793=logs=8eee98ee-a482-5f7c-2c51-b3456453e704=da58e781-88fe-508b-b74c-018210e533cc=38014]
> {code:java}
> Dec 08 01:26:44 java.lang.IllegalStateException: Job has entered CANCELED
> state, but expecting [FINISHED]
> Dec 08 01:26:44 at
> org.apache.flink.runtime.testutils.CommonTestUtils.lambda$waitForJobStatus$7(CommonTestUtils.java:284)
> Dec 08 01:26:44 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:150)
> Dec 08 01:26:44 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:144)
> Dec 08 01:26:44 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitForJobStatus(CommonTestUtils.java:268)
> Dec 08 01:26:44 at
> org.apache.flink.connector.testframe.testsuites.SourceTestSuiteBase.testSourceSingleSplit(SourceTestSuiteBase.java:159)
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
