[jira] [Closed] (FLINK-22137) Execute unaligned checkpoint test on a cluster
[ https://issues.apache.org/jira/browse/FLINK-22137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise closed FLINK-22137. --- Resolution: Done Thanks for reminding. This has been done eons ago ;) > Execute unaligned checkpoint test on a cluster > -- > > Key: FLINK-22137 > URL: https://issues.apache.org/jira/browse/FLINK-22137 > Project: Flink > Issue Type: Sub-task >Reporter: Arvid Heise >Priority: Major > Labels: pull-request-available > > Start application and at some point cancel/induce failure, the user needs to > restart from a retained checkpoint with > * lower > * same > * higher degree of parallelism. > To enable unaligned checkpoints, set > * execution.checkpointing.unaligned: true > * execution.checkpointing.alignment-timeout to 0s, 10s, 1min (for high > backpressure) > The primary objective is to check if all data is recovered properly and if > the semantics is correct (does state match input?). > The secondary objective is to check if Flink UI shows the information > correctly: > * unaligned checkpoint enabled on job level > * timeout on job level > * for each checkpoint, if it's unaligned or not; how much data was written -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6573) Flink MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606504#comment-17606504 ] Arvid Heise commented on FLINK-6573: [~martijnvisser] [~MartijnVisser] is probably the right person for guiding the new connector. He is currently pushing for the different external connector repos. @Martijn, could you please champion this effort? (Side note: you should get your apache accounts merged through INFRA ;)). > Flink MongoDB Connector > --- > > Key: FLINK-6573 > URL: https://issues.apache.org/jira/browse/FLINK-6573 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.2.0 > Environment: Linux Operating System, Mongo DB >Reporter: Nagamallikarjuna >Assignee: ZhuoYu Chen >Priority: Not a Priority > Labels: pull-request-available, stale-assigned > Attachments: image-2021-11-15-14-41-07-514.png > > Original Estimate: 672h > Remaining Estimate: 672h > > Hi Community, > Currently we are using Flink in the current Project. We have huge amount of > data to process using Flink which resides in Mongo DB. We have a requirement > of parallel data connectivity in between Flink and Mongo DB for both > reads/writes. Currently we are planning to create this connector and > contribute to the Community. > I will update the further details once I receive your feedback > Please let us know if you have any concerns. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-26760) The new CSV source (file system source + CSV format) does not support reading files whose file encoding is not UTF-8
[
https://issues.apache.org/jira/browse/FLINK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-26760:
---
Assignee: (was: Arvid Heise)
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8
>
>
> Key: FLINK-26760
> URL: https://issues.apache.org/jira/browse/FLINK-26760
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.13.6, 1.14.4, 1.15.0
>Reporter: Lijie Wang
>Priority: Major
> Fix For: 1.17.0
>
> Attachments: PerformanceTest.java, example.csv
>
>
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8, but the legacy {{CsvTableSource}}
> supports it.
> We provide an {{*example.csv*}} whose file encoding is {{{}ISO-8599-1{}}}.
> When reading it with the legacy {{{}CsvTableSource{}}}, it executes correctly:
> {code:java}
> @Test
> public void testLegacyCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> CsvTableSource.Builder builder = CsvTableSource.builder();
> CsvTableSource source =
> builder.path("example.csv")
> .emptyColumnAsNull()
> .lineDelimiter("\n")
> .fieldDelimiter("|")
> .field("name", DataTypes.STRING())
> .build();
> ConnectorCatalogTable catalogTable =
> ConnectorCatalogTable.source(source, true);
> tEnv.getCatalog(tEnv.getCurrentCatalog())
> .ifPresent(
> catalog -> {
> try {
> catalog.createTable(
> new
> ObjectPath(tEnv.getCurrentDatabase(), "example"),
> catalogTable,
> false);
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> });
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
>
> When reading it with the new CSV source (file system source + CSV format), it
> throws the following error:
> {code:java}
> @Test
> public void testNewCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> String ddl =
> "create table example ("
> + "name string"
> + ") with ("
> + "'connector' = 'filesystem',"
> + "'path' = 'example.csv',"
> + "'format' = 'csv',"
> + "'csv.array-element-delimiter' = '\n',"
> + "'csv.field-delimiter' = '|',"
> + "'csv.null-literal' = ''"
> + ")";
> tEnv.executeSql(ddl);
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
> at
>
[jira] [Commented] (FLINK-23528) stop-with-savepoint can fail with FlinkKinesisConsumer
[
https://issues.apache.org/jira/browse/FLINK-23528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564343#comment-17564343
]
Arvid Heise commented on FLINK-23528:
-
As far as I remember, I needed to fix this for the tests to succeed reliably.
> stop-with-savepoint can fail with FlinkKinesisConsumer
> --
>
> Key: FLINK-23528
> URL: https://issues.apache.org/jira/browse/FLINK-23528
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.11.3, 1.13.1, 1.12.4, 1.14.3, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Krzysztof Dziolak
>Priority: Critical
> Labels: pull-request-available, stale-assigned
> Fix For: 1.15.2
>
>
> {{FlinkKinesisConsumer#cancel()}} (inside
> {{KinesisDataFetcher#shutdownFetcher()}}) shouldn't be interrupting source
> thread. Otherwise, as described in FLINK-23527, network stack can be left in
> an invalid state and downstream tasks can encounter deserialisation errors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-23528) stop-with-savepoint can fail with FlinkKinesisConsumer
Title: Message Title Arvid Heise commented on FLINK-23528 Re: stop-with-savepoint can fail with FlinkKinesisConsumer Hi Danny Cranmer, sorry for the late reply. The main problem with completing the ticket was the lack of test coverage for EFO. See your comment in https://github.com/apache/flink/pull/17189#discussion_r705274129. I'm positive that the PR works for non-EFO cases. Add Comment This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)
[jira] [Updated] (FLINK-26793) Flink Cassandra connector performance issue
[ https://issues.apache.org/jira/browse/FLINK-26793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-26793: Labels: documentation (was: ) > Flink Cassandra connector performance issue > > > Key: FLINK-26793 > URL: https://issues.apache.org/jira/browse/FLINK-26793 > Project: Flink > Issue Type: Improvement > Components: Connectors / Cassandra >Affects Versions: 1.14.4 >Reporter: Jay Ghiya >Assignee: Etienne Chauchot >Priority: Major > Labels: documentation > Attachments: Capture d’écran de 2022-04-14 16-34-59.png, Capture > d’écran de 2022-04-14 16-35-07.png, Capture d’écran de 2022-04-14 > 16-35-30.png, jobmanager_log.txt, taskmanager_127.0.1.1_33251-af56fa_log > > > A warning is observed during long runs of flink job stating “Insertions into > scylla might be suffering. Expect performance problems unless this is > resolved.” > Upon initial analysis - “flink cassandra connector is not keeping instance of > mapping manager that is used to convert a pojo to cassandra row. Ideally the > mapping manager should have the same life time as cluster and session objects > which are also created once when the driver is initialized” > Reference: > https://stackoverflow.com/questions/59203418/cassandra-java-driver-warning -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-27318) KafkaSink: when init transaction, it take too long to get a producerId with epoch=0
[
https://issues.apache.org/jira/browse/FLINK-27318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530623#comment-17530623
]
Arvid Heise commented on FLINK-27318:
-
This is exactly what I meant: You don't restart a task, you start a new one. So
you should use a new transaction prefix for all intents and purposes.
Just imagine, you run the same application twice at the same time on different
input topics. You'd naturally pick two different prefixes, so that the
transactions don't interfere. Now your use case implies you run the same
application twice just at different times. I'd still naturally choose different
transaction prefixes to avoid that the transactions interfere. For example,
imagine that some prefix transactions linger around and need to be canceled
(that is the whole reason for the algorithm that causes issues to you).
Just for clarification: there is no way in Kafka to check how many and which
transactions are open on a given topic so we need to go to such lengths to
cancel all transactions or else you wouldn't see any data in your output until
all transactions timeout. I'd argue that our approach is faster in most
settings. Usually, we can shortcut this whole algorithm by using the last
successful transaction id but we don't have this information if we are not
resuming from a checkpoint. Again, Kafka has no transaction API to speak of.
Kafka assumes that you use the same transaction id for the whole application,
which kind of works for Kafka Streams but doesn't play nicely with Flink: In
the worst case, we can't write any data during a checkpoint at all.
But just to double-check: Do you delete the output topic before restarting?
That seems natural to me and I would have assumed that related transaction data
is deleted on Kafka side but I wouldn't be surprised if it didn't and we see
the exhibited issue.
> KafkaSink: when init transaction, it take too long to get a producerId with
> epoch=0
> ---
>
> Key: FLINK-27318
> URL: https://issues.apache.org/jira/browse/FLINK-27318
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.4
>Reporter: Zhengqi Zhang
>Priority: Major
> Attachments: image-2022-04-20-17-34-48-207.png,
> image-2022-04-20-17-59-27-397.png, image-2022-05-01-16-50-27-264.png
>
>
> as we can see, the new KafkaSink aborts all transactions that have been
> created by a subtask in a previous run, only return when get a producerId was
> unused before(epoch=0). But this can take a long time, especially if the task
> has been started and cancelled many times before. In my tests, it even took
> {*}10 minutes{*}. Is there a better way to solve this problem, or {*}do what
> FlinkKafkaProducer did{*}.
> !image-2022-04-20-17-59-27-397.png|width=534,height=256!
> !image-2022-04-20-17-34-48-207.png|width=556,height=412!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27405) Refactor SourceCoordinator to abstract BaseCoordinator implementation
[ https://issues.apache.org/jira/browse/FLINK-27405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529957#comment-17529957 ] Arvid Heise commented on FLINK-27405: - Hi [~stevenz3wu], I think this approach looks promising, here are a couple of gotchas: - Coordinator API is internal, so you either need a FLIP to make it public or you need to create some public abstraction on top of it which requires a FLIP. This is especially important for Iceberg since it's living in its own repository (as all connectors should). - Coordinators checkpoint independent of the related subtasks at the beginning of a checkpoint at the job manager. For a statistics component that should not matter in practice but you should be careful who owns which part of information and what happens when these parts are checkpointed out of sync and recovered later. One way would be to always checkpoint the collected statistics on subtask level and to not checkpoint any derived information. Upon recovery, all subtasks resend their recovered statics to the coordinator. That makes the coordinator effectively stateless. - This coordinator checkpointing behavior might make releasing coordinator API harder. Alternatively, we could implement some kind of checkpointing alignment in a separate effort that acts as a prerequisite. - Metric groups are still not implemented on coordinator side :( I do believe that a public coordinator API would be beneficial for quite some advanced use cases. I'm not sure if you or someone else have the capacity to lead this effort. > Refactor SourceCoordinator to abstract BaseCoordinator implementation > - > > Key: FLINK-27405 > URL: https://issues.apache.org/jira/browse/FLINK-27405 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: gang ye >Priority: Major > > To solve small files issue caused by data skewness, Flink Iceberg data > shuffling was proposed(design doc > [https://docs.google.com/document/d/13N8cMqPi-ZPSKbkXGOBMPOzbv2Fua59j8bIjjtxLWqo/edit]#). > The basic idea is to use statistics operator to collect local statistics for > traffic distribution at taskmanagers (workers). Local statistics are > periodically sent to the statistics coordinator (running in jobmanager). Once > globally aggregated statistics are ready, the statistics coordinator > broadcasts them to all operator instances. And then a customized partitioner > uses the global statistics which is passed down from statistics operator to > distribute data to Iceberg writers. > In the process of Flink Iceberg data shuffling implementation, we found that, > StatisticsCoordinator can share function with > SourceCoordinator#runInEventLoop, StatisticsCoordinatorContext needs similar > function as SourceCoordinatorConext#callInCoordinatorThread and the > StatisticsCoordinatorProvider ExecutorThreadFactory logic is almost same as > SourceCoordinatorProvider#CoordinatorExecutorThreadFactory. So we would want > to refactor the source coordinator classes to abstract a general coordinator > implementation to reduce the duplicated code when adding other coordinators. > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-26645) Pulsar Source subscribe to a single topic partition will consume all partitions from that topic
[ https://issues.apache.org/jira/browse/FLINK-26645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529221#comment-17529221 ] Arvid Heise commented on FLINK-26645: - Merged into 1.15 as 5de8016a5dcee8a38d21f54e685633809f800c48. > Pulsar Source subscribe to a single topic partition will consume all > partitions from that topic > > > Key: FLINK-26645 > URL: https://issues.apache.org/jira/browse/FLINK-26645 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0, 1.14.4 >Reporter: Yufei Zhang >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0, 1.14.5 > > > Say users subscribe to 4 partitions of a topic with 16 partitions, current > Pulsar source > will actually consume from all 16 partitions. Expect to consume from 4 > partitions only. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-27320) When using KafkaSink EXACTLY_ONCE semantics, frequent OutOfOrderSequenceException anomalies
[
https://issues.apache.org/jira/browse/FLINK-27320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528911#comment-17528911
]
Arvid Heise commented on FLINK-27320:
-
Could you please add which Kafka server version you are using? We only observed
these issues with RedPanda and couldn't reproduce it with any real Kafka server
so far.
> When using KafkaSink EXACTLY_ONCE semantics, frequent
> OutOfOrderSequenceException anomalies
> ---
>
> Key: FLINK-27320
> URL: https://issues.apache.org/jira/browse/FLINK-27320
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.4
>Reporter: Zhengqi Zhang
>Priority: Major
> Attachments: image-2022-04-20-17-48-37-149.png,
> image-2022-04-20-17-49-15-143.png
>
>
> This problem does not occur when using EXACTLY_ONCE semantics in
> FlinkKafkaProducer, but occurs frequently when using KafkaSink.
> !image-2022-04-20-17-48-37-149.png|width=573,height=220!
> !image-2022-04-20-17-49-15-143.png|width=818,height=469!
> This is ProducerConfig when using KafkaSink:
> {code:java}
> acks = 1
> batch.size = 16384
> bootstrap.servers = [localhost:9092]
> buffer.memory = 33554432
> client.dns.lookup = default
> client.id =
> compression.type = none
> connections.max.idle.ms = 54
> delivery.timeout.ms = 12
> enable.idempotence = false
> interceptor.classes = []
> key.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
> linger.ms = 0
> max.block.ms = 6
> max.in.flight.requests.per.connection = 5
> max.request.size = 1048576
> metadata.max.age.ms = 30
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 3
> partitioner.class = class
> org.apache.kafka.clients.producer.internals.DefaultPartitioner
> receive.buffer.bytes = 32768
> reconnect.backoff.max.ms = 1000
> reconnect.backoff.ms = 50
> request.timeout.ms = 3
> retries = 2147483647
> retry.backoff.ms = 100
> sasl.client.callback.handler.class = null
> sasl.jaas.config = null
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.min.time.before.relogin = 6
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> sasl.kerberos.ticket.renew.window.factor = 0.8
> sasl.login.callback.handler.class = null
> sasl.login.class = null
> sasl.login.refresh.buffer.seconds = 300
> sasl.login.refresh.min.period.seconds = 60
> sasl.login.refresh.window.factor = 0.8
> sasl.login.refresh.window.jitter = 0.05
> sasl.mechanism = GSSAPI
> security.protocol = PLAINTEXT
> security.providers = null
> send.buffer.bytes = 131072
> ssl.cipher.suites = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.endpoint.identification.algorithm = https
> ssl.key.password = null
> ssl.keymanager.algorithm = SunX509
> ssl.keystore.location = null
> ssl.keystore.password = null
> ssl.keystore.type = JKS
> ssl.protocol = TLS
> ssl.provider = null
> ssl.secure.random.implementation = null
> ssl.trustmanager.algorithm = PKIX
> ssl.truststore.location = null
> ssl.truststore.password = null
> ssl.truststore.type = JKS
> transaction.timeout.ms = 30
> transactional.id = kafka-sink-0-36
> value.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer{code}
> This is ProducerConfig when using FlinkKafkaProducer:
>
> {code:java}
> acks = 1
> batch.size = 16384
> bootstrap.servers = [localhost:9092]
> buffer.memory = 33554432
> client.dns.lookup = default
> client.id =
> compression.type = none
> connections.max.idle.ms = 54
> delivery.timeout.ms = 12
> enable.idempotence = false
> interceptor.classes = []
> key.serializer = class
> org.apache.kafka.common.serialization.ByteArraySerializer
> linger.ms = 0
> max.block.ms = 6
> max.in.flight.requests.per.connection = 5
> max.request.size = 1048576
> metadata.max.age.ms = 30
> metric.reporters = []
> metrics.num.samples = 2
> metrics.recording.level = INFO
> metrics.sample.window.ms = 3
> partitioner.class = class
> org.apache.kafka.clients.producer.internals.DefaultPartitioner
> receive.buffer.bytes = 32768
> reconnect.backoff.max.ms = 1000
> reconnect.backoff.ms = 50
> request.timeout.ms = 3
> retries = 2147483647
> retry.backoff.ms = 100
> sasl.client.callback.handler.class = null
> sasl.jaas.config = null
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
>
[jira] [Commented] (FLINK-27318) KafkaSink: when init transaction, it take too long to get a producerId with epoch=0
[
https://issues.apache.org/jira/browse/FLINK-27318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527413#comment-17527413
]
Arvid Heise commented on FLINK-27318:
-
Btw your scenario could also happen if many successive checkpoints fail or you
are in a restart loop for a longer time. But then, having a 10 min recovery
wouldn't matter - you have other problems at hand. Once the restart loop is
solved and checkpointing is consistent, recovery should be faster than with
{{FlinkKafkaProducer}}.
> KafkaSink: when init transaction, it take too long to get a producerId with
> epoch=0
> ---
>
> Key: FLINK-27318
> URL: https://issues.apache.org/jira/browse/FLINK-27318
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.4
>Reporter: Zhengqi Zhang
>Priority: Major
> Attachments: image-2022-04-20-17-34-48-207.png,
> image-2022-04-20-17-59-27-397.png
>
>
> as we can see, the new KafkaSink aborts all transactions that have been
> created by a subtask in a previous run, only return when get a producerId was
> unused before(epoch=0). But this can take a long time, especially if the task
> has been started and cancelled many times before. In my tests, it even took
> {*}10 minutes{*}. Is there a better way to solve this problem, or {*}do what
> FlinkKafkaProducer did{*}.
> !image-2022-04-20-17-59-27-397.png|width=534,height=256!
> !image-2022-04-20-17-34-48-207.png|width=556,height=412!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27318) KafkaSink: when init transaction, it take too long to get a producerId with epoch=0
[
https://issues.apache.org/jira/browse/FLINK-27318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527412#comment-17527412
]
Arvid Heise commented on FLINK-27318:
-
Are you sure your test is realistic? In a production setup, you'd have each
transaction id only used once and only on restart, you'd abort 1 transaction.
If you ofc restart your application over and over again without a checkpoint
without using a different transaction id, then you'd get a large accumulation
of started transactions. But this is not how the sink should be used. Use one
transaction id prefix per unique start (and keep for restarts from checkpoints).
> KafkaSink: when init transaction, it take too long to get a producerId with
> epoch=0
> ---
>
> Key: FLINK-27318
> URL: https://issues.apache.org/jira/browse/FLINK-27318
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.4
>Reporter: Zhengqi Zhang
>Priority: Major
> Attachments: image-2022-04-20-17-34-48-207.png,
> image-2022-04-20-17-59-27-397.png
>
>
> as we can see, the new KafkaSink aborts all transactions that have been
> created by a subtask in a previous run, only return when get a producerId was
> unused before(epoch=0). But this can take a long time, especially if the task
> has been started and cancelled many times before. In my tests, it even took
> {*}10 minutes{*}. Is there a better way to solve this problem, or {*}do what
> FlinkKafkaProducer did{*}.
> !image-2022-04-20-17-59-27-397.png|width=534,height=256!
> !image-2022-04-20-17-34-48-207.png|width=556,height=412!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-26793) Flink Cassandra connector performance issue
[ https://issues.apache.org/jira/browse/FLINK-26793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527329#comment-17527329 ] Arvid Heise commented on FLINK-26793: - Just to double-check: Can we optimize anything on Flink side to lower the performance drop on restart? For me, it looks like there is nothing and we should just include the information about the timeout on the Cassandra documentation page (maybe even in the Javadocs). > Flink Cassandra connector performance issue > > > Key: FLINK-26793 > URL: https://issues.apache.org/jira/browse/FLINK-26793 > Project: Flink > Issue Type: Improvement > Components: Connectors / Cassandra >Affects Versions: 1.14.4 >Reporter: Jay Ghiya >Assignee: Etienne Chauchot >Priority: Major > Attachments: Capture d’écran de 2022-04-14 16-34-59.png, Capture > d’écran de 2022-04-14 16-35-07.png, Capture d’écran de 2022-04-14 > 16-35-30.png, jobmanager_log.txt, taskmanager_127.0.1.1_33251-af56fa_log > > > A warning is observed during long runs of flink job stating “Insertions into > scylla might be suffering. Expect performance problems unless this is > resolved.” > Upon initial analysis - “flink cassandra connector is not keeping instance of > mapping manager that is used to convert a pojo to cassandra row. Ideally the > mapping manager should have the same life time as cluster and session objects > which are also created once when the driver is initialized” > Reference: > https://stackoverflow.com/questions/59203418/cassandra-java-driver-warning -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-27137) Remove usage of AdminClient from KafkaSource logic
[
https://issues.apache.org/jira/browse/FLINK-27137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519674#comment-17519674
]
Arvid Heise commented on FLINK-27137:
-
Hi [~gyfora], sorry if you perceived my message as unfriendly that was not my
intention. :(
I just expect a ticket that basically reverts a ticket that went through duly
process to be more descriptive so that everyone can understand the issue and
discuss. The information in the ticket description is still lacking in that
regard.
You are also completely dismissing the valid use case of users not depending on
Java 8, simply bumping Kafka dependency to 3 and use the new functionality. So
please go for an approach that makes both possible.
I'd also raise the possibility that you simply implement a limited
{{AdminClient}} on your end as well or else you may run into issues with other
Java-based frameworks, such as Spark, that may or may not also use the
{{AdminClient}}. Expecting certain frameworks to use one part of Public API of
a client jar and not another is a non-obvious requirement. For context, you
can't run Pulsar without its {{AdminClient}}, so don't read too much into the
name. Lastly, even Redpanda is providing that API to be a full drop-in
replacement.
In general, I'd deem your approach to security as non-standard, so please make
sure to not disable the standard cases by simply reverting.
> Remove usage of AdminClient from KafkaSource logic
> --
>
> Key: FLINK-27137
> URL: https://issues.apache.org/jira/browse/FLINK-27137
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Affects Versions: 1.15.0, 1.14.4
>Reporter: Gyula Fora
>Priority: Major
>
> Parts of the KafkaSource logic (specifically the KafkaSourceEnumerator) uses
> the Kafka AdminClient instead of the KafkaConsumer.
> It seems that the KafkaConsumer already provides all the necessary
> information that the enumerator needs so there is no reason for introducing
> the AdminClient.
> In some environments using the AdminClient can be problematic even if we are
> not using certaing features.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-27137) Remove usage of AdminClient from KafkaSource logic
[ https://issues.apache.org/jira/browse/FLINK-27137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-27137: Priority: Major (was: Critical) > Remove usage of AdminClient from KafkaSource logic > -- > > Key: FLINK-27137 > URL: https://issues.apache.org/jira/browse/FLINK-27137 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.15.0, 1.14.4 >Reporter: Gyula Fora >Priority: Major > > Parts of the KafkaSource logic (specifically the KafkaSourceEnumerator) uses > the Kafka AdminClient instead of the KafkaConsumer. > It seems that the KafkaConsumer already provides all the necessary > information that the enumerator needs so there is no reason for introducing > the AdminClient. > In some environments using the AdminClient can be problematic even if we are > not using certaing features. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-27137) Remove usage of AdminClient from KafkaSource logic
[
https://issues.apache.org/jira/browse/FLINK-27137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519514#comment-17519514
]
Arvid Heise commented on FLINK-27137:
-
Can you please elaborate on {{In some environments using the AdminClient can be
problematic even if we are not using certaing features.}}
There are some pros to using the AdminClient and the drawbacks have not been
clearly stated, so I'd be inclined to close this ticket with "Won't Do".
After you have stated your concerns clearly, I'd probably go for a hybrid
approach, where a newly introduced {{OffsetRetriever}} abstracts from the used
client to serve both use cases.
> Remove usage of AdminClient from KafkaSource logic
> --
>
> Key: FLINK-27137
> URL: https://issues.apache.org/jira/browse/FLINK-27137
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Affects Versions: 1.15.0, 1.14.4
>Reporter: Gyula Fora
>Priority: Critical
>
> Parts of the KafkaSource logic (specifically the KafkaSourceEnumerator) uses
> the Kafka AdminClient instead of the KafkaConsumer.
> It seems that the KafkaConsumer already provides all the necessary
> information that the enumerator needs so there is no reason for introducing
> the AdminClient.
> In some environments using the AdminClient can be problematic even if we are
> not using certaing features.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25916) Using upsert-kafka with a flush buffer results in Null Pointer Exception
[
https://issues.apache.org/jira/browse/FLINK-25916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517087#comment-17517087
]
Arvid Heise commented on FLINK-25916:
-
This looks like FLINK-24608 to me except this was fixed in 1.14.3. [~heaje] can
you double-check if you use a newer version? I also see 1.15.0 in the ticket
description, did anyone actually verify it with 1.15.0?
> Using upsert-kafka with a flush buffer results in Null Pointer Exception
>
>
> Key: FLINK-25916
> URL: https://issues.apache.org/jira/browse/FLINK-25916
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Runtime
>Affects Versions: 1.15.0, 1.14.3
> Environment: CentOS 7.9 x64
> Intel Xeon Gold 6140 CPU
>Reporter: Corey Shaw
>Priority: Major
>
> Flink Version: 1.14.3
> upsert-kafka version: 1.14.3
>
> I have been trying to buffer output from the upsert-kafka connector using the
> documented parameters {{sink.buffer-flush.max-rows}} and
> {{sink.buffer-flush.interval}}
> Whenever I attempt to run an INSERT query with buffering, I receive the
> following error (shortened for brevity):
> {code:java}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.streaming.connectors.kafka.table.ReducingUpsertWriter.flush(ReducingUpsertWriter.java:145)
>
> at
> org.apache.flink.streaming.connectors.kafka.table.ReducingUpsertWriter.lambda$registerFlush$3(ReducingUpsertWriter.java:124)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1693)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$22(StreamTask.java:1684)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:338)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:324)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
>
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
>
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> at java.lang.Thread.run(Thread.java:829) [?:?] {code}
>
> If I remove the parameters related to flush buffering, then everything works
> as expected with no problems at all. For reference, here is the full setup
> with source, destination, and queries. Yes, I realize the INSERT could use
> an overhaul, but that's not the issue at hand :).
> {code:java}
> CREATE TABLE `source_topic` (
> `timeGMT` INT,
> `eventtime` AS TO_TIMESTAMP(FROM_UNIXTIME(`timeGMT`)),
> `visIdHigh` BIGINT,
> `visIdLow` BIGINT,
> `visIdStr` AS CONCAT(IF(`visIdHigh` IS NULL, '', CAST(`visIdHigh` AS
> STRING)), IF(`visIdLow` IS NULL, '', CAST(`visIdLow` AS STRING))),
> WATERMARK FOR eventtime AS eventtime - INTERVAL '25' SECONDS
> ) WITH (
> 'connector' = 'kafka',
> 'properties.group.id' = 'flink_metrics',
> 'properties.bootstrap.servers' = 'brokers.example.com:9093',
> 'topic' = 'source_topic',
> 'scan.startup.mode' = 'earliest-offset',
> 'value.format' = 'avro-confluent',
> 'value.avro-confluent.url' = 'http://schema.example.com',
> 'value.fields-include' = 'EXCEPT_KEY'
> );
> CREATE TABLE dest_topic (
> `messageType` VARCHAR,
> `observationID` BIGINT,
> `obsYear` BIGINT,
> `obsMonth` BIGINT,
> `obsDay` BIGINT,
> `obsHour` BIGINT,
> `obsMinute` BIGINT,
> `obsTz` VARCHAR(5),
> `value` BIGINT,
> PRIMARY KEY (observationID, messageType) NOT ENFORCED
> ) WITH (
> 'connector' = 'upsert-kafka',
> 'key.format' = 'json',
> 'properties.bootstrap.servers' = 'brokers.example.com:9092',
> 'sink.buffer-flush.max-rows' = '5',
> 'sink.buffer-flush.interval' = '1000',
> 'topic' = 'dest_topic ',
> 'value.format' = 'json'
> );
> INSERT INTO adobenow_metrics
> SELECT `messageType`, `observationID`, obsYear, obsMonth, obsDay,
> obsHour, obsMinute, obsTz,
[jira] [Closed] (FLINK-26751) [FLIP-171] Kafka implementation of AsyncSinkBase
[
https://issues.apache.org/jira/browse/FLINK-26751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise closed FLINK-26751.
---
Resolution: Won't Fix
As stated, with the current async sink design, it's not possible to implement
an EOS sink. If we ever extend the async sink, we can reopen the ticket.
Note that async sink is a means to reduce boilerplate for "easy" sinks that do
not require a sophisticated state management, in particular sinks that just do
some API calls to write data in either a fire-and-forget or retry-until-success
fashion. We would probably overengineer the interface if we also want to
capture the various EOS flavors. It's wiser to add more base implementations of
{{Sink}} in the future when multiple EOS sinks share common code (e.g., Pulsar
and Kafka).
> [FLIP-171] Kafka implementation of AsyncSinkBase
>
>
> Key: FLINK-26751
> URL: https://issues.apache.org/jira/browse/FLINK-26751
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Kafka
>Reporter: Almog Tavor
>Priority: Major
>
> *User stories:*
> Standardize the Kafka connector to implement AsyncSinkBase.
> *Scope:*
> * Implement an asynchronous sink for Kafka by inheriting the AsyncSinkBase
> class.
> h2. References
> More details to be found
> [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]
> h4.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (FLINK-26760) The new CSV source (file system source + CSV format) does not support reading files whose file encoding is not UTF-8
[
https://issues.apache.org/jira/browse/FLINK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516711#comment-17516711
]
Arvid Heise edited comment on FLINK-26760 at 4/4/22 9:25 AM:
-
Why are our benchmarks not use UTF-8? I thought that was the default [~twalthr]?
was (Author: arvid):
Why are our benchmarks not use UTF-8? I thought that was the default
[~twalters]?
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8
>
>
> Key: FLINK-26760
> URL: https://issues.apache.org/jira/browse/FLINK-26760
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0, 1.13.6, 1.14.4
>Reporter: Lijie Wang
>Assignee: Arvid Heise
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: PerformanceTest.java, example.csv
>
>
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8, but the legacy {{CsvTableSource}}
> supports it.
> We provide an {{*example.csv*}} whose file encoding is {{{}ISO-8599-1{}}}.
> When reading it with the legacy {{{}CsvTableSource{}}}, it executes correctly:
> {code:java}
> @Test
> public void testLegacyCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> CsvTableSource.Builder builder = CsvTableSource.builder();
> CsvTableSource source =
> builder.path("example.csv")
> .emptyColumnAsNull()
> .lineDelimiter("\n")
> .fieldDelimiter("|")
> .field("name", DataTypes.STRING())
> .build();
> ConnectorCatalogTable catalogTable =
> ConnectorCatalogTable.source(source, true);
> tEnv.getCatalog(tEnv.getCurrentCatalog())
> .ifPresent(
> catalog -> {
> try {
> catalog.createTable(
> new
> ObjectPath(tEnv.getCurrentDatabase(), "example"),
> catalogTable,
> false);
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> });
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
>
> When reading it with the new CSV source (file system source + CSV format), it
> throws the following error:
> {code:java}
> @Test
> public void testNewCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> String ddl =
> "create table example ("
> + "name string"
> + ") with ("
> + "'connector' = 'filesystem',"
> + "'path' = 'example.csv',"
> + "'format' = 'csv',"
> + "'csv.array-element-delimiter' = '\n',"
> + "'csv.field-delimiter' = '|',"
> + "'csv.null-literal' = ''"
> + ")";
> tEnv.executeSql(ddl);
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
> at
>
[jira] [Commented] (FLINK-26760) The new CSV source (file system source + CSV format) does not support reading files whose file encoding is not UTF-8
[
https://issues.apache.org/jira/browse/FLINK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516711#comment-17516711
]
Arvid Heise commented on FLINK-26760:
-
Why are our benchmarks not use UTF-8? I thought that was the default
[~twalters]?
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8
>
>
> Key: FLINK-26760
> URL: https://issues.apache.org/jira/browse/FLINK-26760
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0, 1.13.6, 1.14.4
>Reporter: Lijie Wang
>Assignee: Arvid Heise
>Priority: Major
> Fix For: 1.15.0
>
> Attachments: PerformanceTest.java, example.csv
>
>
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8, but the legacy {{CsvTableSource}}
> supports it.
> We provide an {{*example.csv*}} whose file encoding is {{{}ISO-8599-1{}}}.
> When reading it with the legacy {{{}CsvTableSource{}}}, it executes correctly:
> {code:java}
> @Test
> public void testLegacyCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> CsvTableSource.Builder builder = CsvTableSource.builder();
> CsvTableSource source =
> builder.path("example.csv")
> .emptyColumnAsNull()
> .lineDelimiter("\n")
> .fieldDelimiter("|")
> .field("name", DataTypes.STRING())
> .build();
> ConnectorCatalogTable catalogTable =
> ConnectorCatalogTable.source(source, true);
> tEnv.getCatalog(tEnv.getCurrentCatalog())
> .ifPresent(
> catalog -> {
> try {
> catalog.createTable(
> new
> ObjectPath(tEnv.getCurrentDatabase(), "example"),
> catalogTable,
> false);
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> });
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
>
> When reading it with the new CSV source (file system source + CSV format), it
> throws the following error:
> {code:java}
> @Test
> public void testNewCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> String ddl =
> "create table example ("
> + "name string"
> + ") with ("
> + "'connector' = 'filesystem',"
> + "'path' = 'example.csv',"
> + "'format' = 'csv',"
> + "'csv.array-element-delimiter' = '\n',"
> + "'csv.field-delimiter' = '|',"
> + "'csv.null-literal' = ''"
> + ")";
> tEnv.executeSql(ddl);
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
> at
>
[jira] [Commented] (FLINK-26645) Pulsar Source subscribe to a single topic partition will consume all partitions from that topic
[ https://issues.apache.org/jira/browse/FLINK-26645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516471#comment-17516471 ] Arvid Heise commented on FLINK-26645: - Merged into master as 71a9f317594d8f02cfc0242050ed634bafae873a. > Pulsar Source subscribe to a single topic partition will consume all > partitions from that topic > > > Key: FLINK-26645 > URL: https://issues.apache.org/jira/browse/FLINK-26645 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0, 1.14.4 >Reporter: Yufei Zhang >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.14.5 > > > Say users subscribe to 4 partitions of a topic with 16 partitions, current > Pulsar source > will actually consume from all 16 partitions. Expect to consume from 4 > partitions only. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26931) Pulsar sink's producer name should be unique
[ https://issues.apache.org/jira/browse/FLINK-26931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17516069#comment-17516069 ] Arvid Heise commented on FLINK-26931: - Merged into master as https://github.com/apache/flink/commit/495685970e31085815ba0435322ab44e4504cd55. > Pulsar sink's producer name should be unique > > > Key: FLINK-26931 > URL: https://issues.apache.org/jira/browse/FLINK-26931 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.15.0, 1.16.0 >Reporter: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Pulsar's new sink interface didn't make the producer name unique. Which would > make the pulsar fail to consume messages. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25256) Savepoints do not work with ExternallyInducedSources
[
https://issues.apache.org/jira/browse/FLINK-25256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise updated FLINK-25256:
Fix Version/s: 1.15.0
1.14.5
> Savepoints do not work with ExternallyInducedSources
>
>
> Key: FLINK-25256
> URL: https://issues.apache.org/jira/browse/FLINK-25256
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0, 1.13.3, 1.12.7
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.14.5
>
>
> It is not possible to take a proper savepoint with
> {{ExternallyInducedSource}} or {{ExternallyInducedSourceReader}} (both legacy
> and FLIP-27 versions). The problem is that we're hardcoding
> {{CheckpointOptions}} in the {{triggerHook}}.
> The outcome of current state is that operators would try to take checkpoints
> in the checkpoint location whereas the {{CheckpointCoordinator}} will write
> metadata for those states in the savepoint location.
> Moreover the situation gets even weirder (I have not checked it entirely), if
> we have a mixture of {{ExternallyInducedSource(s)}} and regular sources. In
> such a case the location and format at which the state of a particular task
> is persisted depends on the order of barriers arrival. If a barrier from a
> regular source arrives last the task takes a savepoint, on the other hand if
> last barrier is from an externally induced source it will take a checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25256) Savepoints do not work with ExternallyInducedSources
[
https://issues.apache.org/jira/browse/FLINK-25256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514306#comment-17514306
]
Arvid Heise commented on FLINK-25256:
-
Merged into master as
7aefdf2c6aefe8c24af30a4f28a59f2780503d21..a4d194e4a0981dceb003508a3178c6a4e0ce0e82.
1.14 backport is pending.
> Savepoints do not work with ExternallyInducedSources
>
>
> Key: FLINK-25256
> URL: https://issues.apache.org/jira/browse/FLINK-25256
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0, 1.13.3, 1.12.7
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Major
> Labels: pull-request-available
>
> It is not possible to take a proper savepoint with
> {{ExternallyInducedSource}} or {{ExternallyInducedSourceReader}} (both legacy
> and FLIP-27 versions). The problem is that we're hardcoding
> {{CheckpointOptions}} in the {{triggerHook}}.
> The outcome of current state is that operators would try to take checkpoints
> in the checkpoint location whereas the {{CheckpointCoordinator}} will write
> metadata for those states in the savepoint location.
> Moreover the situation gets even weirder (I have not checked it entirely), if
> we have a mixture of {{ExternallyInducedSource(s)}} and regular sources. In
> such a case the location and format at which the state of a particular task
> is persisted depends on the order of barriers arrival. If a barrier from a
> regular source arrives last the task takes a savepoint, on the other hand if
> last barrier is from an externally induced source it will take a checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25256) Savepoints do not work with ExternallyInducedSources
[
https://issues.apache.org/jira/browse/FLINK-25256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17513910#comment-17513910
]
Arvid Heise commented on FLINK-25256:
-
Merged into 1.15 as
5a5490a49866d02ab5c4761f59ddf94c06f42b41..42e978548f44e98fafbf5f62175e48acba8642e0.
> Savepoints do not work with ExternallyInducedSources
>
>
> Key: FLINK-25256
> URL: https://issues.apache.org/jira/browse/FLINK-25256
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0, 1.13.3, 1.12.7
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Major
> Labels: pull-request-available
>
> It is not possible to take a proper savepoint with
> {{ExternallyInducedSource}} or {{ExternallyInducedSourceReader}} (both legacy
> and FLIP-27 versions). The problem is that we're hardcoding
> {{CheckpointOptions}} in the {{triggerHook}}.
> The outcome of current state is that operators would try to take checkpoints
> in the checkpoint location whereas the {{CheckpointCoordinator}} will write
> metadata for those states in the savepoint location.
> Moreover the situation gets even weirder (I have not checked it entirely), if
> we have a mixture of {{ExternallyInducedSource(s)}} and regular sources. In
> such a case the location and format at which the state of a particular task
> is persisted depends on the order of barriers arrival. If a barrier from a
> regular source arrives last the task takes a savepoint, on the other hand if
> last barrier is from an externally induced source it will take a checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26790) Description image about Unaligned Checkpointing maybe incorrect.
[ https://issues.apache.org/jira/browse/FLINK-26790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511226#comment-17511226 ] Arvid Heise commented on FLINK-26790: - I guess you understand "end of output buffers" differently than I do. I meant to say that it will be added as the buffer that is being sent next downstream. Do you have a better wording? In case of a deque, we would probably call it head. [~pnowojski] FYI. > Description image about Unaligned Checkpointing maybe incorrect. > > > Key: FLINK-26790 > URL: https://issues.apache.org/jira/browse/FLINK-26790 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.13.6, 1.14.4 >Reporter: jinghaihang >Priority: Minor > Attachments: image-2022-03-22-11-12-20-025.png > > > Due to the second description ： > * It immediately forwards the barrier to the downstream operator by adding > it to the end of the output buffers. > so i think the picture maybe misleading readers that barrier will not appear > in the middle of the data stream. > > !image-2022-03-22-11-12-20-025.png|width=599,height=303! -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-26701) Relocation of connector-base might break user jars due to changed imports
[ https://issues.apache.org/jira/browse/FLINK-26701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-26701. - Assignee: Alexander Fedulov Resolution: Fixed > Relocation of connector-base might break user jars due to changed imports > - > > Key: FLINK-26701 > URL: https://issues.apache.org/jira/browse/FLINK-26701 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Assignee: Alexander Fedulov >Priority: Blocker > Labels: pull-request-available > Fix For: 1.15.0 > > > With the introduction of FLINK-25927, every connector now relocates > connector-base to better support connectors compatibility with different > Flink versions. Unfortunately, not all classes in connector-base are only > used by connector but some are supposed to be used inside the user jar > directly i.e. DeliveryGuarantee, HybridSource... > Since the connector now relocates the module the existing imports are broken. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26701) Relocation of connector-base might break user jars due to changed imports
[ https://issues.apache.org/jira/browse/FLINK-26701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511142#comment-17511142 ] Arvid Heise commented on FLINK-26701: - Merged into 1.15 as 1b04ba84222dd44cce9b3241e7a939275557478a, into master as 62defcae6bfe325265c7386542cf44ee179f8385. > Relocation of connector-base might break user jars due to changed imports > - > > Key: FLINK-26701 > URL: https://issues.apache.org/jira/browse/FLINK-26701 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: Fabian Paul >Priority: Blocker > Labels: pull-request-available > Fix For: 1.15.0 > > > With the introduction of FLINK-25927, every connector now relocates > connector-base to better support connectors compatibility with different > Flink versions. Unfortunately, not all classes in connector-base are only > used by connector but some are supposed to be used inside the user jar > directly i.e. DeliveryGuarantee, HybridSource... > Since the connector now relocates the module the existing imports are broken. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26760) The new CSV source (file system source + CSV format) does not support reading files whose file encoding is not UTF-8
[
https://issues.apache.org/jira/browse/FLINK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise updated FLINK-26760:
Issue Type: Improvement (was: Bug)
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8
>
>
> Key: FLINK-26760
> URL: https://issues.apache.org/jira/browse/FLINK-26760
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0, 1.13.6, 1.14.4
>Reporter: Lijie Wang
>Priority: Critical
> Fix For: 1.15.0
>
> Attachments: example.csv
>
>
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8, but the legacy {{CsvTableSource}}
> supports it.
> We provide an {{*example.csv*}} whose file encoding is {{{}ISO-8599-1{}}}.
> When reading it with the legacy {{{}CsvTableSource{}}}, it executes correctly:
> {code:java}
> @Test
> public void testLegacyCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> CsvTableSource.Builder builder = CsvTableSource.builder();
> CsvTableSource source =
> builder.path("example.csv")
> .emptyColumnAsNull()
> .lineDelimiter("\n")
> .fieldDelimiter("|")
> .field("name", DataTypes.STRING())
> .build();
> ConnectorCatalogTable catalogTable =
> ConnectorCatalogTable.source(source, true);
> tEnv.getCatalog(tEnv.getCurrentCatalog())
> .ifPresent(
> catalog -> {
> try {
> catalog.createTable(
> new
> ObjectPath(tEnv.getCurrentDatabase(), "example"),
> catalogTable,
> false);
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> });
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
>
> When reading it with the new CSV source (file system source + CSV format), it
> throws the following error:
> {code:java}
> @Test
> public void testNewCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> String ddl =
> "create table example ("
> + "name string"
> + ") with ("
> + "'connector' = 'filesystem',"
> + "'path' = 'example.csv',"
> + "'format' = 'csv',"
> + "'csv.array-element-delimiter' = '\n',"
> + "'csv.field-delimiter' = '|',"
> + "'csv.null-literal' = ''"
> + ")";
> tEnv.executeSql(ddl);
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
> at
>
[jira] [Commented] (FLINK-26760) The new CSV source (file system source + CSV format) does not support reading files whose file encoding is not UTF-8
[
https://issues.apache.org/jira/browse/FLINK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510020#comment-17510020
]
Arvid Heise commented on FLINK-26760:
-
I prioritized to critical since the behavior was already present in 1.13.
In general, I don't see why this should be a bug: you haven't specified the
encoding and as such expecting anything non-utf to work is a stretch.
However, I'd probably treat it as a critical feature request: I have not found
a way to specify the encoding and I guess it would be nice for Excel
compatibility to also support latin1.
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8
>
>
> Key: FLINK-26760
> URL: https://issues.apache.org/jira/browse/FLINK-26760
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0, 1.13.6, 1.14.4
>Reporter: Lijie Wang
>Priority: Critical
> Fix For: 1.15.0
>
> Attachments: example.csv
>
>
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8, but the legacy {{CsvTableSource}}
> supports it.
> We provide an {{*example.csv*}} whose file encoding is {{{}ISO-8599-1{}}}.
> When reading it with the legacy {{{}CsvTableSource{}}}, it executes correctly:
> {code:java}
> @Test
> public void testLegacyCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> CsvTableSource.Builder builder = CsvTableSource.builder();
> CsvTableSource source =
> builder.path("example.csv")
> .emptyColumnAsNull()
> .lineDelimiter("\n")
> .fieldDelimiter("|")
> .field("name", DataTypes.STRING())
> .build();
> ConnectorCatalogTable catalogTable =
> ConnectorCatalogTable.source(source, true);
> tEnv.getCatalog(tEnv.getCurrentCatalog())
> .ifPresent(
> catalog -> {
> try {
> catalog.createTable(
> new
> ObjectPath(tEnv.getCurrentDatabase(), "example"),
> catalogTable,
> false);
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> });
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
>
> When reading it with the new CSV source (file system source + CSV format), it
> throws the following error:
> {code:java}
> @Test
> public void testNewCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> String ddl =
> "create table example ("
> + "name string"
> + ") with ("
> + "'connector' = 'filesystem',"
> + "'path' = 'example.csv',"
> + "'format' = 'csv',"
> + "'csv.array-element-delimiter' = '\n',"
> + "'csv.field-delimiter' = '|',"
> + "'csv.null-literal' = ''"
> + ")";
> tEnv.executeSql(ddl);
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
[jira] [Updated] (FLINK-26760) The new CSV source (file system source + CSV format) does not support reading files whose file encoding is not UTF-8
[
https://issues.apache.org/jira/browse/FLINK-26760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise updated FLINK-26760:
Priority: Critical (was: Blocker)
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8
>
>
> Key: FLINK-26760
> URL: https://issues.apache.org/jira/browse/FLINK-26760
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile)
>Affects Versions: 1.15.0, 1.13.6, 1.14.4
>Reporter: Lijie Wang
>Priority: Critical
> Fix For: 1.15.0
>
> Attachments: example.csv
>
>
> The new CSV source (file system source + CSV format) does not support reading
> files whose file encoding is not UTF-8, but the legacy {{CsvTableSource}}
> supports it.
> We provide an {{*example.csv*}} whose file encoding is {{{}ISO-8599-1{}}}.
> When reading it with the legacy {{{}CsvTableSource{}}}, it executes correctly:
> {code:java}
> @Test
> public void testLegacyCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> CsvTableSource.Builder builder = CsvTableSource.builder();
> CsvTableSource source =
> builder.path("example.csv")
> .emptyColumnAsNull()
> .lineDelimiter("\n")
> .fieldDelimiter("|")
> .field("name", DataTypes.STRING())
> .build();
> ConnectorCatalogTable catalogTable =
> ConnectorCatalogTable.source(source, true);
> tEnv.getCatalog(tEnv.getCurrentCatalog())
> .ifPresent(
> catalog -> {
> try {
> catalog.createTable(
> new
> ObjectPath(tEnv.getCurrentDatabase(), "example"),
> catalogTable,
> false);
> } catch (Exception e) {
> throw new RuntimeException(e);
> }
> });
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
>
> When reading it with the new CSV source (file system source + CSV format), it
> throws the following error:
> {code:java}
> @Test
> public void testNewCsvSource() {
> EnvironmentSettings environmentSettings =
> EnvironmentSettings.inBatchMode();
> TableEnvironment tEnv = TableEnvironment.create(environmentSettings);
> String ddl =
> "create table example ("
> + "name string"
> + ") with ("
> + "'connector' = 'filesystem',"
> + "'path' = 'example.csv',"
> + "'format' = 'csv',"
> + "'csv.array-element-delimiter' = '\n',"
> + "'csv.field-delimiter' = '|',"
> + "'csv.null-literal' = ''"
> + ")";
> tEnv.executeSql(ddl);
> tEnv.executeSql("select count(name) from example").print();
> }
> {code}
> {code:java}
> Caused by: java.lang.RuntimeException: One or more fetchers have encountered
> exception
> at
> org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:225)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:169)
> at
> org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:130)
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:385)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:519)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:804)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:753)
> at
>
[jira] [Commented] (FLINK-26717) Move s3 common utils to flink-core
[ https://issues.apache.org/jira/browse/FLINK-26717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17508739#comment-17508739 ] Arvid Heise commented on FLINK-26717: - Common utilities could be placed in flink-core as long as there are no additional dependencies involved. Make sure to not migrate anything that contains S3 in the name (that would be weird in flink-core). If it's not really S3-specific, please rename. > Move s3 common utils to flink-core > -- > > Key: FLINK-26717 > URL: https://issues.apache.org/jira/browse/FLINK-26717 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Reporter: wujinhu >Assignee: wujinhu >Priority: Major > Labels: pull-request-available > > Utility classes are common and can be used by other cloud storages(oss) in > flink-s3-fs-base module. I will move them to flink-core module like > RefCountedFile. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25256) Savepoints do not work with ExternallyInducedSources
[
https://issues.apache.org/jira/browse/FLINK-25256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-25256:
---
Assignee: Arvid Heise
> Savepoints do not work with ExternallyInducedSources
>
>
> Key: FLINK-25256
> URL: https://issues.apache.org/jira/browse/FLINK-25256
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0, 1.13.3, 1.12.7
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Major
>
> It is not possible to take a proper savepoint with
> {{ExternallyInducedSource}} or {{ExternallyInducedSourceReader}} (both legacy
> and FLIP-27 versions). The problem is that we're hardcoding
> {{CheckpointOptions}} in the {{triggerHook}}.
> The outcome of current state is that operators would try to take checkpoints
> in the checkpoint location whereas the {{CheckpointCoordinator}} will write
> metadata for those states in the savepoint location.
> Moreover the situation gets even weirder (I have not checked it entirely), if
> we have a mixture of {{ExternallyInducedSource(s)}} and regular sources. In
> such a case the location and format at which the state of a particular task
> is persisted depends on the order of barriers arrival. If a barrier from a
> regular source arrives last the task takes a savepoint, on the other hand if
> last barrier is from an externally induced source it will take a checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25256) Savepoints do not work with ExternallyInducedSources
[
https://issues.apache.org/jira/browse/FLINK-25256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504787#comment-17504787
]
Arvid Heise commented on FLINK-25256:
-
I'll have a look. I'd focus on FLIP-27 interfaces.
> Savepoints do not work with ExternallyInducedSources
>
>
> Key: FLINK-25256
> URL: https://issues.apache.org/jira/browse/FLINK-25256
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0, 1.13.3, 1.12.7
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Major
>
> It is not possible to take a proper savepoint with
> {{ExternallyInducedSource}} or {{ExternallyInducedSourceReader}} (both legacy
> and FLIP-27 versions). The problem is that we're hardcoding
> {{CheckpointOptions}} in the {{triggerHook}}.
> The outcome of current state is that operators would try to take checkpoints
> in the checkpoint location whereas the {{CheckpointCoordinator}} will write
> metadata for those states in the savepoint location.
> Moreover the situation gets even weirder (I have not checked it entirely), if
> we have a mixture of {{ExternallyInducedSource(s)}} and regular sources. In
> such a case the location and format at which the state of a particular task
> is persisted depends on the order of barriers arrival. If a barrier from a
> regular source arrives last the task takes a savepoint, on the other hand if
> last barrier is from an externally induced source it will take a checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25256) Savepoints do not work with ExternallyInducedSources
[
https://issues.apache.org/jira/browse/FLINK-25256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise updated FLINK-25256:
Issue Type: Bug (was: Improvement)
> Savepoints do not work with ExternallyInducedSources
>
>
> Key: FLINK-25256
> URL: https://issues.apache.org/jira/browse/FLINK-25256
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0, 1.13.3, 1.12.7
>Reporter: Dawid Wysakowicz
>Priority: Major
>
> It is not possible to take a proper savepoint with
> {{ExternallyInducedSource}} or {{ExternallyInducedSourceReader}} (both legacy
> and FLIP-27 versions). The problem is that we're hardcoding
> {{CheckpointOptions}} in the {{triggerHook}}.
> The outcome of current state is that operators would try to take checkpoints
> in the checkpoint location whereas the {{CheckpointCoordinator}} will write
> metadata for those states in the savepoint location.
> Moreover the situation gets even weirder (I have not checked it entirely), if
> we have a mixture of {{ExternallyInducedSource(s)}} and regular sources. In
> such a case the location and format at which the state of a particular task
> is persisted depends on the order of barriers arrival. If a barrier from a
> regular source arrives last the task takes a savepoint, on the other hand if
> last barrier is from an externally induced source it will take a checkpoint.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (FLINK-21406) Add AvroParquetFileRecordFormat
[
https://issues.apache.org/jira/browse/FLINK-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-21406.
-
Resolution: Fixed
Merged into master as
b4ca35041988cae9b23affe0441595a25506aaba..cdf3d483e191716ab40bd4185c7a674c7a648b6e.
> Add AvroParquetFileRecordFormat
> ---
>
> Key: FLINK-21406
> URL: https://issues.apache.org/jira/browse/FLINK-21406
> Project: Flink
> Issue Type: New Feature
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Reporter: Chesnay Schepler
>Assignee: Jing Ge
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.15.0
>
>
> There is currently no easy way to read avro GenericRecords from parquet via
> the new {{FileSource}}.
> While helping out a user I started writing FileRecordFormat that could do
> that, but it requires some finalization.
> The implementation is similar to our ParquetAvroWriters class, in that we
> just wrap some parquet classes and bridge our FileSystem with their IO
> abstraction.
> The main goal was to have a format that reads data through our FileSystems,
> and not work directly against Hadoop to prevent a ClassLoader leak from the
> S3AFileSystem (threads in a thread pool can keep references to the user
> classloader).
> According to the user it appears to be working, but it will need some
> cleanup, ideally support for specific records, support for checkpointing
> (which should be fairly easy I believe), maybe splitting files (not sure
> whether this works properly with Parquet) and of course tests + documentation.
> {code}
> public class ParquetAvroFileRecordFormat implements
> FileRecordFormat {
> private final transient Schema schema;
> public ParquetAvroFileRecordFormat(Schema schema) {
> this.schema = schema;
> }
> @Override
> public Reader createReader(
> Configuration config, Path filePath, long splitOffset, long
> splitLength)
> throws IOException {
> final FileSystem fs = filePath.getFileSystem();
> final FileStatus status = fs.getFileStatus(filePath);
> final FSDataInputStream in = fs.open(filePath);
> return new MyReader(
> AvroParquetReader.builder(new
> InputFileWrapper(in, status.getLen()))
> .withDataModel(GenericData.get())
> .build());
> }
> @Override
> public Reader restoreReader(
> Configuration config,
> Path filePath,
> long restoredOffset,
> long splitOffset,
> long splitLength) {
> // not called if checkpointing isn't used
> return null;
> }
> @Override
> public boolean isSplittable() {
> // let's not worry about this for now
> return false;
> }
> @Override
> public TypeInformation getProducedType() {
> return new GenericRecordAvroTypeInfo(schema);
> }
> private static class MyReader implements
> FileRecordFormat.Reader {
> private final ParquetReader parquetReader;
> private MyReader(ParquetReader parquetReader) {
> this.parquetReader = parquetReader;
> }
> @Nullable
> @Override
> public GenericRecord read() throws IOException {
> return parquetReader.read();
> }
> @Override
> public void close() throws IOException {
> parquetReader.close();
> }
> }
> private static class InputFileWrapper implements InputFile {
> private final FSDataInputStream inputStream;
> private final long length;
> private InputFileWrapper(FSDataInputStream inputStream, long length) {
> this.inputStream = inputStream;
> this.length = length;
> }
> @Override
> public long getLength() {
> return length;
> }
> @Override
> public SeekableInputStream newStream() {
> return new SeekableInputStreamAdapter(inputStream);
> }
> }
> private static class SeekableInputStreamAdapter extends
> DelegatingSeekableInputStream {
> private final FSDataInputStream inputStream;
> private SeekableInputStreamAdapter(FSDataInputStream inputStream) {
> super(inputStream);
> this.inputStream = inputStream;
> }
> @Override
> public long getPos() throws IOException {
> return inputStream.getPos();
> }
> @Override
> public void seek(long newPos) throws IOException {
> inputStream.seek(newPos);
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25223) ElasticsearchWriterITCase fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461253#comment-17461253
]
Arvid Heise commented on FLINK-25223:
-
[~alexanderpreuss] could you please double-check if we need a backport? The
related docker image was only used in master afaik but there seems to be
similar issues on old release branches. So maybe we also need to apply a
similar fix to that legacy code.
> ElasticsearchWriterITCase fails on AZP
> --
>
> Key: FLINK-25223
> URL: https://issues.apache.org/jira/browse/FLINK-25223
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Alexander Preuss
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> The {{ElasticsearchWriterITCase}} fails on AZP because
> {code}
> 2021-12-08T13:56:59.5449851Z Dec 08 13:56:59 [ERROR]
> org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase Time
> elapsed: 171.046 s <<< ERROR!
> 2021-12-08T13:56:59.5450680Z Dec 08 13:56:59
> org.testcontainers.containers.ContainerLaunchException: Container startup
> failed
> 2021-12-08T13:56:59.5451652Z Dec 08 13:56:59 at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
> 2021-12-08T13:56:59.5452677Z Dec 08 13:56:59 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
> 2021-12-08T13:56:59.5453637Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:242)
> 2021-12-08T13:56:59.5454757Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.access$200(TestcontainersExtension.java:229)
> 2021-12-08T13:56:59.5455946Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$null$1(TestcontainersExtension.java:59)
> 2021-12-08T13:56:59.5457322Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$getOrComputeIfAbsent$4(ExtensionValuesStore.java:86)
> 2021-12-08T13:56:59.5458571Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.computeValue(ExtensionValuesStore.java:223)
> 2021-12-08T13:56:59.5459771Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.get(ExtensionValuesStore.java:211)
> 2021-12-08T13:56:59.5460693Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.evaluate(ExtensionValuesStore.java:191)
> 2021-12-08T13:56:59.5461437Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.access$100(ExtensionValuesStore.java:171)
> 2021-12-08T13:56:59.5462198Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore.getOrComputeIfAbsent(ExtensionValuesStore.java:89)
> 2021-12-08T13:56:59.5467999Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.NamespaceAwareStore.getOrComputeIfAbsent(NamespaceAwareStore.java:53)
> 2021-12-08T13:56:59.5468791Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$beforeAll$2(TestcontainersExtension.java:59)
> 2021-12-08T13:56:59.5469436Z Dec 08 13:56:59 at
> java.util.ArrayList.forEach(ArrayList.java:1259)
> 2021-12-08T13:56:59.5470058Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension.beforeAll(TestcontainersExtension.java:59)
> 2021-12-08T13:56:59.5470846Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$10(ClassBasedTestDescriptor.java:381)
> 2021-12-08T13:56:59.5471641Z Dec 08 13:56:59 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-08T13:56:59.5472403Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:381)
> 2021-12-08T13:56:59.5473190Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:205)
> 2021-12-08T13:56:59.5474001Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:80)
> 2021-12-08T13:56:59.5474759Z Dec 08 13:56:59 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
> 2021-12-08T13:56:59.5475833Z Dec 08 13:56:59 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-08T13:56:59.5476739Z Dec 08 13:56:59 at
>
[jira] [Resolved] (FLINK-25223) ElasticsearchWriterITCase fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-25223.
-
Resolution: Fixed
Reactivated tests and provided memory limit in master as
98a74baeaf45350fb665558cdbed5fb72fb310dd.
> ElasticsearchWriterITCase fails on AZP
> --
>
> Key: FLINK-25223
> URL: https://issues.apache.org/jira/browse/FLINK-25223
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Alexander Preuss
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> The {{ElasticsearchWriterITCase}} fails on AZP because
> {code}
> 2021-12-08T13:56:59.5449851Z Dec 08 13:56:59 [ERROR]
> org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase Time
> elapsed: 171.046 s <<< ERROR!
> 2021-12-08T13:56:59.5450680Z Dec 08 13:56:59
> org.testcontainers.containers.ContainerLaunchException: Container startup
> failed
> 2021-12-08T13:56:59.5451652Z Dec 08 13:56:59 at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
> 2021-12-08T13:56:59.5452677Z Dec 08 13:56:59 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
> 2021-12-08T13:56:59.5453637Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.start(TestcontainersExtension.java:242)
> 2021-12-08T13:56:59.5454757Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension$StoreAdapter.access$200(TestcontainersExtension.java:229)
> 2021-12-08T13:56:59.5455946Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$null$1(TestcontainersExtension.java:59)
> 2021-12-08T13:56:59.5457322Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore.lambda$getOrComputeIfAbsent$4(ExtensionValuesStore.java:86)
> 2021-12-08T13:56:59.5458571Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.computeValue(ExtensionValuesStore.java:223)
> 2021-12-08T13:56:59.5459771Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore$MemoizingSupplier.get(ExtensionValuesStore.java:211)
> 2021-12-08T13:56:59.5460693Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.evaluate(ExtensionValuesStore.java:191)
> 2021-12-08T13:56:59.5461437Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore$StoredValue.access$100(ExtensionValuesStore.java:171)
> 2021-12-08T13:56:59.5462198Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.ExtensionValuesStore.getOrComputeIfAbsent(ExtensionValuesStore.java:89)
> 2021-12-08T13:56:59.5467999Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.execution.NamespaceAwareStore.getOrComputeIfAbsent(NamespaceAwareStore.java:53)
> 2021-12-08T13:56:59.5468791Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension.lambda$beforeAll$2(TestcontainersExtension.java:59)
> 2021-12-08T13:56:59.5469436Z Dec 08 13:56:59 at
> java.util.ArrayList.forEach(ArrayList.java:1259)
> 2021-12-08T13:56:59.5470058Z Dec 08 13:56:59 at
> org.testcontainers.junit.jupiter.TestcontainersExtension.beforeAll(TestcontainersExtension.java:59)
> 2021-12-08T13:56:59.5470846Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.lambda$invokeBeforeAllCallbacks$10(ClassBasedTestDescriptor.java:381)
> 2021-12-08T13:56:59.5471641Z Dec 08 13:56:59 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-08T13:56:59.5472403Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.invokeBeforeAllCallbacks(ClassBasedTestDescriptor.java:381)
> 2021-12-08T13:56:59.5473190Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:205)
> 2021-12-08T13:56:59.5474001Z Dec 08 13:56:59 at
> org.junit.jupiter.engine.descriptor.ClassBasedTestDescriptor.before(ClassBasedTestDescriptor.java:80)
> 2021-12-08T13:56:59.5474759Z Dec 08 13:56:59 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:148)
> 2021-12-08T13:56:59.5475833Z Dec 08 13:56:59 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2021-12-08T13:56:59.5476739Z Dec 08 13:56:59 at
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
> 2021-12-08T13:56:59.5477520Z Dec 08 13:56:59 at
>
[jira] [Commented] (FLINK-25167) Support user-defined `StreamOperatorFactory` in `ConnectedStreams`#transform
[ https://issues.apache.org/jira/browse/FLINK-25167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461250#comment-17461250 ] Arvid Heise commented on FLINK-25167: - I assigned. [~MartijnVisser] could you please keep an eye on it? > Support user-defined `StreamOperatorFactory` in `ConnectedStreams`#transform > > > Key: FLINK-25167 > URL: https://issues.apache.org/jira/browse/FLINK-25167 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Lsw_aka_laplace >Assignee: Lsw_aka_laplace >Priority: Minor > > From my side, it is necessary to set my custom `StreamOperatorFactory` when > I ’m calling `ConnectedStreams`#transform so that I can set up my own > `OperatorCoordinator`. > Well, currently, `ConnectStreams` seems not to give the access, the default > behavior is using `SimpleOperatorFactory`. After checking the code, I think > it is a trivial change to support that. If no one is working on it, I'm > willing to doing that. : ) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25167) Support user-defined `StreamOperatorFactory` in `ConnectedStreams`#transform
[ https://issues.apache.org/jira/browse/FLINK-25167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise reassigned FLINK-25167: --- Assignee: Lsw_aka_laplace > Support user-defined `StreamOperatorFactory` in `ConnectedStreams`#transform > > > Key: FLINK-25167 > URL: https://issues.apache.org/jira/browse/FLINK-25167 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Lsw_aka_laplace >Assignee: Lsw_aka_laplace >Priority: Minor > > From my side, it is necessary to set my custom `StreamOperatorFactory` when > I ’m calling `ConnectedStreams`#transform so that I can set up my own > `OperatorCoordinator`. > Well, currently, `ConnectStreams` seems not to give the access, the default > behavior is using `SimpleOperatorFactory`. After checking the code, I think > it is a trivial change to support that. If no one is working on it, I'm > willing to doing that. : ) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-11576) FLIP-33: Standardize connector metrics
[ https://issues.apache.org/jira/browse/FLINK-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-11576. - Fix Version/s: 1.14.0 Assignee: Arvid Heise Resolution: Fixed > FLIP-33: Standardize connector metrics > -- > > Key: FLINK-11576 > URL: https://issues.apache.org/jira/browse/FLINK-11576 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Reporter: Jiangjie Qin >Assignee: Arvid Heise >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > Fix For: 1.14.0 > > > This is a umbrella ticket for standardize connector metrics. Subtasks will be > created for each individual connector. The FLIP link is following: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-20160) Implement standard metrics in JDBC connectors.
[ https://issues.apache.org/jira/browse/FLINK-20160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-20160: Parent: (was: FLINK-11576) Issue Type: New Feature (was: Sub-task) > Implement standard metrics in JDBC connectors. > -- > > Key: FLINK-20160 > URL: https://issues.apache.org/jira/browse/FLINK-20160 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC, Table SQL / Ecosystem >Reporter: Lijie Wang >Priority: Major > Labels: auto-unassigned > > Implement the standard metrics proposed in > [FLIP-33|https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics] > in JDBC connectors. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25330) Flink SQL doesn't retract all versions of Hbase data
[ https://issues.apache.org/jira/browse/FLINK-25330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise reassigned FLINK-25330: --- Assignee: Jing Ge > Flink SQL doesn't retract all versions of Hbase data > > > Key: FLINK-25330 > URL: https://issues.apache.org/jira/browse/FLINK-25330 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase >Affects Versions: 1.14.0 >Reporter: Bruce Wong >Assignee: Jing Ge >Priority: Critical > Labels: pull-request-available > Attachments: image-2021-12-15-20-05-18-236.png > > > h2. Background > When we use CDC to synchronize mysql data to HBase, we find that HBase > deletes only the last version of the specified rowkey when deleting mysql > data. The data of the old version still exists. You end up using the wrong > data. And I think its a bug of HBase connector. > The following figure shows Hbase data changes before and after mysql data is > deleted. > !image-2021-12-15-20-05-18-236.png|width=910,height=669! > > h2. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25293) Option to let fail if KafkaSource keeps failing to commit offset
[
https://issues.apache.org/jira/browse/FLINK-25293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459844#comment-17459844
]
Arvid Heise commented on FLINK-25293:
-
[~renqs], could you PTAL?
> Option to let fail if KafkaSource keeps failing to commit offset
>
>
> Key: FLINK-25293
> URL: https://issues.apache.org/jira/browse/FLINK-25293
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Affects Versions: 1.14.0
> Environment: Flink 1.14.0
>Reporter: rerorero
>Priority: Major
>
> Is it possible to let KafkaSource fail if it keeps failing to commit offset?
>
> I faced an issue where KafkaSource keeps failing and never recover, while
> it's logging like these logs:
> {code:java}
> 2021-12-08 22:18:34,155 INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator [] -
> [Consumer clientId=dbz-cg-1, groupId=dbz-cg] Group coordinator
> b4-pkc-x.asia-northeast1.gcp.confluent.cloud:9092 (id: 2147483643 rack:
> null) is unavailable or invalid due to cause: null.isDisconnected: true.
> Rediscovery will be attempted.
> 2021-12-08 22:18:34,157 WARN
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReader [] - Failed
> to commit consumer offsets for checkpoint 13 {code}
> This is happening not just once, but a couple of times a week (it happens
> when the Kafka broker performs rolling restart). It can be recovered by
> restarting the Flink Job.
> I found other people reporting the similar thing:
> [https://lists.apache.org/thread/8l4f2yb4qwysdn1cj1wjk99tfb79kgs2]. This
> could possibly be a problem with the Kafka client, and of course, the problem
> should be fixed on Kafka side if so.
> However, Flink Kafka connector doesn't provide an automatic way to save this
> situation. KafkaSource keeps retrying forever when a retriable error occurs,
> even if it is not retriable actually:
> [https://github.com/apache/flink/blob/afb29d92c4e76ec6a453459c3d8a08304efec549/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaSourceReader.java#L144-L148]
> Since it sends metrics of the number of times a commit fails, it could be
> automated by monitoring it and restarting the job, but that would mean we
> need to have a new process to be managed.
> Does it make sense to have KafkaSource have the option like, let the source
> task fail if it keeps failing to commit an offset more than X times?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24489) The size of entryCache & eventsBufferCache in the SharedBuffer should be defined with a threshold to limit the number of the elements in the cache
[ https://issues.apache.org/jira/browse/FLINK-24489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-24489: Release Note: CEP now allows users to configure the cache size used while processing records. > The size of entryCache & eventsBufferCache in the SharedBuffer should be > defined with a threshold to limit the number of the elements in the cache > -- > > Key: FLINK-24489 > URL: https://issues.apache.org/jira/browse/FLINK-24489 > Project: Flink > Issue Type: Improvement > Components: Library / CEP >Affects Versions: 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0 >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > Attachments: details of the big map object.png, incoming-reference-to > the-big-map-obj.png, 截屏2021-10-13 20.28.17.png > > > source code : > [here|https://github.com/apache/flink/blob/c3cb886ee73b5fee23b2bccff0f5e4d45a30b3a1/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBuffer.java#L79] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-24489) The size of entryCache & eventsBufferCache in the SharedBuffer should be defined with a threshold to limit the number of the elements in the cache
[ https://issues.apache.org/jira/browse/FLINK-24489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-24489. - Fix Version/s: 1.15.0 Assignee: RocMarshal Resolution: Fixed > The size of entryCache & eventsBufferCache in the SharedBuffer should be > defined with a threshold to limit the number of the elements in the cache > -- > > Key: FLINK-24489 > URL: https://issues.apache.org/jira/browse/FLINK-24489 > Project: Flink > Issue Type: Improvement > Components: Library / CEP >Affects Versions: 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0 >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > Attachments: details of the big map object.png, incoming-reference-to > the-big-map-obj.png, 截屏2021-10-13 20.28.17.png > > > source code : > [here|https://github.com/apache/flink/blob/c3cb886ee73b5fee23b2bccff0f5e4d45a30b3a1/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBuffer.java#L79] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25263) KafkaSourceE2ECase is broken
[
https://issues.apache.org/jira/browse/FLINK-25263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458294#comment-17458294
]
Arvid Heise commented on FLINK-25263:
-
[~renqs] could you PTAL?
> KafkaSourceE2ECase is broken
>
>
> Key: FLINK-25263
> URL: https://issues.apache.org/jira/browse/FLINK-25263
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.15.0
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27963=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a
> {code}
> ec 10 16:16:05 java.lang.RuntimeException: Failed to delete path
> "/tmp/flink-checkpoint5687460701536838268"
> Dec 10 16:16:05 at
> org.apache.flink.tests.util.flink.container.FlinkContainersBuilder.lambda$deleteTemporaryPaths$1(FlinkContainersBuilder.java:315)
> Dec 10 16:16:05 at java.util.ArrayList.forEach(ArrayList.java:1259)
> Dec 10 16:16:05 at
> org.apache.flink.tests.util.flink.container.FlinkContainersBuilder.deleteTemporaryPaths(FlinkContainersBuilder.java:309)
> Dec 10 16:16:05 at
> org.apache.flink.tests.util.flink.container.FlinkContainersBuilder.lambda$build$0(FlinkContainersBuilder.java:203)
> Dec 10 16:16:05 at
> org.apache.flink.tests.util.flink.container.FlinkContainers.stop(FlinkContainers.java:167)
> Dec 10 16:16:05 at
> org.apache.flink.tests.util.flink.FlinkContainerTestEnvironment.tearDown(FlinkContainerTestEnvironment.java:74)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (FLINK-9050) Expose operator IO counter metrics
[ https://issues.apache.org/jira/browse/FLINK-9050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-9050. Fix Version/s: 1.14.0 Resolution: Duplicate > Expose operator IO counter metrics > -- > > Key: FLINK-9050 > URL: https://issues.apache.org/jira/browse/FLINK-9050 > Project: Flink > Issue Type: New Feature > Components: API / DataSet, API / DataStream, Runtime / Metrics >Reporter: Chesnay Schepler >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > Attachments: image-2018-11-06-15-40-28-205.png > > > To properly expose the number of records read by sources / emitted by sink we > have to expose the operator counter metrics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-19743) Add Source metrics definitions
[
https://issues.apache.org/jira/browse/FLINK-19743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-19743.
-
Fix Version/s: 1.14.0
(was: 1.12.6)
(was: 1.15.0)
Resolution: Duplicate
> Add Source metrics definitions
> --
>
> Key: FLINK-19743
> URL: https://issues.apache.org/jira/browse/FLINK-19743
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Common
>Affects Versions: 1.11.2
>Reporter: Jiangjie Qin
>Priority: Major
> Labels: auto-unassigned, pull-request-available
> Fix For: 1.14.0
>
>
> Add the metrics defined in
> [FLIP-33|https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics]
> to \{{OperatorMetricsGroup}} and {{SourceReaderContext}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-11682) Add abstract source and sink metric classes.
[ https://issues.apache.org/jira/browse/FLINK-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-11682: Fix Version/s: 1.14.0 > Add abstract source and sink metric classes. > > > Key: FLINK-11682 > URL: https://issues.apache.org/jira/browse/FLINK-11682 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Jiangjie Qin >Priority: Major > Labels: auto-unassigned > Fix For: 1.14.0 > > > This is ticket will introduce a new flink-connectors-common module to > host the common classes shared by the connectors. In this case, standard > source and sink metrics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-11682) Add abstract source and sink metric classes.
[ https://issues.apache.org/jira/browse/FLINK-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-11682. - Resolution: Duplicate > Add abstract source and sink metric classes. > > > Key: FLINK-11682 > URL: https://issues.apache.org/jira/browse/FLINK-11682 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Common >Reporter: Jiangjie Qin >Priority: Major > Labels: auto-unassigned > > This is ticket will introduce a new flink-connectors-common module to > host the common classes shared by the connectors. In this case, standard > source and sink metrics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-11683) Implement standard metrics in Kafka connectors.
[ https://issues.apache.org/jira/browse/FLINK-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-11683. - Fix Version/s: 1.14.0 Resolution: Fixed > Implement standard metrics in Kafka connectors. > --- > > Key: FLINK-11683 > URL: https://issues.apache.org/jira/browse/FLINK-11683 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Reporter: Jiangjie Qin >Priority: Major > Labels: auto-unassigned > Fix For: 1.14.0 > > > Implement the standard metrics in Kafka connectors. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24077) HBaseConnectorITCase.testTableSink
[
https://issues.apache.org/jira/browse/FLINK-24077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17457648#comment-17457648
]
Arvid Heise commented on FLINK-24077:
-
Merged
4e962d9a7980dc143a2e4ace1211887314648d7d..cc78095923508299437eab0b39c78576b6a69c07
into 1.14,
2efb0df1fb2d2a7bf081ae205c78698f3244c625..9d9842fb610a95d9f31ad432e2546d8ce61b106e
into 1.13,
8e3b291fb5b04b8e774d603a517f2dfadc6095ed..b6ae90384b47a6274e2f746c76aa31387c2d5ee8
into 1.12.
> HBaseConnectorITCase.testTableSink
> --
>
> Key: FLINK-24077
> URL: https://issues.apache.org/jira/browse/FLINK-24077
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Xintong Song
>Assignee: Jing Ge
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23160=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d=12962
> {code}
> Aug 31 05:10:58 Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 73.758 sec <<< FAILURE! - in
> org.apache.flink.connector.hbase2.HBaseConnectorITCase
> Aug 31 05:10:58
> testTableSink(org.apache.flink.connector.hbase2.HBaseConnectorITCase) Time
> elapsed: 6.516 sec <<< FAILURE!
> Aug 31 05:10:58 java.lang.AssertionError:
> Aug 31 05:10:58 Different elements in arrays: expected 8 elements and
> received 3
> Aug 31 05:10:58 expected: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3], +I[4, 40, null, 400, 4.04, true, Welt-4], +I[5, 50, Hello-5,
> 500, 5.05, false, Welt-5], +I[6, 60, Hello-6, 600, 6.06, true, Welt-6], +I[7,
> 70, Hello-7, 700, 7.07, false, Welt-7], +I[8, 80, null, 800, 8.08, true,
> Welt-8]]
> Aug 31 05:10:58 received: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3]] expected:<8> but was:<3>
> Aug 31 05:10:58 at org.junit.Assert.fail(Assert.java:89)
> Aug 31 05:10:58 at org.junit.Assert.failNotEquals(Assert.java:835)
> Aug 31 05:10:58 at org.junit.Assert.assertEquals(Assert.java:647)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResult(TestBaseUtils.java:395)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResultAsText(TestBaseUtils.java:347)
> Aug 31 05:10:58 at
> org.apache.flink.connector.hbase2.HBaseConnectorITCase.testTableSink(HBaseConnectorITCase.java:284)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (FLINK-24077) HBaseConnectorITCase.testTableSink
[
https://issues.apache.org/jira/browse/FLINK-24077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-24077.
-
Fix Version/s: 1.12.6
1.13.4
Resolution: Fixed
> HBaseConnectorITCase.testTableSink
> --
>
> Key: FLINK-24077
> URL: https://issues.apache.org/jira/browse/FLINK-24077
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Xintong Song
>Assignee: Jing Ge
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.6, 1.15.0, 1.14.1, 1.13.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23160=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d=12962
> {code}
> Aug 31 05:10:58 Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 73.758 sec <<< FAILURE! - in
> org.apache.flink.connector.hbase2.HBaseConnectorITCase
> Aug 31 05:10:58
> testTableSink(org.apache.flink.connector.hbase2.HBaseConnectorITCase) Time
> elapsed: 6.516 sec <<< FAILURE!
> Aug 31 05:10:58 java.lang.AssertionError:
> Aug 31 05:10:58 Different elements in arrays: expected 8 elements and
> received 3
> Aug 31 05:10:58 expected: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3], +I[4, 40, null, 400, 4.04, true, Welt-4], +I[5, 50, Hello-5,
> 500, 5.05, false, Welt-5], +I[6, 60, Hello-6, 600, 6.06, true, Welt-6], +I[7,
> 70, Hello-7, 700, 7.07, false, Welt-7], +I[8, 80, null, 800, 8.08, true,
> Welt-8]]
> Aug 31 05:10:58 received: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3]] expected:<8> but was:<3>
> Aug 31 05:10:58 at org.junit.Assert.fail(Assert.java:89)
> Aug 31 05:10:58 at org.junit.Assert.failNotEquals(Assert.java:835)
> Aug 31 05:10:58 at org.junit.Assert.assertEquals(Assert.java:647)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResult(TestBaseUtils.java:395)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResultAsText(TestBaseUtils.java:347)
> Aug 31 05:10:58 at
> org.apache.flink.connector.hbase2.HBaseConnectorITCase.testTableSink(HBaseConnectorITCase.java:284)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24077) HBaseConnectorITCase.testTableSink
[
https://issues.apache.org/jira/browse/FLINK-24077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456285#comment-17456285
]
Arvid Heise commented on FLINK-24077:
-
Merged
7976be0f8675a8753a5bb7e7a44dda6b4a347247..fca04c3aaf6346d61cf9fe022a7ac77ab4d66c91
into master. [~jingge], could you please also create backports to 1.14, 1.13,
and if it's an easy cherry-pick also to 1.12?
> HBaseConnectorITCase.testTableSink
> --
>
> Key: FLINK-24077
> URL: https://issues.apache.org/jira/browse/FLINK-24077
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Xintong Song
>Assignee: Jing Ge
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23160=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d=12962
> {code}
> Aug 31 05:10:58 Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 73.758 sec <<< FAILURE! - in
> org.apache.flink.connector.hbase2.HBaseConnectorITCase
> Aug 31 05:10:58
> testTableSink(org.apache.flink.connector.hbase2.HBaseConnectorITCase) Time
> elapsed: 6.516 sec <<< FAILURE!
> Aug 31 05:10:58 java.lang.AssertionError:
> Aug 31 05:10:58 Different elements in arrays: expected 8 elements and
> received 3
> Aug 31 05:10:58 expected: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3], +I[4, 40, null, 400, 4.04, true, Welt-4], +I[5, 50, Hello-5,
> 500, 5.05, false, Welt-5], +I[6, 60, Hello-6, 600, 6.06, true, Welt-6], +I[7,
> 70, Hello-7, 700, 7.07, false, Welt-7], +I[8, 80, null, 800, 8.08, true,
> Welt-8]]
> Aug 31 05:10:58 received: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3]] expected:<8> but was:<3>
> Aug 31 05:10:58 at org.junit.Assert.fail(Assert.java:89)
> Aug 31 05:10:58 at org.junit.Assert.failNotEquals(Assert.java:835)
> Aug 31 05:10:58 at org.junit.Assert.assertEquals(Assert.java:647)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResult(TestBaseUtils.java:395)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResultAsText(TestBaseUtils.java:347)
> Aug 31 05:10:58 at
> org.apache.flink.connector.hbase2.HBaseConnectorITCase.testTableSink(HBaseConnectorITCase.java:284)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24489) The size of entryCache & eventsBufferCache in the SharedBuffer should be defined with a threshold to limit the number of the elements in the cache
[ https://issues.apache.org/jira/browse/FLINK-24489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456213#comment-17456213 ] Arvid Heise commented on FLINK-24489: - Merged into master as 4d142b3a417c57e4ed8af4c886fae68d927da067. I'd say that this is an improvement (and not just a simple bug fix), so we shouldn't backport. WDYT? > The size of entryCache & eventsBufferCache in the SharedBuffer should be > defined with a threshold to limit the number of the elements in the cache > -- > > Key: FLINK-24489 > URL: https://issues.apache.org/jira/browse/FLINK-24489 > Project: Flink > Issue Type: Improvement > Components: Library / CEP >Affects Versions: 1.10.0, 1.11.0, 1.12.0, 1.13.0, 1.14.0 >Reporter: Yuepeng Pan >Priority: Major > Labels: pull-request-available > Attachments: details of the big map object.png, incoming-reference-to > the-big-map-obj.png, 截屏2021-10-13 20.28.17.png > > > source code : > [here|https://github.com/apache/flink/blob/c3cb886ee73b5fee23b2bccff0f5e4d45a30b3a1/flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBuffer.java#L79] > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24859) Document new File formats
[ https://issues.apache.org/jira/browse/FLINK-24859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17455336#comment-17455336 ] Arvid Heise commented on FLINK-24859: - Renaming was breaking API in 1.14 and master. 1.14 partially reverted in 94f269d2736ea8ad3860f8dde534f7fb075f0197. Master amended in c18e5b91b6b578bd935b4006447da92acb2645dd. > Document new File formats > - > > Key: FLINK-24859 > URL: https://issues.apache.org/jira/browse/FLINK-24859 > Project: Flink > Issue Type: Technical Debt > Components: Documentation >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.14.1 > > > The project recently introduced new formats: _BulkFormat_ and _StreamFormat_ > interfaces. > There are already implementations of these formats: hive, parquet, orc and > textLine formats that need to be documented. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-20928) KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270 » Timeout
[
https://issues.apache.org/jira/browse/FLINK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17455093#comment-17455093
]
Arvid Heise commented on FLINK-20928:
-
I think the proper solution is to migrate the tests to docker instead. If we
are missing the test utils, we can also consider using
https://mguenther.github.io/kafka-junit/ for tests.
> KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270
> » Timeout
> ---
>
> Key: FLINK-20928
> URL: https://issues.apache.org/jira/browse/FLINK-20928
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0, 1.14.0, 1.15.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11861=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 93.992 s <<< FAILURE! - in
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest
> [ERROR]
> testOffsetCommitOnCheckpointComplete(org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest)
> Time elapsed: 60.086 s <<< ERROR!
> java.util.concurrent.TimeoutException: The offset commit did not finish
> before timeout.
> at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:210)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.pollUntil(KafkaSourceReaderTest.java:270)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete(KafkaSourceReaderTest.java:189)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-20928) KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270 » Timeout
[
https://issues.apache.org/jira/browse/FLINK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454748#comment-17454748
]
Arvid Heise commented on FLINK-20928:
-
[~renqs], is this test still using the proxy setup of the legacy consumer and
producer? Maybe we should just harden the test with retry rules?
> KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270
> » Timeout
> ---
>
> Key: FLINK-20928
> URL: https://issues.apache.org/jira/browse/FLINK-20928
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0, 1.14.0, 1.15.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11861=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 93.992 s <<< FAILURE! - in
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest
> [ERROR]
> testOffsetCommitOnCheckpointComplete(org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest)
> Time elapsed: 60.086 s <<< ERROR!
> java.util.concurrent.TimeoutException: The offset commit did not finish
> before timeout.
> at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:210)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.pollUntil(KafkaSourceReaderTest.java:270)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete(KafkaSourceReaderTest.java:189)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24077) HBaseConnectorITCase.testTableSink
[
https://issues.apache.org/jira/browse/FLINK-24077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454670#comment-17454670
]
Arvid Heise commented on FLINK-24077:
-
Note that the cluster could also be non-static. We could of course, also change
that through the rule.
> HBaseConnectorITCase.testTableSink
> --
>
> Key: FLINK-24077
> URL: https://issues.apache.org/jira/browse/FLINK-24077
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Xintong Song
>Assignee: Jing Ge
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23160=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=ed165f3f-d0f6-524b-5279-86f8ee7d0e2d=12962
> {code}
> Aug 31 05:10:58 Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time
> elapsed: 73.758 sec <<< FAILURE! - in
> org.apache.flink.connector.hbase2.HBaseConnectorITCase
> Aug 31 05:10:58
> testTableSink(org.apache.flink.connector.hbase2.HBaseConnectorITCase) Time
> elapsed: 6.516 sec <<< FAILURE!
> Aug 31 05:10:58 java.lang.AssertionError:
> Aug 31 05:10:58 Different elements in arrays: expected 8 elements and
> received 3
> Aug 31 05:10:58 expected: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3], +I[4, 40, null, 400, 4.04, true, Welt-4], +I[5, 50, Hello-5,
> 500, 5.05, false, Welt-5], +I[6, 60, Hello-6, 600, 6.06, true, Welt-6], +I[7,
> 70, Hello-7, 700, 7.07, false, Welt-7], +I[8, 80, null, 800, 8.08, true,
> Welt-8]]
> Aug 31 05:10:58 received: [+I[1, 10, Hello-1, 100, 1.01, false, Welt-1],
> +I[2, 20, Hello-2, 200, 2.02, true, Welt-2], +I[3, 30, Hello-3, 300, 3.03,
> false, Welt-3]] expected:<8> but was:<3>
> Aug 31 05:10:58 at org.junit.Assert.fail(Assert.java:89)
> Aug 31 05:10:58 at org.junit.Assert.failNotEquals(Assert.java:835)
> Aug 31 05:10:58 at org.junit.Assert.assertEquals(Assert.java:647)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResult(TestBaseUtils.java:395)
> Aug 31 05:10:58 at
> org.apache.flink.test.util.TestBaseUtils.compareResultAsText(TestBaseUtils.java:347)
> Aug 31 05:10:58 at
> org.apache.flink.connector.hbase2.HBaseConnectorITCase.testTableSink(HBaseConnectorITCase.java:284)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25167) Support user-defined `StreamOperatorFactory` in `ConnectedStreams`#transform
[ https://issues.apache.org/jira/browse/FLINK-25167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454500#comment-17454500 ] Arvid Heise commented on FLINK-25167: - Let me include [~pnowojski]. I think he wanted to overhaul `ProcessFunction` to remove the need to expose `StreamOperatorFactory`. But I'm also open for your suggestion for the meantime. A quick reminder that `OperatorCoordinator` is an internal concept and may break with any release. Also note that state management of the coordinator is currently not fleshed out and only works for exactly once use cases at sources. > Support user-defined `StreamOperatorFactory` in `ConnectedStreams`#transform > > > Key: FLINK-25167 > URL: https://issues.apache.org/jira/browse/FLINK-25167 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: Lsw_aka_laplace >Priority: Minor > > From my side, it is necessary to set my custom `StreamOperatorFactory` when > I ’m calling `ConnectedStreams`#transform so that I can set up my own > `OperatorCoordinator`. > Well, currently, `ConnectStreams` seems not to give the access, the default > behavior is using `SimpleOperatorFactory`. After checking the code, I think > it is a trivial change to support that. If no one is working on it, I'm > willing to doing that. : ) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-25167) Support user-defined `StreamOperatorFactory` in `ConnectedStreams`#transform
[
https://issues.apache.org/jira/browse/FLINK-25167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454500#comment-17454500
]
Arvid Heise edited comment on FLINK-25167 at 12/7/21, 9:23 AM:
---
Let me include [~pnowojski]. I think he wanted to overhaul {{ProcessFunction}}
to remove the need to expose {{StreamOperatorFactory}}. But I'm also open for
your suggestion for the meantime.
A quick reminder that {{OperatorCoordinator}} is an internal concept and may
break with any release. Also note that state management of the coordinator is
currently not fleshed out and only works for exactly once use cases at sources.
was (Author: arvid):
Let me include [~pnowojski]. I think he wanted to overhaul `ProcessFunction` to
remove the need to expose `StreamOperatorFactory`. But I'm also open for your
suggestion for the meantime.
A quick reminder that `OperatorCoordinator` is an internal concept and may
break with any release. Also note that state management of the coordinator is
currently not fleshed out and only works for exactly once use cases at sources.
> Support user-defined `StreamOperatorFactory` in `ConnectedStreams`#transform
>
>
> Key: FLINK-25167
> URL: https://issues.apache.org/jira/browse/FLINK-25167
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Lsw_aka_laplace
>Priority: Minor
>
> From my side, it is necessary to set my custom `StreamOperatorFactory` when
> I ’m calling `ConnectedStreams`#transform so that I can set up my own
> `OperatorCoordinator`.
> Well, currently, `ConnectStreams` seems not to give the access, the default
> behavior is using `SimpleOperatorFactory`. After checking the code, I think
> it is a trivial change to support that. If no one is working on it, I'm
> willing to doing that. : )
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24859) Document new File formats
[ https://issues.apache.org/jira/browse/FLINK-24859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise updated FLINK-24859: Fix Version/s: 1.14.1 > Document new File formats > - > > Key: FLINK-24859 > URL: https://issues.apache.org/jira/browse/FLINK-24859 > Project: Flink > Issue Type: Technical Debt > Components: Documentation >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.14.1 > > > The project recently introduced new formats: _BulkFormat_ and _StreamFormat_ > interfaces. > There are already implementations of these formats: hive, parquet, orc and > textLine formats that need to be documented. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24859) Document new File formats
[ https://issues.apache.org/jira/browse/FLINK-24859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454190#comment-17454190 ] Arvid Heise commented on FLINK-24859: - Merged into 1.14 as 3b9aaca3c96a79c241568f4493f6b308be1eb4ce..a87ca1e04889a5c655d58fea1df54f9925c10409. > Document new File formats > - > > Key: FLINK-24859 > URL: https://issues.apache.org/jira/browse/FLINK-24859 > Project: Flink > Issue Type: Technical Debt > Components: Documentation >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > The project recently introduced new formats: _BulkFormat_ and _StreamFormat_ > interfaces. > There are already implementations of these formats: hive, parquet, orc and > textLine formats that need to be documented. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24821) FlinkKafkaProducerMigrationOperatorTest.testRestoreProducer fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-24821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454144#comment-17454144
]
Arvid Heise commented on FLINK-24821:
-
https://github.com/apache/flink/pull/16108 is currently addressing the issue by
introducing file locks.
> FlinkKafkaProducerMigrationOperatorTest.testRestoreProducer fails on AZP
>
>
> Key: FLINK-24821
> URL: https://issues.apache.org/jira/browse/FLINK-24821
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> The test {{FlinkKafkaProducerMigrationOperatorTest.testRestoreProducer}}
> fails on AZP with
> {code}
> Nov 07 23:02:14 [ERROR] testRestoreProducer[Migration Savepoint: 1.8] Time
> elapsed: 2.008 s <<< ERROR!
> Nov 07 23:02:14 java.net.BindException: Address already in use
> Nov 07 23:02:14 at sun.nio.ch.Net.bind0(Native Method)
> Nov 07 23:02:14 at sun.nio.ch.Net.bind(Net.java:461)
> Nov 07 23:02:14 at sun.nio.ch.Net.bind(Net.java:453)
> Nov 07 23:02:14 at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
> Nov 07 23:02:14 at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
> Nov 07 23:02:14 at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:78)
> Nov 07 23:02:14 at
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:90)
> Nov 07 23:02:14 at
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:120)
> Nov 07 23:02:14 at
> org.apache.curator.test.TestingZooKeeperMain.runFromConfig(TestingZooKeeperMain.java:93)
> Nov 07 23:02:14 at
> org.apache.curator.test.TestingZooKeeperServer$1.run(TestingZooKeeperServer.java:148)
> Nov 07 23:02:14 at java.lang.Thread.run(Thread.java:748)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26096=logs=72d4811f-9f0d-5fd0-014a-0bc26b72b642=e424005a-b16e-540f-196d-da062cc19bdf=7302
> It looks that there is a race condition for avaiable ports.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24686) Make doc clear on AsyncFunction::timeout() overriding
[
https://issues.apache.org/jira/browse/FLINK-24686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454123#comment-17454123
]
Arvid Heise commented on FLINK-24686:
-
Merged into master as
7166ef1c08e3e3831a2539c1311e3caa150b97ba..0f44d2d80180f41658cc21f04d68e517c526e843.
> Make doc clear on AsyncFunction::timeout() overriding
> -
>
> Key: FLINK-24686
> URL: https://issues.apache.org/jira/browse/FLINK-24686
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 1.14.0, 1.13.3
>Reporter: Jun Qin
>Assignee: Jun Qin
>Priority: Major
> Labels: pull-request-available, stale-assigned
>
> Sometimes, a user overrides {{AsyncFunction::timeout()}} with an empty method
> or with only logging code. This causes the timeout does not signal back to
> the framework and job stuck especially when using {{orderedWait()}}. Opening
> this Jira to make the doc clear on this.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24626) Flink JDBC Sink may lose data in left join
[
https://issues.apache.org/jira/browse/FLINK-24626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise updated FLINK-24626:
Labels: correctness (was: )
> Flink JDBC Sink may lose data in left join
> --
>
> Key: FLINK-24626
> URL: https://issues.apache.org/jira/browse/FLINK-24626
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Reporter: Kenyore
>Priority: Major
> Labels: correctness
>
> The JDBC sink will lose some data while using
> TableBufferReducedStatementExecutor and left join.
> Here are a snippet of executeBatch.
> {code:java}
> @Override
> public void executeBatch() throws SQLException {
> for (Map.Entry> entry :
> reduceBuffer.entrySet()) {
> if (entry.getValue().f0) {
> upsertExecutor.addToBatch(entry.getValue().f1);
> } else {
> // delete by key
> deleteExecutor.addToBatch(entry.getKey());
> }
> }
> upsertExecutor.executeBatch();
> deleteExecutor.executeBatch();
> reduceBuffer.clear();
> }
> {code}
> Left join will generate a DETETE row before upsert row and the executeBatch
> will excute them in a wrong order.Whitch may causes data lose.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (FLINK-21407) Clarify which sources and APIs support which formats
[ https://issues.apache.org/jira/browse/FLINK-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-21407. - Fix Version/s: 1.14.1 Resolution: Fixed > Clarify which sources and APIs support which formats > > > Key: FLINK-21407 > URL: https://issues.apache.org/jira/browse/FLINK-21407 > Project: Flink > Issue Type: New Feature > Components: API / DataSet, API / DataStream, Documentation >Reporter: Chesnay Schepler >Assignee: Etienne Chauchot >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Fix For: 1.15.0, 1.14.1, 1.13.4 > > > The DataSet connectors documentation is essentially an empty desert amounting > to "you can read files". > The DataStream connectors documentation do not mention formats like > avro/parquet anywhere, nor the possibility to read from filesystems (only the > sinks are documented). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-21407) Clarify which sources and APIs support which formats
[ https://issues.apache.org/jira/browse/FLINK-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452592#comment-17452592 ] Arvid Heise commented on FLINK-21407: - Merged into 1.14 as db8be1d355ebac1cfe552c10aaa33c661dc00d7b..a4953120c7c93d9e2e8784829f523808859325d7. > Clarify which sources and APIs support which formats > > > Key: FLINK-21407 > URL: https://issues.apache.org/jira/browse/FLINK-21407 > Project: Flink > Issue Type: New Feature > Components: API / DataSet, API / DataStream, Documentation >Reporter: Chesnay Schepler >Assignee: Etienne Chauchot >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Fix For: 1.15.0, 1.13.4 > > > The DataSet connectors documentation is essentially an empty desert amounting > to "you can read files". > The DataStream connectors documentation do not mention formats like > avro/parquet anywhere, nor the possibility to read from filesystems (only the > sinks are documented). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-24921) FileSourceSplit should not be visible in the user API in ParquetColumnarRowInputFormat
[ https://issues.apache.org/jira/browse/FLINK-24921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-24921. - Resolution: Abandoned > FileSourceSplit should not be visible in the user API in > ParquetColumnarRowInputFormat > -- > > Key: FLINK-24921 > URL: https://issues.apache.org/jira/browse/FLINK-24921 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > _FileSourceSplit_ is an internal class that should not be visible in the user > API like > [here|https://github.com/apache/flink/blob/6f2d8fe3007464343c5312e27612be448b415148/flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormatTest.java#L235]. > The fact that _FileSourceSplit_ surfaces in the API also influences the user > to do a raw use of the parametrized class like > [here|https://github.com/apache/flink/blob/6f2d8fe3007464343c5312e27612be448b415148/flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormatTest.java#L407] > It could be better to make parquet format a not parametrized class as it is > done for hive connector > _class_ HiveBulkFormatAdapter > _implements BulkFormat_ > rather than > _class ParquetColumnarRowInputFormat_ > _extends ParquetVectorizedInputFormat_ > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-24392) Upgrade presto s3 fs implementation to Trino >= 348
[ https://issues.apache.org/jira/browse/FLINK-24392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-24392. - Resolution: Later > Upgrade presto s3 fs implementation to Trino >= 348 > --- > > Key: FLINK-24392 > URL: https://issues.apache.org/jira/browse/FLINK-24392 > Project: Flink > Issue Type: Improvement > Components: FileSystems >Affects Versions: 1.14.0 >Reporter: Robert Metzger >Priority: Major > Fix For: 1.15.0 > > > The Presto s3 filesystem implementation currently shipped with Flink doesn't > support streaming uploads. All data needs to be materialized to a single file > on disk, before it can be uploaded. > This can lead to situations where TaskManagers are running out of disk when > creating a savepoint. > The Hadoop filesystem implementation supports streaming uploads (by using > multipart uploads of smaller (say 100mb) files locally), but it does more API > calls, leading to other issues. > Trino version >= 348 supports streaming uploads. > During experiments, I also noticed that the current presto s3 fs > implementation seems to allocate a lot of memory outside the heap (when > shipping large data, for example when creating a savepoint). On a K8s pod > with a memory limit of 4000Mi, I was not able to run Flink with a > "taskmanager.memory.flink.size" above 3000m. This means that an additional > 1gb of memory needs to be allocated just for the peaks in memory allocation > when presto s3 is taking a savepoint. It would be good to confirm this > behavior, and then either adjust the default memory configuration or the > documentation. > As part of this upgrade, we also need to make sure that the new presto / > Trino version is not doing substantially more S3 API calls than the current > version. After switching away from the presto s3 to hadoop s3, I noticed that > disposing an old checkpoint (~100gb) can take up to 15 minutes. The upgraded > presto s3 fs should still be able to quickly dispose state. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24392) Upgrade presto s3 fs implementation to Trino >= 348
[ https://issues.apache.org/jira/browse/FLINK-24392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451122#comment-17451122 ] Arvid Heise commented on FLINK-24392: - This is blocked on dropping Java 8 on which the community has yet to decide. > Upgrade presto s3 fs implementation to Trino >= 348 > --- > > Key: FLINK-24392 > URL: https://issues.apache.org/jira/browse/FLINK-24392 > Project: Flink > Issue Type: Improvement > Components: FileSystems >Affects Versions: 1.14.0 >Reporter: Robert Metzger >Priority: Major > Fix For: 1.15.0 > > > The Presto s3 filesystem implementation currently shipped with Flink doesn't > support streaming uploads. All data needs to be materialized to a single file > on disk, before it can be uploaded. > This can lead to situations where TaskManagers are running out of disk when > creating a savepoint. > The Hadoop filesystem implementation supports streaming uploads (by using > multipart uploads of smaller (say 100mb) files locally), but it does more API > calls, leading to other issues. > Trino version >= 348 supports streaming uploads. > During experiments, I also noticed that the current presto s3 fs > implementation seems to allocate a lot of memory outside the heap (when > shipping large data, for example when creating a savepoint). On a K8s pod > with a memory limit of 4000Mi, I was not able to run Flink with a > "taskmanager.memory.flink.size" above 3000m. This means that an additional > 1gb of memory needs to be allocated just for the peaks in memory allocation > when presto s3 is taking a savepoint. It would be good to confirm this > behavior, and then either adjust the default memory configuration or the > documentation. > As part of this upgrade, we also need to make sure that the new presto / > Trino version is not doing substantially more S3 API calls than the current > version. After switching away from the presto s3 to hadoop s3, I noticed that > disposing an old checkpoint (~100gb) can take up to 15 minutes. The upgraded > presto s3 fs should still be able to quickly dispose state. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24348) Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container startup failed"
[
https://issues.apache.org/jira/browse/FLINK-24348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450981#comment-17450981
]
Arvid Heise commented on FLINK-24348:
-
Please do, since that is a test-only backport I expect little implications.
> Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container
> startup failed"
> --
>
> Key: FLINK-24348
> URL: https://issues.apache.org/jira/browse/FLINK-24348
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Dawid Wysakowicz
>Assignee: Martijn Visser
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24338=logs=b0097207-033c-5d9a-b48c-6d4796fbe60d=8338a7d2-16f7-52e5-f576-4b7b3071eb3d=7140
> {code}
> Sep 21 02:44:33 org.testcontainers.containers.ContainerLaunchException:
> Container startup failed
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
> Sep 21 02:44:33 at
> org.testcontainers.containers.KafkaContainer.doStart(KafkaContainer.java:97)
> Sep 21 02:44:33 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase$1.doStart(KafkaTableTestBase.java:71)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:315)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1060)
> Sep 21 02:44:33 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Sep 21 02:44:33 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Sep 21 02:44:33 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> Sep 21 02:44:33 at
>
[jira] [Commented] (FLINK-24348) Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container startup failed"
[
https://issues.apache.org/jira/browse/FLINK-24348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450930#comment-17450930
]
Arvid Heise commented on FLINK-24348:
-
Okay the script "testcontainers_start.sh" is generated in KafkaContainer in
test containers 1.16.0. With
https://github.com/testcontainers/testcontainers-java/pull/2078 that was
changed and either it fixes the issue or should provide us better logs, so I'm
proposing to upgrade to 1.16.2.
> Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container
> startup failed"
> --
>
> Key: FLINK-24348
> URL: https://issues.apache.org/jira/browse/FLINK-24348
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24338=logs=b0097207-033c-5d9a-b48c-6d4796fbe60d=8338a7d2-16f7-52e5-f576-4b7b3071eb3d=7140
> {code}
> Sep 21 02:44:33 org.testcontainers.containers.ContainerLaunchException:
> Container startup failed
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
> Sep 21 02:44:33 at
> org.testcontainers.containers.KafkaContainer.doStart(KafkaContainer.java:97)
> Sep 21 02:44:33 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase$1.doStart(KafkaTableTestBase.java:71)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:315)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1060)
> Sep 21 02:44:33 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Sep 21 02:44:33 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Sep 21 02:44:33 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
>
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[
https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450921#comment-17450921
]
Arvid Heise commented on FLINK-22300:
-
Okay I see, so Flink would never evict the data. That is indeed a bug.
What do you think is the proper fix? I have a hard time to come up with proper
semantics beyond saying that users can't use {{TimeEvictor}} with processing
time. For a window in processing time, a user probably wants to retain all
elements before trigger and evict all elements after the trigger in all cases.
For other cases, they would use ingestion or event time.
> Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of
> TimeCharacteristic
>
>
> Key: FLINK-22300
> URL: https://issues.apache.org/jira/browse/FLINK-22300
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.11.3
>Reporter: Bo Huang
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime).
> The data after Windows process can not be evicted by TimeEvictor of Keyed
> Windows Beause TimestampedValue have no timestamp value.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24348) Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container startup failed"
[
https://issues.apache.org/jira/browse/FLINK-24348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450912#comment-17450912
]
Arvid Heise commented on FLINK-24348:
-
I have not found out much yesterday. It is a standard Unix error but I'm unsure
if it is a general issue with test containers or with the specific kafka
container.
> Kafka ITCases (e.g. KafkaTableITCase) fail with "ContainerLaunch Container
> startup failed"
> --
>
> Key: FLINK-24348
> URL: https://issues.apache.org/jira/browse/FLINK-24348
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24338=logs=b0097207-033c-5d9a-b48c-6d4796fbe60d=8338a7d2-16f7-52e5-f576-4b7b3071eb3d=7140
> {code}
> Sep 21 02:44:33 org.testcontainers.containers.ContainerLaunchException:
> Container startup failed
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
> Sep 21 02:44:33 at
> org.testcontainers.containers.KafkaContainer.doStart(KafkaContainer.java:97)
> Sep 21 02:44:33 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase$1.doStart(KafkaTableTestBase.java:71)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.start(GenericContainer.java:315)
> Sep 21 02:44:33 at
> org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1060)
> Sep 21 02:44:33 at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> Sep 21 02:44:33 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> Sep 21 02:44:33 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> Sep 21 02:44:33 at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Sep 21 02:44:33 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Sep 21 02:44:33 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Sep 21 02:44:33 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> Sep 21 02:44:33 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Sep 21 02:44:33 at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
> Sep 21 02:44:33 at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> Sep 21 02:44:33 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
> Sep 21 02:44:33 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
> Sep 21 02:44:33 at
>
[jira] [Assigned] (FLINK-19863) SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process failed due to timeout"
[
https://issues.apache.org/jira/browse/FLINK-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-19863:
---
Assignee: Martijn Visser (was: Leonard Xu)
> SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process
> failed due to timeout"
> ---
>
> Key: FLINK-19863
> URL: https://issues.apache.org/jira/browse/FLINK-19863
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
>Affects Versions: 1.12.0, 1.12.3, 1.15.0
>Reporter: Dian Fu
>Assignee: Martijn Visser
>Priority: Major
> Labels: auto-unassigned, pull-request-available, test-stability
> Fix For: 1.12.0, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8541=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6
> {code}
> 00:50:02,589 [main] INFO
> org.apache.flink.tests.util.flink.FlinkDistribution [] - Stopping
> Flink cluster.
> 00:50:04,106 [main] INFO
> org.apache.flink.tests.util.flink.FlinkDistribution [] - Stopping
> Flink cluster.
> 00:50:04,741 [main] INFO
> org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource [] - Backed up
> logs to
> /home/vsts/work/1/s/flink-end-to-end-tests/artifacts/flink-b3924665-1ac9-4309-8255-20f0dc94e7b9.
> 00:50:04,788 [main] INFO
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource [] - Stopping
> HBase Cluster
> 00:50:16,243 [main] ERROR
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase [] -
>
> Test testHBase[0:
> hbase-version:1.4.3](org.apache.flink.tests.util.hbase.SQLClientHBaseITCase)
> failed with:
> java.io.IOException: Process failed due to timeout.
> at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:130)
> at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:108)
> at
> org.apache.flink.tests.util.flink.FlinkDistribution.submitSQLJob(FlinkDistribution.java:221)
> at
> org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource$StandaloneClusterController.submitSQLJob(LocalStandaloneFlinkResource.java:196)
> at
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase.executeSqlStatements(SQLClientHBaseITCase.java:215)
> at
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase.testHBase(SQLClientHBaseITCase.java:152)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24921) FileSourceSplit should not be visible in the user API in ParquetColumnarRowInputFormat
[ https://issues.apache.org/jira/browse/FLINK-24921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450534#comment-17450534 ] Arvid Heise commented on FLINK-24921: - Should we close this ticket or do we expect any work on this end (in the foreseeable future)? Note that we plan a larger cleanup in Parquet in 1.16. > FileSourceSplit should not be visible in the user API in > ParquetColumnarRowInputFormat > -- > > Key: FLINK-24921 > URL: https://issues.apache.org/jira/browse/FLINK-24921 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > _FileSourceSplit_ is an internal class that should not be visible in the user > API like > [here|https://github.com/apache/flink/blob/6f2d8fe3007464343c5312e27612be448b415148/flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormatTest.java#L235]. > The fact that _FileSourceSplit_ surfaces in the API also influences the user > to do a raw use of the parametrized class like > [here|https://github.com/apache/flink/blob/6f2d8fe3007464343c5312e27612be448b415148/flink-formats/flink-parquet/src/test/java/org/apache/flink/formats/parquet/ParquetColumnarRowInputFormatTest.java#L407] > It could be better to make parquet format a not parametrized class as it is > done for hive connector > _class_ HiveBulkFormatAdapter > _implements BulkFormat_ > rather than > _class ParquetColumnarRowInputFormat_ > _extends ParquetVectorizedInputFormat_ > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450532#comment-17450532 ] Arvid Heise commented on FLINK-22300: - Sorry for not following it up. I have a hard time to understand what you exactly expect. The timestamp of a record in processing time is always `now()` by definition and from your description it sounds like indeed returns it (I may be wrong here). I think what you are looking at is to use [ingestion time|https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/streaming_analytics/]. Then a record gets a proper timestamp assigned on first arrival in the system and windowing and eviction are deterministic tasks. However, I might have misunderstood your needs, so please reiterate them if ingestion time is not working for you and why. > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24859) Document new File formats
[ https://issues.apache.org/jira/browse/FLINK-24859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450526#comment-17450526 ] Arvid Heise commented on FLINK-24859: - You are right that they currently cannot be used in DataStream. That was something that we originally planned for 1.15 but that has been pushed back. > Document new File formats > - > > Key: FLINK-24859 > URL: https://issues.apache.org/jira/browse/FLINK-24859 > Project: Flink > Issue Type: Technical Debt > Components: Documentation >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > Labels: pull-request-available > > The project recently introduced new formats: _BulkFormat_ and _StreamFormat_ > interfaces. > There are already implementations of these formats: hive, parquet, orc and > textLine formats that need to be documented. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-20928) KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270 » Timeout
[
https://issues.apache.org/jira/browse/FLINK-20928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450523#comment-17450523
]
Arvid Heise commented on FLINK-20928:
-
[~renqs], [~lindong] could you please take another look?
> KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270
> » Timeout
> ---
>
> Key: FLINK-20928
> URL: https://issues.apache.org/jira/browse/FLINK-20928
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0, 1.14.0, 1.15.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11861=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 93.992 s <<< FAILURE! - in
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest
> [ERROR]
> testOffsetCommitOnCheckpointComplete(org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest)
> Time elapsed: 60.086 s <<< ERROR!
> java.util.concurrent.TimeoutException: The offset commit did not finish
> before timeout.
> at
> org.apache.flink.core.testutils.CommonTestUtils.waitUtil(CommonTestUtils.java:210)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.pollUntil(KafkaSourceReaderTest.java:270)
> at
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete(KafkaSourceReaderTest.java:189)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-21407) Clarify which sources and APIs support which formats
[ https://issues.apache.org/jira/browse/FLINK-21407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450521#comment-17450521 ] Arvid Heise commented on FLINK-21407: - Merged into master as dd31d03dbc3f29522ec01328b7a93121137df7d7..cbedff73193c90f2324951273baa71cdfd8c23f1. Could you please reopen 1.14? > Clarify which sources and APIs support which formats > > > Key: FLINK-21407 > URL: https://issues.apache.org/jira/browse/FLINK-21407 > Project: Flink > Issue Type: New Feature > Components: API / DataSet, API / DataStream, Documentation >Reporter: Chesnay Schepler >Assignee: Etienne Chauchot >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Fix For: 1.15.0, 1.13.4 > > > The DataSet connectors documentation is essentially an empty desert amounting > to "you can read files". > The DataStream connectors documentation do not mention formats like > avro/parquet anywhere, nor the possibility to read from filesystems (only the > sinks are documented). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-24108) Update kafka documentation in Chinese
[ https://issues.apache.org/jira/browse/FLINK-24108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-24108. - Resolution: Duplicate > Update kafka documentation in Chinese > - > > Key: FLINK-24108 > URL: https://issues.apache.org/jira/browse/FLINK-24108 > Project: Flink > Issue Type: Improvement > Components: chinese-translation >Affects Versions: 1.14.0, 1.15.0 >Reporter: Arvid Heise >Priority: Major > Labels: documentation, pull-request-available, stale-major > > A few things have changed that should all be synced. In particular: > - Metrics of Kafka source have been added > - KafkaSink was completely added > - Metrics in ops/metrics were removed and a link to the Kafka page has been > added. > Please ping me if something is not clear. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24108) Update kafka documentation in Chinese
[ https://issues.apache.org/jira/browse/FLINK-24108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450516#comment-17450516 ] Arvid Heise commented on FLINK-24108: - [~liangkw16], [~monster#12] sorry for not noticing your comments and thank you very much. In the meantime, FLINK-24308 already addressed it. > Update kafka documentation in Chinese > - > > Key: FLINK-24108 > URL: https://issues.apache.org/jira/browse/FLINK-24108 > Project: Flink > Issue Type: Improvement > Components: chinese-translation >Affects Versions: 1.14.0, 1.15.0 >Reporter: Arvid Heise >Priority: Major > Labels: documentation, pull-request-available, stale-major > > A few things have changed that should all be synced. In particular: > - Metrics of Kafka source have been added > - KafkaSink was completely added > - Metrics in ops/metrics were removed and a link to the Kafka page has been > added. > Please ping me if something is not clear. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-24308) Translate KafkaSink docs to chinese
[ https://issues.apache.org/jira/browse/FLINK-24308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise reassigned FLINK-24308: --- Assignee: Qingsheng Ren > Translate KafkaSink docs to chinese > --- > > Key: FLINK-24308 > URL: https://issues.apache.org/jira/browse/FLINK-24308 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.14.0 >Reporter: Fabian Paul >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > > With https://issues.apache.org/jira/browse/FLINK-23664 only the English > documentation was updated. We also have to update the Chinese docs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25029) Hadoop Caller Context Setting In Flink
[ https://issues.apache.org/jira/browse/FLINK-25029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450509#comment-17450509 ] Arvid Heise commented on FLINK-25029: - Just FYI, the HDFS Filesystem lives in a separate class loader. It has access to all Flink core classes but it would be difficult to add a context class just to the Filesystem. But I haven't understood which components are involved. Does it also apply to hive-specific things? Note also that the filesystem will not have access to any job-specific information; that's why I thought it has to be added to the coordinator. I could imagine that we need to add a general context facility (e.g. service interface on JobMaster and service implementation in filesystem). Be also aware that we constantly need to swap context classloaders for various things. So this is not a reliable hook. Thread locals seem to be a good way at first glance. > Hadoop Caller Context Setting In Flink > -- > > Key: FLINK-25029 > URL: https://issues.apache.org/jira/browse/FLINK-25029 > Project: Flink > Issue Type: Improvement > Components: FileSystems >Reporter: 刘方奇 >Assignee: 刘方奇 >Priority: Major > > For a given HDFS operation (e.g. delete file), it's very helpful to track > which upper level job issues it. The upper level callers may be specific > Oozie tasks, MR jobs, and hive queries. One scenario is that the namenode > (NN) is abused/spammed, the operator may want to know immediately which MR > job should be blamed so that she can kill it. To this end, the caller context > contains at least the application-dependent "tracking id". > The above is the main effect of the Caller Context. HDFS Client set Caller > Context, then name node get it in audit log to do some work. > Now the Spark and hive have the Caller Context to meet the HDFS Job Audit > requirement. > In my company, flink jobs often cause some problems for HDFS, so we did it > for preventing some cases. > If the feature is general enough. Should we support it, then I can submit a > PR for this. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-21214) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed
[ https://issues.apache.org/jira/browse/FLINK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450435#comment-17450435 ] Arvid Heise commented on FLINK-21214: - Yes, I wrote the reason for closing O_o. [~fpaul] apparently decided to forward port the fix but I don't know if the ticket number is valid. > FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed > -- > > Key: FLINK-21214 > URL: https://issues.apache.org/jira/browse/FLINK-21214 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.13.3 >Reporter: Guowei Ma >Assignee: Fabian Paul >Priority: Major > Labels: pull-request-available, test-stability > Fix For: 1.15.0, 1.14.1, 1.13.4 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12687=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > > [ERROR] > testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase) > Time elapsed: 62.857 s <<< ERROR! > org.apache.kafka.common.errors.TimeoutException: > org.apache.kafka.common.errors.TimeoutException: Timeout expired after > 6milliseconds while awaiting InitProducerId > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired > after 6milliseconds while awaiting InitProducerId > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-25088) KafkaSinkITCase failed on azure due to Container did not start correctly
[
https://issues.apache.org/jira/browse/FLINK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-25088.
-
Assignee: Arvid Heise
Resolution: Duplicate
This is the same issue as. I will continue investigation there as it's the
earlier ticket.
> KafkaSinkITCase failed on azure due to Container did not start correctly
>
>
> Key: FLINK-25088
> URL: https://issues.apache.org/jira/browse/FLINK-25088
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Connectors / Kafka
>Affects Versions: 1.14.0
>Reporter: Yun Gao
>Assignee: Arvid Heise
>Priority: Major
> Labels: test-stability
> Fix For: 1.14.1
>
>
> {code:java}
> Nov 29 03:38:00 at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
> Nov 29 03:38:00 at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
> Nov 29 03:38:00 at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:150)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:120)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> Nov 29 03:38:00 at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Nov 29 03:38:00 Caused by: org.rnorth.ducttape.RetryCountExceededException:
> Retry limit hit with exception
> Nov 29 03:38:00 at
> org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
> Nov 29 03:38:00 at
> org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:327)
> Nov 29 03:38:00 ... 33 more
> Nov 29 03:38:00 Caused by:
> org.testcontainers.containers.ContainerLaunchException: Could not
> create/start container
> Nov 29 03:38:00 at
> org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:523)
> Nov 29 03:38:00 at
> org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:329)
> Nov 29 03:38:00 at
> org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
> Nov 29 03:38:00 ... 34 more
> Nov 29 03:38:00 Caused by: java.lang.IllegalStateException: Container did not
> start correctly.
> Nov 29 03:38:00 at
> org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:461)
> Nov 29 03:38:00 ... 36 more
> Nov 29 03:38:00
> Nov 29 03:38:01 [INFO] Running
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase
> Nov 29 03:38:16 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time
> elapsed: 14.442 s - in
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase
> Nov 29 03:38:16 [INFO]
> Nov 29 03:38:16 [INFO] Results:
> Nov 29 03:38:16 [INFO]
> Nov 29 03:38:16 [ERROR] Errors:
> Nov 29 03:38:16 [ERROR] KafkaSinkITCase » ContainerLaunch Container startup
> failed
> Nov 29 03:38:16 [INFO]
> Nov 29 03:38:16 [ERROR] Tests run: 186, Failures: 0, Errors: 1, Skipped: 0
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=27181=logs=b0097207-033c-5d9a-b48c-6d4796fbe60d=8338a7d2-16f7-52e5-f576-4b7b3071eb3d=7151
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-19863) SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process failed due to timeout"
[
https://issues.apache.org/jira/browse/FLINK-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450420#comment-17450420
]
Arvid Heise commented on FLINK-19863:
-
[~MartijnVisser] you are currently hardening the shell commands against
timeouts, could you PTAL?
The last 3 failures were 2x
{noformat}
Aug 24 15:11:30 java.io.IOException: Process ([wget, -q, -P,
/home/vsts/work/1/e2e_cache/downloads/1782322924,
https://archive.apache.org/dist/hbase/2.2.3/hbase-2.2.3-bin.tar.gz]) exceeded
timeout (60) or number of retries (3).
{noformat}
and 1 time
{noformat}
Aug 14 23:41:26 [ERROR] testHBase[1:
hbase-version:2.2.3](org.apache.flink.tests.util.hbase.SQLClientHBaseITCase)
Time elapsed: 297.333 s <<< ERROR!
Aug 14 23:41:26 java.io.IOException:
Aug 14 23:41:26 Process execution failed due error. Error output:
Aug 14 23:41:26 gzip: stdin: unexpected end of file
Aug 14 23:41:26 at
org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.setupHBaseDist(LocalStandaloneHBaseResource.java:86)
Aug 14 23:41:26 at
org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.before(LocalStandaloneHBaseResource.java:76)
Aug 14 23:41:26 at
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)
Aug 14 23:41:26 at
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
Aug 14 23:41:26 at
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
Aug 14 23:41:26 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
Aug 14 23:41:26 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
Aug 14 23:41:26 at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
Aug 14 23:41:26 at org.junit.runners.Suite.runChild(Suite.java:128)
Aug 14 23:41:26 at org.junit.runners.Suite.runChild(Suite.java:27)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
Aug 14 23:41:26 at
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
Aug 14 23:41:26 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
Aug 14 23:41:26 at org.junit.runners.Suite.runChild(Suite.java:128)
Aug 14 23:41:26 at org.junit.runners.Suite.runChild(Suite.java:27)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
Aug 14 23:41:26 at
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
Aug 14 23:41:26 at
org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
Aug 14 23:41:26 at
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
Aug 14 23:41:26 at
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
Aug 14 23:41:26 at
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
Aug 14 23:41:26 at
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
Aug 14 23:41:26 at
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
Aug 14 23:41:26 at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
Aug 14 23:41:26 at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
Aug 14 23:41:26 at
[jira] [Assigned] (FLINK-19863) SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process failed due to timeout"
[
https://issues.apache.org/jira/browse/FLINK-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-19863:
---
Assignee: Leonard Xu (was: Arvid Heise)
> SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process
> failed due to timeout"
> ---
>
> Key: FLINK-19863
> URL: https://issues.apache.org/jira/browse/FLINK-19863
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
>Affects Versions: 1.12.0, 1.12.3, 1.15.0
>Reporter: Dian Fu
>Assignee: Leonard Xu
>Priority: Major
> Labels: auto-unassigned, pull-request-available, test-stability
> Fix For: 1.12.0, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8541=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6
> {code}
> 00:50:02,589 [main] INFO
> org.apache.flink.tests.util.flink.FlinkDistribution [] - Stopping
> Flink cluster.
> 00:50:04,106 [main] INFO
> org.apache.flink.tests.util.flink.FlinkDistribution [] - Stopping
> Flink cluster.
> 00:50:04,741 [main] INFO
> org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource [] - Backed up
> logs to
> /home/vsts/work/1/s/flink-end-to-end-tests/artifacts/flink-b3924665-1ac9-4309-8255-20f0dc94e7b9.
> 00:50:04,788 [main] INFO
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource [] - Stopping
> HBase Cluster
> 00:50:16,243 [main] ERROR
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase [] -
>
> Test testHBase[0:
> hbase-version:1.4.3](org.apache.flink.tests.util.hbase.SQLClientHBaseITCase)
> failed with:
> java.io.IOException: Process failed due to timeout.
> at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:130)
> at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:108)
> at
> org.apache.flink.tests.util.flink.FlinkDistribution.submitSQLJob(FlinkDistribution.java:221)
> at
> org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource$StandaloneClusterController.submitSQLJob(LocalStandaloneFlinkResource.java:196)
> at
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase.executeSqlStatements(SQLClientHBaseITCase.java:215)
> at
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase.testHBase(SQLClientHBaseITCase.java:152)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-22621) HBaseConnectorITCase.testTableSourceSinkWithDDL unstable on azure
[
https://issues.apache.org/jira/browse/FLINK-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-22621:
---
Assignee: Jing Ge (was: Arvid Heise)
> HBaseConnectorITCase.testTableSourceSinkWithDDL unstable on azure
> -
>
> Key: FLINK-22621
> URL: https://issues.apache.org/jira/browse/FLINK-22621
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.13.1
>Reporter: Roman Khachatryan
>Assignee: Jing Ge
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17763=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=12317]
>
> {code:java}
> 2021-05-10T00:19:41.1703846Z May 10 00:19:41
> testTableSourceSinkWithDDL[planner = BLINK_PLANNER, legacy =
> false](org.apache.flink.connector.hbase2.HBaseConnectorITCase) Time elapsed:
> 2.907 sec <<< FAILURE!
> 2021-05-10T00:19:41.1711710Z May 10 00:19:41 java.lang.AssertionError:
> expected:<[+I[1, 10, Hello-1, 100, 1.01, false, Welt-1, 2019-08-18T19:00,
> 2019-08-18, 19:00, 12345678.0001], +I[2, 20, Hello-2, 200, 2.02, true,
> Welt-2, 2019-08-18T19:01, 2019-08-18, 19:01, 12345678.0002], +I[3, 30,
> Hello-3, 300, 3.03, false, Welt-3, 2019-08-18T19:02, 2019-08-18, 19:02,
> 12345678.0003], +I[4, 40, null, 400, 4.04, true, Welt-4, 2019-08-18T19:03,
> 2019-08-18, 19:03, 12345678.0004], +I[5, 50, Hello-5, 500, 5.05, false,
> Welt-5, 2019-08-19T19:10, 2019-08-19, 19:10, 12345678.0005], +I[6, 60,
> Hello-6, 600, 6.06, true, Welt-6, 2019-08-19T19:20, 2019-08-19, 19:20,
> 12345678.0006], +I[7, 70, Hello-7, 700, 7.07, false, Welt-7,
> 2019-08-19T19:30, 201 9-08-19, 19:30, 12345678.0007], +I[8, 80, null, 800,
> 8.08, true, Welt-8, 2019-08-19T19:40, 2019-08-19, 19:40, 12345678.0008]]> but
> was:<[+I[1, 10, Hello-1, 100, 1.01, false, Welt-1, 2019-08-18T19:00,
> 2019-08-18, 19:00, 12345678.0001], +I[2, 20, Hello-2, 200, 2.02, true,
> Welt-2, 2019-08-18T19 :01, 2019-08-18, 19:01, 12345678.0002], +I[3, 30,
> Hello-3, 300, 3.03, false, Welt-3, 2019-08-18T19:02, 2019-08-18, 19:02,
> 12345678.0003]]>
> 2021-05-10T00:19:41.1716769Z May 10 00:19:41at
> org.junit.Assert.fail(Assert.java:88)
> 2021-05-10T00:19:41.1717997Z May 10 00:19:41at
> org.junit.Assert.failNotEquals(Assert.java:834)
> 2021-05-10T00:19:41.1718744Z May 10 00:19:41at
> org.junit.Assert.assertEquals(Assert.java:118)
> 2021-05-10T00:19:41.1719472Z May 10 00:19:41at
> org.junit.Assert.assertEquals(Assert.java:144)
> 2021-05-10T00:19:41.1720270Z May 10 00:19:41at
> org.apache.flink.connector.hbase2.HBaseConnectorITCase.testTableSourceSinkWithDDL(HBaseConnecto
> rITCase.java:506)
> {code}
> Probably the same or similar to FLINK-19615
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-22621) HBaseConnectorITCase.testTableSourceSinkWithDDL unstable on azure
[
https://issues.apache.org/jira/browse/FLINK-22621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450414#comment-17450414
]
Arvid Heise commented on FLINK-22621:
-
[~jingge] is currently investigating, I'll reassign.
> HBaseConnectorITCase.testTableSourceSinkWithDDL unstable on azure
> -
>
> Key: FLINK-22621
> URL: https://issues.apache.org/jira/browse/FLINK-22621
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.14.0, 1.13.1
>Reporter: Roman Khachatryan
>Assignee: Arvid Heise
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17763=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20=12317]
>
> {code:java}
> 2021-05-10T00:19:41.1703846Z May 10 00:19:41
> testTableSourceSinkWithDDL[planner = BLINK_PLANNER, legacy =
> false](org.apache.flink.connector.hbase2.HBaseConnectorITCase) Time elapsed:
> 2.907 sec <<< FAILURE!
> 2021-05-10T00:19:41.1711710Z May 10 00:19:41 java.lang.AssertionError:
> expected:<[+I[1, 10, Hello-1, 100, 1.01, false, Welt-1, 2019-08-18T19:00,
> 2019-08-18, 19:00, 12345678.0001], +I[2, 20, Hello-2, 200, 2.02, true,
> Welt-2, 2019-08-18T19:01, 2019-08-18, 19:01, 12345678.0002], +I[3, 30,
> Hello-3, 300, 3.03, false, Welt-3, 2019-08-18T19:02, 2019-08-18, 19:02,
> 12345678.0003], +I[4, 40, null, 400, 4.04, true, Welt-4, 2019-08-18T19:03,
> 2019-08-18, 19:03, 12345678.0004], +I[5, 50, Hello-5, 500, 5.05, false,
> Welt-5, 2019-08-19T19:10, 2019-08-19, 19:10, 12345678.0005], +I[6, 60,
> Hello-6, 600, 6.06, true, Welt-6, 2019-08-19T19:20, 2019-08-19, 19:20,
> 12345678.0006], +I[7, 70, Hello-7, 700, 7.07, false, Welt-7,
> 2019-08-19T19:30, 201 9-08-19, 19:30, 12345678.0007], +I[8, 80, null, 800,
> 8.08, true, Welt-8, 2019-08-19T19:40, 2019-08-19, 19:40, 12345678.0008]]> but
> was:<[+I[1, 10, Hello-1, 100, 1.01, false, Welt-1, 2019-08-18T19:00,
> 2019-08-18, 19:00, 12345678.0001], +I[2, 20, Hello-2, 200, 2.02, true,
> Welt-2, 2019-08-18T19 :01, 2019-08-18, 19:01, 12345678.0002], +I[3, 30,
> Hello-3, 300, 3.03, false, Welt-3, 2019-08-18T19:02, 2019-08-18, 19:02,
> 12345678.0003]]>
> 2021-05-10T00:19:41.1716769Z May 10 00:19:41at
> org.junit.Assert.fail(Assert.java:88)
> 2021-05-10T00:19:41.1717997Z May 10 00:19:41at
> org.junit.Assert.failNotEquals(Assert.java:834)
> 2021-05-10T00:19:41.1718744Z May 10 00:19:41at
> org.junit.Assert.assertEquals(Assert.java:118)
> 2021-05-10T00:19:41.1719472Z May 10 00:19:41at
> org.junit.Assert.assertEquals(Assert.java:144)
> 2021-05-10T00:19:41.1720270Z May 10 00:19:41at
> org.apache.flink.connector.hbase2.HBaseConnectorITCase.testTableSourceSinkWithDDL(HBaseConnecto
> rITCase.java:506)
> {code}
> Probably the same or similar to FLINK-19615
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25088) KafkaSinkITCase failed on azure due to Container did not start correctly
[
https://issues.apache.org/jira/browse/FLINK-25088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450367#comment-17450367
]
Arvid Heise commented on FLINK-25088:
-
I found in the logs
{noformat}
03:37:59,944 [main] INFO [confluentinc/cp-kafka:5.5.2]
[] - Creating container for image:
confluentinc/cp-kafka:5.5.2
03:38:00,065 [main] INFO [confluentinc/cp-kafka:5.5.2]
[] - Starting container with ID:
9f6a0e7664040345d6dc9a4189aea0bfcc2f191b31545e0ea0d191382454138c
03:38:00,537 [main] INFO [confluentinc/cp-kafka:5.5.2]
[] - Container confluentinc/cp-kafka:5.5.2 is starting:
9f6a0e7664040345d6dc9a4189aea0bfcc2f191b31545e0ea0d191382454138c
03:38:00,640 [docker-java-stream-1817108049] INFO
org.apache.flink.connector.kafka.sink.KafkaSinkITCase[] - STDERR: sh:
1: /testcontainers_start.sh: Text file busy
03:38:00,800 [main] ERROR [confluentinc/cp-kafka:5.5.2]
[] - Could not start container
java.lang.IllegalStateException: Container did not start correctly.
at
org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:461)
~[testcontainers-1.16.0.jar:?]
at
org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:329)
~[testcontainers-1.16.0.jar:?]
at
org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
~[duct-tape-1.0.8.jar:?]
at
org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:327)
~[testcontainers-1.16.0.jar:?]
at
org.testcontainers.containers.KafkaContainer.doStart(KafkaContainer.java:97)
~[kafka-1.16.0.jar:?]
at
org.testcontainers.containers.GenericContainer.start(GenericContainer.java:315)
~[testcontainers-1.16.0.jar:?]
at
org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1060)
~[testcontainers-1.16.0.jar:?]
at
org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
~[testcontainers-1.16.0.jar:?]
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
~[junit-4.13.2.jar:4.13.2]
at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
~[junit-4.13.2.jar:4.13.2]
at
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
~[junit-vintage-engine-5.7.2.jar:5.7.2]
at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
~[?:1.8.0_292]
at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
~[?:1.8.0_292]
at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_292]
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
~[?:1.8.0_292]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
~[?:1.8.0_292]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_292]
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
~[?:1.8.0_292]
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
~[?:1.8.0_292]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_292]
at
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
~[?:1.8.0_292]
at
org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
~[junit-vintage-engine-5.7.2.jar:5.7.2]
at
org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
~[junit-vintage-engine-5.7.2.jar:5.7.2]
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
~[junit-platform-launcher-1.3.1.jar:1.3.1]
at
org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
~[junit-platform-launcher-1.3.1.jar:1.3.1]
at
org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
[junit-platform-launcher-1.3.1.jar:1.3.1]
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
[junit-platform-launcher-1.3.1.jar:1.3.1]
at
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
[junit-platform-launcher-1.3.1.jar:1.3.1]
at
[jira] [Resolved] (FLINK-24325) Create ElasticSearch 6.8 Sink
[ https://issues.apache.org/jira/browse/FLINK-24325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-24325. - Fix Version/s: 1.15.0 Resolution: Fixed Merged into master as 9649f45dab374c5b5a18ba12cc0526b37a504fd8..35e90eb0ca612e52d0de15a7a18dad23a459cd42. > Create ElasticSearch 6.8 Sink > - > > Key: FLINK-24325 > URL: https://issues.apache.org/jira/browse/FLINK-24325 > Project: Flink > Issue Type: Sub-task > Components: Connectors / ElasticSearch >Reporter: Alexander Preuss >Assignee: Alexander Preuss >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25044) Add More Unit Test For Pulsar Source
[ https://issues.apache.org/jira/browse/FLINK-25044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise reassigned FLINK-25044: --- Assignee: Yufei Zhang > Add More Unit Test For Pulsar Source > > > Key: FLINK-25044 > URL: https://issues.apache.org/jira/browse/FLINK-25044 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Reporter: Yufei Zhang >Assignee: Yufei Zhang >Priority: Minor > Labels: pull-request-available, testing > > We should enhance the pulsar source connector tests by adding more unit tests. > > * SourceReader > * SplitReader > * Enumerator > * SourceBuilder -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-21214) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed
[ https://issues.apache.org/jira/browse/FLINK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise resolved FLINK-21214. - Fix Version/s: 1.13.4 Resolution: Fixed > FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed > -- > > Key: FLINK-21214 > URL: https://issues.apache.org/jira/browse/FLINK-21214 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.13.3 >Reporter: Guowei Ma >Assignee: Fabian Paul >Priority: Major > Labels: pull-request-available, test-stability > Fix For: 1.15.0, 1.14.1, 1.13.4 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12687=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > > [ERROR] > testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase) > Time elapsed: 62.857 s <<< ERROR! > org.apache.kafka.common.errors.TimeoutException: > org.apache.kafka.common.errors.TimeoutException: Timeout expired after > 6milliseconds while awaiting InitProducerId > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired > after 6milliseconds while awaiting InitProducerId > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-21214) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed
[ https://issues.apache.org/jira/browse/FLINK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449618#comment-17449618 ] Arvid Heise commented on FLINK-21214: - Merged into 1.14 as 29c288993ce7e24af70063cf7d645f4ff83909d6. Closing for now as we assume that Kafka 2.8.1 solves the issue on master. Please reopen otherwise. > FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed > -- > > Key: FLINK-21214 > URL: https://issues.apache.org/jira/browse/FLINK-21214 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.13.3 >Reporter: Guowei Ma >Assignee: Fabian Paul >Priority: Major > Labels: pull-request-available, test-stability > Fix For: 1.15.0, 1.14.1 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12687=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > > [ERROR] > testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase) > Time elapsed: 62.857 s <<< ERROR! > org.apache.kafka.common.errors.TimeoutException: > org.apache.kafka.common.errors.TimeoutException: Timeout expired after > 6milliseconds while awaiting InitProducerId > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired > after 6milliseconds while awaiting InitProducerId > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-24703) Add CSV format support for filesystem based on StreamFormat and BulkWriter interfaces.
[ https://issues.apache.org/jira/browse/FLINK-24703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arvid Heise reassigned FLINK-24703: --- Assignee: Alexander Fedulov > Add CSV format support for filesystem based on StreamFormat and BulkWriter > interfaces. > -- > > Key: FLINK-24703 > URL: https://issues.apache.org/jira/browse/FLINK-24703 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream, Formats (JSON, Avro, Parquet, ORC, > SequenceFile) >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-25023) ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of user code
[
https://issues.apache.org/jira/browse/FLINK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-25023:
---
Assignee: David Morávek
> ClassLoader leak on JM/TM through indirectly-started Hadoop threads out of
> user code
>
>
> Key: FLINK-25023
> URL: https://issues.apache.org/jira/browse/FLINK-25023
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.14.0, 1.12.5, 1.13.3
>Reporter: Nico Kruber
>Assignee: David Morávek
>Priority: Major
> Labels: pull-request-available
>
> If a Flink job is using HDFS through Flink's filesystem abstraction (either
> on the JM or TM), that code may actually spawn a few threads, e.g. from
> static class members:
> *
> {{org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner}}
> * {{IPC Parameter Sending Thread#*}}
> These threads are started as soon as the classes are loaded which may be in
> the context of the user code. In this specific scenario, however, the created
> threads may contain references to the context class loader (I did not see
> that though) or, as happened here, it may inherit thread contexts such as the
> {{ProtectionDomain}} (from an {{{}AccessController{}}}).
> Hence user contexts and user class loaders are leaked into long-running
> threads that are run in Flink's (parent) classloader.
> Fortunately, it seems to only *leak a single* {{ChildFirstClassLoader}} in
> this concrete example but that may depend on which code paths each client
> execution is walking.
>
> A *proper solution* doesn't seem so simple:
> * We could try to proactively initialize available file systems in the hope
> to start all threads in the parent classloader with parent context.
> * We could create a default {{ProtectionDomain}} for spawned threads as
> discussed at [https://dzone.com/articles/javalangoutofmemory-permgen],
> however, the {{StatisticsDataReferenceCleaner}} isn't actually actively
> spawned from any callback but as a static variable and this with the class
> loading itself (but maybe this is still possible somehow).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-21214) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed
[ https://issues.apache.org/jira/browse/FLINK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449009#comment-17449009 ] Arvid Heise commented on FLINK-21214: - Merged into 1.13 as 408ba03902621f69b996a19af11fe9e8e8f13008. > FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed > -- > > Key: FLINK-21214 > URL: https://issues.apache.org/jira/browse/FLINK-21214 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.13.3 >Reporter: Guowei Ma >Assignee: Fabian Paul >Priority: Major > Labels: pull-request-available, test-stability > Fix For: 1.15.0, 1.14.1 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12687=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > > [ERROR] > testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase) > Time elapsed: 62.857 s <<< ERROR! > org.apache.kafka.common.errors.TimeoutException: > org.apache.kafka.common.errors.TimeoutException: Timeout expired after > 6milliseconds while awaiting InitProducerId > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired > after 6milliseconds while awaiting InitProducerId > -- This message was sent by Atlassian Jira (v8.20.1#820001)
