[jira] [Commented] (FLINK-26033) In KafkaConnector, when 'sink.partitioner' is configured as 'round-robin', it does not take effect
[
https://issues.apache.org/jira/browse/FLINK-26033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489327#comment-17489327
]
Martijn Visser commented on FLINK-26033:
[~renqs] What are your thoughts on this?
> In KafkaConnector, when 'sink.partitioner' is configured as 'round-robin', it
> does not take effect
> --
>
> Key: FLINK-26033
> URL: https://issues.apache.org/jira/browse/FLINK-26033
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.3, 1.14.3
>Reporter: shizhengchao
>Priority: Major
>
> In KafkaConnector, when 'sink.partitioner' is configured as 'round-robin', it
> does not take effect. Flink treats 'default' and 'round-robin' as the same
> strategy.
> {code:java}
> //代码占位符
> public static Optional>
> getFlinkKafkaPartitioner(
> ReadableConfig tableOptions, ClassLoader classLoader) {
> return tableOptions
> .getOptional(SINK_PARTITIONER)
> .flatMap(
> (String partitioner) -> {
> switch (partitioner) {
> case SINK_PARTITIONER_VALUE_FIXED:
> return Optional.of(new
> FlinkFixedPartitioner<>());
> case SINK_PARTITIONER_VALUE_DEFAULT:
> case SINK_PARTITIONER_VALUE_ROUND_ROBIN:
> return Optional.empty();
> // Default fallback to full class name of the
> partitioner.
> default:
> return Optional.of(
> initializePartitioner(partitioner,
> classLoader));
> }
> });
> } {code}
> They both use kafka's default partitioner, but the actual There are two
> scenarios for the partition on DefaultPartitioner:
> 1. Random when there is no key
> 2. When there is a key, take the modulo according to the key
> {code:java}
> // org.apache.kafka.clients.producer.internals.DefaultPartitioner
> public int partition(String topic, Object key, byte[] keyBytes, Object value,
> byte[] valueBytes, Cluster cluster) {
> if (keyBytes == null) {
> // Random when there is no key
> return stickyPartitionCache.partition(topic, cluster);
> }
> List partitions = cluster.partitionsForTopic(topic);
> int numPartitions = partitions.size();
> // hash the keyBytes to choose a partition
> return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
> } {code}
> Therefore, KafkaConnector does not have a round-robin strategy.But we can
> borrow from kafka's RoundRobinPartitioner
> {code:java}
> //代码占位符
> public class RoundRobinPartitioner implements Partitioner {
> private final ConcurrentMap topicCounterMap = new
> ConcurrentHashMap<>();
> public void configure(Map configs) {}
> /**
> * Compute the partition for the given record.
> *
> * @param topic The topic name
> * @param key The key to partition on (or null if no key)
> * @param keyBytes serialized key to partition on (or null if no key)
> * @param value The value to partition on or null
> * @param valueBytes serialized value to partition on or null
> * @param cluster The current cluster metadata
> */
> @Override
> public int partition(String topic, Object key, byte[] keyBytes, Object
> value, byte[] valueBytes, Cluster cluster) {
> List partitions = cluster.partitionsForTopic(topic);
> int numPartitions = partitions.size();
> int nextValue = nextValue(topic);
> List availablePartitions =
> cluster.availablePartitionsForTopic(topic);
> if (!availablePartitions.isEmpty()) {
> int part = Utils.toPositive(nextValue) %
> availablePartitions.size();
> return availablePartitions.get(part).partition();
> } else {
> // no partitions are available, give a non-available partition
> return Utils.toPositive(nextValue) % numPartitions;
> }
> }
> private int nextValue(String topic) {
> AtomicInteger counter = topicCounterMap.computeIfAbsent(topic, k -> {
> return new AtomicInteger(0);
> });
> return counter.getAndIncrement();
> }
> public void close() {}
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18676: [FLINK-26035][build][planner] Add table-planner-loader-helper module
flinkbot commented on pull request #18676: URL: https://github.com/apache/flink/pull/18676#issuecomment-1033442314 ## CI report: * dad0e032399b3b422d20e018161c0030ca84d08e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18356) Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489321#comment-17489321
]
Yun Gao commented on FLINK-18356:
-
Hi [~dmvk]~ do we have already had jira issue for the aws thread leaking~?
> Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Piotr Nowojski
>Assignee: Chesnay Schepler
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
> Attachments: 1234.jpg, app-profiling_4.gif
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18677: [FLINK-26032][client] add log info for the status of job submission and execution result.
flinkbot commented on pull request #18677: URL: https://github.com/apache/flink/pull/18677#issuecomment-1033455918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18676: [FLINK-26035][build][planner] Add table-planner-loader-helper module
flinkbot edited a comment on pull request #18676: URL: https://github.com/apache/flink/pull/18676#issuecomment-1033442314 ## CI report: * dad0e032399b3b422d20e018161c0030ca84d08e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30977) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18675: [FLINK-25491][table-planner] Fix bug: generated code for a large IN filter can't be compiled
flinkbot edited a comment on pull request #18675: URL: https://github.com/apache/flink/pull/18675#issuecomment-1033442210 ## CI report: * d13e6f599e562fb155120cbc60f9a4a983356f2e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30976) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26036) LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory timeout on azure
[
https://issues.apache.org/jira/browse/FLINK-26036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489320#comment-17489320
]
Yun Gao commented on FLINK-26036:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30947=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=24198
> LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> timeout on azure
> ---
>
> Key: FLINK-26036
> URL: https://issues.apache.org/jira/browse/FLINK-26036
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 022-02-09T02:18:17.1827314Z Feb 09 02:18:14 [ERROR]
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> Time elapsed: 62.252 s <<< ERROR!
> 2022-02-09T02:18:17.1827940Z Feb 09 02:18:14
> java.util.concurrent.TimeoutException
> 2022-02-09T02:18:17.1828450Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> 2022-02-09T02:18:17.1829040Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2022-02-09T02:18:17.1829752Z Feb 09 02:18:14 at
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory(LocalRecoveryITCase.java:115)
> 2022-02-09T02:18:17.1830407Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-02-09T02:18:17.1830954Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-02-09T02:18:17.1831582Z Feb 09 02:18:14 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-09T02:18:17.1832135Z Feb 09 02:18:14 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-02-09T02:18:17.1832697Z Feb 09 02:18:14 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> 2022-02-09T02:18:17.1833566Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2022-02-09T02:18:17.1834394Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-02-09T02:18:17.1835125Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2022-02-09T02:18:17.1835875Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> 2022-02-09T02:18:17.1836565Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> 2022-02-09T02:18:17.1837294Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2022-02-09T02:18:17.1838007Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-02-09T02:18:17.1838743Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-02-09T02:18:17.1839499Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-02-09T02:18:17.1840224Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2022-02-09T02:18:17.1840952Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2022-02-09T02:18:17.1841616Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> 2022-02-09T02:18:17.1842257Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> 2022-02-09T02:18:17.1842951Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
> 2022-02-09T02:18:17.1843681Z Feb 09 02:18:14 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2022-02-09T02:18:17.1844782Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
> 2022-02-09T02:18:17.1845603Z Feb 09 02:18:14 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18434: [FLINK-25742][akka] Remove the serialization of rpc invocation at Fli…
flinkbot edited a comment on pull request #18434: URL: https://github.com/apache/flink/pull/18434#issuecomment-1018078894 ## CI report: * 9cb1fddc375c0b7583b529ebd83faa0ee700a983 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30267) * 972db53783b183c8f0f829b0510e599f140787af Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30962) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-26040) PrometheusReporterEndToEndITCase hang on azure pipeline
Yun Gao created FLINK-26040:
---
Summary: PrometheusReporterEndToEndITCase hang on azure pipeline
Key: FLINK-26040
URL: https://issues.apache.org/jira/browse/FLINK-26040
Project: Flink
Issue Type: Bug
Components: Runtime / Metrics
Affects Versions: 1.13.5
Reporter: Yun Gao
{code:java}
"main" #1 prio=5 os_prio=0 tid=0x7ffb1400b800 nid=0x9590b in Object.wait()
[0x7ffb199a8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x8a2fd4b8> (a java.lang.UNIXProcess)
at java.lang.Object.wait(Object.java:460)
at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:348)
at java.lang.UNIXProcess.waitFor(UNIXProcess.java:410)
- locked <0x8a2fd4b8> (a java.lang.UNIXProcess)
at
org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:134)
at
org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:115)
at
org.apache.flink.tests.util.flink.FlinkDistribution.startFlinkCluster(FlinkDistribution.java:119)
at
org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource.startCluster(LocalStandaloneFlinkResource.java:133)
at
org.apache.flink.metrics.prometheus.tests.PrometheusReporterEndToEndITCase.testReporter(PrometheusReporterEndToEndITCase.java:231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
at
org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30955=logs=08866332-78f7-59e4-4f7e-49a56faa3179=7f606211-1454-543c-70ab-c7a028a1ce8c=30397
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26039) Incorrect value getter in map unnest table function
Han created FLINK-26039:
---
Summary: Incorrect value getter in map unnest table function
Key: FLINK-26039
URL: https://issues.apache.org/jira/browse/FLINK-26039
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.14.3
Reporter: Han
Fix For: 1.15.0
Suppose we have a map field that needs to be expanded.
{code:java}
CREATE TABLE t (
id INT,
map_field MAP
) WITH (
-- ...
);
SELECT id, k, v FROM t, unnest(map_field) as A(k, v);{code}
We will get the following runtime exception:
{code:java}
Caused by: java.lang.ClassCastException:
org.apache.flink.table.data.binary.BinaryStringData cannot be cast to
java.lang.Integer
at
org.apache.flink.table.data.GenericRowData.getInt(GenericRowData.java:149)
at
org.apache.flink.table.data.utils.JoinedRowData.getInt(JoinedRowData.java:149)
at
org.apache.flink.table.data.RowData.lambda$createFieldGetter$245ca7d1$6(RowData.java:245)
at
org.apache.flink.table.data.RowData.lambda$createFieldGetter$25774257$1(RowData.java:296)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copyRowData(RowDataSerializer.java:170)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:131)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:48)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:80)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
at
org.apache.flink.table.runtime.util.StreamRecordCollector.collect(StreamRecordCollector.java:44)
at
org.apache.flink.table.runtime.collector.TableFunctionCollector.outputResult(TableFunctionCollector.java:68)
at StreamExecCorrelate$10$TableFunctionCollector$4.collect(Unknown Source)
at
org.apache.flink.table.runtime.collector.WrappingCollector.outputResult(WrappingCollector.java:39)
at
StreamExecCorrelate$10$TableFunctionResultConverterCollector$8.collect(Unknown
Source)
at
org.apache.flink.table.functions.TableFunction.collect(TableFunction.java:197)
at
org.apache.flink.table.runtime.functions.SqlUnnestUtils$MapUnnestTableFunction.eval(SqlUnnestUtils.java:169)
at StreamExecCorrelate$10.processElement(Unknown Source)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-26032) log job info in the ContextEnvironment
[
https://issues.apache.org/jira/browse/FLINK-26032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-26032:
---
Labels: pull-request-available (was: )
> log job info in the ContextEnvironment
> --
>
> Key: FLINK-26032
> URL: https://issues.apache.org/jira/browse/FLINK-26032
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission
>Reporter: Jing Ge
>Assignee: Jing Ge
>Priority: Minor
> Labels: pull-request-available
>
> In Flink codebase, the {{org.apache.flink.client.program.ContextEnvironment}}
> has a static
> [Logger|https://github.com/apache/flink/blob/7f9587c723057e2b6cbaf748181c8c80a7f6703d/flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java#L47]
> defined in the class, however, it doesn’t use it to print any logs. Instead,
> it prints logs with {{System.out}} and passes the Logger to
> {{ShutdownHookUtil.addShutdownHook}} and
> {{jobExecutionResultFuture.whenComplete}} for logging any hook errors. If
> customer integrated the CLI (‘FlinkYarnSessionCli’ in their case) into a
> multi-threaded program to submit jobs in parallel, does it lead to any logs
> missing/override/disorder problems?
> It is always helpful to log the status information during the job submit
> process.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26036) LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory timeout on azure
[
https://issues.apache.org/jira/browse/FLINK-26036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489317#comment-17489317
]
Yun Gao commented on FLINK-26036:
-
Perhaps cc [~trohrmann]~
> LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> timeout on azure
> ---
>
> Key: FLINK-26036
> URL: https://issues.apache.org/jira/browse/FLINK-26036
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 022-02-09T02:18:17.1827314Z Feb 09 02:18:14 [ERROR]
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> Time elapsed: 62.252 s <<< ERROR!
> 2022-02-09T02:18:17.1827940Z Feb 09 02:18:14
> java.util.concurrent.TimeoutException
> 2022-02-09T02:18:17.1828450Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> 2022-02-09T02:18:17.1829040Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2022-02-09T02:18:17.1829752Z Feb 09 02:18:14 at
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory(LocalRecoveryITCase.java:115)
> 2022-02-09T02:18:17.1830407Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-02-09T02:18:17.1830954Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-02-09T02:18:17.1831582Z Feb 09 02:18:14 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-09T02:18:17.1832135Z Feb 09 02:18:14 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-02-09T02:18:17.1832697Z Feb 09 02:18:14 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> 2022-02-09T02:18:17.1833566Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2022-02-09T02:18:17.1834394Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-02-09T02:18:17.1835125Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2022-02-09T02:18:17.1835875Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> 2022-02-09T02:18:17.1836565Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> 2022-02-09T02:18:17.1837294Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2022-02-09T02:18:17.1838007Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-02-09T02:18:17.1838743Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-02-09T02:18:17.1839499Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-02-09T02:18:17.1840224Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2022-02-09T02:18:17.1840952Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2022-02-09T02:18:17.1841616Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> 2022-02-09T02:18:17.1842257Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> 2022-02-09T02:18:17.1842951Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
> 2022-02-09T02:18:17.1843681Z Feb 09 02:18:14 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2022-02-09T02:18:17.1844782Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
> 2022-02-09T02:18:17.1845603Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
> 2022-02-09T02:18:17.1846375Z Feb 09 02:18:14 at
>
[GitHub] [flink] JingGe opened a new pull request #18677: [FLINK-26032][client] add log info for the status of job submission and execution result.
JingGe opened a new pull request #18677: URL: https://github.com/apache/flink/pull/18677 ## What is the purpose of the change Add log info for ContextEnvironment. ## Brief change log - Add log info of job submission - Add log info of job execution result ## Verifying this change This change is a trivial rework of adding log info without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26038) Support delay message on PulsarSink
[ https://issues.apache.org/jira/browse/FLINK-26038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufan Sheng updated FLINK-26038: Description: A delayed message is a type of message that would be sent to the consumer after waiting for a fixed interval. We should support this feature in both sources and sink API. > Support delay message on PulsarSink > --- > > Key: FLINK-26038 > URL: https://issues.apache.org/jira/browse/FLINK-26038 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Priority: Major > > A delayed message is a type of message that would be sent to the consumer > after waiting for a fixed interval. We should support this feature in both > sources and sink API. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25233) UpsertKafkaTableITCase.testAggregate fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489316#comment-17489316
]
Yun Gao commented on FLINK-25233:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30956=logs=c5612577-f1f7-5977-6ff6-7432788526f7=ffa8837a-b445-534e-cdf4-db364cf8235d
> UpsertKafkaTableITCase.testAggregate fails on AZP
> -
>
> Key: FLINK-25233
> URL: https://issues.apache.org/jira/browse/FLINK-25233
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> {{UpsertKafkaTableITCase.testAggregate}} fails on AZP with
> {code}
> 2021-12-09T01:41:49.8038402Z Dec 09 01:41:49 [ERROR]
> UpsertKafkaTableITCase.testAggregate Time elapsed: 90.624 s <<< ERROR!
> 2021-12-09T01:41:49.8039372Z Dec 09 01:41:49
> java.util.concurrent.ExecutionException:
> org.apache.flink.table.api.TableException: Failed to wait job finish
> 2021-12-09T01:41:49.8040303Z Dec 09 01:41:49 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2021-12-09T01:41:49.8040956Z Dec 09 01:41:49 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2021-12-09T01:41:49.8041862Z Dec 09 01:41:49 at
> org.apache.flink.table.api.internal.TableResultImpl.awaitInternal(TableResultImpl.java:118)
> 2021-12-09T01:41:49.8042939Z Dec 09 01:41:49 at
> org.apache.flink.table.api.internal.TableResultImpl.await(TableResultImpl.java:81)
> 2021-12-09T01:41:49.8044130Z Dec 09 01:41:49 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.wordCountToUpsertKafka(UpsertKafkaTableITCase.java:436)
> 2021-12-09T01:41:49.8045308Z Dec 09 01:41:49 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.testAggregate(UpsertKafkaTableITCase.java:79)
> 2021-12-09T01:41:49.8045940Z Dec 09 01:41:49 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-12-09T01:41:49.8052892Z Dec 09 01:41:49 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-12-09T01:41:49.8053812Z Dec 09 01:41:49 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-12-09T01:41:49.8054458Z Dec 09 01:41:49 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-12-09T01:41:49.8055027Z Dec 09 01:41:49 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2021-12-09T01:41:49.8055649Z Dec 09 01:41:49 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-12-09T01:41:49.8056644Z Dec 09 01:41:49 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2021-12-09T01:41:49.8057911Z Dec 09 01:41:49 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-12-09T01:41:49.8058858Z Dec 09 01:41:49 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2021-12-09T01:41:49.8059907Z Dec 09 01:41:49 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2021-12-09T01:41:49.8060871Z Dec 09 01:41:49 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-12-09T01:41:49.8061847Z Dec 09 01:41:49 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-12-09T01:41:49.8062898Z Dec 09 01:41:49 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-12-09T01:41:49.8063804Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-12-09T01:41:49.8064963Z Dec 09 01:41:49 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-12-09T01:41:49.8065992Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-12-09T01:41:49.8066940Z Dec 09 01:41:49 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-12-09T01:41:49.8067939Z Dec 09 01:41:49 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-12-09T01:41:49.8068904Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-12-09T01:41:49.8069837Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-12-09T01:41:49.8070715Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-12-09T01:41:49.8071587Z Dec 09 01:41:49 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-12-09T01:41:49.8072582Z Dec 09 01:41:49 at
>
[jira] [Updated] (FLINK-26038) Support delay message on PulsarSink
[ https://issues.apache.org/jira/browse/FLINK-26038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufan Sheng updated FLINK-26038: Summary: Support delay message on PulsarSink (was: Support delay message on Pulsar sink) > Support delay message on PulsarSink > --- > > Key: FLINK-26038 > URL: https://issues.apache.org/jira/browse/FLINK-26038 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26036) LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory timeout on azure
[
https://issues.apache.org/jira/browse/FLINK-26036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489315#comment-17489315
]
Yangze Guo commented on FLINK-26036:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30962=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=21108
> LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> timeout on azure
> ---
>
> Key: FLINK-26036
> URL: https://issues.apache.org/jira/browse/FLINK-26036
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 022-02-09T02:18:17.1827314Z Feb 09 02:18:14 [ERROR]
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> Time elapsed: 62.252 s <<< ERROR!
> 2022-02-09T02:18:17.1827940Z Feb 09 02:18:14
> java.util.concurrent.TimeoutException
> 2022-02-09T02:18:17.1828450Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> 2022-02-09T02:18:17.1829040Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2022-02-09T02:18:17.1829752Z Feb 09 02:18:14 at
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory(LocalRecoveryITCase.java:115)
> 2022-02-09T02:18:17.1830407Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-02-09T02:18:17.1830954Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-02-09T02:18:17.1831582Z Feb 09 02:18:14 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-09T02:18:17.1832135Z Feb 09 02:18:14 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-02-09T02:18:17.1832697Z Feb 09 02:18:14 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> 2022-02-09T02:18:17.1833566Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2022-02-09T02:18:17.1834394Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-02-09T02:18:17.1835125Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2022-02-09T02:18:17.1835875Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> 2022-02-09T02:18:17.1836565Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> 2022-02-09T02:18:17.1837294Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2022-02-09T02:18:17.1838007Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-02-09T02:18:17.1838743Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-02-09T02:18:17.1839499Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-02-09T02:18:17.1840224Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2022-02-09T02:18:17.1840952Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2022-02-09T02:18:17.1841616Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> 2022-02-09T02:18:17.1842257Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> 2022-02-09T02:18:17.1842951Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
> 2022-02-09T02:18:17.1843681Z Feb 09 02:18:14 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2022-02-09T02:18:17.1844782Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
> 2022-02-09T02:18:17.1845603Z Feb 09 02:18:14 at
>
[GitHub] [flink] KarmaGYZ commented on pull request #18434: [FLINK-25742][akka] Remove the serialization of rpc invocation at Fli…
KarmaGYZ commented on pull request #18434: URL: https://github.com/apache/flink/pull/18434#issuecomment-1033442692 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18676: [FLINK-26035][build][planner] Add table-planner-loader-helper module
flinkbot commented on pull request #18676: URL: https://github.com/apache/flink/pull/18676#issuecomment-1033442771 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit dad0e032399b3b422d20e018161c0030ca84d08e (Wed Feb 09 07:48:34 UTC 2022) **Warnings:** * **4 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18675: [FLINK-25491][table-planner] Fix bug: generated code for a large IN filter can't be compiled
flinkbot commented on pull request #18675: URL: https://github.com/apache/flink/pull/18675#issuecomment-1033442795 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d13e6f599e562fb155120cbc60f9a4a983356f2e (Wed Feb 09 07:48:35 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25940) pyflink/datastream/tests/test_data_stream.py::StreamingModeDataStreamTests::test_keyed_process_function_with_state failed on AZP
[
https://issues.apache.org/jira/browse/FLINK-25940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489314#comment-17489314
]
Yun Gao commented on FLINK-25940:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30956=logs=3e4dd1a2-fe2f-5e5d-a581-48087e718d53=b4612f28-e3b5-5853-8a8b-610ae894217a=24773
> pyflink/datastream/tests/test_data_stream.py::StreamingModeDataStreamTests::test_keyed_process_function_with_state
> failed on AZP
>
>
> Key: FLINK-25940
> URL: https://issues.apache.org/jira/browse/FLINK-25940
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> The test
> {{pyflink/datastream/tests/test_data_stream.py::StreamingModeDataStreamTests::test_keyed_process_function_with_state}}
> fails on AZP:
> {code}
> 2022-02-02T17:44:12.1898582Z Feb 02 17:44:12
> === FAILURES
> ===
> 2022-02-02T17:44:12.1899860Z Feb 02 17:44:12 _
> StreamingModeDataStreamTests.test_keyed_process_function_with_state __
> 2022-02-02T17:44:12.1900493Z Feb 02 17:44:12
> 2022-02-02T17:44:12.1901218Z Feb 02 17:44:12 self =
> testMethod=test_keyed_process_function_with_state>
> 2022-02-02T17:44:12.1901948Z Feb 02 17:44:12
> 2022-02-02T17:44:12.1902745Z Feb 02 17:44:12 def
> test_keyed_process_function_with_state(self):
> 2022-02-02T17:44:12.1903722Z Feb 02 17:44:12
> self.env.get_config().set_auto_watermark_interval(2000)
> 2022-02-02T17:44:12.1904473Z Feb 02 17:44:12
> self.env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
> 2022-02-02T17:44:12.1906780Z Feb 02 17:44:12 data_stream =
> self.env.from_collection([(1, 'hi', '1603708211000'),
> 2022-02-02T17:44:12.1908034Z Feb 02 17:44:12
>(2, 'hello', '1603708224000'),
> 2022-02-02T17:44:12.1909166Z Feb 02 17:44:12
>(3, 'hi', '1603708226000'),
> 2022-02-02T17:44:12.1910122Z Feb 02 17:44:12
>(4, 'hello', '1603708289000'),
> 2022-02-02T17:44:12.1911099Z Feb 02 17:44:12
>(5, 'hi', '1603708291000'),
> 2022-02-02T17:44:12.1912451Z Feb 02 17:44:12
>(6, 'hello', '1603708293000')],
> 2022-02-02T17:44:12.1913456Z Feb 02 17:44:12
> type_info=Types.ROW([Types.INT(), Types.STRING(),
> 2022-02-02T17:44:12.1914338Z Feb 02 17:44:12
>Types.STRING()]))
> 2022-02-02T17:44:12.1914811Z Feb 02 17:44:12
> 2022-02-02T17:44:12.1915317Z Feb 02 17:44:12 class
> MyTimestampAssigner(TimestampAssigner):
> 2022-02-02T17:44:12.1915724Z Feb 02 17:44:12
> 2022-02-02T17:44:12.1916782Z Feb 02 17:44:12 def
> extract_timestamp(self, value, record_timestamp) -> int:
> 2022-02-02T17:44:12.1917621Z Feb 02 17:44:12 return
> int(value[2])
> 2022-02-02T17:44:12.1918262Z Feb 02 17:44:12
> 2022-02-02T17:44:12.1918855Z Feb 02 17:44:12 class
> MyProcessFunction(KeyedProcessFunction):
> 2022-02-02T17:44:12.1919363Z Feb 02 17:44:12
> 2022-02-02T17:44:12.1919744Z Feb 02 17:44:12 def __init__(self):
> 2022-02-02T17:44:12.1920143Z Feb 02 17:44:12 self.value_state
> = None
> 2022-02-02T17:44:12.1920648Z Feb 02 17:44:12 self.list_state
> = None
> 2022-02-02T17:44:12.1921298Z Feb 02 17:44:12 self.map_state =
> None
> 2022-02-02T17:44:12.1921864Z Feb 02 17:44:12
> 2022-02-02T17:44:12.1922479Z Feb 02 17:44:12 def open(self,
> runtime_context: RuntimeContext):
> 2022-02-02T17:44:12.1923907Z Feb 02 17:44:12
> value_state_descriptor = ValueStateDescriptor('value_state', Types.INT())
> 2022-02-02T17:44:12.1924922Z Feb 02 17:44:12 self.value_state
> = runtime_context.get_state(value_state_descriptor)
> 2022-02-02T17:44:12.1925741Z Feb 02 17:44:12
> list_state_descriptor = ListStateDescriptor('list_state', Types.INT())
> 2022-02-02T17:44:12.1926482Z Feb 02 17:44:12 self.list_state
> = runtime_context.get_list_state(list_state_descriptor)
> 2022-02-02T17:44:12.1927465Z Feb 02 17:44:12
> map_state_descriptor = MapStateDescriptor('map_state', Types.INT(),
> Types.STRING())
> 2022-02-02T17:44:12.1927998Z Feb 02 17:44:12 state_ttl_config
> = StateTtlConfig \
[jira] [Created] (FLINK-26038) Support delay message on Pulsar sink
Yufan Sheng created FLINK-26038: --- Summary: Support delay message on Pulsar sink Key: FLINK-26038 URL: https://issues.apache.org/jira/browse/FLINK-26038 Project: Flink Issue Type: Sub-task Components: Connectors / Pulsar Reporter: Yufan Sheng -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18675: [FLINK-25491][table-planner] Fix bug: generated code for a large IN filter can't be compiled
flinkbot commented on pull request #18675: URL: https://github.com/apache/flink/pull/18675#issuecomment-1033442210 ## CI report: * d13e6f599e562fb155120cbc60f9a4a983356f2e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18672: [FLINK-25996][runtime] Introduce job property isDynamicGraph to ExecutionConfig
flinkbot edited a comment on pull request #18672: URL: https://github.com/apache/flink/pull/18672#issuecomment-1033272608 ## CI report: * 154cd56e0f3298e2190cffe947e2f283f5c7d2ec Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30961) * 684a97f9456209609f00381df489b6dc17e329f4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30965) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26036) LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory timeout on azure
[
https://issues.apache.org/jira/browse/FLINK-26036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489312#comment-17489312
]
Yun Gao commented on FLINK-26036:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30956=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=b78d9d30-509a-5cea-1fef-db7abaa325ae=23911
> LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> timeout on azure
> ---
>
> Key: FLINK-26036
> URL: https://issues.apache.org/jira/browse/FLINK-26036
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 022-02-09T02:18:17.1827314Z Feb 09 02:18:14 [ERROR]
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> Time elapsed: 62.252 s <<< ERROR!
> 2022-02-09T02:18:17.1827940Z Feb 09 02:18:14
> java.util.concurrent.TimeoutException
> 2022-02-09T02:18:17.1828450Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> 2022-02-09T02:18:17.1829040Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2022-02-09T02:18:17.1829752Z Feb 09 02:18:14 at
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory(LocalRecoveryITCase.java:115)
> 2022-02-09T02:18:17.1830407Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-02-09T02:18:17.1830954Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-02-09T02:18:17.1831582Z Feb 09 02:18:14 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-09T02:18:17.1832135Z Feb 09 02:18:14 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-02-09T02:18:17.1832697Z Feb 09 02:18:14 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> 2022-02-09T02:18:17.1833566Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2022-02-09T02:18:17.1834394Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-02-09T02:18:17.1835125Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2022-02-09T02:18:17.1835875Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> 2022-02-09T02:18:17.1836565Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> 2022-02-09T02:18:17.1837294Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2022-02-09T02:18:17.1838007Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-02-09T02:18:17.1838743Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-02-09T02:18:17.1839499Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-02-09T02:18:17.1840224Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2022-02-09T02:18:17.1840952Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2022-02-09T02:18:17.1841616Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> 2022-02-09T02:18:17.1842257Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> 2022-02-09T02:18:17.1842951Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
> 2022-02-09T02:18:17.1843681Z Feb 09 02:18:14 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2022-02-09T02:18:17.1844782Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
> 2022-02-09T02:18:17.1845603Z Feb 09 02:18:14 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) * 529674ae6b4a679fc4c5a45fd86f9e83919cb62d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25825) MySqlCatalogITCase fails on azure
[
https://issues.apache.org/jira/browse/FLINK-25825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489311#comment-17489311
]
Yun Gao commented on FLINK-25825:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30956=logs=e9af9cde-9a65-5281-a58e-2c8511d36983=c520d2c3-4d17-51f1-813b-4b0b74a0c307=14836
> MySqlCatalogITCase fails on azure
> -
>
> Key: FLINK-25825
> URL: https://issues.apache.org/jira/browse/FLINK-25825
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC, Table SQL / API
>Affects Versions: 1.15.0
>Reporter: Roman Khachatryan
>Assignee: RocMarshal
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30189=logs=e9af9cde-9a65-5281-a58e-2c8511d36983=c520d2c3-4d17-51f1-813b-4b0b74a0c307=13677
>
> {code}
> 2022-01-26T06:04:42.8019913Z Jan 26 06:04:42 [ERROR]
> org.apache.flink.connector.jdbc.catalog.MySqlCatalogITCase.testFullPath Time
> elapsed: 2.166 *s <<< FAILURE!
> 2022-01-26T06:04:42.8025522Z Jan 26 06:04:42 java.lang.AssertionError:
> expected: java.util.ArrayList<[+I[1, -1, 1, null, true, null, hello, 2021-0
> 8-04, 2021-08-04T01:54:16, -1, 1, -1.0, 1.0, enum2, -9.1, 9.1, -1, 1, -1, 1,
> \{"k1": "v1"}, null, col_longtext, null, -1, 1, col_mediumtext, -99, 9 9,
> -1.0, 1.0, set_ele1, -1, 1, col_text, 10:32:34, 2021-08-04T01:54:16,
> col_tinytext, -1, 1, null, col_varchar, 2021-08-04T01:54:16.463, 09:33:43,
> 2021-08-04T01:54:16.463, null], +I[2, -1, 1, null, true, null, hello,
> 2021-08-04, 2021-08-04T01:53:19, -1, 1, -1.0, 1.0, enum2, -9.1, 9.1, -1, 1,
> -1, 1, \{"k1": "v1"}, null, col_longtext, null, -1, 1, col_mediumtext, -99,
> 99, -1.0, 1.0, set_ele1,set_ele12, -1, 1, col_text, 10:32:34, 2021-08-
> 04T01:53:19, col_tinytext, -1, 1, null, col_varchar, 2021-08-04T01:53:19.098,
> 09:33:43, 2021-08-04T01:53:19.098, null]]> but was: java.util.ArrayL
> ist<[+I[1, -1, 1, null, true, null, hello, 2021-08-04, 2021-08-04T01:54:16,
> -1, 1, -1.0, 1.0, enum2, -9.1, 9.1, -1, 1, -1, 1, \{"k1": "v1"}, null,
> col_longtext, null, -1, 1, col_mediumtext, -99, 99, -1.0, 1.0, set_ele1, -1,
> 1, col_text, 10:32:34, 2021-08-04T01:54:16, col_tinytext, -1, 1, null ,
> col_varchar, 2021-08-04T01:54:16.463, 09:33:43, 2021-08-04T01:54:16.463,
> null], +I[2, -1, 1, null, true, null, hello, 2021-08-04, 2021-08-04T01:
> 53:19, -1, 1, -1.0, 1.0, enum2, -9.1, 9.1, -1, 1, -1, 1, \{"k1": "v1"}, null,
> col_longtext, null, -1, 1, col_mediumtext, -99, 99, -1.0, 1.0, set_el
> e1,set_ele12, -1, 1, col_text, 10:32:34, 2021-08-04T01:53:19, col_tinytext,
> -1, 1, null, col_varchar, 2021-08-04T01:53:19.098, 09:33:43, 2021-08-0
> 4T01:53:19.098, null]]>
> 2022-01-26T06:04:42.8029336Z Jan 26 06:04:42 at
> org.junit.Assert.fail(Assert.java:89)
> 2022-01-26T06:04:42.8029824Z Jan 26 06:04:42 at
> org.junit.Assert.failNotEquals(Assert.java:835)
> 2022-01-26T06:04:42.8030319Z Jan 26 06:04:42 at
> org.junit.Assert.assertEquals(Assert.java:120)
> 2022-01-26T06:04:42.8030815Z Jan 26 06:04:42 at
> org.junit.Assert.assertEquals(Assert.java:146)
> 2022-01-26T06:04:42.8031419Z Jan 26 06:04:42 at
> org.apache.flink.connector.jdbc.catalog.MySqlCatalogITCase.testFullPath(MySqlCatalogITCase.java*:306)
> {code}
>
> {code}
> 2022-01-26T06:04:43.2899378Z Jan 26 06:04:43 [ERROR] Failures:
> 2022-01-26T06:04:43.2907942Z Jan 26 06:04:43 [ERROR]
> MySqlCatalogITCase.testFullPath:306 expected: java.util.ArrayList<[+I[1, -1,
> 1, null, true,
> 2022-01-26T06:04:43.2914065Z Jan 26 06:04:43 [ERROR]
> MySqlCatalogITCase.testGetTable:253 expected:<(
> 2022-01-26T06:04:43.2983567Z Jan 26 06:04:43 [ERROR]
> MySqlCatalogITCase.testSelectToInsert:323 expected:
> java.util.ArrayList<[+I[1, -1, 1, null,
> 2022-01-26T06:04:43.2997373Z Jan 26 06:04:43 [ERROR]
> MySqlCatalogITCase.testWithoutCatalog:291 expected:
> java.util.ArrayList<[+I[1, -1, 1, null,
> 2022-01-26T06:04:43.3010450Z Jan 26 06:04:43 [ERROR]
> MySqlCatalogITCase.testWithoutCatalogDB:278 expected:
> java.util.ArrayList<[+I[1, -1, 1, nul
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26036) LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory timeout on azure
[
https://issues.apache.org/jira/browse/FLINK-26036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489310#comment-17489310
]
Yun Gao commented on FLINK-26036:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30956=logs=a549b384-c55a-52c0-c451-00e0477ab6db=eef5922c-08d9-5ba3-7299-8393476594e7=31047
> LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> timeout on azure
> ---
>
> Key: FLINK-26036
> URL: https://issues.apache.org/jira/browse/FLINK-26036
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> 022-02-09T02:18:17.1827314Z Feb 09 02:18:14 [ERROR]
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> Time elapsed: 62.252 s <<< ERROR!
> 2022-02-09T02:18:17.1827940Z Feb 09 02:18:14
> java.util.concurrent.TimeoutException
> 2022-02-09T02:18:17.1828450Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> 2022-02-09T02:18:17.1829040Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2022-02-09T02:18:17.1829752Z Feb 09 02:18:14 at
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory(LocalRecoveryITCase.java:115)
> 2022-02-09T02:18:17.1830407Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-02-09T02:18:17.1830954Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-02-09T02:18:17.1831582Z Feb 09 02:18:14 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-09T02:18:17.1832135Z Feb 09 02:18:14 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-02-09T02:18:17.1832697Z Feb 09 02:18:14 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> 2022-02-09T02:18:17.1833566Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2022-02-09T02:18:17.1834394Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-02-09T02:18:17.1835125Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2022-02-09T02:18:17.1835875Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> 2022-02-09T02:18:17.1836565Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> 2022-02-09T02:18:17.1837294Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2022-02-09T02:18:17.1838007Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-02-09T02:18:17.1838743Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-02-09T02:18:17.1839499Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-02-09T02:18:17.1840224Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2022-02-09T02:18:17.1840952Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2022-02-09T02:18:17.1841616Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> 2022-02-09T02:18:17.1842257Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> 2022-02-09T02:18:17.1842951Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
> 2022-02-09T02:18:17.1843681Z Feb 09 02:18:14 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2022-02-09T02:18:17.1844782Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
> 2022-02-09T02:18:17.1845603Z Feb 09 02:18:14 at
>
[jira] [Commented] (FLINK-25937) SQL Client end-to-end test e2e fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-25937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489308#comment-17489308
]
Yun Gao commented on FLINK-25937:
-
SQL Client end-to-end test:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30956=logs=fb37c667-81b7-5c22-dd91-846535e99a97=39a035c3-c65e-573c-fb66-104c66c28912=5005
> SQL Client end-to-end test e2e fails on AZP
> ---
>
> Key: FLINK-25937
> URL: https://issues.apache.org/jira/browse/FLINK-25937
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Table SQL / API
>Affects Versions: 1.15.0
>Reporter: Till Rohrmann
>Assignee: David Morávek
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> The {{SQL Client end-to-end test}} e2e tests fails on AZP when using the
> {{AdaptiveScheduler}} because the scheduler expects that the parallelism is
> set for all vertices:
> {code}
> Feb 03 03:45:13 org.apache.flink.runtime.client.JobInitializationException:
> Could not start the JobMaster.
> Feb 03 03:45:13 at
> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
> Feb 03 03:45:13 at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> Feb 03 03:45:13 at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> Feb 03 03:45:13 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> Feb 03 03:45:13 at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1609)
> Feb 03 03:45:13 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> Feb 03 03:45:13 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> Feb 03 03:45:13 at java.lang.Thread.run(Thread.java:748)
> Feb 03 03:45:13 Caused by: java.util.concurrent.CompletionException:
> java.lang.IllegalStateException: The adaptive scheduler expects the
> parallelism being set for each JobVertex (violated JobVertex:
> f74b775b58627a33e46b8c155b320255).
> Feb 03 03:45:13 at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> Feb 03 03:45:13 at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> Feb 03 03:45:13 at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
> Feb 03 03:45:13 ... 3 more
> Feb 03 03:45:13 Caused by: java.lang.IllegalStateException: The adaptive
> scheduler expects the parallelism being set for each JobVertex (violated
> JobVertex: f74b775b58627a33e46b8c155b320255).
> Feb 03 03:45:13 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:215)
> Feb 03 03:45:13 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.assertPreconditions(AdaptiveScheduler.java:296)
> Feb 03 03:45:13 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.(AdaptiveScheduler.java:230)
> Feb 03 03:45:13 at
> org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerFactory.createInstance(AdaptiveSchedulerFactory.java:122)
> Feb 03 03:45:13 at
> org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:115)
> Feb 03 03:45:13 at
> org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:345)
> Feb 03 03:45:13 at
> org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:322)
> Feb 03 03:45:13 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:106)
> Feb 03 03:45:13 at
> org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:94)
> Feb 03 03:45:13 at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> Feb 03 03:45:13 at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> Feb 03 03:45:13 ... 3 more
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30662=logs=fb37c667-81b7-5c22-dd91-846535e99a97=39a035c3-c65e-573c-fb66-104c66c28912=5782
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-26036) LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory timeout on azure
[
https://issues.apache.org/jira/browse/FLINK-26036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-26036:
Priority: Critical (was: Major)
> LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> timeout on azure
> ---
>
> Key: FLINK-26036
> URL: https://issues.apache.org/jira/browse/FLINK-26036
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 022-02-09T02:18:17.1827314Z Feb 09 02:18:14 [ERROR]
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
> Time elapsed: 62.252 s <<< ERROR!
> 2022-02-09T02:18:17.1827940Z Feb 09 02:18:14
> java.util.concurrent.TimeoutException
> 2022-02-09T02:18:17.1828450Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> 2022-02-09T02:18:17.1829040Z Feb 09 02:18:14 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2022-02-09T02:18:17.1829752Z Feb 09 02:18:14 at
> org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory(LocalRecoveryITCase.java:115)
> 2022-02-09T02:18:17.1830407Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-02-09T02:18:17.1830954Z Feb 09 02:18:14 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-02-09T02:18:17.1831582Z Feb 09 02:18:14 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-09T02:18:17.1832135Z Feb 09 02:18:14 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-02-09T02:18:17.1832697Z Feb 09 02:18:14 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> 2022-02-09T02:18:17.1833566Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2022-02-09T02:18:17.1834394Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-02-09T02:18:17.1835125Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2022-02-09T02:18:17.1835875Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> 2022-02-09T02:18:17.1836565Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> 2022-02-09T02:18:17.1837294Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2022-02-09T02:18:17.1838007Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-02-09T02:18:17.1838743Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-02-09T02:18:17.1839499Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-02-09T02:18:17.1840224Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2022-02-09T02:18:17.1840952Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2022-02-09T02:18:17.1841616Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> 2022-02-09T02:18:17.1842257Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> 2022-02-09T02:18:17.1842951Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
> 2022-02-09T02:18:17.1843681Z Feb 09 02:18:14 at
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
> 2022-02-09T02:18:17.1844782Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
> 2022-02-09T02:18:17.1845603Z Feb 09 02:18:14 at
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
> 2022-02-09T02:18:17.1846375Z Feb 09 02:18:14 at
>
[jira] [Created] (FLINK-26037) TaskManagerRunnerTest JVM crash with exit code 239
Yun Gao created FLINK-26037:
---
Summary: TaskManagerRunnerTest JVM crash with exit code 239
Key: FLINK-26037
URL: https://issues.apache.org/jira/browse/FLINK-26037
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Yun Gao
{code:java}
Feb 09 02:22:52 [ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on
project flink-runtime: There are test failures.
Feb 09 02:22:52 [ERROR]
Feb 09 02:22:52 [ERROR] Please refer to
/__w/2/s/flink-runtime/target/surefire-reports for the individual test results.
Feb 09 02:22:52 [ERROR] Please refer to dump files (if any exist) [date].dump,
[date]-jvmRun[N].dump and [date].dumpstream.
Feb 09 02:22:52 [ERROR] ExecutionException The forked VM terminated without
properly saying goodbye. VM crash or System.exit called?
Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
-Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar
/__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar
/__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2
surefire8313405181432833392tmp surefire_2318122906826894495431tmp
Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
Feb 09 02:22:52 [ERROR] Process Exit Code: 239
Feb 09 02:22:52 [ERROR] Crashed tests:
Feb 09 02:22:52 [ERROR]
org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
Feb 09 02:22:52 [ERROR]
org.apache.maven.surefire.booter.SurefireBooterForkException:
ExecutionException The forked VM terminated without properly saying goodbye. VM
crash or System.exit called?
Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
-Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar
/__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar
/__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2
surefire8313405181432833392tmp surefire_2318122906826894495431tmp
Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
Feb 09 02:22:52 [ERROR] Process Exit Code: 239
Feb 09 02:22:52 [ERROR] Crashed tests:
Feb 09 02:22:52 [ERROR]
org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
Feb 09 02:22:52 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
Feb 09 02:22:52 [ERROR] at
org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30956=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7=9164
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-26035) Rework loader-bundle into separate module
[ https://issues.apache.org/jira/browse/FLINK-26035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26035: --- Labels: pull-request-available (was: ) > Rework loader-bundle into separate module > - > > Key: FLINK-26035 > URL: https://issues.apache.org/jira/browse/FLINK-26035 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Table SQL / Planner >Affects Versions: 1.15.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > The flink-table-planner currently creates 2 artifacts. 1 jar containing the > planner and various dependencies for the cases where the planner is used > directly, and another jar that additionally bundles scala for cases where the > loader is used. > The latter artifact is purely an intermediate build artifact, and as such we > usually wouldn't want to publish it. This is particularly important because > this jar doesn't have a correct NOTICE, and having different NOTICE files for > different artifacts is surprisingly tricky. > We should just rework this into a separate module. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25491) Code generation: init method exceeds 64 KB when there is a large IN filter with Table API
[
https://issues.apache.org/jira/browse/FLINK-25491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-25491:
---
Labels: pull-request-available (was: )
> Code generation: init method exceeds 64 KB when there is a large IN filter
> with Table API
> -
>
> Key: FLINK-25491
> URL: https://issues.apache.org/jira/browse/FLINK-25491
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API, Table SQL / Runtime
>Affects Versions: 1.14.2
>Reporter: Daniel Cheng
>Assignee: Caizhi Weng
>Priority: Major
> Labels: pull-request-available
>
> When using Table API (Blink planner), if you are filtering using an IN filter
> with a lot of values, e.g. {{$(colName).in()}}, it will result
> in the error
>
> {{Code of method "(...)V" of class "BatchExecCal$3006" grows beyond 64
> KB}}
>
> The size of the init method mainly comes from the below method, which
> initializes the hash set with all the values in the filter.
> addReusableHashSet in CodeGeneratorContext.scala
> [https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/CodeGeneratorContext.scala#L409]
>
> This affects older versions as well, with 1.14.2 being the latest version
> that exhibits this issue.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] zentol opened a new pull request #18676: [FLINK-26035][build][planner] Add table-planner-loader-helper module
zentol opened a new pull request #18676: URL: https://github.com/apache/flink/pull/18676 The flink-table-planner currently creates 2 artifacts. 1 jar containing the planner and various dependencies for the cases where the planner is used directly, and another jar that additionally bundles scala for cases where the loader is used. The latter artifact is purely an intermediate build artifact, and as such we usually wouldn't want to publish it. This is particularly important because this jar doesn't have a correct NOTICE, and having different NOTICE files for different artifacts is surprisingly tricky. We should just rework this into a separate module. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tsreaper opened a new pull request #18675: [FLINK-25491][table-planner] Fix bug: generated code for a large IN filter can't be compiled
tsreaper opened a new pull request #18675: URL: https://github.com/apache/flink/pull/18675 ## What is the purpose of the change Although we've introduced a dedicated code splitter module in FLINK-23007 there are still some corner cases. For example, current code splitter cannot handle long constructor code. `IN` filters with a constant set will generate an operator which constructs the set in the class constructor, so for a large IN filter its generator code may become too long to be compiled. This PR fixes this issue by moving the code for set constructing to a dedicated method. I'm not going to handle long constructor code for the code splitter because it is sort of hard to pick out which part of the constructor code can be split out. For example, `final` class members must be assigned in the constructor, not in other methods, so we just can't move all constructor code into another method. ## Brief change log - Fix bug: generated code for a large IN filter can't be compiled ## Verifying this change This change added tests and can be verified by running `CodeSplitITCase#testManyIns`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-26036) LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory timeout on azure
Yun Gao created FLINK-26036:
---
Summary:
LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
timeout on azure
Key: FLINK-26036
URL: https://issues.apache.org/jira/browse/FLINK-26036
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.15.0
Reporter: Yun Gao
{code:java}
022-02-09T02:18:17.1827314Z Feb 09 02:18:14 [ERROR]
org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory
Time elapsed: 62.252 s <<< ERROR!
2022-02-09T02:18:17.1827940Z Feb 09 02:18:14
java.util.concurrent.TimeoutException
2022-02-09T02:18:17.1828450Z Feb 09 02:18:14at
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
2022-02-09T02:18:17.1829040Z Feb 09 02:18:14at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
2022-02-09T02:18:17.1829752Z Feb 09 02:18:14at
org.apache.flink.test.recovery.LocalRecoveryITCase.testRecoverLocallyFromProcessCrashWithWorkingDirectory(LocalRecoveryITCase.java:115)
2022-02-09T02:18:17.1830407Z Feb 09 02:18:14at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-02-09T02:18:17.1830954Z Feb 09 02:18:14at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022-02-09T02:18:17.1831582Z Feb 09 02:18:14at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-02-09T02:18:17.1832135Z Feb 09 02:18:14at
java.lang.reflect.Method.invoke(Method.java:498)
2022-02-09T02:18:17.1832697Z Feb 09 02:18:14at
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
2022-02-09T02:18:17.1833566Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
2022-02-09T02:18:17.1834394Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
2022-02-09T02:18:17.1835125Z Feb 09 02:18:14at
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
2022-02-09T02:18:17.1835875Z Feb 09 02:18:14at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
2022-02-09T02:18:17.1836565Z Feb 09 02:18:14at
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
2022-02-09T02:18:17.1837294Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
2022-02-09T02:18:17.1838007Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
2022-02-09T02:18:17.1838743Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
2022-02-09T02:18:17.1839499Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
2022-02-09T02:18:17.1840224Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
2022-02-09T02:18:17.1840952Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
2022-02-09T02:18:17.1841616Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
2022-02-09T02:18:17.1842257Z Feb 09 02:18:14at
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
2022-02-09T02:18:17.1842951Z Feb 09 02:18:14at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:214)
2022-02-09T02:18:17.1843681Z Feb 09 02:18:14at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2022-02-09T02:18:17.1844782Z Feb 09 02:18:14at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:210)
2022-02-09T02:18:17.1845603Z Feb 09 02:18:14at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
2022-02-09T02:18:17.1846375Z Feb 09 02:18:14at
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
2022-02-09T02:18:17.1847084Z Feb 09 02:18:14at
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
2022-02-09T02:18:17.1847785Z Feb 09 02:18:14at
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
2022-02-09T02:18:17.1848490Z Feb 09 02:18:14
[jira] [Created] (FLINK-26035) Rework loader-bundle into separate module
Chesnay Schepler created FLINK-26035: Summary: Rework loader-bundle into separate module Key: FLINK-26035 URL: https://issues.apache.org/jira/browse/FLINK-26035 Project: Flink Issue Type: Technical Debt Components: Build System, Table SQL / Planner Affects Versions: 1.15.0 Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.15.0 The flink-table-planner currently creates 2 artifacts. 1 jar containing the planner and various dependencies for the cases where the planner is used directly, and another jar that additionally bundles scala for cases where the loader is used. The latter artifact is purely an intermediate build artifact, and as such we usually wouldn't want to publish it. This is particularly important because this jar doesn't have a correct NOTICE, and having different NOTICE files for different artifacts is surprisingly tricky. We should just rework this into a separate module. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18603: [FLINK-25610][connector/firehose] Adding table api base for kinesis firehose
flinkbot edited a comment on pull request #18603: URL: https://github.com/apache/flink/pull/18603#issuecomment-1027853740 ## CI report: * 8b6adf5e05234f15048390cae6f97355d286944d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30972) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-26027) Add FLIP-33 metrics to new PulsarSink
[ https://issues.apache.org/jira/browse/FLINK-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26027: -- Assignee: Yufei Zhang > Add FLIP-33 metrics to new PulsarSink > - > > Key: FLINK-26027 > URL: https://issues.apache.org/jira/browse/FLINK-26027 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufei Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26028) Write documentation for new PulsarSink
[ https://issues.apache.org/jira/browse/FLINK-26028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26028: -- Assignee: Yufei Zhang > Write documentation for new PulsarSink > -- > > Key: FLINK-26028 > URL: https://issues.apache.org/jira/browse/FLINK-26028 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufei Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26027) Add FLIP-33 metrics to new PulsarSink
[ https://issues.apache.org/jira/browse/FLINK-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26027: -- Assignee: (was: Yufei Gu) > Add FLIP-33 metrics to new PulsarSink > - > > Key: FLINK-26027 > URL: https://issues.apache.org/jira/browse/FLINK-26027 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26022) Implement at-least-once and exactly-once Pulsar Sink
[ https://issues.apache.org/jira/browse/FLINK-26022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26022: -- Assignee: Yufan Sheng > Implement at-least-once and exactly-once Pulsar Sink > > > Key: FLINK-26022 > URL: https://issues.apache.org/jira/browse/FLINK-26022 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > > Support both three types of DeliveryGuarantee in Pulsar sink. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26027) Add FLIP-33 metrics to new PulsarSink
[ https://issues.apache.org/jira/browse/FLINK-26027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26027: -- Assignee: Yufei Gu > Add FLIP-33 metrics to new PulsarSink > - > > Key: FLINK-26027 > URL: https://issues.apache.org/jira/browse/FLINK-26027 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufei Gu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26025) Replace MockPulsar with new Pulsar test tools based on PulsarStandalone
[ https://issues.apache.org/jira/browse/FLINK-26025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26025: -- Assignee: Yufan Sheng > Replace MockPulsar with new Pulsar test tools based on PulsarStandalone > --- > > Key: FLINK-26025 > URL: https://issues.apache.org/jira/browse/FLINK-26025 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > > The old Pulsar connector tests are based on a mocked Pulsar broker which is > kinda wired in some behavior. The transaction isn't supported in this mocked > Pulsar. So we have to use PulsarStandalone directly. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26024) Create a PulsarSerializationSchema for better records serialization
[ https://issues.apache.org/jira/browse/FLINK-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26024: -- Assignee: Yufan Sheng > Create a PulsarSerializationSchema for better records serialization > --- > > Key: FLINK-26024 > URL: https://issues.apache.org/jira/browse/FLINK-26024 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26026) Test FLIP-191 PulsarSink
[ https://issues.apache.org/jira/browse/FLINK-26026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26026: -- Assignee: Yufan Sheng > Test FLIP-191 PulsarSink > > > Key: FLINK-26026 > URL: https://issues.apache.org/jira/browse/FLINK-26026 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-26023) Create a Pulsar sink config model for matching ProducerConfigurationData
[ https://issues.apache.org/jira/browse/FLINK-26023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26023: -- Assignee: Yufan Sheng > Create a Pulsar sink config model for matching ProducerConfigurationData > > > Key: FLINK-26023 > URL: https://issues.apache.org/jira/browse/FLINK-26023 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18658: [FLINK-26002][tests] Added RestoreUpgradedJobITCase for checking situ…
flinkbot edited a comment on pull request #18658: URL: https://github.com/apache/flink/pull/18658#issuecomment-1032365173 ## CI report: * 753ba1b6b5423fd899393f243ac7ab35587aa91c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30934) * 4d975e55579e651490ef78eddad73a1f12914bc7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30975) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-26021) Pulsar topic deduplicated in both sink and source connector
[ https://issues.apache.org/jira/browse/FLINK-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26021: -- Assignee: Yufan Sheng > Pulsar topic deduplicated in both sink and source connector > --- > > Key: FLINK-26021 > URL: https://issues.apache.org/jira/browse/FLINK-26021 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > > Both topics and partitions are regarded as topics in Pulsar. We have to make > the topic configuration more robust for deduplicating the partitions and > topics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26030) Set FLINK_LIB_DIR to 'lib' under working dir in YARN containers
[
https://issues.apache.org/jira/browse/FLINK-26030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489303#comment-17489303
]
Yang Wang commented on FLINK-26030:
---
{{ClusterEntrypointUtils#tryFindUserLibDirectory}} will return a wrong usr lib
if environment {{FLINK_LIB_DIR}} is pre-configured in the YARN cluster. So I
think this ticket is a minor bug, not an improvement. Right?
> Set FLINK_LIB_DIR to 'lib' under working dir in YARN containers
> ---
>
> Key: FLINK-26030
> URL: https://issues.apache.org/jira/browse/FLINK-26030
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Biao Geng
>Priority: Major
>
> Currently, we utilize
> {{org.apache.flink.runtime.entrypoint.ClusterEntrypointUtils#tryFindUserLibDirectory}}
> to locate usrlib in both flink client and cluster side.
> This method relies on the value of environment variable {{FLINK_LIB_DIR}} to
> find the {{usrlib}}.
> It makes sense in client side since in {{bin/config.sh}}, {{FLINK_LIB_DIR}}
> will be set by default(i.e. {{FLINK_HOME/lib}} if not exists. But in YARN
> cluster's containers, when we want to reuse this method to find {{usrlib}},
> as the YARN usually starts the process using commands like
> bq. /bin/bash -c /usr/lib/jvm/java-1.8.0/bin/java -Xmx1073741824
> -Xms1073741824
> -XX:MaxMetaspaceSize=268435456org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint
> -D jobmanager.memory.off-heap.size=134217728b -D
> jobmanager.memory.jvm-overhead.min=201326592b -D
> jobmanager.memory.jvm-metaspace.size=268435456b -D
> jobmanager.memory.heap.size=1073741824b -D
> jobmanager.memory.jvm-overhead.max=201326592b ...
> {{FLINK_LIB_DIR}} is not guaranteed to be set in such case. Current codes
> will use current working dir to locate the {{usrlib}} which is correct in
> most cases. But bad things can happen if the machine which the YARN container
> resides in has already set {{FLINK_LIB_DIR}} to a different folder. In that
> case, codes will try to find {{usrlib}} in a undesired place.
> One possible solution would be overriding the {{FLINK_LIB_DIR}} in YARN
> container env to the {{lib}} dir under YARN's workding dir.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-26020) Unified Pulsar Connector config model
[ https://issues.apache.org/jira/browse/FLINK-26020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-26020: -- Assignee: Yufan Sheng > Unified Pulsar Connector config model > - > > Key: FLINK-26020 > URL: https://issues.apache.org/jira/browse/FLINK-26020 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Pulsar >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > > PulsarClient has built-in config model named ClientConfigurationData, > ConsumerConfigurationData and ProducerConfigurationData. We don't want to > expose all the config options. And some config options could conflict with > each other. > We decide to design a new config model based on Flink's Configuration. Which > could provide type checks and better integration with Flink Table. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) * 529674ae6b4a679fc4c5a45fd86f9e83919cb62d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
flinkbot edited a comment on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1012981724 ## CI report: * c6b8f500af6adadfa82d1ee404e18c6ec78ece04 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30959) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #18642: [FLINK-25572][connectors/filesystem] Update File Sink to use decomposed interfaces.
gaoyunhaii commented on pull request #18642: URL: https://github.com/apache/flink/pull/18642#issuecomment-1033434323 Thanks @pltbkd for the updating! LGTM. For the fix I think it could be put as a hot fix in a separate commit in this PR. I'll change it when merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18658: [FLINK-26002][tests] Added RestoreUpgradedJobITCase for checking situ…
flinkbot edited a comment on pull request #18658: URL: https://github.com/apache/flink/pull/18658#issuecomment-1032365173 ## CI report: * 753ba1b6b5423fd899393f243ac7ab35587aa91c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30934) * 4d975e55579e651490ef78eddad73a1f12914bc7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18657: [FLINK-26001][avro] Implement ProjectableDecodingFormat for avro BulkDecodingFormat
flinkbot edited a comment on pull request #18657: URL: https://github.com/apache/flink/pull/18657#issuecomment-1032365045 ## CI report: * c3a56aca00474d59d326645d489dc5fccb67f2ad Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30960) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) * b398b5b5c922f18680c159b0051e98d1961b2518 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
flinkbot edited a comment on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033040800 ## CI report: * d025b76c02f35a4c6e56311a7dfed49b5d0dc9d6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30953) * c09bd68f357ba6dc0243892a4cbde5b0e4da05fe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30974) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17452: [FLINK-20732][connector/pulsar] Introduction of Pulsar Sink
flinkbot edited a comment on pull request #17452: URL: https://github.com/apache/flink/pull/17452#issuecomment-940136217 ## CI report: * 68b185ec99cc4db97323f2b5b09b4d17e09ac80d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30796) * 50c4d3ae81692fd5d7fb1d07758da8cc0aca0234 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30973) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-26034) Add maven wapper for flink
Aiden Gong created FLINK-26034: -- Summary: Add maven wapper for flink Key: FLINK-26034 URL: https://issues.apache.org/jira/browse/FLINK-26034 Project: Flink Issue Type: Improvement Components: Build System Affects Versions: 1.15.0 Reporter: Aiden Gong Fix For: 1.15.0 Idea just support this feature now. It is very helpful for us. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
flinkbot edited a comment on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033040800 ## CI report: * d025b76c02f35a4c6e56311a7dfed49b5d0dc9d6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30953) * c09bd68f357ba6dc0243892a4cbde5b0e4da05fe UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18603: [FLINK-25610][connector/firehose] Adding table api base for kinesis firehose
flinkbot edited a comment on pull request #18603: URL: https://github.com/apache/flink/pull/18603#issuecomment-1027853740 ## CI report: * 9c269c64ad7e0cb3e24dbaf1b9e5da3d5df5a6cd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30723) * 8b6adf5e05234f15048390cae6f97355d286944d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30972) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) * b398b5b5c922f18680c159b0051e98d1961b2518 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17452: [FLINK-20732][connector/pulsar] Introduction of Pulsar Sink
flinkbot edited a comment on pull request #17452: URL: https://github.com/apache/flink/pull/17452#issuecomment-940136217 ## CI report: * 68b185ec99cc4db97323f2b5b09b4d17e09ac80d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30796) * 50c4d3ae81692fd5d7fb1d07758da8cc0aca0234 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26033) In KafkaConnector, when 'sink.partitioner' is configured as 'round-robin', it does not take effect
[
https://issues.apache.org/jira/browse/FLINK-26033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489298#comment-17489298
]
shizhengchao commented on FLINK-26033:
--
[~jark] [~libenchao] Can you check this question?
> In KafkaConnector, when 'sink.partitioner' is configured as 'round-robin', it
> does not take effect
> --
>
> Key: FLINK-26033
> URL: https://issues.apache.org/jira/browse/FLINK-26033
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.3, 1.14.3
>Reporter: shizhengchao
>Priority: Major
>
> In KafkaConnector, when 'sink.partitioner' is configured as 'round-robin', it
> does not take effect. Flink treats 'default' and 'round-robin' as the same
> strategy.
> {code:java}
> //代码占位符
> public static Optional>
> getFlinkKafkaPartitioner(
> ReadableConfig tableOptions, ClassLoader classLoader) {
> return tableOptions
> .getOptional(SINK_PARTITIONER)
> .flatMap(
> (String partitioner) -> {
> switch (partitioner) {
> case SINK_PARTITIONER_VALUE_FIXED:
> return Optional.of(new
> FlinkFixedPartitioner<>());
> case SINK_PARTITIONER_VALUE_DEFAULT:
> case SINK_PARTITIONER_VALUE_ROUND_ROBIN:
> return Optional.empty();
> // Default fallback to full class name of the
> partitioner.
> default:
> return Optional.of(
> initializePartitioner(partitioner,
> classLoader));
> }
> });
> } {code}
> They both use kafka's default partitioner, but the actual There are two
> scenarios for the partition on DefaultPartitioner:
> 1. Random when there is no key
> 2. When there is a key, take the modulo according to the key
> {code:java}
> // org.apache.kafka.clients.producer.internals.DefaultPartitioner
> public int partition(String topic, Object key, byte[] keyBytes, Object value,
> byte[] valueBytes, Cluster cluster) {
> if (keyBytes == null) {
> // Random when there is no key
> return stickyPartitionCache.partition(topic, cluster);
> }
> List partitions = cluster.partitionsForTopic(topic);
> int numPartitions = partitions.size();
> // hash the keyBytes to choose a partition
> return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
> } {code}
> Therefore, KafkaConnector does not have a round-robin strategy.But we can
> borrow from kafka's RoundRobinPartitioner
> {code:java}
> //代码占位符
> public class RoundRobinPartitioner implements Partitioner {
> private final ConcurrentMap topicCounterMap = new
> ConcurrentHashMap<>();
> public void configure(Map configs) {}
> /**
> * Compute the partition for the given record.
> *
> * @param topic The topic name
> * @param key The key to partition on (or null if no key)
> * @param keyBytes serialized key to partition on (or null if no key)
> * @param value The value to partition on or null
> * @param valueBytes serialized value to partition on or null
> * @param cluster The current cluster metadata
> */
> @Override
> public int partition(String topic, Object key, byte[] keyBytes, Object
> value, byte[] valueBytes, Cluster cluster) {
> List partitions = cluster.partitionsForTopic(topic);
> int numPartitions = partitions.size();
> int nextValue = nextValue(topic);
> List availablePartitions =
> cluster.availablePartitionsForTopic(topic);
> if (!availablePartitions.isEmpty()) {
> int part = Utils.toPositive(nextValue) %
> availablePartitions.size();
> return availablePartitions.get(part).partition();
> } else {
> // no partitions are available, give a non-available partition
> return Utils.toPositive(nextValue) % numPartitions;
> }
> }
> private int nextValue(String topic) {
> AtomicInteger counter = topicCounterMap.computeIfAbsent(topic, k -> {
> return new AtomicInteger(0);
> });
> return counter.getAndIncrement();
> }
> public void close() {}
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-25992) JobDispatcherITCase.testRecoverFromCheckpointAfterLosingAndRegainingLeadership fails on azure
[
https://issues.apache.org/jira/browse/FLINK-25992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489297#comment-17489297
]
Liu commented on FLINK-25992:
-
[~roman] I am afraid that the reason is still uncertain from the log. From the
log, the processing is as following:
# When the first checkpoint expired, the UnsupportedOperationException is
thrown and the job restarts.
# The job is running and checkpoint 2 is done.
# The leadership changes and the job restores from checkpoint 2.
# We check that the restored checkpoint is not null.
I reproduce the processing in my local host but the test is still ok. So the
UnsupportedOperationException will not cause the test fail. Other two things
can be optimized as following:
# Implement the method notifyCheckpointAbortAsync.
# Tune the checkpoint timeout to 1000 to avoid the failed checkpoint.
But I am not sure whether the optimization can resolve the issue.
> JobDispatcherITCase.testRecoverFromCheckpointAfterLosingAndRegainingLeadership
> fails on azure
> -
>
> Key: FLINK-25992
> URL: https://issues.apache.org/jira/browse/FLINK-25992
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.15.0
>Reporter: Roman Khachatryan
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0
>
> Attachments: mvn-2.log
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=30871=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=9154
> {code}
> 19:41:35,515 [flink-akka.actor.default-dispatcher-9] WARN
> org.apache.flink.runtime.taskmanager.Task[] - jobVertex
> (1/1)#0 (7efdea21f5f95490e02117063ce8a314) switched from RUNNING to FAILED
> with failure cause: java.lang.RuntimeException: Error while notify checkpoint
> ABORT.
> at
> org.apache.flink.runtime.taskmanager.Task.notifyCheckpoint(Task.java:1457)
> at
> org.apache.flink.runtime.taskmanager.Task.notifyCheckpointAborted(Task.java:1407)
> at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.abortCheckpoint(TaskExecutor.java:1021)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:316)
> at
> org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:314)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
> at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
> at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
> at akka.actor.Actor.aroundReceive(Actor.scala:537)
> at akka.actor.Actor.aroundReceive$(Actor.scala:535)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
> at akka.actor.ActorCell.invoke(ActorCell.scala:548)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
> at akka.dispatch.Mailbox.run(Mailbox.scala:231)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Caused by: java.lang.UnsupportedOperationException:
> notifyCheckpointAbortAsync not supported by
> org.apache.flink.runtime.dispatcher.JobDispatcherITCase$AtLeastOneCheckpointInvokable
> at
> org.apache.flink.runtime.jobgraph.tasks.AbstractInvokable.notifyCheckpointAbortAsync(AbstractInvokable.java:205)
> at
>
[GitHub] [flink] flinkbot edited a comment on pull request #18603: [FLINK-25610][connector/firehose] Adding table api base for kinesis firehose
flinkbot edited a comment on pull request #18603: URL: https://github.com/apache/flink/pull/18603#issuecomment-1027853740 ## CI report: * 9c269c64ad7e0cb3e24dbaf1b9e5da3d5df5a6cd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30723) * 8b6adf5e05234f15048390cae6f97355d286944d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
flinkbot edited a comment on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033040800 ## CI report: * d025b76c02f35a4c6e56311a7dfed49b5d0dc9d6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30953) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-26033) In KafkaConnector, when 'sink.partitioner' is configured as 'round-robin', it does not take effect
shizhengchao created FLINK-26033:
Summary: In KafkaConnector, when 'sink.partitioner' is configured
as 'round-robin', it does not take effect
Key: FLINK-26033
URL: https://issues.apache.org/jira/browse/FLINK-26033
Project: Flink
Issue Type: Bug
Components: Connectors / Kafka
Affects Versions: 1.14.3, 1.13.3
Reporter: shizhengchao
In KafkaConnector, when 'sink.partitioner' is configured as 'round-robin', it
does not take effect. Flink treats 'default' and 'round-robin' as the same
strategy.
{code:java}
//代码占位符
public static Optional> getFlinkKafkaPartitioner(
ReadableConfig tableOptions, ClassLoader classLoader) {
return tableOptions
.getOptional(SINK_PARTITIONER)
.flatMap(
(String partitioner) -> {
switch (partitioner) {
case SINK_PARTITIONER_VALUE_FIXED:
return Optional.of(new
FlinkFixedPartitioner<>());
case SINK_PARTITIONER_VALUE_DEFAULT:
case SINK_PARTITIONER_VALUE_ROUND_ROBIN:
return Optional.empty();
// Default fallback to full class name of the
partitioner.
default:
return Optional.of(
initializePartitioner(partitioner,
classLoader));
}
});
} {code}
They both use kafka's default partitioner, but the actual There are two
scenarios for the partition on DefaultPartitioner:
1. Random when there is no key
2. When there is a key, take the modulo according to the key
{code:java}
// org.apache.kafka.clients.producer.internals.DefaultPartitioner
public int partition(String topic, Object key, byte[] keyBytes, Object value,
byte[] valueBytes, Cluster cluster) {
if (keyBytes == null) {
// Random when there is no key
return stickyPartitionCache.partition(topic, cluster);
}
List partitions = cluster.partitionsForTopic(topic);
int numPartitions = partitions.size();
// hash the keyBytes to choose a partition
return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions;
} {code}
Therefore, KafkaConnector does not have a round-robin strategy.But we can
borrow from kafka's RoundRobinPartitioner
{code:java}
//代码占位符
public class RoundRobinPartitioner implements Partitioner {
private final ConcurrentMap topicCounterMap = new
ConcurrentHashMap<>();
public void configure(Map configs) {}
/**
* Compute the partition for the given record.
*
* @param topic The topic name
* @param key The key to partition on (or null if no key)
* @param keyBytes serialized key to partition on (or null if no key)
* @param value The value to partition on or null
* @param valueBytes serialized value to partition on or null
* @param cluster The current cluster metadata
*/
@Override
public int partition(String topic, Object key, byte[] keyBytes, Object
value, byte[] valueBytes, Cluster cluster) {
List partitions = cluster.partitionsForTopic(topic);
int numPartitions = partitions.size();
int nextValue = nextValue(topic);
List availablePartitions =
cluster.availablePartitionsForTopic(topic);
if (!availablePartitions.isEmpty()) {
int part = Utils.toPositive(nextValue) % availablePartitions.size();
return availablePartitions.get(part).partition();
} else {
// no partitions are available, give a non-available partition
return Utils.toPositive(nextValue) % numPartitions;
}
}
private int nextValue(String topic) {
AtomicInteger counter = topicCounterMap.computeIfAbsent(topic, k -> {
return new AtomicInteger(0);
});
return counter.getAndIncrement();
}
public void close() {}
} {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18674: WIP : Review Only: Add monitor metrics to Pulsar sink
flinkbot commented on pull request #18674: URL: https://github.com/apache/flink/pull/18674#issuecomment-1033419841 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e61a9e002712c8d0b7233b19e2b0eb14d507bd92 (Wed Feb 09 07:11:28 UTC 2022) **Warnings:** * **2 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] vahmed-hamdy commented on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
vahmed-hamdy commented on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033419382 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30971) * b398b5b5c922f18680c159b0051e98d1961b2518 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] imaffe commented on pull request #18674: WIP : Review Only: Add monitor metrics to Pulsar sink
imaffe commented on pull request #18674: URL: https://github.com/apache/flink/pull/18674#issuecomment-1033418753 Close because the target fork is wrong. Sorry for any inconvenience -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18220: [FLINK-25410] Flink CLI should exit when app is accepted with detach …
flinkbot edited a comment on pull request #18220: URL: https://github.com/apache/flink/pull/18220#issuecomment-1001978981 ## CI report: * 8dc3a423728c27a335cd6f2f393ef79fb801e9c7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30820) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] imaffe closed pull request #18674: WIP : Review Only: Add monitor metrics to Pulsar sink
imaffe closed pull request #18674: URL: https://github.com/apache/flink/pull/18674 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] bgeng777 commented on a change in pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
bgeng777 commented on a change in pull request #18531:
URL: https://github.com/apache/flink/pull/18531#discussion_r802321843
##
File path:
flink-yarn/src/test/java/org/apache/flink/yarn/YarnClusterDescriptorTest.java
##
@@ -699,6 +701,32 @@ public void
testDisableSystemClassPathIncludeUserJarAndWithIllegalShipDirectoryN
}
}
+/** Tests that the usrlib will be automatically shipped. */
+@Test
+public void testShipUsrLib() throws IOException {
+final Map oldEnv = System.getenv();
+final Map env = new HashMap<>(1);
+File homeFolder = temporaryFolder.newFolder().getAbsoluteFile();
+File libFolder = new File(homeFolder.getAbsolutePath(), "lib");
+assertTrue(libFolder.createNewFile());
+File usrLibFolder =
+new File(homeFolder.getAbsolutePath(),
ConfigConstants.DEFAULT_FLINK_USR_LIB_DIR);
+assertTrue(usrLibFolder.mkdirs());
+File usrLibFile = new File(usrLibFolder, "usrLibFile.jar");
+assertTrue(usrLibFile.createNewFile());
+env.put(ConfigConstants.ENV_FLINK_LIB_DIR,
libFolder.getAbsolutePath());
+CommonTestUtils.setEnv(env);
+
+try (YarnClusterDescriptor descriptor = createYarnClusterDescriptor())
{
+Set effectiveShipFiles = new HashSet<>();
+descriptor.addUsrLibFolderToShipFiles(effectiveShipFiles);
+assertThat(effectiveShipFiles, containsInAnyOrder(usrLibFolder));
+assertThat(effectiveShipFiles,
not(containsInAnyOrder(usrLibFile)));
Review comment:
You are right. Our `tryFindUserLibDirectory` will only return a
directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] imaffe commented on a change in pull request #18674: WIP : Review Only: Add monitor metrics to Pulsar sink
imaffe commented on a change in pull request #18674:
URL: https://github.com/apache/flink/pull/18674#discussion_r802321257
##
File path:
flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/PulsarWriter.java
##
@@ -143,11 +152,15 @@ public void write(IN element, Context context) throws
IOException, InterruptedEx
} else {
// Waiting for permits to write message.
pendingMessages.acquire();
+long sendStartTime = timeService.getCurrentProcessingTime();
CompletableFuture sender = builder.sendAsync();
sender.whenComplete(
(id, ex) -> {
+long sendEndTime =
timeService.getCurrentProcessingTime();
+this.lastSendTime = sendEndTime - sendStartTime;
Review comment:
update to lastSendTime should be thread-safe ?
##
File path:
flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/sink/writer/topic/TopicProducerRegister.java
##
@@ -189,4 +198,16 @@ private void clearTransactions() {
// Clear the transactions, we would create new transaction when new
message comes.
transactionRegister.clear();
}
+
+/** Retrieve the producer metrics and map to Flink SinkWriterMetrics. */
+private void updateProducerStats() {
+long numBytesOutSum = 0;
+long numRecordsOutSum = 0;
+for (Producer producer : producerRegister.values()) {
+numBytesOutSum += producer.getStats().getNumBytesSent();
+numRecordsOutSum += producer.getStats().getNumMsgsSent();
Review comment:
recordsOut can be collected on sendAsync() callback complete, but I
think current implementation is ok ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] imaffe opened a new pull request #18674: WIP : Review Only: Add monitor metrics to Pulsar sink
imaffe opened a new pull request #18674: URL: https://github.com/apache/flink/pull/18674 Since the sink code is still evolving, for now I create this PR for temporary discussion purposes. Will create new PR to flink repo after the major code is merged. 1. Test code I think the test code is better put in the PulsarWriterTest (unit test), since @syhily is working on the unit test, I'll add the corresponding unit test after this work. 2. How to retrieve `currentSendTime` In [FLIP-33](https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics) `currentSendTime` is an optional metric. At the same time it should reflect the time spent to send out the last record. However Kafka sink does not strictly follow the description. Instead Kafka uses the average send latency from Kafka client metrics as the `currentSendTime` [link](https://github.com/apache/flink/blob/7e43674abc2281af51ad83b2e3972f7ffb3d2c7b/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/KafkaWriter.java#L358). Currently I use a volatile long to store the send time for each message. Here I believe using volatile should be safe as we set the `lastSendTime` without relying on values from a different thread but I think this part worth notice. A more general design doc on how to retrieve the metrics can be found here : https://docs.google.com/document/d/1rEFs-_9LRK_g5fcYl6GECjaR57xFpZ2PnfJu5093gsg/edit# -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-26032) log job info in the ContextEnvironment
Jing Ge created FLINK-26032:
---
Summary: log job info in the ContextEnvironment
Key: FLINK-26032
URL: https://issues.apache.org/jira/browse/FLINK-26032
Project: Flink
Issue Type: Improvement
Components: Client / Job Submission
Reporter: Jing Ge
Assignee: Jing Ge
In Flink codebase, the {{org.apache.flink.client.program.ContextEnvironment}}
has a static
[Logger|https://github.com/apache/flink/blob/7f9587c723057e2b6cbaf748181c8c80a7f6703d/flink-clients/src/main/java/org/apache/flink/client/program/ContextEnvironment.java#L47]
defined in the class, however, it doesn’t use it to print any logs. Instead,
it prints logs with {{System.out}} and passes the Logger to
{{ShutdownHookUtil.addShutdownHook}} and
{{jobExecutionResultFuture.whenComplete}} for logging any hook errors. If
customer integrated the CLI (‘FlinkYarnSessionCli’ in their case) into a
multi-threaded program to submit jobs in parallel, does it lead to any logs
missing/override/disorder problems?
It is always helpful to log the status information during the job submit
process.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] bgeng777 commented on a change in pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
bgeng777 commented on a change in pull request #18531:
URL: https://github.com/apache/flink/pull/18531#discussion_r802311817
##
File path:
flink-yarn/src/test/java/org/apache/flink/yarn/YarnClusterDescriptorTest.java
##
@@ -699,6 +701,32 @@ public void
testDisableSystemClassPathIncludeUserJarAndWithIllegalShipDirectoryN
}
}
+/** Tests that the usrlib will be automatically shipped. */
+@Test
+public void testShipUsrLib() throws IOException {
+final Map oldEnv = System.getenv();
+final Map env = new HashMap<>(1);
+File homeFolder = temporaryFolder.newFolder().getAbsoluteFile();
+File libFolder = new File(homeFolder.getAbsolutePath(), "lib");
+assertTrue(libFolder.createNewFile());
+File usrLibFolder =
+new File(homeFolder.getAbsolutePath(),
ConfigConstants.DEFAULT_FLINK_USR_LIB_DIR);
+assertTrue(usrLibFolder.mkdirs());
+File usrLibFile = new File(usrLibFolder, "usrLibFile.jar");
+assertTrue(usrLibFile.createNewFile());
+env.put(ConfigConstants.ENV_FLINK_LIB_DIR,
libFolder.getAbsolutePath());
+CommonTestUtils.setEnv(env);
+
+try (YarnClusterDescriptor descriptor = createYarnClusterDescriptor())
{
+Set effectiveShipFiles = new HashSet<>();
+descriptor.addUsrLibFolderToShipFiles(effectiveShipFiles);
+assertThat(effectiveShipFiles, containsInAnyOrder(usrLibFolder));
+assertThat(effectiveShipFiles,
not(containsInAnyOrder(usrLibFile)));
Review comment:
The `assertThat(effectiveShipFiles,
not(containsInAnyOrder(usrLibFile)));` just follows the check logic in
testEnvironmentDirectoryShipping to make sure single file is not added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 9ba86019cbc260ed4371207a48baf12e8a6c5afe UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tsreaper commented on pull request #18657: [FLINK-26001][avro] Implement ProjectableDecodingFormat for avro BulkDecodingFormat
tsreaper commented on pull request #18657: URL: https://github.com/apache/flink/pull/18657#issuecomment-1033411514 Azure passes in https://dev.azure.com/tsreaper96/Flink/_build/results?buildId=324=results -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * 5d9511b68eeb445ab13a84e98a01f5ef6fa16522 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18671: [WIP][FLINK-26019] Don't reorder key groups
flinkbot edited a comment on pull request #18671: URL: https://github.com/apache/flink/pull/18671#issuecomment-1033207374 ## CI report: * 97226580a6e71c78492c3ae2bf48cccf84e2a81c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30958) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
flinkbot edited a comment on pull request #18531: URL: https://github.com/apache/flink/pull/18531#issuecomment-1022985542 ## CI report: * 2952a6c6af10494f1ad1f30c18ac3cbc39693330 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30910) * 7942048b31c8d18d36367fb319627b80f79944e3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30963) * 59c6dfebd1f5bdd4ac17ede1c8270aa779afc252 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30967) * 2833dba1facd9f745c9c713cb017319947a85f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30970) * ac8f4cadaa8eccbdcb6f1a96ce63a6549082815a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] bgeng777 commented on a change in pull request #18531: [FLINK-24897] Enable application mode on YARN to use usrlib
bgeng777 commented on a change in pull request #18531:
URL: https://github.com/apache/flink/pull/18531#discussion_r802311817
##
File path:
flink-yarn/src/test/java/org/apache/flink/yarn/YarnClusterDescriptorTest.java
##
@@ -699,6 +701,32 @@ public void
testDisableSystemClassPathIncludeUserJarAndWithIllegalShipDirectoryN
}
}
+/** Tests that the usrlib will be automatically shipped. */
+@Test
+public void testShipUsrLib() throws IOException {
+final Map oldEnv = System.getenv();
+final Map env = new HashMap<>(1);
+File homeFolder = temporaryFolder.newFolder().getAbsoluteFile();
+File libFolder = new File(homeFolder.getAbsolutePath(), "lib");
+assertTrue(libFolder.createNewFile());
+File usrLibFolder =
+new File(homeFolder.getAbsolutePath(),
ConfigConstants.DEFAULT_FLINK_USR_LIB_DIR);
+assertTrue(usrLibFolder.mkdirs());
+File usrLibFile = new File(usrLibFolder, "usrLibFile.jar");
+assertTrue(usrLibFile.createNewFile());
+env.put(ConfigConstants.ENV_FLINK_LIB_DIR,
libFolder.getAbsolutePath());
+CommonTestUtils.setEnv(env);
+
+try (YarnClusterDescriptor descriptor = createYarnClusterDescriptor())
{
+Set effectiveShipFiles = new HashSet<>();
+descriptor.addUsrLibFolderToShipFiles(effectiveShipFiles);
+assertThat(effectiveShipFiles, containsInAnyOrder(usrLibFolder));
+assertThat(effectiveShipFiles,
not(containsInAnyOrder(usrLibFile)));
Review comment:
The `assertThat(effectiveShipFiles,
not(containsInAnyOrder(usrLibFile)));` just follows the check logic in
testEnvironmentDirectoryShipping to make sure single is not added.
##
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
##
@@ -918,6 +915,18 @@ private ApplicationReport startAppMaster(
: Path.CUR_DIR,
LocalResourceType.FILE);
+// usrlib will be automatically shipped if it exists.
+final Set usrLibShipFiles = new HashSet<>();
Review comment:
I remove this check because I double checked the codes and it should
work well as we will do this check in addUsrLibFolderToShipFiles.
For readability and less object allocation/function call , I agree to get it
back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-26031) Support projection pushdown on keys and values in sst file readers
Caizhi Weng created FLINK-26031: --- Summary: Support projection pushdown on keys and values in sst file readers Key: FLINK-26031 URL: https://issues.apache.org/jira/browse/FLINK-26031 Project: Flink Issue Type: Sub-task Components: Table Store Reporter: Caizhi Weng Fix For: 1.15.0 Projection pushdown is an optimization for sources. With this optimization, we can avoid reading useless columns and thus improve performance. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24439) Introduce CoordinatorStore
[
https://issues.apache.org/jira/browse/FLINK-24439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17489279#comment-17489279
]
Jiangjie Qin commented on FLINK-24439:
--
[~pnowojski] This effectively introduces a cross-operator communication
mechanism for control purpose. Can you articulate a bit on the intended usage
of this {{{}CoordinatorStore{}}}? Will it be used as an ordinary shared
hashmap? If so, will there be any convention of the keys to avoid conflicts
between different operators?
> Introduce CoordinatorStore
> --
>
> Key: FLINK-24439
> URL: https://issues.apache.org/jira/browse/FLINK-24439
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / Common
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> In order to allow {{SourceCoordinators}}s from different {{Sources}} (for
> example two different Kafka sources, or Kafka and Kinesis) to align
> watermarks, they have to be able to exchange information/aggregate watermarks
> from those different Sources. To enable this, we need to provide some
> {{CoordinatorStore}} concept, that would be a thread safe singleton.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-table-store] tsreaper opened a new pull request #18: Support projection pushdown on keys and values in sst file readers
tsreaper opened a new pull request #18: URL: https://github.com/apache/flink-table-store/pull/18 Projection pushdown is an optimization for sources. With this optimization, we can avoid reading useless columns and thus improve performance. This PR is waiting for #16 and #17 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
flinkbot edited a comment on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033040800 ## CI report: * d025b76c02f35a4c6e56311a7dfed49b5d0dc9d6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30953) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
flinkbot edited a comment on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033040800 ## CI report: * d025b76c02f35a4c6e56311a7dfed49b5d0dc9d6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30953) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24607) SourceCoordinator may miss to close SplitEnumerator when failover frequently
[
https://issues.apache.org/jira/browse/FLINK-24607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiangjie Qin updated FLINK-24607:
-
Fix Version/s: 1.15.0
1.13.6
1.14.4
> SourceCoordinator may miss to close SplitEnumerator when failover frequently
>
>
> Key: FLINK-24607
> URL: https://issues.apache.org/jira/browse/FLINK-24607
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.3
>Reporter: Jark Wu
>Assignee: Jiangjie Qin
>Priority: Critical
> Fix For: 1.15.0, 1.13.6, 1.14.4
>
> Attachments: jobmanager.log
>
>
> We are having a connection leak problem when using mysql-cdc [1] source. We
> observed that many enumerators are not closed from the JM log.
> {code}
> ➜ test123 cat jobmanager.log | grep "SourceCoordinator \[\] - Restoring
> SplitEnumerator" | wc -l
> 264
> ➜ test123 cat jobmanager.log | grep "SourceCoordinator \[\] - Starting split
> enumerator" | wc -l
> 264
> ➜ test123 cat jobmanager.log | grep "MySqlSourceEnumerator \[\] - Starting
> enumerator" | wc -l
> 263
> ➜ test123 cat jobmanager.log | grep "SourceCoordinator \[\] - Closing
> SourceCoordinator" | wc -l
> 264
> ➜ test123 cat jobmanager.log | grep "MySqlSourceEnumerator \[\] - Closing
> enumerator" | wc -l
> 195
> {code}
> We added "Closing enumerator" log in {{MySqlSourceEnumerator#close()}}, and
> "Starting enumerator" in {{MySqlSourceEnumerator#start()}}. From the above
> result you can see that SourceCoordinator is restored and closed 264 times,
> split enumerator is started 264 but only closed 195 times. It seems that
> {{SourceCoordinator}} misses to close enumerator when job failover
> frequently.
> I also went throught the code of {{SourceCoordinator}} and found some
> suspicious point:
> The {{started}} flag and {{enumerator}} is assigned in the main thread,
> however {{SourceCoordinator#close()}} is executed async by
> {{DeferrableCoordinator#closeAsync}}. That means the close method will check
> the {{started}} and {{enumerator}} variable async. Is there any concurrency
> problem here which mean lead to dirty read and miss to close the
> {{enumerator}}?
> I'm still not sure, because it's hard to reproduce locally, and we can't
> deploy a custom flink version to production env.
> [1]: https://github.com/ververica/flink-cdc-connectors
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
