[jira] [Commented] (FLINK-32315) Support local file upload in K8s mode
[ https://issues.apache.org/jira/browse/FLINK-32315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810742#comment-17810742 ] Paul Lin commented on FLINK-32315: -- [~mbalassi] Please go ahead. Thanks! > Support local file upload in K8s mode > - > > Key: FLINK-32315 > URL: https://issues.apache.org/jira/browse/FLINK-32315 > Project: Flink > Issue Type: New Feature > Components: Client / Job Submission, Deployment / Kubernetes >Reporter: Paul Lin >Priority: Major > > Currently, Flink assumes all resources are locally accessible in the pods, > which requires users to prepare the resources by mounting storages, > downloading resources with init containers, or rebuilding images for each > execution. > We could make things much easier by introducing a built-in file distribution > mechanism based on Flink-supported filesystems. It's implemented in two steps: > > 1. KubernetesClusterDescripter uploads all local resources to remote storage > via Flink filesystem (skips if the resources are already remote). > 2. KubernetesApplicationClusterEntrypoint and KubernetesTaskExecutorRunner > download the resources and put them in the classpath during startup. > > The 2nd step is mostly done by > [FLINK-28915|https://issues.apache.org/jira/browse/FLINK-28915], thus this > issue is focused on the upload part. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32143) Automatic cleanup of terminated flinkdeployments
[ https://issues.apache.org/jira/browse/FLINK-32143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760752#comment-17760752 ] Paul Lin commented on FLINK-32143: -- Looks great! Thanks [~gyfora] > Automatic cleanup of terminated flinkdeployments > > > Key: FLINK-32143 > URL: https://issues.apache.org/jira/browse/FLINK-32143 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: Paul Lin >Priority: Major > > Currently, if a job turns into terminated status (e.g. FINISHED or FAILED), > the flinkdeployment remains until a manual cleanup is performed. > We could add a configuration named > `kubernetes.operator.deployment.cleanup.delay`, and clean up terminated > deployments after the delay. The delay is default to `-1` which means the > cleanup is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32315) Support local file upload in K8s mode
Paul Lin created FLINK-32315: Summary: Support local file upload in K8s mode Key: FLINK-32315 URL: https://issues.apache.org/jira/browse/FLINK-32315 Project: Flink Issue Type: New Feature Components: Client / Job Submission, Deployment / Kubernetes Reporter: Paul Lin Currently, Flink assumes all resources are locally accessible in the pods, which requires users to prepare the resources by mounting storages, downloading resources with init containers, or rebuilding images for each execution. We could make things much easier by introducing a built-in file distribution mechanism based on Flink-supported filesystems. It's implemented in two steps: 1. KubernetesClusterDescripter uploads all local resources to remote storage via Flink filesystem (skips if the resources are already remote). 2. KubernetesApplicationClusterEntrypoint and KubernetesTaskExecutorRunner download the resources and put them in the classpath during startup. The 2nd step is mostly done by [FLINK-28915|https://issues.apache.org/jira/browse/FLINK-28915], thus this issue is focused on the upload part. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-26541) SQL Client should support submitting SQL jobs in application mode
[ https://issues.apache.org/jira/browse/FLINK-26541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17727239#comment-17727239 ] Paul Lin commented on FLINK-26541: -- I've created [FLIP-316|https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver], please join the [discussion thread|https://lists.apache.org/thread/7zyojksml5xj77fdokcpttf550qflyqm] if you're interested. > SQL Client should support submitting SQL jobs in application mode > - > > Key: FLINK-26541 > URL: https://issues.apache.org/jira/browse/FLINK-26541 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Deployment / YARN, Table SQL / > Client >Reporter: Jark Wu >Priority: Major > > Currently, the SQL Client only supports submitting jobs in session mode and > per-job mode. As the community going to drop the per-job mode (FLINK-26000), > SQL Client should support application mode as well. Otherwise, SQL Client can > only submit SQL in session mode then, but streaming jobs should be submitted > in per-job or application mode to have bettter resource isolation. > Disucssions: https://lists.apache.org/thread/2yq351nb721x23rz1q8qlyf2tqrk147r -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32143) Automatic cleanup of terminated flinkdeployments
Paul Lin created FLINK-32143: Summary: Automatic cleanup of terminated flinkdeployments Key: FLINK-32143 URL: https://issues.apache.org/jira/browse/FLINK-32143 Project: Flink Issue Type: New Feature Components: Kubernetes Operator Reporter: Paul Lin Currently, if a job turns into terminated status (e.g. FINISHED or FAILED), the flinkdeployment remains until a manual cleanup is performed. We could add a configuration named `kubernetes.operator.deployment.cleanup.delay`, and clean up terminated deployments after the delay. The delay is default to `-1` which means the cleanup is disabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31965) Fix ClassNotFoundException in benchmarks
[ https://issues.apache.org/jira/browse/FLINK-31965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-31965: - Priority: Minor (was: Major) > Fix ClassNotFoundException in benchmarks > > > Key: FLINK-31965 > URL: https://issues.apache.org/jira/browse/FLINK-31965 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.18.0 >Reporter: Paul Lin >Priority: Minor > Labels: pull-request-available > > The benchmarks rely on the test jar of `flink-streaming-java`. However, the > jar is set to test scope, thus not included in the packaged jar. Therefore > ClassNotFoundException occurs while running the benchmarks with `java --jar > xxx` command. > ``` > java.lang.NoClassDefFoundError: > org/apache/flink/streaming/util/KeyedOneInputStreamOperatorTestHarness > at > org.apache.flink.contrib.streaming.state.benchmark.RescalingBenchmark.prepareState(RescalingBenchmark.java:111) > at > org.apache.flink.contrib.streaming.state.benchmark.RescalingBenchmark.setUp(RescalingBenchmark.java:78) > at > org.apache.flink.state.benchmark.RocksdbStateBackendRescalingBenchmarkExecutor.setUp(RocksdbStateBackendRescalingBenchmarkExecutor.java:66) > at > org.apache.flink.state.benchmark.generated.RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest._jmh_tryInit_f_rocksdbstatebackendrescalingbenchmarkexecutor0_0(RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.java:370) > at > org.apache.flink.state.benchmark.generated.RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.rescaleRocksDB_AverageTime(RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.java:147) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453) > at > org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundException: > org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 17 more > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31965) Fix ClassNotFoundException in benchmarks
[ https://issues.apache.org/jira/browse/FLINK-31965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717573#comment-17717573 ] Paul Lin commented on FLINK-31965: -- Yes, it runs fine with source codes. > Fix ClassNotFoundException in benchmarks > > > Key: FLINK-31965 > URL: https://issues.apache.org/jira/browse/FLINK-31965 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.18.0 >Reporter: Paul Lin >Priority: Major > Labels: pull-request-available > > The benchmarks rely on the test jar of `flink-streaming-java`. However, the > jar is set to test scope, thus not included in the packaged jar. Therefore > ClassNotFoundException occurs while running the benchmarks with `java --jar > xxx` command. > ``` > java.lang.NoClassDefFoundError: > org/apache/flink/streaming/util/KeyedOneInputStreamOperatorTestHarness > at > org.apache.flink.contrib.streaming.state.benchmark.RescalingBenchmark.prepareState(RescalingBenchmark.java:111) > at > org.apache.flink.contrib.streaming.state.benchmark.RescalingBenchmark.setUp(RescalingBenchmark.java:78) > at > org.apache.flink.state.benchmark.RocksdbStateBackendRescalingBenchmarkExecutor.setUp(RocksdbStateBackendRescalingBenchmarkExecutor.java:66) > at > org.apache.flink.state.benchmark.generated.RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest._jmh_tryInit_f_rocksdbstatebackendrescalingbenchmarkexecutor0_0(RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.java:370) > at > org.apache.flink.state.benchmark.generated.RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.rescaleRocksDB_AverageTime(RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.java:147) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453) > at > org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundException: > org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 17 more > ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31965) Fix ClassNotFoundException in benchmarks
[ https://issues.apache.org/jira/browse/FLINK-31965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-31965: - Description: The benchmarks rely on the test jar of `flink-streaming-java`. However, the jar is set to test scope, thus not included in the packaged jar. Therefore ClassNotFoundException occurs while running the benchmarks with `java --jar xxx` command. ``` java.lang.NoClassDefFoundError: org/apache/flink/streaming/util/KeyedOneInputStreamOperatorTestHarness at org.apache.flink.contrib.streaming.state.benchmark.RescalingBenchmark.prepareState(RescalingBenchmark.java:111) at org.apache.flink.contrib.streaming.state.benchmark.RescalingBenchmark.setUp(RescalingBenchmark.java:78) at org.apache.flink.state.benchmark.RocksdbStateBackendRescalingBenchmarkExecutor.setUp(RocksdbStateBackendRescalingBenchmarkExecutor.java:66) at org.apache.flink.state.benchmark.generated.RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest._jmh_tryInit_f_rocksdbstatebackendrescalingbenchmarkexecutor0_0(RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.java:370) at org.apache.flink.state.benchmark.generated.RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.rescaleRocksDB_AverageTime(RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.java:147) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:453) at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:437) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.util.KeyedOneInputStreamOperatorTestHarness at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 17 more ``` was:The benchmarks rely on the test jar of `flink-streaming-java`. However, the jar is set to test scope, thus not included in the packaged jar. Therefore ClassNotFoundException occurs while running the benchmarks with `java --jar xxx` command. > Fix ClassNotFoundException in benchmarks > > > Key: FLINK-31965 > URL: https://issues.apache.org/jira/browse/FLINK-31965 > Project: Flink > Issue Type: Bug > Components: Benchmarks >Affects Versions: 1.18.0 >Reporter: Paul Lin >Priority: Major > > The benchmarks rely on the test jar of `flink-streaming-java`. However, the > jar is set to test scope, thus not included in the packaged jar. Therefore > ClassNotFoundException occurs while running the benchmarks with `java --jar > xxx` command. > ``` > java.lang.NoClassDefFoundError: > org/apache/flink/streaming/util/KeyedOneInputStreamOperatorTestHarness > at > org.apache.flink.contrib.streaming.state.benchmark.RescalingBenchmark.prepareState(RescalingBenchmark.java:111) > at > org.apache.flink.contrib.streaming.state.benchmark.RescalingBenchmark.setUp(RescalingBenchmark.java:78) > at > org.apache.flink.state.benchmark.RocksdbStateBackendRescalingBenchmarkExecutor.setUp(RocksdbStateBackendRescalingBenchmarkExecutor.java:66) > at > org.apache.flink.state.benchmark.generated.RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest._jmh_tryInit_f_rocksdbstatebackendrescalingbenchmarkexecutor0_0(RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.java:370) > at > org.apache.flink.state.benchmark.generated.RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.rescaleRocksDB_AverageTime(RocksdbStateBackendRescalingBenchmarkExecutor_rescaleRocksDB_jmhTest.java:147) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at >
[jira] [Created] (FLINK-31965) Fix ClassNotFoundException in benchmarks
Paul Lin created FLINK-31965: Summary: Fix ClassNotFoundException in benchmarks Key: FLINK-31965 URL: https://issues.apache.org/jira/browse/FLINK-31965 Project: Flink Issue Type: Bug Components: Benchmarks Affects Versions: 1.18.0 Reporter: Paul Lin The benchmarks rely on the test jar of `flink-streaming-java`. However, the jar is set to test scope, thus not included in the packaged jar. Therefore ClassNotFoundException occurs while running the benchmarks with `java --jar xxx` command. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-26541) SQL Client should support submitting SQL jobs in application mode
[ https://issues.apache.org/jira/browse/FLINK-26541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17690985#comment-17690985 ] Paul Lin commented on FLINK-26541: -- Is there any ongoing effort currently? I'm happy to take over this FYI. [~jark] [~martijnvisser] > SQL Client should support submitting SQL jobs in application mode > - > > Key: FLINK-26541 > URL: https://issues.apache.org/jira/browse/FLINK-26541 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Deployment / YARN, Table SQL / > Client >Reporter: Jark Wu >Priority: Major > > Currently, the SQL Client only supports submitting jobs in session mode and > per-job mode. As the community going to drop the per-job mode (FLINK-26000), > SQL Client should support application mode as well. Otherwise, SQL Client can > only submit SQL in session mode then, but streaming jobs should be submitted > in per-job or application mode to have bettter resource isolation. > Disucssions: https://lists.apache.org/thread/2yq351nb721x23rz1q8qlyf2tqrk147r -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30811) Fix sql gateway can not stop job correctly
[ https://issues.apache.org/jira/browse/FLINK-30811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682881#comment-17682881 ] Paul Lin edited comment on FLINK-30811 at 2/1/23 8:00 AM: -- [~qingyue] [~fsk119] Sorry for introducing this bug, I should have fixed this via FLINK-30538. was (Author: paul lin): [~qingyue] [~fsk119] Sorry for introducing this bug, I should have fixed this via https://issues.apache.org/jira/browse/FLINK-30538. > Fix sql gateway can not stop job correctly > -- > > Key: FLINK-30811 > URL: https://issues.apache.org/jira/browse/FLINK-30811 > Project: Flink > Issue Type: Bug > Components: Table SQL / Gateway >Affects Versions: 1.17 >Reporter: Shengkai Fang >Assignee: Jane Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.17 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30811) Fix sql gateway can not stop job correctly
[ https://issues.apache.org/jira/browse/FLINK-30811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682881#comment-17682881 ] Paul Lin commented on FLINK-30811: -- [~qingyue] [~fsk119] Sorry for introducing this bug, I should have fixed this via https://issues.apache.org/jira/browse/FLINK-30538. > Fix sql gateway can not stop job correctly > -- > > Key: FLINK-30811 > URL: https://issues.apache.org/jira/browse/FLINK-30811 > Project: Flink > Issue Type: Bug > Components: Table SQL / Gateway >Affects Versions: 1.17 >Reporter: Shengkai Fang >Assignee: Jane Chan >Priority: Major > Labels: pull-request-available > Fix For: 1.17 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30677) SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
[
https://issues.apache.org/jira/browse/FLINK-30677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17677720#comment-17677720
]
Paul Lin edited comment on FLINK-30677 at 1/17/23 11:28 AM:
[~fsk119] I prefer the old approach too. It's intuitive and efficient.
was (Author: paul lin):
[~fsk119] I prefer the old approach too. It's instinctive and efficient.
> SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
> -
>
> Key: FLINK-30677
> URL: https://issues.apache.org/jira/browse/FLINK-30677
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> We're observing a test instability with
> {{SqlGatewayServiceStatementITCase.testFlinkSqlStatements}} in the following
> builds:
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=14251]
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=14608
> {code:java}
> Jan 13 02:46:10 [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 27.279 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase
> Jan 13 02:46:10 [ERROR]
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase.testFlinkSqlStatements(String)[5]
> Time elapsed: 1.573 s <<< FAILURE!
> Jan 13 02:46:10 org.opentest4j.AssertionFailedError:
> Jan 13 02:46:10
> Jan 13 02:46:10 expected:
> Jan 13 02:46:10 "# table.q - CREATE/DROP/SHOW/ALTER/DESCRIBE TABLE
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Licensed to the Apache Software Foundation (ASF) under
> one or more
> Jan 13 02:46:10 # contributor license agreements. See the NOTICE file
> distributed with
> Jan 13 02:46:10 # this work for additional information regarding copyright
> ownership.
> Jan 13 02:46:10 # The ASF licenses this file to you under the Apache
> License, Version 2.0
> Jan 13 02:46:10 # (the "License"); you may not use this file except in
> compliance with
> Jan 13 02:46:10 # the License. You may obtain a copy of the License at
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # http://www.apache.org/licenses/LICENSE-2.0
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Unless required by applicable law or agreed to in
> writing, software
> Jan 13 02:46:10 # distributed under the License is distributed on an "AS
> IS" BASIS,
> Jan 13 02:46:10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
> express or implied.
> Jan 13 02:46:10 # See the License for the specific language governing
> permissions and
> Jan 13 02:46:10 # limitations under the License.
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30677) SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
[
https://issues.apache.org/jira/browse/FLINK-30677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17677720#comment-17677720
]
Paul Lin commented on FLINK-30677:
--
[~fsk119] I prefer the old approach too. It's instinctive and efficient.
> SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
> -
>
> Key: FLINK-30677
> URL: https://issues.apache.org/jira/browse/FLINK-30677
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> We're observing a test instability with
> {{SqlGatewayServiceStatementITCase.testFlinkSqlStatements}} in the following
> builds:
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=14251]
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=14608
> {code:java}
> Jan 13 02:46:10 [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 27.279 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase
> Jan 13 02:46:10 [ERROR]
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase.testFlinkSqlStatements(String)[5]
> Time elapsed: 1.573 s <<< FAILURE!
> Jan 13 02:46:10 org.opentest4j.AssertionFailedError:
> Jan 13 02:46:10
> Jan 13 02:46:10 expected:
> Jan 13 02:46:10 "# table.q - CREATE/DROP/SHOW/ALTER/DESCRIBE TABLE
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Licensed to the Apache Software Foundation (ASF) under
> one or more
> Jan 13 02:46:10 # contributor license agreements. See the NOTICE file
> distributed with
> Jan 13 02:46:10 # this work for additional information regarding copyright
> ownership.
> Jan 13 02:46:10 # The ASF licenses this file to you under the Apache
> License, Version 2.0
> Jan 13 02:46:10 # (the "License"); you may not use this file except in
> compliance with
> Jan 13 02:46:10 # the License. You may obtain a copy of the License at
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # http://www.apache.org/licenses/LICENSE-2.0
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Unless required by applicable law or agreed to in
> writing, software
> Jan 13 02:46:10 # distributed under the License is distributed on an "AS
> IS" BASIS,
> Jan 13 02:46:10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
> express or implied.
> Jan 13 02:46:10 # See the License for the specific language governing
> permissions and
> Jan 13 02:46:10 # limitations under the License.
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30677) SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
[
https://issues.apache.org/jira/browse/FLINK-30677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676872#comment-17676872
]
Paul Lin edited comment on FLINK-30677 at 1/17/23 9:13 AM:
---
-I couldn't reproduce the error locally. Is that fixed already?-
Managed to reproduce the problem, like a 1 in 10 chance.
was (Author: paul lin):
~~I couldn't reproduce the error locally. Is that fixed already?~~
Managed to reproduce the problem, like a 1 in 10 chance.
> SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
> -
>
> Key: FLINK-30677
> URL: https://issues.apache.org/jira/browse/FLINK-30677
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> We're observing a test instability with
> {{SqlGatewayServiceStatementITCase.testFlinkSqlStatements}} in the following
> builds:
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=14251]
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=14608
> {code:java}
> Jan 13 02:46:10 [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 27.279 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase
> Jan 13 02:46:10 [ERROR]
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase.testFlinkSqlStatements(String)[5]
> Time elapsed: 1.573 s <<< FAILURE!
> Jan 13 02:46:10 org.opentest4j.AssertionFailedError:
> Jan 13 02:46:10
> Jan 13 02:46:10 expected:
> Jan 13 02:46:10 "# table.q - CREATE/DROP/SHOW/ALTER/DESCRIBE TABLE
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Licensed to the Apache Software Foundation (ASF) under
> one or more
> Jan 13 02:46:10 # contributor license agreements. See the NOTICE file
> distributed with
> Jan 13 02:46:10 # this work for additional information regarding copyright
> ownership.
> Jan 13 02:46:10 # The ASF licenses this file to you under the Apache
> License, Version 2.0
> Jan 13 02:46:10 # (the "License"); you may not use this file except in
> compliance with
> Jan 13 02:46:10 # the License. You may obtain a copy of the License at
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # http://www.apache.org/licenses/LICENSE-2.0
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Unless required by applicable law or agreed to in
> writing, software
> Jan 13 02:46:10 # distributed under the License is distributed on an "AS
> IS" BASIS,
> Jan 13 02:46:10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
> express or implied.
> Jan 13 02:46:10 # See the License for the specific language governing
> permissions and
> Jan 13 02:46:10 # limitations under the License.
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30677) SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
[
https://issues.apache.org/jira/browse/FLINK-30677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676872#comment-17676872
]
Paul Lin edited comment on FLINK-30677 at 1/17/23 9:12 AM:
---
~~I couldn't reproduce the error locally. Is that fixed already?~~
Managed to reproduce the problem, like a 1 in 10 chance.
was (Author: paul lin):
I couldn't reproduce the error locally. Is that fixed already?
> SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
> -
>
> Key: FLINK-30677
> URL: https://issues.apache.org/jira/browse/FLINK-30677
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> We're observing a test instability with
> {{SqlGatewayServiceStatementITCase.testFlinkSqlStatements}} in the following
> builds:
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=14251]
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=14608
> {code:java}
> Jan 13 02:46:10 [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 27.279 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase
> Jan 13 02:46:10 [ERROR]
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase.testFlinkSqlStatements(String)[5]
> Time elapsed: 1.573 s <<< FAILURE!
> Jan 13 02:46:10 org.opentest4j.AssertionFailedError:
> Jan 13 02:46:10
> Jan 13 02:46:10 expected:
> Jan 13 02:46:10 "# table.q - CREATE/DROP/SHOW/ALTER/DESCRIBE TABLE
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Licensed to the Apache Software Foundation (ASF) under
> one or more
> Jan 13 02:46:10 # contributor license agreements. See the NOTICE file
> distributed with
> Jan 13 02:46:10 # this work for additional information regarding copyright
> ownership.
> Jan 13 02:46:10 # The ASF licenses this file to you under the Apache
> License, Version 2.0
> Jan 13 02:46:10 # (the "License"); you may not use this file except in
> compliance with
> Jan 13 02:46:10 # the License. You may obtain a copy of the License at
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # http://www.apache.org/licenses/LICENSE-2.0
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Unless required by applicable law or agreed to in
> writing, software
> Jan 13 02:46:10 # distributed under the License is distributed on an "AS
> IS" BASIS,
> Jan 13 02:46:10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
> express or implied.
> Jan 13 02:46:10 # See the License for the specific language governing
> permissions and
> Jan 13 02:46:10 # limitations under the License.
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30698) SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
[
https://issues.apache.org/jira/browse/FLINK-30698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17677278#comment-17677278
]
Paul Lin commented on FLINK-30698:
--
Duplicate of https://issues.apache.org/jira/browse/FLINK-30677
> SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
> -
>
> Key: FLINK-30698
> URL: https://issues.apache.org/jira/browse/FLINK-30698
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
>
> We're seeing an assertion in
> {{SqlGatewayServiceStatementITCase.testFlinkSqlStatements}} in the following
> build:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44824=logs=ce3801ad-3bd5-5f06-d165-34d37e757d90=5e4d9387-1dcc-5885-a901-90469b7e6d2f=16651
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30677) SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
[
https://issues.apache.org/jira/browse/FLINK-30677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676872#comment-17676872
]
Paul Lin commented on FLINK-30677:
--
I couldn't reproduce the error locally. Is that fixed already?
> SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
> -
>
> Key: FLINK-30677
> URL: https://issues.apache.org/jira/browse/FLINK-30677
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
>
> We're observing a test instability with
> {{SqlGatewayServiceStatementITCase.testFlinkSqlStatements}} in the following
> builds:
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=14251]
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=14608
> {code:java}
> Jan 13 02:46:10 [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 27.279 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase
> Jan 13 02:46:10 [ERROR]
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase.testFlinkSqlStatements(String)[5]
> Time elapsed: 1.573 s <<< FAILURE!
> Jan 13 02:46:10 org.opentest4j.AssertionFailedError:
> Jan 13 02:46:10
> Jan 13 02:46:10 expected:
> Jan 13 02:46:10 "# table.q - CREATE/DROP/SHOW/ALTER/DESCRIBE TABLE
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Licensed to the Apache Software Foundation (ASF) under
> one or more
> Jan 13 02:46:10 # contributor license agreements. See the NOTICE file
> distributed with
> Jan 13 02:46:10 # this work for additional information regarding copyright
> ownership.
> Jan 13 02:46:10 # The ASF licenses this file to you under the Apache
> License, Version 2.0
> Jan 13 02:46:10 # (the "License"); you may not use this file except in
> compliance with
> Jan 13 02:46:10 # the License. You may obtain a copy of the License at
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # http://www.apache.org/licenses/LICENSE-2.0
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Unless required by applicable law or agreed to in
> writing, software
> Jan 13 02:46:10 # distributed under the License is distributed on an "AS
> IS" BASIS,
> Jan 13 02:46:10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
> express or implied.
> Jan 13 02:46:10 # See the License for the specific language governing
> permissions and
> Jan 13 02:46:10 # limitations under the License.
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30677) SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
[
https://issues.apache.org/jira/browse/FLINK-30677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676835#comment-17676835
]
Paul Lin edited comment on FLINK-30677 at 1/14/23 4:16 AM:
---
I think the problem lies here:
{code:java}
2023-01-13T01:32:32.9258077Z Jan 13 01:32:32 # test explain select with
CHANGELOG_MODE
2023-01-13T01:32:32.9258568Z Jan 13 01:32:32 explain changelog_mode select
`user`, product from orders;
2023-01-13T01:32:32.9259004Z Jan 13 01:32:32 !output
2023-01-13T01:32:32.9259402Z Jan 13 01:32:32
java.util.NoSuchElementException: null
2023-01-13T01:32:32.9259800Z Jan 13 01:32:32 !error {code}
It seems not directly related to f1770892 (FLINK-28655), but I will take a look.
was (Author: paul lin):
I think the problem lies here:
{code:java}
2023-01-13T01:32:32.9258077Z Jan 13 01:32:32 # test explain select with
CHANGELOG_MODE
2023-01-13T01:32:32.9258568Z Jan 13 01:32:32 explain changelog_mode select
`user`, product from orders;
2023-01-13T01:32:32.9259004Z Jan 13 01:32:32 !output
2023-01-13T01:32:32.9259402Z Jan 13 01:32:32
java.util.NoSuchElementException: null
2023-01-13T01:32:32.9259800Z Jan 13 01:32:32 !error {code}
It seems not directly related to f1770892 (FLINK-28655), but I will take a look.
> SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
> -
>
> Key: FLINK-30677
> URL: https://issues.apache.org/jira/browse/FLINK-30677
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
>
> We're observing a test instability with
> {{SqlGatewayServiceStatementITCase.testFlinkSqlStatements}} in the following
> builds:
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=14251]
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=14608
> {code:java}
> Jan 13 02:46:10 [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 27.279 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase
> Jan 13 02:46:10 [ERROR]
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase.testFlinkSqlStatements(String)[5]
> Time elapsed: 1.573 s <<< FAILURE!
> Jan 13 02:46:10 org.opentest4j.AssertionFailedError:
> Jan 13 02:46:10
> Jan 13 02:46:10 expected:
> Jan 13 02:46:10 "# table.q - CREATE/DROP/SHOW/ALTER/DESCRIBE TABLE
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Licensed to the Apache Software Foundation (ASF) under
> one or more
> Jan 13 02:46:10 # contributor license agreements. See the NOTICE file
> distributed with
> Jan 13 02:46:10 # this work for additional information regarding copyright
> ownership.
> Jan 13 02:46:10 # The ASF licenses this file to you under the Apache
> License, Version 2.0
> Jan 13 02:46:10 # (the "License"); you may not use this file except in
> compliance with
> Jan 13 02:46:10 # the License. You may obtain a copy of the License at
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # http://www.apache.org/licenses/LICENSE-2.0
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Unless required by applicable law or agreed to in
> writing, software
> Jan 13 02:46:10 # distributed under the License is distributed on an "AS
> IS" BASIS,
> Jan 13 02:46:10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
> express or implied.
> Jan 13 02:46:10 # See the License for the specific language governing
> permissions and
> Jan 13 02:46:10 # limitations under the License.
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30677) SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
[
https://issues.apache.org/jira/browse/FLINK-30677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676835#comment-17676835
]
Paul Lin commented on FLINK-30677:
--
I think the problem lies here:
{code:java}
2023-01-13T01:32:32.9258077Z Jan 13 01:32:32 # test explain select with
CHANGELOG_MODE
2023-01-13T01:32:32.9258568Z Jan 13 01:32:32 explain changelog_mode select
`user`, product from orders;
2023-01-13T01:32:32.9259004Z Jan 13 01:32:32 !output
2023-01-13T01:32:32.9259402Z Jan 13 01:32:32
java.util.NoSuchElementException: null
2023-01-13T01:32:32.9259800Z Jan 13 01:32:32 !error {code}
It seems not directly related to f1770892 (FLINK-28655), but I will take a look.
> SqlGatewayServiceStatementITCase.testFlinkSqlStatements fails
> -
>
> Key: FLINK-30677
> URL: https://issues.apache.org/jira/browse/FLINK-30677
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
>
> We're observing a test instability with
> {{SqlGatewayServiceStatementITCase.testFlinkSqlStatements}} in the following
> builds:
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=14251]
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44775=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=14608
> {code:java}
> Jan 13 02:46:10 [ERROR] Tests run: 9, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 27.279 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase
> Jan 13 02:46:10 [ERROR]
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase.testFlinkSqlStatements(String)[5]
> Time elapsed: 1.573 s <<< FAILURE!
> Jan 13 02:46:10 org.opentest4j.AssertionFailedError:
> Jan 13 02:46:10
> Jan 13 02:46:10 expected:
> Jan 13 02:46:10 "# table.q - CREATE/DROP/SHOW/ALTER/DESCRIBE TABLE
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Licensed to the Apache Software Foundation (ASF) under
> one or more
> Jan 13 02:46:10 # contributor license agreements. See the NOTICE file
> distributed with
> Jan 13 02:46:10 # this work for additional information regarding copyright
> ownership.
> Jan 13 02:46:10 # The ASF licenses this file to you under the Apache
> License, Version 2.0
> Jan 13 02:46:10 # (the "License"); you may not use this file except in
> compliance with
> Jan 13 02:46:10 # the License. You may obtain a copy of the License at
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # http://www.apache.org/licenses/LICENSE-2.0
> Jan 13 02:46:10 #
> Jan 13 02:46:10 # Unless required by applicable law or agreed to in
> writing, software
> Jan 13 02:46:10 # distributed under the License is distributed on an "AS
> IS" BASIS,
> Jan 13 02:46:10 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
> express or implied.
> Jan 13 02:46:10 # See the License for the specific language governing
> permissions and
> Jan 13 02:46:10 # limitations under the License.
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28655) Support show jobs statement in SqlGatewayService
[ https://issues.apache.org/jira/browse/FLINK-28655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-28655: - Summary: Support show jobs statement in SqlGatewayService (was: Support show jobs statement in SqlGatewatService) > Support show jobs statement in SqlGatewayService > > > Key: FLINK-28655 > URL: https://issues.apache.org/jira/browse/FLINK-28655 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Gateway >Reporter: Paul Lin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30538) Improve error handling of stop job operation
Paul Lin created FLINK-30538: Summary: Improve error handling of stop job operation Key: FLINK-30538 URL: https://issues.apache.org/jira/browse/FLINK-30538 Project: Flink Issue Type: Sub-task Components: Table SQL / Client, Table SQL / Gateway Reporter: Paul Lin Currently, the stop-job operation produces some verbose error msg and doesn't handle exceptions in stop-without-savepoint gracefully. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27344) FLIP-222: Support full job lifecycle statements in SQL client
[ https://issues.apache.org/jira/browse/FLINK-27344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17616929#comment-17616929 ] Paul Lin commented on FLINK-27344: -- [~AlexeyLV] Thanks a lot for your input! It's very informative. Providing only system stored procedures without user-defined ones is indeed an approach we had not thought about or discussed. But given that the community has voted through the job statement syntax and the related efforts are currently underway, IMHO it'd be better to stick with the plan if it has no obvious drawbacks. WDYT? WRT INFORMATION_SCHEMA tables, I find there might be some problem with Flink's pluggable catalog design which allows access to external systems (which is different from common database scenarios). For catalogs like HiveCatalog or JdbcCatalog, it would be kind of weird that users could find the INFORMATION_SCHEMA (or maybe another name that is less likely to conflict) tables only through Flink SQL client. > FLIP-222: Support full job lifecycle statements in SQL client > - > > Key: FLINK-27344 > URL: https://issues.apache.org/jira/browse/FLINK-27344 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Reporter: Paul Lin >Assignee: Paul Lin >Priority: Major > > With the efforts in FLIP-24 and FLIP-91, Flink SQL client supports submitting > SQL jobs but lacks further support for their lifecycles afterward which is > crucial for streaming use cases. That means Flink SQL client users have to > turn to other clients (e.g. CLI) or APIs (e.g. REST API) to manage the jobs, > like triggering savepoints or canceling queries, which makes the user > experience of SQL client incomplete. > Therefore, this proposal aims to complete the capability of SQL client by > adding job lifecycle statements. With these statements, users could manage > jobs and savepoints through pure SQL in SQL client. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27344) FLIP-222: Support full job lifecycle statements in SQL client
[ https://issues.apache.org/jira/browse/FLINK-27344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17615441#comment-17615441 ] Paul Lin commented on FLINK-27344: -- [~ekoblov] Sure. It's documented in the FLIP [https://cwiki.apache.org/confluence/display/FLINK/FLIP-222%3A+Support+full+job+lifecycle+statements+in+SQL+client]. Also you can find the discussion in the mail list [https://lists.apache.org/thread/qkvh9p5w9b12s7ykh3l7lv7m9dbgnf1g]. > FLIP-222: Support full job lifecycle statements in SQL client > - > > Key: FLINK-27344 > URL: https://issues.apache.org/jira/browse/FLINK-27344 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Reporter: Paul Lin >Assignee: Paul Lin >Priority: Major > > With the efforts in FLIP-24 and FLIP-91, Flink SQL client supports submitting > SQL jobs but lacks further support for their lifecycles afterward which is > crucial for streaming use cases. That means Flink SQL client users have to > turn to other clients (e.g. CLI) or APIs (e.g. REST API) to manage the jobs, > like triggering savepoints or canceling queries, which makes the user > experience of SQL client incomplete. > Therefore, this proposal aims to complete the capability of SQL client by > adding job lifecycle statements. With these statements, users could manage > jobs and savepoints through pure SQL in SQL client. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27344) FLIP-222: Support full job lifecycle statements in SQL client
[ https://issues.apache.org/jira/browse/FLINK-27344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17614320#comment-17614320 ] Paul Lin commented on FLINK-27344: -- [~ekoblov] Yes, we've roughly considered the stored procedures approach, but given that Flink currently doesn't support stored procedures and it's not very straightforward from the users' point of view, we then lean toward the new syntax approach (as many databases also provide similar SQL statements to control jobs/tasks, like Hive/CrukroachDB/KSQL). WDYT? > FLIP-222: Support full job lifecycle statements in SQL client > - > > Key: FLINK-27344 > URL: https://issues.apache.org/jira/browse/FLINK-27344 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Reporter: Paul Lin >Assignee: Paul Lin >Priority: Major > > With the efforts in FLIP-24 and FLIP-91, Flink SQL client supports submitting > SQL jobs but lacks further support for their lifecycles afterward which is > crucial for streaming use cases. That means Flink SQL client users have to > turn to other clients (e.g. CLI) or APIs (e.g. REST API) to manage the jobs, > like triggering savepoints or canceling queries, which makes the user > experience of SQL client incomplete. > Therefore, this proposal aims to complete the capability of SQL client by > adding job lifecycle statements. With these statements, users could manage > jobs and savepoints through pure SQL in SQL client. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28685) Optimize CatalogManager getTable implementation
[
https://issues.apache.org/jira/browse/FLINK-28685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Lin updated FLINK-28685:
-
Priority: Minor (was: Major)
> Optimize CatalogManager getTable implementation
> ---
>
> Key: FLINK-28685
> URL: https://issues.apache.org/jira/browse/FLINK-28685
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.16.0
>Reporter: Paul Lin
>Priority: Minor
>
> Currently, Catalog#getTable(ObjectPath) might be abused in CatalogManager.
> CatalogManager gets an ObjectPath directly, instead of checking the existence
> of the table first, which could surprise Catalog developers and may get
> unexpected exceptions other than TableNotFoundExceptions.
> {code:java}
> private Optional getUnresolvedTable(ObjectIdentifier
> objectIdentifier) {
> Catalog currentCatalog = catalogs.get(objectIdentifier.getCatalogName());
> ObjectPath objectPath = objectIdentifier.toObjectPath();
> if (currentCatalog != null) {
> try
> { final CatalogBaseTable table = currentCatalog.getTable(objectPath); return
> Optional.of(table); }
> catch (TableNotExistException e)
> { // Ignore. }
> }
> return Optional.empty();
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28685) Optimize CatalogManager getTable implementation
[
https://issues.apache.org/jira/browse/FLINK-28685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571188#comment-17571188
]
Paul Lin commented on FLINK-28685:
--
Please assign this issue to me. I have fixed it in our in-house branch. Thanks!
> Optimize CatalogManager getTable implementation
> ---
>
> Key: FLINK-28685
> URL: https://issues.apache.org/jira/browse/FLINK-28685
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.16.0
>Reporter: Paul Lin
>Priority: Major
>
> Currently, Catalog#getTable(ObjectPath) might be abused in CatalogManager.
> CatalogManager gets an ObjectPath directly, instead of checking the existence
> of the table first, which could surprise Catalog developers and may get
> unexpected exceptions other than TableNotFoundExceptions.
> {code:java}
> private Optional getUnresolvedTable(ObjectIdentifier
> objectIdentifier) {
> Catalog currentCatalog = catalogs.get(objectIdentifier.getCatalogName());
> ObjectPath objectPath = objectIdentifier.toObjectPath();
> if (currentCatalog != null) {
> try
> { final CatalogBaseTable table = currentCatalog.getTable(objectPath); return
> Optional.of(table); }
> catch (TableNotExistException e)
> { // Ignore. }
> }
> return Optional.empty();
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28685) Optimize CatalogManager getTable implementation
[
https://issues.apache.org/jira/browse/FLINK-28685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Lin updated FLINK-28685:
-
Description:
Currently, Catalog#getTable(ObjectPath) might be abused in CatalogManager.
CatalogManager gets an ObjectPath directly, instead of checking the existence
of the table first, which could surprise Catalog developers and may get
unexpected exceptions other than TableNotFoundExceptions.
{code:java}
private Optional getUnresolvedTable(ObjectIdentifier
objectIdentifier) {
Catalog currentCatalog = catalogs.get(objectIdentifier.getCatalogName());
ObjectPath objectPath = objectIdentifier.toObjectPath();
if (currentCatalog != null) {
try
{ final CatalogBaseTable table = currentCatalog.getTable(objectPath); return
Optional.of(table); }
catch (TableNotExistException e)
{ // Ignore. }
}
return Optional.empty();
}{code}
was:
Currently, Catalog#getTable(ObjectPath) might be abused in CatalogManager.
CatalogManager gets an ObjectPath directly, instead of checking the existence
of the table first, which could surprise Catalog developers and may get
unexpected exceptions other than TableNotFoundExceptions.
```
private Optional getUnresolvedTable(ObjectIdentifier
objectIdentifier) {
Catalog currentCatalog =
catalogs.get(objectIdentifier.getCatalogName());
ObjectPath objectPath = objectIdentifier.toObjectPath();
if (currentCatalog != null) {
try {
final CatalogBaseTable table =
currentCatalog.getTable(objectPath);
return Optional.of(table);
} catch (TableNotExistException e) {
// Ignore.
}
}
return Optional.empty();
}
```
> Optimize CatalogManager getTable implementation
> ---
>
> Key: FLINK-28685
> URL: https://issues.apache.org/jira/browse/FLINK-28685
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.16.0
>Reporter: Paul Lin
>Priority: Major
>
> Currently, Catalog#getTable(ObjectPath) might be abused in CatalogManager.
> CatalogManager gets an ObjectPath directly, instead of checking the existence
> of the table first, which could surprise Catalog developers and may get
> unexpected exceptions other than TableNotFoundExceptions.
> {code:java}
> private Optional getUnresolvedTable(ObjectIdentifier
> objectIdentifier) {
> Catalog currentCatalog = catalogs.get(objectIdentifier.getCatalogName());
> ObjectPath objectPath = objectIdentifier.toObjectPath();
> if (currentCatalog != null) {
> try
> { final CatalogBaseTable table = currentCatalog.getTable(objectPath); return
> Optional.of(table); }
> catch (TableNotExistException e)
> { // Ignore. }
> }
> return Optional.empty();
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-28685) Optimize CatalogManager getTable implementation
Paul Lin created FLINK-28685:
Summary: Optimize CatalogManager getTable implementation
Key: FLINK-28685
URL: https://issues.apache.org/jira/browse/FLINK-28685
Project: Flink
Issue Type: Improvement
Components: Table SQL / API
Affects Versions: 1.16.0
Reporter: Paul Lin
Currently, Catalog#getTable(ObjectPath) might be abused in CatalogManager.
CatalogManager gets an ObjectPath directly, instead of checking the existence
of the table first, which could surprise Catalog developers and may get
unexpected exceptions other than TableNotFoundExceptions.
```
private Optional getUnresolvedTable(ObjectIdentifier
objectIdentifier) {
Catalog currentCatalog =
catalogs.get(objectIdentifier.getCatalogName());
ObjectPath objectPath = objectIdentifier.toObjectPath();
if (currentCatalog != null) {
try {
final CatalogBaseTable table =
currentCatalog.getTable(objectPath);
return Optional.of(table);
} catch (TableNotExistException e) {
// Ignore.
}
}
return Optional.empty();
}
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28656) Support savepoint statements in SqlGatewayService
[ https://issues.apache.org/jira/browse/FLINK-28656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-28656: - Component/s: Table SQL / API Table SQL / Gateway > Support savepoint statements in SqlGatewayService > - > > Key: FLINK-28656 > URL: https://issues.apache.org/jira/browse/FLINK-28656 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Gateway >Reporter: Paul Lin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28657) Support drop savepoint statement in SqlGatewayService
[ https://issues.apache.org/jira/browse/FLINK-28657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-28657: - Component/s: Table SQL / API Table SQL / Gateway > Support drop savepoint statement in SqlGatewayService > - > > Key: FLINK-28657 > URL: https://issues.apache.org/jira/browse/FLINK-28657 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Gateway >Reporter: Paul Lin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28658) Add document for job lifecycle statements
Paul Lin created FLINK-28658: Summary: Add document for job lifecycle statements Key: FLINK-28658 URL: https://issues.apache.org/jira/browse/FLINK-28658 Project: Flink Issue Type: Sub-task Components: Documentation, Table SQL / API Reporter: Paul Lin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28657) Support drop savepoint statement in SqlGatewayService
Paul Lin created FLINK-28657: Summary: Support drop savepoint statement in SqlGatewayService Key: FLINK-28657 URL: https://issues.apache.org/jira/browse/FLINK-28657 Project: Flink Issue Type: Sub-task Reporter: Paul Lin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28656) Support savepoint statements in SqlGatewayService
Paul Lin created FLINK-28656: Summary: Support savepoint statements in SqlGatewayService Key: FLINK-28656 URL: https://issues.apache.org/jira/browse/FLINK-28656 Project: Flink Issue Type: Sub-task Reporter: Paul Lin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28655) Support show jobs statement in SqlGatewatService
Paul Lin created FLINK-28655: Summary: Support show jobs statement in SqlGatewatService Key: FLINK-28655 URL: https://issues.apache.org/jira/browse/FLINK-28655 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Gateway Reporter: Paul Lin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28617) Support stop job statement in SqlGatewayService
Paul Lin created FLINK-28617: Summary: Support stop job statement in SqlGatewayService Key: FLINK-28617 URL: https://issues.apache.org/jira/browse/FLINK-28617 Project: Flink Issue Type: Sub-task Components: Table SQL / Gateway Reporter: Paul Lin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28391) DefaultBlocklistHandlerTest unstable
Paul Lin created FLINK-28391: Summary: DefaultBlocklistHandlerTest unstable Key: FLINK-28391 URL: https://issues.apache.org/jira/browse/FLINK-28391 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 1.16.0 Reporter: Paul Lin Test org.apache.flink.runtime.blocklist.DefaultBlocklistHandlerTest.testRemoveTimeoutNodes unstable. see https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_apis/build/builds/37624/logs/111 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28363) Support drop savepoint statement in SQL client
Paul Lin created FLINK-28363: Summary: Support drop savepoint statement in SQL client Key: FLINK-28363 URL: https://issues.apache.org/jira/browse/FLINK-28363 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Reporter: Paul Lin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28362) Support savepoint statements in SQL client
Paul Lin created FLINK-28362: Summary: Support savepoint statements in SQL client Key: FLINK-28362 URL: https://issues.apache.org/jira/browse/FLINK-28362 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Reporter: Paul Lin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28360) Support stop job statement in SQL client
[ https://issues.apache.org/jira/browse/FLINK-28360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-28360: - Summary: Support stop job statement in SQL client (was: Support stop job operation in SQL client) > Support stop job statement in SQL client > > > Key: FLINK-28360 > URL: https://issues.apache.org/jira/browse/FLINK-28360 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Client >Reporter: Paul Lin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28361) Support show jobs statement in SQL client
[ https://issues.apache.org/jira/browse/FLINK-28361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-28361: - Summary: Support show jobs statement in SQL client (was: Support show jobs operation in SQL client) > Support show jobs statement in SQL client > - > > Key: FLINK-28361 > URL: https://issues.apache.org/jira/browse/FLINK-28361 > Project: Flink > Issue Type: Sub-task >Reporter: Paul Lin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28361) Support show jobs operation in SQL client
Paul Lin created FLINK-28361: Summary: Support show jobs operation in SQL client Key: FLINK-28361 URL: https://issues.apache.org/jira/browse/FLINK-28361 Project: Flink Issue Type: Sub-task Reporter: Paul Lin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-28360) Support stop job operation in SQL client
Paul Lin created FLINK-28360: Summary: Support stop job operation in SQL client Key: FLINK-28360 URL: https://issues.apache.org/jira/browse/FLINK-28360 Project: Flink Issue Type: Sub-task Components: Table SQL / Client Reporter: Paul Lin -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27344) FLIP-222: Support full job lifecycle statements in SQL client
[ https://issues.apache.org/jira/browse/FLINK-27344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558714#comment-17558714 ] Paul Lin commented on FLINK-27344: -- [~jark][~fsk119][~martijnvisser] please assign this ticket to me, thx! > FLIP-222: Support full job lifecycle statements in SQL client > - > > Key: FLINK-27344 > URL: https://issues.apache.org/jira/browse/FLINK-27344 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Reporter: Paul Lin >Priority: Major > > With the efforts in FLIP-24 and FLIP-91, Flink SQL client supports submitting > SQL jobs but lacks further support for their lifecycles afterward which is > crucial for streaming use cases. That means Flink SQL client users have to > turn to other clients (e.g. CLI) or APIs (e.g. REST API) to manage the jobs, > like triggering savepoints or canceling queries, which makes the user > experience of SQL client incomplete. > Therefore, this proposal aims to complete the capability of SQL client by > adding job lifecycle statements. With these statements, users could manage > jobs and savepoints through pure SQL in SQL client. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-27344) FLIP-222: Support full job lifecycle statements in SQL client
[ https://issues.apache.org/jira/browse/FLINK-27344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-27344: - Description: With the efforts in FLIP-24 and FLIP-91, Flink SQL client supports submitting SQL jobs but lacks further support for their lifecycles afterward which is crucial for streaming use cases. That means Flink SQL client users have to turn to other clients (e.g. CLI) or APIs (e.g. REST API) to manage the jobs, like triggering savepoints or canceling queries, which makes the user experience of SQL client incomplete. Therefore, this proposal aims to complete the capability of SQL client by adding job lifecycle statements. With these statements, users could manage jobs and savepoints through pure SQL in SQL client. was: With the efforts in FLIP-24 and FLIP-91, Flink SQL client supports submitting queries but lacks further support for their lifecycles afterward which is crucial for streaming use cases. That means Flink SQL client users have to turn to other clients (e.g. CLI) or APIs (e.g. REST API) to manage the queries, like triggering savepoints or canceling queries, which makes the user experience of SQL client incomplete. Therefore, this proposal aims to complete the capability of SQL client by adding query lifecycle statements. With these statements, users could manage queries and savepoints through pure SQL in SQL client. Summary: FLIP-222: Support full job lifecycle statements in SQL client (was: FLIP-222: Support full query lifecycle statements in SQL client) > FLIP-222: Support full job lifecycle statements in SQL client > - > > Key: FLINK-27344 > URL: https://issues.apache.org/jira/browse/FLINK-27344 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Client >Reporter: Paul Lin >Priority: Major > > With the efforts in FLIP-24 and FLIP-91, Flink SQL client supports submitting > SQL jobs but lacks further support for their lifecycles afterward which is > crucial for streaming use cases. That means Flink SQL client users have to > turn to other clients (e.g. CLI) or APIs (e.g. REST API) to manage the jobs, > like triggering savepoints or canceling queries, which makes the user > experience of SQL client incomplete. > Therefore, this proposal aims to complete the capability of SQL client by > adding job lifecycle statements. With these statements, users could manage > jobs and savepoints through pure SQL in SQL client. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (FLINK-25558) TableResult OK may be empty if returned by multiple queries
[
https://issues.apache.org/jira/browse/FLINK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Lin resolved FLINK-25558.
--
Resolution: Fixed
> TableResult OK may be empty if returned by multiple queries
> ---
>
> Key: FLINK-25558
> URL: https://issues.apache.org/jira/browse/FLINK-25558
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.14.0
>Reporter: Paul Lin
>Priority: Major
>
> Many queries return `TABLE_RESULT_OK`, which is a static member for reducing
> memory overhead. However, TableResult contains an iterator that can be read
> once. That means if a query returns `TABLE_RESULT_OK` and it is read,
> then`TABLE_RESULT_OK` returned by the following queries would be considered
> empty.
> It can be reproduced by simply printing two query results.
> ```
> tEnv.executeSql("create table tbl_a (a string)").print();
> tEnv.executeSql("create table tbl_b (a string)").print();
> ```
> The output would be
> ```
> ++
> | result |
> ++
> | OK |
> ++
> 1 row in set
> Empty set
> ```
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Updated] (FLINK-27585) JdbcOutpuFormat fails to flush data on checkpoint and causes data loss
[ https://issues.apache.org/jira/browse/FLINK-27585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-27585: - Description: Currently, JdbcOutpuFormat may periodically flush the batches, and store the exception if there's one. The exception would not be thrown out to flink runtime immediately, instead, it would be checked when a new record in written or flush() is called on checkpoint snapshot. However, in a job I see, when there's an exception already, the flush() called by checkpointed function would not recognize the exception, and the checkpoint would still succeed regardless of the flush exception, which makes the failed batch silently lost. I'm upgrading it to 1.14 to see if it happens still. was: Currently, JdbcOutpuFormat may periodically flush the batches, and store the exception if there's one. The exception would not be thrown out to flink runtime immediately, instead, it would be checked when a new record in written or flush() is called on checkpoint snapshot. However, in a job I see, when there's an exception already, the flush() called by checkpointed function would not recognize the exception, and the checkpoint would still succeed regardless of the flush exception, which makes the failed batch silently lost. I'm upgrading it to 1.14 to see if it happens still. > JdbcOutpuFormat fails to flush data on checkpoint and causes data loss > -- > > Key: FLINK-27585 > URL: https://issues.apache.org/jira/browse/FLINK-27585 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Affects Versions: 1.12.3 >Reporter: Paul Lin >Priority: Critical > > Currently, JdbcOutpuFormat may periodically flush the batches, and store the > exception if there's one. The exception would not be thrown out to flink > runtime immediately, instead, it would be checked when a new record in > written or flush() is called on checkpoint snapshot. > However, in a job I see, when there's an exception already, the flush() > called by checkpointed function would not recognize the exception, and the > checkpoint would still succeed regardless of the flush exception, which makes > the failed batch silently lost. > I'm upgrading it to 1.14 to see if it happens still. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27585) JdbcOutpuFormat fails to flush data on checkpoint and causes data loss
Paul Lin created FLINK-27585: Summary: JdbcOutpuFormat fails to flush data on checkpoint and causes data loss Key: FLINK-27585 URL: https://issues.apache.org/jira/browse/FLINK-27585 Project: Flink Issue Type: New Feature Components: Connectors / JDBC Affects Versions: 1.12.3 Reporter: Paul Lin Currently, JdbcOutpuFormat may periodically flush the batches, and store the exception if there's one. The exception would not be thrown out to flink runtime immediately, instead, it would be checked when a new record in written or flush() is called on checkpoint snapshot. However, in a job I see, when there's an exception already, the flush() called by checkpointed function would not recognize the exception, and the checkpoint would still succeed regardless of the flush exception, which makes the failed batch silently lost. I'm upgrading it to 1.14 to see if it happens still. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27344) FLIP-222: Support full query lifecycle statements in SQL client
Paul Lin created FLINK-27344: Summary: FLIP-222: Support full query lifecycle statements in SQL client Key: FLINK-27344 URL: https://issues.apache.org/jira/browse/FLINK-27344 Project: Flink Issue Type: New Feature Components: Table SQL / Client Reporter: Paul Lin With the efforts in FLIP-24 and FLIP-91, Flink SQL client supports submitting queries but lacks further support for their lifecycles afterward which is crucial for streaming use cases. That means Flink SQL client users have to turn to other clients (e.g. CLI) or APIs (e.g. REST API) to manage the queries, like triggering savepoints or canceling queries, which makes the user experience of SQL client incomplete. Therefore, this proposal aims to complete the capability of SQL client by adding query lifecycle statements. With these statements, users could manage queries and savepoints through pure SQL in SQL client. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-26651) The connector CI build timeout in azure
[
https://issues.apache.org/jira/browse/FLINK-26651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510246#comment-17510246
]
Paul Lin commented on FLINK-26651:
--
[~leonard] Looks like it's related to this bug
https://github.com/apache/flink/pull/17318
> The connector CI build timeout in azure
> ---
>
> Key: FLINK-26651
> URL: https://issues.apache.org/jira/browse/FLINK-26651
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Leonard Xu
>Priority: Major
> Fix For: 1.16.0
>
>
> {code:java}
> his task took already 95% of the available time budget of 234 minutes
> ...
> =
> === WARNING: Killing task ===
> =
> ./tools/azure-pipelines/uploading_watchdog.sh: line 76: 304 Terminated
> $COMMAND
> {code}
> Failed instance:
> https://dev.azure.com/leonardBang/Azure_CI/_build/results?buildId=676=logs=dafbab6d-4616-5d7b-ee37-3c54e4828fd7=e204f081-e6cd-5c04-4f4c-919639b63be9
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26720) Support stop-with-savepoint on web UI
Paul Lin created FLINK-26720: Summary: Support stop-with-savepoint on web UI Key: FLINK-26720 URL: https://issues.apache.org/jira/browse/FLINK-26720 Project: Flink Issue Type: New Feature Components: Runtime / Web Frontend Affects Versions: 1.15.0 Reporter: Paul Lin Currently, web UI supports cancel of the running jobs, but no with-savepoint option. Adding that would be convenient for web UI users. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-23513) Remove old connectors and formats stack around descriptors
[
https://issues.apache.org/jira/browse/FLINK-23513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Lin updated FLINK-23513:
-
Attachment: Catch(03-16-16-22-14).jpg
> Remove old connectors and formats stack around descriptors
> --
>
> Key: FLINK-23513
> URL: https://issues.apache.org/jira/browse/FLINK-23513
> Project: Flink
> Issue Type: Sub-task
>Reporter: Ingo Bürk
>Assignee: Ingo Bürk
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0
>
> Attachments: Catch(03-16-16-22-14).jpg
>
>
> This removes the old connectors and formats stack that has been deprecated as
> part of FLIP-95. The stack used the outdated
> {{org.apache.flink.table.factories.TableFactory}} which has been replaced
> with subclasses of {{org.apache.flink.table.factories.Factory}}. Also the
> corresponding descriptors from {{org.apache.flink.table.descriptors}} have
> been removed due to the removal of {{TableEnvironment#connect}}. It has been
> replaced with the new {{org.apache.flink.table.api.TableDescriptor}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26618) Remove jar statement not aligned with pipleline.jars
[ https://issues.apache.org/jira/browse/FLINK-26618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17506662#comment-17506662 ] Paul Lin commented on FLINK-26618: -- cc [~jark][~twalthr] > Remove jar statement not aligned with pipleline.jars > > > Key: FLINK-26618 > URL: https://issues.apache.org/jira/browse/FLINK-26618 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.14.3 >Reporter: Paul Lin >Priority: Major > Labels: pull-request-available > > Currently, `remove jar` statement doesn't remove the corresponding jars in > pipeline.jars. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26618) Remove jar statement not aligned with pipleline.jars
[ https://issues.apache.org/jira/browse/FLINK-26618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-26618: - Description: Currently, `remove jar` statement doesn't remove the corresponding jars in pipeline.jars. (was: Currently, `remove jar` statement doesn't remove the corresponding jars in pipeline options.) > Remove jar statement not aligned with pipleline.jars > > > Key: FLINK-26618 > URL: https://issues.apache.org/jira/browse/FLINK-26618 > Project: Flink > Issue Type: Bug >Affects Versions: 1.14.3 >Reporter: Paul Lin >Priority: Major > > Currently, `remove jar` statement doesn't remove the corresponding jars in > pipeline.jars. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26618) Remove jar statement not aligned with pipleline.jars
[ https://issues.apache.org/jira/browse/FLINK-26618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-26618: - Component/s: Table SQL / Client > Remove jar statement not aligned with pipleline.jars > > > Key: FLINK-26618 > URL: https://issues.apache.org/jira/browse/FLINK-26618 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.14.3 >Reporter: Paul Lin >Priority: Major > > Currently, `remove jar` statement doesn't remove the corresponding jars in > pipeline.jars. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26618) Remove jar statement not aligned with pipleline.jars
Paul Lin created FLINK-26618: Summary: Remove jar statement not aligned with pipleline.jars Key: FLINK-26618 URL: https://issues.apache.org/jira/browse/FLINK-26618 Project: Flink Issue Type: Bug Affects Versions: 1.14.3 Reporter: Paul Lin Currently, `remove jar` statement doesn't remove the corresponding jars in pipeline options. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25558) TableResult OK may be empty if returned by multiple queries
[
https://issues.apache.org/jira/browse/FLINK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17500575#comment-17500575
]
Paul Lin commented on FLINK-25558:
--
Seems that the issue is fixed by FLINK-24685 in Flink 1.15.0.
> TableResult OK may be empty if returned by multiple queries
> ---
>
> Key: FLINK-25558
> URL: https://issues.apache.org/jira/browse/FLINK-25558
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.14.0
>Reporter: Paul Lin
>Priority: Major
>
> Many queries return `TABLE_RESULT_OK`, which is a static member for reducing
> memory overhead. However, TableResult contains an iterator that can be read
> once. That means if a query returns `TABLE_RESULT_OK` and it is read,
> then`TABLE_RESULT_OK` returned by the following queries would be considered
> empty.
> It can be reproduced by simply printing two query results.
> ```
> tEnv.executeSql("create table tbl_a (a string)").print();
> tEnv.executeSql("create table tbl_b (a string)").print();
> ```
> The output would be
> ```
> ++
> | result |
> ++
> | OK |
> ++
> 1 row in set
> Empty set
> ```
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-26204) set table.optimizer.distinct-agg.split.enabled to true and using Window TVF CUMULATE to count users, It will appear that the value of the current step window time is gre
[ https://issues.apache.org/jira/browse/FLINK-26204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-26204: - Attachment: output-of-window-tvf-test.png > set table.optimizer.distinct-agg.split.enabled to true and using Window TVF > CUMULATE to count users, It will appear that the value of the current step > window time is greater than the value of the next step window time > - > > Key: FLINK-26204 > URL: https://issues.apache.org/jira/browse/FLINK-26204 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.14.0, 1.14.3 >Reporter: Bo Huang >Priority: Blocker > Attachments: TestApp.java, WindowTvfTest.java, > output-of-window-tvf-test.png, test.png > > > set table.optimizer.distinct-agg.split.enabled to true > using Window TVF CUMULATE to count users > It will appear that the value of the current step window time is greater than > the value of the next step window time -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-26204) set table.optimizer.distinct-agg.split.enabled to true and using Window TVF CUMULATE to count users, It will appear that the value of the current step window time
[ https://issues.apache.org/jira/browse/FLINK-26204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497257#comment-17497257 ] Paul Lin edited comment on FLINK-26204 at 2/24/22, 8:43 AM: I've reproduced the problem with minimum codes and dependencies. Please see [^WindowTvfTest.java]. It would take about 15 minutes before the wrong result shows up. [~jark] [~leonard] was (Author: paul lin): I've reproduced the problem with minimum external dependencies. Please see [^WindowTvfTest.java]. It would take about 15 minutes before the wrong result shows up. [~jark] [~leonard] > set table.optimizer.distinct-agg.split.enabled to true and using Window TVF > CUMULATE to count users, It will appear that the value of the current step > window time is greater than the value of the next step window time > - > > Key: FLINK-26204 > URL: https://issues.apache.org/jira/browse/FLINK-26204 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.14.0, 1.14.3 >Reporter: Bo Huang >Priority: Blocker > Attachments: TestApp.java, WindowTvfTest.java, > output-of-window-tvf-test.png, test.png > > > set table.optimizer.distinct-agg.split.enabled to true > using Window TVF CUMULATE to count users > It will appear that the value of the current step window time is greater than > the value of the next step window time -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-26204) set table.optimizer.distinct-agg.split.enabled to true and using Window TVF CUMULATE to count users, It will appear that the value of the current step window time
[ https://issues.apache.org/jira/browse/FLINK-26204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497257#comment-17497257 ] Paul Lin edited comment on FLINK-26204 at 2/24/22, 8:41 AM: I've reproduced the problem with minimum external dependencies. Please see [^WindowTvfTest.java]. It would take about 15 minutes before the wrong result shows up. [~jark] [~leonard] was (Author: paul lin): I've reproduced the problem with minimum external dependencies. Please see [^WindowTvfTest.java]. It would take about 15 minutes before the wrong result shows up. [~jark] > set table.optimizer.distinct-agg.split.enabled to true and using Window TVF > CUMULATE to count users, It will appear that the value of the current step > window time is greater than the value of the next step window time > - > > Key: FLINK-26204 > URL: https://issues.apache.org/jira/browse/FLINK-26204 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.14.0, 1.14.3 >Reporter: Bo Huang >Priority: Blocker > Attachments: TestApp.java, WindowTvfTest.java, > output-of-window-tvf-test.png, test.png > > > set table.optimizer.distinct-agg.split.enabled to true > using Window TVF CUMULATE to count users > It will appear that the value of the current step window time is greater than > the value of the next step window time -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26204) set table.optimizer.distinct-agg.split.enabled to true and using Window TVF CUMULATE to count users, It will appear that the value of the current step window time is g
[ https://issues.apache.org/jira/browse/FLINK-26204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17497257#comment-17497257 ] Paul Lin commented on FLINK-26204: -- I've reproduced the problem with minimum external dependencies. Please see [^WindowTvfTest.java]. It would take about 15 minutes before the wrong result shows up. [~jark] > set table.optimizer.distinct-agg.split.enabled to true and using Window TVF > CUMULATE to count users, It will appear that the value of the current step > window time is greater than the value of the next step window time > - > > Key: FLINK-26204 > URL: https://issues.apache.org/jira/browse/FLINK-26204 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.14.0, 1.14.3 >Reporter: Bo Huang >Priority: Blocker > Attachments: TestApp.java, WindowTvfTest.java, test.png > > > set table.optimizer.distinct-agg.split.enabled to true > using Window TVF CUMULATE to count users > It will appear that the value of the current step window time is greater than > the value of the next step window time -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26204) set table.optimizer.distinct-agg.split.enabled to true and using Window TVF CUMULATE to count users, It will appear that the value of the current step window time is gre
[ https://issues.apache.org/jira/browse/FLINK-26204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-26204: - Attachment: WindowTvfTest.java > set table.optimizer.distinct-agg.split.enabled to true and using Window TVF > CUMULATE to count users, It will appear that the value of the current step > window time is greater than the value of the next step window time > - > > Key: FLINK-26204 > URL: https://issues.apache.org/jira/browse/FLINK-26204 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.14.0, 1.14.3 >Reporter: Bo Huang >Priority: Blocker > Attachments: TestApp.java, WindowTvfTest.java, test.png > > > set table.optimizer.distinct-agg.split.enabled to true > using Window TVF CUMULATE to count users > It will appear that the value of the current step window time is greater than > the value of the next step window time -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26204) set table.optimizer.distinct-agg.split.enabled to true and using Window TVF CUMULATE to count users, It will appear that the value of the current step window time is g
[ https://issues.apache.org/jira/browse/FLINK-26204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17495963#comment-17495963 ] Paul Lin commented on FLINK-26204: -- [~jark] I reproduced the problem locally. It seems that it only affects event time aggregates. I guess there's something wrong with the interpretation of retract streams when `table.optimizer.distinct-agg.split.enabled` is on? > set table.optimizer.distinct-agg.split.enabled to true and using Window TVF > CUMULATE to count users, It will appear that the value of the current step > window time is greater than the value of the next step window time > - > > Key: FLINK-26204 > URL: https://issues.apache.org/jira/browse/FLINK-26204 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.14.0, 1.14.3 >Reporter: Bo Huang >Priority: Blocker > Attachments: TestApp.java, test.png > > > set table.optimizer.distinct-agg.split.enabled to true > using Window TVF CUMULATE to count users > It will appear that the value of the current step window time is greater than > the value of the next step window time -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25558) TableResult OK may be empty if returned by multiple queries
[
https://issues.apache.org/jira/browse/FLINK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470039#comment-17470039
]
Paul Lin commented on FLINK-25558:
--
Working on the fix. Please assign this issue to me.
> TableResult OK may be empty if returned by multiple queries
> ---
>
> Key: FLINK-25558
> URL: https://issues.apache.org/jira/browse/FLINK-25558
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.14.0
>Reporter: Paul Lin
>Priority: Major
>
> Many queries return `TABLE_RESULT_OK`, which is a static member for reducing
> memory overhead. However, TableResult contains an iterator that can be read
> once. That means if a query returns `TABLE_RESULT_OK` and it is read,
> then`TABLE_RESULT_OK` returned by the following queries would be considered
> empty.
> It can be reproduced by simply printing two query results.
> ```
> tEnv.executeSql("create table tbl_a (a string)").print();
> tEnv.executeSql("create table tbl_b (a string)").print();
> ```
> The output would be
> ```
> ++
> | result |
> ++
> | OK |
> ++
> 1 row in set
> Empty set
> ```
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25558) TableResult OK may be empty if returned by multiple queries
[
https://issues.apache.org/jira/browse/FLINK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Lin updated FLINK-25558:
-
Component/s: Table SQL / API
Affects Version/s: 1.14.0
> TableResult OK may be empty if returned by multiple queries
> ---
>
> Key: FLINK-25558
> URL: https://issues.apache.org/jira/browse/FLINK-25558
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.14.0
>Reporter: Paul Lin
>Priority: Major
>
> Many queries return `TABLE_RESULT_OK`, which is a static member for reducing
> memory overhead. However, TableResult contains an iterator that can be read
> once. That means if a query returns `TABLE_RESULT_OK` and it is read,
> then`TABLE_RESULT_OK` returned by the following queries would be considered
> empty.
> It can be reproduced by simply printing two query results.
> ```
> tEnv.executeSql("create table tbl_a (a string)").print();
> tEnv.executeSql("create table tbl_b (a string)").print();
> ```
> The output would be
> ```
> ++
> | result |
> ++
> | OK |
> ++
> 1 row in set
> Empty set
> ```
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25558) TableResult OK may be empty if returned by multiple queries
[
https://issues.apache.org/jira/browse/FLINK-25558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Lin updated FLINK-25558:
-
Description:
Many queries return `TABLE_RESULT_OK`, which is a static member for reducing
memory overhead. However, TableResult contains an iterator that can be read
once. That means if a query returns `TABLE_RESULT_OK` and it is read,
then`TABLE_RESULT_OK` returned by the following queries would be considered
empty.
It can be reproduced by simply printing two query results.
```
tEnv.executeSql("create table tbl_a (a string)").print();
tEnv.executeSql("create table tbl_b (a string)").print();
```
The output would be
```
++
| result |
++
| OK |
++
1 row in set
Empty set
```
was:
Many queries return `TABLE_RESULT_OK`, which is a static member for reducing
memory overhead. However, TableResult contains an iterator that can be read
once. That means if a query returns `TABLE_RESULT_OK` and it is read,
then`TABLE_RESULT_OK` returned by the following queries would be considered
empty.
It can be reproduced by simply printing two query results.
```
tEnv.executeSql("create table tbl_a (a string)").print();
tEnv.executeSql("create table tbl_b (a string)").print();
```
The output would be
```
++
| result |
++
| OK |
++
1 row in set
Empty set
```
> TableResult OK may be empty if returned by multiple queries
> ---
>
> Key: FLINK-25558
> URL: https://issues.apache.org/jira/browse/FLINK-25558
> Project: Flink
> Issue Type: Bug
>Reporter: Paul Lin
>Priority: Major
>
> Many queries return `TABLE_RESULT_OK`, which is a static member for reducing
> memory overhead. However, TableResult contains an iterator that can be read
> once. That means if a query returns `TABLE_RESULT_OK` and it is read,
> then`TABLE_RESULT_OK` returned by the following queries would be considered
> empty.
> It can be reproduced by simply printing two query results.
> ```
> tEnv.executeSql("create table tbl_a (a string)").print();
> tEnv.executeSql("create table tbl_b (a string)").print();
> ```
> The output would be
> ```
> ++
> | result |
> ++
> | OK |
> ++
> 1 row in set
> Empty set
> ```
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25558) TableResult OK may be empty if returned by multiple queries
Paul Lin created FLINK-25558:
Summary: TableResult OK may be empty if returned by multiple
queries
Key: FLINK-25558
URL: https://issues.apache.org/jira/browse/FLINK-25558
Project: Flink
Issue Type: Bug
Reporter: Paul Lin
Many queries return `TABLE_RESULT_OK`, which is a static member for reducing
memory overhead. However, TableResult contains an iterator that can be read
once. That means if a query returns `TABLE_RESULT_OK` and it is read,
then`TABLE_RESULT_OK` returned by the following queries would be considered
empty.
It can be reproduced by simply printing two query results.
```
tEnv.executeSql("create table tbl_a (a string)").print();
tEnv.executeSql("create table tbl_b (a string)").print();
```
The output would be
```
++
| result |
++
| OK |
++
1 row in set
Empty set
```
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-12022) Enable StreamWriter to update file length on sync flush
[ https://issues.apache.org/jira/browse/FLINK-12022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin closed FLINK-12022. Resolution: Won't Fix > Enable StreamWriter to update file length on sync flush > --- > > Key: FLINK-12022 > URL: https://issues.apache.org/jira/browse/FLINK-12022 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem >Affects Versions: 1.6.4, 1.7.2 >Reporter: Paul Lin >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, stale-minor > > Currently, users of file systems that do not support truncating have to > struggle with BucketingSink and use its valid length file to indicate the > checkpointed data position. The problem is that by default the file length > will only be updated when a block is full or the file is closed, but when the > job crashes and the file is not closed properly, the file length is still > behind its actual value and the checkpointed file length. When the job > restarts, it looks like data loss, because the valid length is bigger than > the file. This situation lasts until namenode notices the change of block > size of the file, and it could be half an hour or more. > So I propose to add an option to StreamWriterBase to update file lengths on > each flush. This can be expensive because it involves namenode and should be > used when strong consistency is needed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24948) Special character in column names breaks JDBC statement parsing
[ https://issues.apache.org/jira/browse/FLINK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459681#comment-17459681 ] Paul Lin commented on FLINK-24948: -- [~jark] I've prepared a PR, PTAL. Thanks! > Special character in column names breaks JDBC statement parsing > --- > > Key: FLINK-24948 > URL: https://issues.apache.org/jira/browse/FLINK-24948 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.12.4 >Reporter: Paul Lin >Assignee: Paul Lin >Priority: Major > Labels: pull-request-available > > Currently, JDBC connector assumes columns names respect Java identifier > naming restrictions, but Databases that support JDBC may have different > naming restrictions. For example, MySQL allows dots and colons in column > names. In that case, JDBC connector would have trouble parsing the SQL. > We could fix this by validating field names in `JdbcDmlOptions`. In addition, > it'd be good to clarify the naming restrictions of Flink SQL, so users and > connector developers would know the standard. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-22300: - Issue Type: Bug (was: Improvement) > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450970#comment-17450970 ] Paul Lin commented on FLINK-22300: -- [~arvid] For a usage case, how about a global window (no window size is defined) that relies on a processing time TimeEvictor to clean up the elements? RWT the solution, I think maybe in case of processing time we can attach the processing timestamp to the TimestampedValue in window assigners and erase them after evictors. WDYT? > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450847#comment-17450847 ] Paul Lin commented on FLINK-22300: -- [~arvid] Please let me clarify a bit more. The problem is that TimeEvictor would not be effective with processing time, because instead of getting the current system timestamp (as `now()` you mentioned), it would extract timestamps from TimestampedValue(backed by StreamRecord), see [code|https://github.com/apache/flink/blob/6f05e6fdadd5cd8f27983bc8140d05f1c3557bb6/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/evictors/TimeEvictor.java#L70]. However, with processing time, TimestampedValue has no timestamp, so `evict()` would always return directly. > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448954#comment-17448954 ] Paul Lin commented on FLINK-22300: -- Any follow-ups? I would be happy to take this issue if we reach a consensus on the solution. [~trohrmann] [~arvid] > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24948) Special character in column names breaks JDBC statement parsing
[ https://issues.apache.org/jira/browse/FLINK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447789#comment-17447789 ] Paul Lin commented on FLINK-24948: -- Yes, I'm working on it. May I ask your opinion on the solution? Should we forbid column names with special characters or just fix the parsing of named parameters? > Special character in column names breaks JDBC statement parsing > --- > > Key: FLINK-24948 > URL: https://issues.apache.org/jira/browse/FLINK-24948 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.12.4 >Reporter: Paul Lin >Assignee: Paul Lin >Priority: Major > > Currently, JDBC connector assumes columns names respect Java identifier > naming restrictions, but Databases that support JDBC may have different > naming restrictions. For example, MySQL allows dots and colons in column > names. In that case, JDBC connector would have trouble parsing the SQL. > We could fix this by validating field names in `JdbcDmlOptions`. In addition, > it'd be good to clarify the naming restrictions of Flink SQL, so users and > connector developers would know the standard. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24948) Special character in column names breaks JDBC statement parsing
[ https://issues.apache.org/jira/browse/FLINK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447746#comment-17447746 ] Paul Lin commented on FLINK-24948: -- [~jark] Sure. For example, a table name tbl_a with schema `(.a varchar(64))` would result in a statement like `insert into tbl_a (`.a`) values (:.a)` (in which the columns names are quoted, for you 2nd question). The statement would be further processed by `FieldNamedPreparedStatementImpl`, in which the named parameter would be extracted by `:` (the delimiter) and characters that are no part of Java identifiers, see [code|https://github.com/apache/flink/blob/e431e6b686e904b6b6f6dabfed8796c67898461b/flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/statement/FieldNamedPreparedStatementImpl.java#L223]. I think the problem lies in the parsing of named parameters, as it simply extracts the names via some reserved characters, but these characters can also be part of a valid column name. > Special character in column names breaks JDBC statement parsing > --- > > Key: FLINK-24948 > URL: https://issues.apache.org/jira/browse/FLINK-24948 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.12.4 >Reporter: Paul Lin >Assignee: Paul Lin >Priority: Major > > Currently, JDBC connector assumes columns names respect Java identifier > naming restrictions, but Databases that support JDBC may have different > naming restrictions. For example, MySQL allows dots and colons in column > names. In that case, JDBC connector would have trouble parsing the SQL. > We could fix this by validating field names in `JdbcDmlOptions`. In addition, > it'd be good to clarify the naming restrictions of Flink SQL, so users and > connector developers would know the standard. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24948) Special character in column names breaks JDBC statement parsing
[ https://issues.apache.org/jira/browse/FLINK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445685#comment-17445685 ] Paul Lin commented on FLINK-24948: -- Working on the fix. Please assign this issue to me. > Special character in column names breaks JDBC statement parsing > --- > > Key: FLINK-24948 > URL: https://issues.apache.org/jira/browse/FLINK-24948 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.12.4 >Reporter: Paul Lin >Priority: Major > > Currently, JDBC connector assumes columns names respect Java identifier > naming restrictions, but Databases that support JDBC may have different > naming restrictions. For example, MySQL allows dots and colons in column > names. In that case, JDBC connector would have trouble parsing the SQL. > We could fix this by validating field names in `JdbcDmlOptions`. In addition, > it'd be good to clarify the naming restrictions of Flink SQL, so users and > connector developers would know the standard. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24948) Special character in column names breaks JDBC statement parsing
Paul Lin created FLINK-24948: Summary: Special character in column names breaks JDBC statement parsing Key: FLINK-24948 URL: https://issues.apache.org/jira/browse/FLINK-24948 Project: Flink Issue Type: Bug Components: Connectors / JDBC Affects Versions: 1.12.4 Reporter: Paul Lin Currently, JDBC connector assumes columns names respect Java identifier naming restrictions, but Databases that support JDBC may have different naming restrictions. For example, MySQL allows dots and colons in column names. In that case, JDBC connector would have trouble parsing the SQL. We could fix this by validating field names in `JdbcDmlOptions`. In addition, it'd be good to clarify the naming restrictions of Flink SQL, so users and connector developers would know the standard. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24628) Invalid JDBC query template when no fields are selected
[ https://issues.apache.org/jira/browse/FLINK-24628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17433601#comment-17433601 ] Paul Lin commented on FLINK-24628: -- I'm working on the fix. Please assign this to me. > Invalid JDBC query template when no fields are selected > --- > > Key: FLINK-24628 > URL: https://issues.apache.org/jira/browse/FLINK-24628 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: 1.12.3 >Reporter: Paul Lin >Priority: Minor > > A query like `select uuid() from mysql_table` will result in an invalid query > template like `select from mysql_table` in JdbcDynamicTableSource. > We should avoid making a TableScan when there're no relevant fields are > actually used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24628) Invalid JDBC query template when no fields are selected
Paul Lin created FLINK-24628: Summary: Invalid JDBC query template when no fields are selected Key: FLINK-24628 URL: https://issues.apache.org/jira/browse/FLINK-24628 Project: Flink Issue Type: Bug Components: Connectors / JDBC Affects Versions: 1.12.3 Reporter: Paul Lin A query like `select uuid() from mysql_table` will result in an invalid query template like `select from mysql_table` in JdbcDynamicTableSource. We should avoid making a TableScan when there're no relevant fields are actually used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24319) Jar not found in ClassLoaderITCase
Paul Lin created FLINK-24319: Summary: Jar not found in ClassLoaderITCase Key: FLINK-24319 URL: https://issues.apache.org/jira/browse/FLINK-24319 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 1.13.2 Reporter: Paul Lin This commit ( [https://github.com/apache/flink/commit/7aa510c9e5dad2f0d2bcdcf1abc6b4f9e0374470#diff-9c5fb3d1b7e3b0f54bc5c4182965c4fe1f9023d449017cece3005d3f90e8e4d8)] changed the working dir of surefire plugin to project base dir, which leads to jar not found exception in ClassLoaderITCase, because it locates jar files via relative paths. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19816) Flink restored from a wrong checkpoint (a very old one and not the last completed one)
[
https://issues.apache.org/jira/browse/FLINK-19816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17408693#comment-17408693
]
Paul Lin commented on FLINK-19816:
--
Still getting this error with Flink 1.12.3. The jobmanager logs are attached,
please take a look. [~trohrmann] [^jm.log]
> Flink restored from a wrong checkpoint (a very old one and not the last
> completed one)
> --
>
> Key: FLINK-19816
> URL: https://issues.apache.org/jira/browse/FLINK-19816
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Steven Zhen Wu
>Assignee: Till Rohrmann
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.11.3, 1.12.0
>
> Attachments: jm.log
>
>
> h2. Summary
> Upon failure, it seems that Flink didn't restore from the last completed
> checkpoint. Instead, it restored from a very old checkpoint. As a result,
> Kafka offsets are invalid and caused the job to replay from the beginning as
> Kafka consumer "auto.offset.reset" was set to "EARLIEST".
> This is an embarrassingly parallel stateless job. Parallelism is over 1,000.
> I have the full log file from jobmanager at INFO level available upon request.
> h2. Sequence of events from the logs
> Just before the failure, checkpoint *210768* completed.
> {code}
> 2020-10-25 02:35:05,970 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
> [jobmanager-future-thread-5] - Completed checkpoint 210768 for job
> 233b4938179c06974e4535ac8a868675 (4623776 bytes in 120402 ms).
> {code}
> During restart, somehow it decided to restore from a very old checkpoint
> *203531*.
> {code:java}
> 2020-10-25 02:36:03,301 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> [cluster-io-thread-3] - Start SessionDispatcherLeaderProcess.
> 2020-10-25 02:36:03,302 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> [cluster-io-thread-5] - Recover all persisted job graphs.
> 2020-10-25 02:36:03,304 INFO com.netflix.bdp.s3fs.BdpS3FileSystem
> [cluster-io-thread-25] - Deleting path:
> s3:///checkpoints/XM3B/clapp_avro-clapp_avro_nontvui/1593/233b4938179c06974e4535ac8a868675/chk-210758/c31aec1e-07a7-4193-aa00-3fbe83f9e2e6
> 2020-10-25 02:36:03,307 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> [cluster-io-thread-5] - Trying to recover job with job id
> 233b4938179c06974e4535ac8a868675.
> 2020-10-25 02:36:03,381 INFO com.netflix.bdp.s3fs.BdpS3FileSystem
> [cluster-io-thread-25] - Deleting path:
> s3:///checkpoints/Hh86/clapp_avro-clapp_avro_nontvui/1593/233b4938179c06974e4535ac8a868675/chk-210758/4ab92f70-dfcd-4212-9b7f-bdbecb9257fd
> ...
> 2020-10-25 02:36:03,427 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
> [flink-akka.actor.default-dispatcher-82003] - Recovering checkpoints from
> ZooKeeper.
> 2020-10-25 02:36:03,432 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
> [flink-akka.actor.default-dispatcher-82003] - Found 0 checkpoints in
> ZooKeeper.
> 2020-10-25 02:36:03,432 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
> [flink-akka.actor.default-dispatcher-82003] - Trying to fetch 0 checkpoints
> from storage.
> 2020-10-25 02:36:03,432 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
> [flink-akka.actor.default-dispatcher-82003] - Starting job
> 233b4938179c06974e4535ac8a868675 from savepoint
> s3:///checkpoints/metadata/clapp_avro-clapp_avro_nontvui/1113/47e2a25a8d0b696c7d0d423722bb6f54/chk-203531/_metadata
> ()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19816) Flink restored from a wrong checkpoint (a very old one and not the last completed one)
[
https://issues.apache.org/jira/browse/FLINK-19816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Lin updated FLINK-19816:
-
Attachment: jm.log
> Flink restored from a wrong checkpoint (a very old one and not the last
> completed one)
> --
>
> Key: FLINK-19816
> URL: https://issues.apache.org/jira/browse/FLINK-19816
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Steven Zhen Wu
>Assignee: Till Rohrmann
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.11.3, 1.12.0
>
> Attachments: jm.log
>
>
> h2. Summary
> Upon failure, it seems that Flink didn't restore from the last completed
> checkpoint. Instead, it restored from a very old checkpoint. As a result,
> Kafka offsets are invalid and caused the job to replay from the beginning as
> Kafka consumer "auto.offset.reset" was set to "EARLIEST".
> This is an embarrassingly parallel stateless job. Parallelism is over 1,000.
> I have the full log file from jobmanager at INFO level available upon request.
> h2. Sequence of events from the logs
> Just before the failure, checkpoint *210768* completed.
> {code}
> 2020-10-25 02:35:05,970 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
> [jobmanager-future-thread-5] - Completed checkpoint 210768 for job
> 233b4938179c06974e4535ac8a868675 (4623776 bytes in 120402 ms).
> {code}
> During restart, somehow it decided to restore from a very old checkpoint
> *203531*.
> {code:java}
> 2020-10-25 02:36:03,301 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> [cluster-io-thread-3] - Start SessionDispatcherLeaderProcess.
> 2020-10-25 02:36:03,302 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> [cluster-io-thread-5] - Recover all persisted job graphs.
> 2020-10-25 02:36:03,304 INFO com.netflix.bdp.s3fs.BdpS3FileSystem
> [cluster-io-thread-25] - Deleting path:
> s3:///checkpoints/XM3B/clapp_avro-clapp_avro_nontvui/1593/233b4938179c06974e4535ac8a868675/chk-210758/c31aec1e-07a7-4193-aa00-3fbe83f9e2e6
> 2020-10-25 02:36:03,307 INFO
> org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess
> [cluster-io-thread-5] - Trying to recover job with job id
> 233b4938179c06974e4535ac8a868675.
> 2020-10-25 02:36:03,381 INFO com.netflix.bdp.s3fs.BdpS3FileSystem
> [cluster-io-thread-25] - Deleting path:
> s3:///checkpoints/Hh86/clapp_avro-clapp_avro_nontvui/1593/233b4938179c06974e4535ac8a868675/chk-210758/4ab92f70-dfcd-4212-9b7f-bdbecb9257fd
> ...
> 2020-10-25 02:36:03,427 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
> [flink-akka.actor.default-dispatcher-82003] - Recovering checkpoints from
> ZooKeeper.
> 2020-10-25 02:36:03,432 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
> [flink-akka.actor.default-dispatcher-82003] - Found 0 checkpoints in
> ZooKeeper.
> 2020-10-25 02:36:03,432 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
> [flink-akka.actor.default-dispatcher-82003] - Trying to fetch 0 checkpoints
> from storage.
> 2020-10-25 02:36:03,432 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator
> [flink-akka.actor.default-dispatcher-82003] - Starting job
> 233b4938179c06974e4535ac8a868675 from savepoint
> s3:///checkpoints/metadata/clapp_avro-clapp_avro_nontvui/1113/47e2a25a8d0b696c7d0d423722bb6f54/chk-203531/_metadata
> ()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403704#comment-17403704
]
Paul Lin commented on FLINK-23725:
--
[~sewen] In the cases I met, the users performed a clean restart without
cleaning the directory. A common purpose is to reset the Kafka topic offsets to
skip some input.
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17403519#comment-17403519
]
Paul Lin commented on FLINK-23725:
--
[~todd5167] You're right. The commit operations should be retryable and
idempotent. IIUC, you mean overriding the already existed target file if
there's a conflict? I think that would be a problem if two files are written by
different executions without consistent states (e.g start a new job without
savepoints while keeping the files of previous executions).
How about this:
* if either the source file or the target file exists, ignore the results;
* if none of them exists, there's a data loss, thus throw an exception;
* if both of them exist, there's a name conflict, thus throw an exception.
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18592) StreamingFileSink fails due to truncating HDFS file failure
[
https://issues.apache.org/jira/browse/FLINK-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Lin updated FLINK-18592:
-
Priority: Major (was: Minor)
> StreamingFileSink fails due to truncating HDFS file failure
> ---
>
> Key: FLINK-18592
> URL: https://issues.apache.org/jira/browse/FLINK-18592
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.10.1
>Reporter: JIAN WANG
>Priority: Major
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.10.4, 1.14.0, 1.11.5
>
>
> I meet the issue on flink-1.10.1. I use flink on YARN(3.0.0-cdh6.3.2) with
> StreamingFileSink.
> code part like this:
> {code}
> public static StreamingFileSink build(String dir,
> BucketAssigner assigner, String prefix) {
> return StreamingFileSink.forRowFormat(new Path(dir), new
> SimpleStringEncoder())
> .withRollingPolicy(
> DefaultRollingPolicy
> .builder()
>
> .withRolloverInterval(TimeUnit.HOURS.toMillis(2))
>
> .withInactivityInterval(TimeUnit.MINUTES.toMillis(10))
> .withMaxPartSize(1024L * 1024L * 1024L
> * 50) // Max 50GB
> .build())
> .withBucketAssigner(assigner)
>
> .withOutputFileConfig(OutputFileConfig.builder().withPartPrefix(prefix).build())
> .build();
> }
> {code}
> The error is
> {noformat}
> java.io.IOException:
> Problem while truncating file:
> hdfs:///business_log/hashtag/2020-06-25/.hashtag-122-37.inprogress.8e65f69c-b5ba-4466-a844-ccc0a5a93de2
> {noformat}
> Due to this issue, it can not restart from the latest checkpoint and
> savepoint.
> Currently, my workaround is that we keep latest 3 checkpoint, and if it
> fails, I manually restart from penult checkpoint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402652#comment-17402652
]
Paul Lin commented on FLINK-23725:
--
[~todd5167] Would you like to provide a fix? If not I could take this issue.
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402650#comment-17402650
]
Paul Lin edited comment on FLINK-23725 at 8/21/21, 5:18 PM:
I've also met this issue. If the file name already exists, FileCommiter would
silently skip the commit, which may lead to data loss.
The root cause is that #rename would not throw exceptions if the target file
already exists or the src file doesn't exist, instead it returns false to
indicate the operation is failed, as [Hadoop
ClientProtocal|https://github.com/apache/hadoop/blob/b6d19718204af02da6e2ed0b83d5936824371fc0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java#L520)]
mentioned.
I think in both cases we should throw an exception.
was (Author: paul lin):
I've also met this issue. If the file name already exists, FileCommiter would
silently skip the commit, which may lead to data loss.
The root cause is that #rename would not throw exceptions if the target file
already exists or the src file doesn't exist, instead it returns false to
indicate the operation is failed, as [Hadoop
ClientProtocal|[https://github.com/apache/hadoop/blob/b6d19718204af02da6e2ed0b83d5936824371fc0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java#L520]|https://github.com/apache/hadoop/blob/b6d19718204af02da6e2ed0b83d5936824371fc0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java#L520)]
mentioned.
I think in both cases we should throw an exception.
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402650#comment-17402650
]
Paul Lin edited comment on FLINK-23725 at 8/21/21, 5:17 PM:
I've also met this issue. If the file name already exists, FileCommiter would
silently skip the commit, which may lead to data loss.
The root cause is that #rename would not throw exceptions if the target file
already exists or the src file doesn't exist, instead it returns false to
indicate the operation is failed, as [Hadoop
ClientProtocal|[https://github.com/apache/hadoop/blob/b6d19718204af02da6e2ed0b83d5936824371fc0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java#L520]|https://github.com/apache/hadoop/blob/b6d19718204af02da6e2ed0b83d5936824371fc0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java#L520)]
mentioned.
I think in both cases we should throw an exception.
was (Author: paul lin):
I've also met this issue. If the file name already exists, FileCommiter would
silently skip the commit, which may lead to data loss.
The root cause is that #rename would not throw exceptions if the target file
already exists or the src file doesn't exist, instead it returns false to
indicate the operation is failed, as [Hadoop
ClientProtocal]([https://github.com/apache/hadoop/blob/b6d19718204af02da6e2ed0b83d5936824371fc0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java#L520)]
mentioned.
I think in both cases we should throw an exception.
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-23725) HadoopFsCommitter, file rename failure
[
https://issues.apache.org/jira/browse/FLINK-23725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17402650#comment-17402650
]
Paul Lin commented on FLINK-23725:
--
I've also met this issue. If the file name already exists, FileCommiter would
silently skip the commit, which may lead to data loss.
The root cause is that #rename would not throw exceptions if the target file
already exists or the src file doesn't exist, instead it returns false to
indicate the operation is failed, as [Hadoop
ClientProtocal]([https://github.com/apache/hadoop/blob/b6d19718204af02da6e2ed0b83d5936824371fc0/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java#L520)]
mentioned.
I think in both cases we should throw an exception.
> HadoopFsCommitter, file rename failure
> --
>
> Key: FLINK-23725
> URL: https://issues.apache.org/jira/browse/FLINK-23725
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hadoop
> Compatibility, FileSystems
>Affects Versions: 1.11.1, 1.12.1
>Reporter: todd
>Priority: Major
>
> When the HDFS file is written, if the part file exists, only false will be
> returned if the duplicate name fails.Whether to throw an exception that
> already exists in the part, or print related logs.
>
> ```
> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.HadoopFsCommitter#commit
> public void commit() throws IOException {
> final Path src = recoverable.tempFile();
> final Path dest = recoverable.targetFile();
> final long expectedLength = recoverable.offset();
> try {
> //always return false or ture
> fs.rename(src, dest);
> } catch (IOException e) {
> throw new IOException(
> "Committing file by rename failed: " + src + " to " + dest, e);
> }
> }
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23868) JobExecutionResult printed even if suppressSysout is on
[ https://issues.apache.org/jira/browse/FLINK-23868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-23868: - Summary: JobExecutionResult printed even if suppressSysout is on (was: JobExecutionResult printed event if suppressSysout is on) > JobExecutionResult printed even if suppressSysout is on > --- > > Key: FLINK-23868 > URL: https://issues.apache.org/jira/browse/FLINK-23868 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.14.0, 1.13.2 >Reporter: Paul Lin >Priority: Minor > Labels: pull-request-available > > Environments prints job execution results to stdout by default and provides a > flag `suppressSysout` to disable the behavior. This flag is useful when > submitting jobs through REST API or other programmatic approaches. However, > JobExecutionResult is still printed when this flag is on, which looks like a > bug to me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23868) JobExecutionResult printed event if suppressSysout is on
[ https://issues.apache.org/jira/browse/FLINK-23868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17401491#comment-17401491 ] Paul Lin commented on FLINK-23868: -- Since this is a minor fix, to facilitate the discussion, I made a PR without issue assignment. > JobExecutionResult printed event if suppressSysout is on > > > Key: FLINK-23868 > URL: https://issues.apache.org/jira/browse/FLINK-23868 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.14.0, 1.13.2 >Reporter: Paul Lin >Priority: Minor > Labels: pull-request-available > > Environments prints job execution results to stdout by default and provides a > flag `suppressSysout` to disable the behavior. This flag is useful when > submitting jobs through REST API or other programmatic approaches. However, > JobExecutionResult is still printed when this flag is on, which looks like a > bug to me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23868) JobExecutionResult printed event if suppressSysout is on
[ https://issues.apache.org/jira/browse/FLINK-23868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-23868: - Description: Environments prints job execution results to stdout by default and provides a flag `suppressSysout` to disable the behavior. This flag is useful when submitting jobs through REST API or other programmatic approaches. However, JobExecutionResult is still printed when this flag is on, which looks like a bug to me. (was: Environments prints job execution results to stdout by default and provided a flag `suppressSysout` to disable the behavior. This flag is useful when submitting jobs through REST API or other programmatic approaches. However, JobExecutionResult is still printed when this flag is on, which looks like a bug to me.) > JobExecutionResult printed event if suppressSysout is on > > > Key: FLINK-23868 > URL: https://issues.apache.org/jira/browse/FLINK-23868 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.14.0, 1.13.2 >Reporter: Paul Lin >Priority: Minor > Labels: pull-request-available > > Environments prints job execution results to stdout by default and provides a > flag `suppressSysout` to disable the behavior. This flag is useful when > submitting jobs through REST API or other programmatic approaches. However, > JobExecutionResult is still printed when this flag is on, which looks like a > bug to me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23868) JobExecutionResult printed event if suppressSysout is on
[ https://issues.apache.org/jira/browse/FLINK-23868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-23868: - Description: Environments prints job execution results to stdout by default and provided a flag `suppressSysout` to disable the behavior. This flag is useful when submitting jobs through REST API or other programmatic approaches. However, JobExecutionResult is still printed when this flag is on, which looks like a bug to me. (was: Environments prints job execution results to stdout by default and provided a flag `suppressSysout` to disable the behavior. This flag is useful when submitting jobs through REST API or by programmatic approaches. However, JobExecutionResult is still printed when this flag is on, which looks like a bug to me.) > JobExecutionResult printed event if suppressSysout is on > > > Key: FLINK-23868 > URL: https://issues.apache.org/jira/browse/FLINK-23868 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.14.0, 1.13.2 >Reporter: Paul Lin >Priority: Minor > > Environments prints job execution results to stdout by default and provided a > flag `suppressSysout` to disable the behavior. This flag is useful when > submitting jobs through REST API or other programmatic approaches. However, > JobExecutionResult is still printed when this flag is on, which looks like a > bug to me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-23868) JobExecutionResult printed event if suppressSysout is on
Paul Lin created FLINK-23868: Summary: JobExecutionResult printed event if suppressSysout is on Key: FLINK-23868 URL: https://issues.apache.org/jira/browse/FLINK-23868 Project: Flink Issue Type: Bug Components: Client / Job Submission Affects Versions: 1.13.2, 1.14.0 Reporter: Paul Lin Environments prints job execution results to stdout by default and provided a flag `suppressSysout` to disable the behavior. This flag is useful when submitting jobs through REST API or by programmatic approaches. However, JobExecutionResult is still printed when this flag is on, which looks like a bug to me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23727) Skip null values in SimpleStringSchema#deserialize
[ https://issues.apache.org/jira/browse/FLINK-23727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17401428#comment-17401428 ] Paul Lin commented on FLINK-23727: -- [~paul8263] Thanks for your contribution, but the PR involves serialization which is beyond the scope of this issue. I'm not sure if it's valid to pass nulls to a serializer, because stream elements should not be null. Plus, I've prepared PR for this issue, just waiting for the issue to be assigned as the contribution guide requires. > Skip null values in SimpleStringSchema#deserialize > -- > > Key: FLINK-23727 > URL: https://issues.apache.org/jira/browse/FLINK-23727 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.13.2 >Reporter: Paul Lin >Priority: Major > Labels: pull-request-available > > In Kafka use cases, it's valid to send a message with a key and a null > payload as a tombstone. But SimpleStringSchema, which is frequently used as a > message value deserializer, throws NPE when the input value is null. We > should tolerate null values in SimpleStringSchema (simply return null to skip > the records), otherwise users need to implement a custom one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-23727) Skip null values in SimpleStringSchema#deserialize
[ https://issues.apache.org/jira/browse/FLINK-23727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400787#comment-17400787 ] Paul Lin edited comment on FLINK-23727 at 8/18/21, 5:13 AM: [~paul8263] WRT solutions, I've considered adding a flag as you suggested, but I doubt in what cases users would need NPE? That's obviously not a best practice. Moreover, Flink deserializer interface states that implementations should return null instead of throwing exceptions if the record can not be deserialized. So I think we could just fix the NPE, no need for an extra parameter. was (Author: paul lin): [~paul8263] WRT solutions, I've considered adding a flag as you suggested, but I doubt in what cases users would need NPE? That's obviously not a best practice. Moreover, Flink deserializer interface states that implementations should return null if the record can not be deserialized instead of throwing exceptions. So I think we could just fix the NPE, no need for an extra parameter. > Skip null values in SimpleStringSchema#deserialize > -- > > Key: FLINK-23727 > URL: https://issues.apache.org/jira/browse/FLINK-23727 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.13.2 >Reporter: Paul Lin >Priority: Major > > In Kafka use cases, it's valid to send a message with a key and a null > payload as a tombstone. But SimpleStringSchema, which is frequently used as a > message value deserializer, throws NPE when the input value is null. We > should tolerate null values in SimpleStringSchema (simply return null to skip > the records), otherwise users need to implement a custom one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23727) Skip null values in SimpleStringSchema#deserialize
[ https://issues.apache.org/jira/browse/FLINK-23727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17400787#comment-17400787 ] Paul Lin commented on FLINK-23727: -- [~paul8263] WRT solutions, I've considered adding a flag as you suggested, but I doubt in what cases users would need NPE? That's obviously not a best practice. Moreover, Flink deserializer interface states that implementations should return null if the record can not be deserialized instead of throwing exceptions. So I think we could just fix the NPE, no need for an extra parameter. > Skip null values in SimpleStringSchema#deserialize > -- > > Key: FLINK-23727 > URL: https://issues.apache.org/jira/browse/FLINK-23727 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.13.2 >Reporter: Paul Lin >Priority: Major > > In Kafka use cases, it's valid to send a message with a key and a null > payload as a tombstone. But SimpleStringSchema, which is frequently used as a > message value deserializer, throws NPE when the input value is null. We > should tolerate null values in SimpleStringSchema (simply return null to skip > the records), otherwise users need to implement a custom one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-23703) Support unrecoverable exception class list in configuration
[ https://issues.apache.org/jira/browse/FLINK-23703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17398420#comment-17398420 ] Paul Lin commented on FLINK-23703: -- cc [~arvid] [~trohrmann] WDYT? > Support unrecoverable exception class list in configuration > --- > > Key: FLINK-23703 > URL: https://issues.apache.org/jira/browse/FLINK-23703 > Project: Flink > Issue Type: New Feature > Components: Runtime / Configuration >Reporter: Paul Lin >Priority: Minor > > Currently, users can use `@ThrowableAnnotation` to denote a custom exception > is unrecoverable to avoid unnecessary retries, but it is not possible to > annotate an exception from the standard libs or third-party libs. > Thus, I propose to add a configuration option, such as > `unrecoverable.exceptions`, of which values are a comma-separated list of > fully-qualified names of exceptions, with a default value covering the most > common unrecoverable exceptions like NPE, ClassNotFoundException, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23727) Skip null values in SimpleStringSchema#deserialize
[ https://issues.apache.org/jira/browse/FLINK-23727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-23727: - Description: In Kafka use cases, it's valid to send a message with a key and a null payload as a tombstone. But SimpleStringSchema, which is frequently used as a message value deserializer, throws NPE when the input value is null. We should tolerate null values in SimpleStringSchema (simply return null to skip the records), otherwise users need to implement a custom one. (was: In Kafka user cases, it's valid to send a message with a key and a null payload as a tombstone. But SimpleStringSchema, which is frequently used as a message value deserializer, throws NPE when the input value is null. We should tolerate null values in SimpleStringSchema (simply return null to skip the records), otherwise users need to implement a custom one.) > Skip null values in SimpleStringSchema#deserialize > -- > > Key: FLINK-23727 > URL: https://issues.apache.org/jira/browse/FLINK-23727 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.13.2 >Reporter: Paul Lin >Priority: Major > > In Kafka use cases, it's valid to send a message with a key and a null > payload as a tombstone. But SimpleStringSchema, which is frequently used as a > message value deserializer, throws NPE when the input value is null. We > should tolerate null values in SimpleStringSchema (simply return null to skip > the records), otherwise users need to implement a custom one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23727) Skip null values in SimpleStringSchema#deserialize
[ https://issues.apache.org/jira/browse/FLINK-23727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Lin updated FLINK-23727: - Summary: Skip null values in SimpleStringSchema#deserialize (was: Skip null values in SimpleStringSchema) > Skip null values in SimpleStringSchema#deserialize > -- > > Key: FLINK-23727 > URL: https://issues.apache.org/jira/browse/FLINK-23727 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.13.2 >Reporter: Paul Lin >Priority: Major > > In Kafka user cases, it's valid to send a message with a key and a null > payload as a tombstone. But SimpleStringSchema, which is frequently used as a > message value deserializer, throws NPE when the input value is null. We > should tolerate null values in SimpleStringSchema (simply return null to skip > the records), otherwise users need to implement a custom one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
