[jira] [Commented] (FLINK-29860) Pulsar connector bug when using Hybrid.Builder
[ https://issues.apache.org/jira/browse/FLINK-29860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630251#comment-17630251 ] Noel O'Connor commented on FLINK-29860: --- [~syhily] yep I do have a patch so assign me. I assume this is just for 1.17 snapshot and not other versions ? > Pulsar connector bug when using Hybrid.Builder > -- > > Key: FLINK-29860 > URL: https://issues.apache.org/jira/browse/FLINK-29860 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.6.4 >Reporter: Noel O'Connor >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > > When using a HybridSource with a set of pulsar sources submitting a job to a > flink cluster results in the following error > > The program finished with the following exception: > The implementation of the BlockElement is not serializable. The object > probably contains or references non serializable fields. > org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164) > org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) > org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) > org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) > org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132) > org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69) > > org.apache.flink.connector.base.source.hybrid.HybridSource$HybridSourceBuilder.addSource(HybridSource.java:246) > > org.apache.flink.connector.base.source.hybrid.HybridSource$HybridSourceBuilder.addSource(HybridSource.java:233) > > org.apache.flink.connector.base.source.hybrid.HybridSource.builder(HybridSource.java:104) > > I think this is related to https://issues.apache.org/jira/browse/FLINK-25444 > From a pulsar connector perspective it's simple fixed, just mark the "private > final InlineElement desc" attribute in > flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/config/CursorVerification.java > as transient to avoid the serialisation process. > > I've tested this and it seems to solve the issue. I can submit a PR with this > fix. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29925) table ui of configure value is strange
[ https://issues.apache.org/jira/browse/FLINK-29925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiadong.lu updated FLINK-29925: --- Description: As shown in the figure below, when the configure value is very large, the ui of the table is a bit strange !截屏2022-11-08 15.37.04.png! Environment: (was: As shown in the figure below, when the configure value is very large, the ui of the table is a bit strange) > table ui of configure value is strange > -- > > Key: FLINK-29925 > URL: https://issues.apache.org/jira/browse/FLINK-29925 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.15.2 >Reporter: jiadong.lu >Priority: Minor > Attachments: 截屏2022-11-08 15.37.04.png > > > As shown in the figure below, when the configure value is very large, the ui > of the table is a bit strange !截屏2022-11-08 15.37.04.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29925) table ui of configure value is strange
jiadong.lu created FLINK-29925: -- Summary: table ui of configure value is strange Key: FLINK-29925 URL: https://issues.apache.org/jira/browse/FLINK-29925 Project: Flink Issue Type: Improvement Components: Runtime / Web Frontend Affects Versions: 1.15.2 Environment: As shown in the figure below, when the configure value is very large, the ui of the table is a bit strange Reporter: jiadong.lu Attachments: 截屏2022-11-08 15.37.04.png -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29923) Hybrid Shuffle may face deadlock when running a task need to execute big size data
[ https://issues.apache.org/jira/browse/FLINK-29923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630244#comment-17630244 ] Weijie Guo commented on FLINK-29923: Through offline discussion with [~AlexXXX] , it is true that the task are stuck forever. Further, the cause of the problem should be the same as FLINK-29298 previously reported. It is a bug in the `LocalBufferPool`, and hybrid shuffle does increase the competition of network buffers, which makes it difficult to reproduce this bug under blocking shuffle, but it almost repeats under the specific query of hybrid shuffle, so I think it should be considered as a very serious bug. > Hybrid Shuffle may face deadlock when running a task need to execute big size > data > -- > > Key: FLINK-29923 > URL: https://issues.apache.org/jira/browse/FLINK-29923 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0 >Reporter: AlexHu >Priority: Major > Attachments: 性能差距.png, 死锁2-select.png, 死锁检测.png > > > The flink 1.16 offers hybrid shuffle to combine the superiority of blocking > shuffle and pipeline shuffle. But when I want to test this new feature I face > a problem that it may cause deadlock when it running. > Actually, it will run well at beginning. However, when it runs to a certain > number it may failure for the buffer size and if I set a bigger size it may > running without data execution like the picture. So I want to ask the cause > of this problem and a solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29835) BlobServer fails to provide artifacts
[
https://issues.apache.org/jira/browse/FLINK-29835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630243#comment-17630243
]
Matthias Pohl commented on FLINK-29835:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42910=logs=fb37c667-81b7-5c22-dd91-846535e99a97=39a035c3-c65e-573c-fb66-104c66c28912=13776
> BlobServer fails to provide artifacts
> -
>
> Key: FLINK-29835
> URL: https://issues.apache.org/jira/browse/FLINK-29835
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: test-stability
> Attachments:
> PulsarSourceUnorderedE2ECase.testSavepoint.FileNotFoundException.log
>
>
> [This
> build|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42680=logs=87489130-75dc-54e4-1f45-80c30aa367a3=73da6d75-f30d-5d5a-acbe-487a9dcff678=16001]
> failed in \{{PulsarSourceUnorderedE2ECase.testSavepoint}} due to some job
> timeout. The issue seems to be related to the BlobServer failing to provide
> some artifacts (\{{java.io.FileNotFoundException}}) that consequently causes
> classes not being found:
> {code:java}
> java.lang.NoClassDefFoundError:
> org/apache/pulsar/shade/org/jvnet/hk2/internal/ServiceLocatorImpl$7
> at
> org.apache.pulsar.shade.org.jvnet.hk2.internal.ServiceLocatorImpl.shutdown(ServiceLocatorImpl.java:894)
> ~[?:?]
> at
> org.apache.pulsar.shade.org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.shutdown(AbstractHk2InjectionManager.java:183)
> ~[?:?]
> at
> org.apache.pulsar.shade.org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.shutdown(ImmediateHk2InjectionManager.java:30)
> ~[?:?]
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.close(ClientRuntime.java:371)
> ~[?:?]
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.onShutdown(ClientRuntime.java:353)
> ~[?:?]
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyClient.release(JerseyClient.java:190)
> ~[?:?]
> at
> org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyClient.close(JerseyClient.java:180)
> ~[?:?]
> at
> org.apache.pulsar.client.admin.internal.PulsarAdminImpl.close(PulsarAdminImpl.java:488)
> ~[?:?]
> at
> org.apache.flink.connector.pulsar.source.enumerator.PulsarSourceEnumerator.close(PulsarSourceEnumerator.java:164)
> ~[?:?]
> at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:255)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.runtime.source.coordinator.SourceCoordinator.close(SourceCoordinator.java:265)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.runtime.operators.coordination.ComponentClosingUtils.lambda$closeAsyncWithTimeout$0(ComponentClosingUtils.java:76)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
> Caused by: java.lang.ClassNotFoundException:
> org.apache.pulsar.shade.org.jvnet.hk2.internal.ServiceLocatorImpl$7
> at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
> ~[?:1.8.0_342]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_342]
> at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
> ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_342]
> ... 13 more{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29427) LookupJoinITCase failed with classloader problem
[
https://issues.apache.org/jira/browse/FLINK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-29427:
--
Affects Version/s: 1.17.0
> LookupJoinITCase failed with classloader problem
>
>
> Key: FLINK-29427
> URL: https://issues.apache.org/jira/browse/FLINK-29427
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Huang Xingbo
>Assignee: Alexander Smirnov
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-27T02:49:20.9501313Z Sep 27 02:49:20 Caused by:
> org.codehaus.janino.InternalCompilerException: Compiling
> "KeyProjection$108341": Trying to access closed classloader. Please check if
> you store classloaders directly or indirectly in static fields. If the
> stacktrace suggests that the leak occurs in a third party library and cannot
> be fixed immediately, you can disable this check with the configuration
> 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9502654Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> 2022-09-27T02:49:20.9503366Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> 2022-09-27T02:49:20.9504044Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> 2022-09-27T02:49:20.9504704Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216)
> 2022-09-27T02:49:20.9505341Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> 2022-09-27T02:49:20.9505965Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
> 2022-09-27T02:49:20.9506584Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)
> 2022-09-27T02:49:20.9507261Z Sep 27 02:49:20 at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:104)
> 2022-09-27T02:49:20.9507883Z Sep 27 02:49:20 ... 30 more
> 2022-09-27T02:49:20.9509266Z Sep 27 02:49:20 Caused by:
> java.lang.IllegalStateException: Trying to access closed classloader. Please
> check if you store classloaders directly or indirectly in static fields. If
> the stacktrace suggests that the leak occurs in a third party library and
> cannot be fixed immediately, you can disable this check with the
> configuration 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9510835Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:184)
> 2022-09-27T02:49:20.9511760Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:192)
> 2022-09-27T02:49:20.9512456Z Sep 27 02:49:20 at
> java.lang.Class.forName0(Native Method)
> 2022-09-27T02:49:20.9513014Z Sep 27 02:49:20 at
> java.lang.Class.forName(Class.java:348)
> 2022-09-27T02:49:20.9513649Z Sep 27 02:49:20 at
> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:89)
> 2022-09-27T02:49:20.9514339Z Sep 27 02:49:20 at
> org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:312)
> 2022-09-27T02:49:20.9514990Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8556)
> 2022-09-27T02:49:20.9515659Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6749)
> 2022-09-27T02:49:20.9516337Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6594)
> 2022-09-27T02:49:20.9516989Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6573)
> 2022-09-27T02:49:20.9517632Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.access$13900(UnitCompiler.java:215)
> 2022-09-27T02:49:20.9518319Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6481)
> 2022-09-27T02:49:20.9519018Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9519680Z Sep 27 02:49:20 at
> org.codehaus.janino.Java$ReferenceType.accept(Java.java:3928)
> 2022-09-27T02:49:20.9520386Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9521042Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6469)
> 2022-09-27T02:49:20.9521677Z Sep 27 02:49:20 at
> org.codehaus.janino.Java$ReferenceType.accept(Java.java:3927)
> 2022-09-27T02:49:20.9522299Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:6469)
>
[jira] [Commented] (FLINK-29427) LookupJoinITCase failed with classloader problem
[
https://issues.apache.org/jira/browse/FLINK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630242#comment-17630242
]
Matthias Pohl commented on FLINK-29427:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42910=logs=f2c100be-250b-5e85-7bbe-176f68fcddc5=05efd11e-5400-54a4-0d27-a4663be008a9=17971
> LookupJoinITCase failed with classloader problem
>
>
> Key: FLINK-29427
> URL: https://issues.apache.org/jira/browse/FLINK-29427
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Alexander Smirnov
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-27T02:49:20.9501313Z Sep 27 02:49:20 Caused by:
> org.codehaus.janino.InternalCompilerException: Compiling
> "KeyProjection$108341": Trying to access closed classloader. Please check if
> you store classloaders directly or indirectly in static fields. If the
> stacktrace suggests that the leak occurs in a third party library and cannot
> be fixed immediately, you can disable this check with the configuration
> 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9502654Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> 2022-09-27T02:49:20.9503366Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> 2022-09-27T02:49:20.9504044Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> 2022-09-27T02:49:20.9504704Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216)
> 2022-09-27T02:49:20.9505341Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> 2022-09-27T02:49:20.9505965Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
> 2022-09-27T02:49:20.9506584Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)
> 2022-09-27T02:49:20.9507261Z Sep 27 02:49:20 at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:104)
> 2022-09-27T02:49:20.9507883Z Sep 27 02:49:20 ... 30 more
> 2022-09-27T02:49:20.9509266Z Sep 27 02:49:20 Caused by:
> java.lang.IllegalStateException: Trying to access closed classloader. Please
> check if you store classloaders directly or indirectly in static fields. If
> the stacktrace suggests that the leak occurs in a third party library and
> cannot be fixed immediately, you can disable this check with the
> configuration 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9510835Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:184)
> 2022-09-27T02:49:20.9511760Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:192)
> 2022-09-27T02:49:20.9512456Z Sep 27 02:49:20 at
> java.lang.Class.forName0(Native Method)
> 2022-09-27T02:49:20.9513014Z Sep 27 02:49:20 at
> java.lang.Class.forName(Class.java:348)
> 2022-09-27T02:49:20.9513649Z Sep 27 02:49:20 at
> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:89)
> 2022-09-27T02:49:20.9514339Z Sep 27 02:49:20 at
> org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:312)
> 2022-09-27T02:49:20.9514990Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8556)
> 2022-09-27T02:49:20.9515659Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6749)
> 2022-09-27T02:49:20.9516337Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6594)
> 2022-09-27T02:49:20.9516989Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6573)
> 2022-09-27T02:49:20.9517632Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.access$13900(UnitCompiler.java:215)
> 2022-09-27T02:49:20.9518319Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6481)
> 2022-09-27T02:49:20.9519018Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9519680Z Sep 27 02:49:20 at
> org.codehaus.janino.Java$ReferenceType.accept(Java.java:3928)
> 2022-09-27T02:49:20.9520386Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9521042Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6469)
> 2022-09-27T02:49:20.9521677Z Sep 27 02:49:20 at
>
[jira] [Commented] (FLINK-26037) TaskManagerRunnerTest JVM crash with exit code 239
[
https://issues.apache.org/jira/browse/FLINK-26037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630241#comment-17630241
]
Matthias Pohl commented on FLINK-26037:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42908=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=8140
> TaskManagerRunnerTest JVM crash with exit code 239
> --
>
> Key: FLINK-26037
> URL: https://issues.apache.org/jira/browse/FLINK-26037
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0, 1.17.0
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> Feb 09 02:22:52 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test)
> on project flink-runtime: There are test failures.
> Feb 09 02:22:52 [ERROR]
> Feb 09 02:22:52 [ERROR] Please refer to
> /__w/2/s/flink-runtime/target/surefire-reports for the individual test
> results.
> Feb 09 02:22:52 [ERROR] Please refer to dump files (if any exist)
> [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> Feb 09 02:22:52 [ERROR] ExecutionException The forked VM terminated without
> properly saying goodbye. VM crash or System.exit called?
> Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar
> /__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2
> surefire8313405181432833392tmp surefire_2318122906826894495431tmp
> Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
> Feb 09 02:22:52 [ERROR] Process Exit Code: 239
> Feb 09 02:22:52 [ERROR] Crashed tests:
> Feb 09 02:22:52 [ERROR]
> org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
> Feb 09 02:22:52 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar
> /__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2
> surefire8313405181432833392tmp surefire_2318122906826894495431tmp
> Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
> Feb 09 02:22:52 [ERROR] Process Exit Code: 239
> Feb 09 02:22:52 [ERROR] Crashed tests:
> Feb 09 02:22:52 [ERROR]
> org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> Feb 09 02:22:52 [ERROR] at
>
[jira] [Commented] (FLINK-29461) ProcessDataStreamStreamingTests.test_process_function unstable
[
https://issues.apache.org/jira/browse/FLINK-29461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630240#comment-17630240
]
Matthias Pohl commented on FLINK-29461:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42908=logs=3e4dd1a2-fe2f-5e5d-a581-48087e718d53=b4612f28-e3b5-5853-8a8b-610ae894217a=27019
> ProcessDataStreamStreamingTests.test_process_function unstable
> --
>
> Key: FLINK-29461
> URL: https://issues.apache.org/jira/browse/FLINK-29461
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-29T02:10:45.3571648Z Sep 29 02:10:45 self =
> testMethod=test_process_function>
> 2022-09-29T02:10:45.3572279Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3572810Z Sep 29 02:10:45 def
> test_process_function(self):
> 2022-09-29T02:10:45.3573495Z Sep 29 02:10:45
> self.env.set_parallelism(1)
> 2022-09-29T02:10:45.3574148Z Sep 29 02:10:45
> self.env.get_config().set_auto_watermark_interval(2000)
> 2022-09-29T02:10:45.3580634Z Sep 29 02:10:45
> self.env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
> 2022-09-29T02:10:45.3583194Z Sep 29 02:10:45 data_stream =
> self.env.from_collection([(1, '1603708211000'),
> 2022-09-29T02:10:45.3584515Z Sep 29 02:10:45
>(2, '1603708224000'),
> 2022-09-29T02:10:45.3585957Z Sep 29 02:10:45
>(3, '1603708226000'),
> 2022-09-29T02:10:45.3587132Z Sep 29 02:10:45
>(4, '1603708289000')],
> 2022-09-29T02:10:45.3588094Z Sep 29 02:10:45
> type_info=Types.ROW([Types.INT(), Types.STRING()]))
> 2022-09-29T02:10:45.3589090Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3589949Z Sep 29 02:10:45 class
> MyProcessFunction(ProcessFunction):
> 2022-09-29T02:10:45.3590710Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3591856Z Sep 29 02:10:45 def
> process_element(self, value, ctx):
> 2022-09-29T02:10:45.3592873Z Sep 29 02:10:45
> current_timestamp = ctx.timestamp()
> 2022-09-29T02:10:45.3593862Z Sep 29 02:10:45
> current_watermark = ctx.timer_service().current_watermark()
> 2022-09-29T02:10:45.3594915Z Sep 29 02:10:45 yield "current
> timestamp: {}, current watermark: {}, current_value: {}"\
> 2022-09-29T02:10:45.3596201Z Sep 29 02:10:45
> .format(str(current_timestamp), str(current_watermark), str(value))
> 2022-09-29T02:10:45.3597089Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3597942Z Sep 29 02:10:45 watermark_strategy =
> WatermarkStrategy.for_monotonous_timestamps()\
> 2022-09-29T02:10:45.3599260Z Sep 29 02:10:45
> .with_timestamp_assigner(SecondColumnTimestampAssigner())
> 2022-09-29T02:10:45.3600611Z Sep 29 02:10:45
> data_stream.assign_timestamps_and_watermarks(watermark_strategy)\
> 2022-09-29T02:10:45.3601877Z Sep 29 02:10:45
> .process(MyProcessFunction(),
> output_type=Types.STRING()).add_sink(self.test_sink)
> 2022-09-29T02:10:45.3603527Z Sep 29 02:10:45 self.env.execute('test
> process function')
> 2022-09-29T02:10:45.3604445Z Sep 29 02:10:45 results =
> self.test_sink.get_results()
> 2022-09-29T02:10:45.3605684Z Sep 29 02:10:45 expected = ["current
> timestamp: 1603708211000, current watermark: "
> 2022-09-29T02:10:45.3607157Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=1, f1='1603708211000')",
> 2022-09-29T02:10:45.3608256Z Sep 29 02:10:45 "current
> timestamp: 1603708224000, current watermark: "
> 2022-09-29T02:10:45.3609650Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=2, f1='1603708224000')",
> 2022-09-29T02:10:45.3610854Z Sep 29 02:10:45 "current
> timestamp: 1603708226000, current watermark: "
> 2022-09-29T02:10:45.3612279Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=3, f1='1603708226000')",
> 2022-09-29T02:10:45.3613382Z Sep 29 02:10:45 "current
> timestamp: 1603708289000, current watermark: "
> 2022-09-29T02:10:45.3615683Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=4, f1='1603708289000')"]
> 2022-09-29T02:10:45.3617687Z Sep 29 02:10:45 >
> self.assert_equals_sorted(expected, results)
> 2022-09-29T02:10:45.3618620Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3619425Z Sep 29 02:10:45
>
[jira] [Commented] (FLINK-29594) RMQSourceITCase.testMessageDelivery timed out
[
https://issues.apache.org/jira/browse/FLINK-29594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630239#comment-17630239
]
Matthias Pohl commented on FLINK-29594:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42908=logs=fa307d6d-91b1-5ab6-d460-ef50f552b1fe=21eae189-b04c-5c04-662b-17dc80ffc83a=38551
> RMQSourceITCase.testMessageDelivery timed out
> -
>
> Key: FLINK-29594
> URL: https://issues.apache.org/jira/browse/FLINK-29594
> Project: Flink
> Issue Type: Bug
> Components: Connectors/ RabbitMQ
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: test-stability
>
> [This
> build|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41843=logs=fc7981dc-d266-55b0-5fff-f0d0a2294e36=1a9b228a-3e0e-598f-fc81-c321539dfdbf=38211]
> failed (not exclusively) due to {{RMQSourceITCase.testMessageDelivery}}
> timing out.
> I wasn't able to reproduce it locally with 200 test runs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo updated FLINK-28786:
-
Fix Version/s: 1.17.0
1.16.1
> Cannot run PyFlink 1.16 on MacOS with M1 chip
> -
>
> Key: FLINK-28786
> URL: https://issues.apache.org/jira/browse/FLINK-28786
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Assignee: Huang Xingbo
>Priority: Major
> Fix For: 1.17.0, 1.16.1
>
>
> I have tested it with 2 m1 machines. i will reproduce the bug 100%.
> 1.m1 machine
> macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> 1.m1 machine
> macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> reproduce step:
> 1.python -m pip install -r flink-python/dev/dev-requirements.txt
> 2.cd flink-python; python setup.py sdist bdist_wheel; cd
> apache-flink-libraries; python setup.py sdist; cd ..;
> 3.python -m pip install apache-flink-libraries/dist/*.tar.gz
> 4.python -m pip install dist/*.whl
> when run
> [word_count.py|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/table_api_tutorial/]
> it will cause
> {code:java}
> :219: RuntimeWarning:
> apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate
> binary incompatibility. Expected 24 from C header, got 32 from PyObject
> Traceback (most recent call last):
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 129, in
> word_count(known_args.input, known_args.output)
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 49, in word_count
> t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 121, in create
> return TableEnvironment(j_tenv)
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 100, in __init__
> self._open()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1637, in _open
> startup_loopback_server()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1628, in startup_loopback_server
> from pyflink.fn_execution.beam.beam_worker_pool_service import \
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
> line 44, in
> from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
> pylint: disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
> line 21, in
> import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
> disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
> line 27, in
> from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
> RemoteOperatorStateBackend
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
> line 33, in
> from pyflink.fn_execution.beam.beam_coders import FlinkCoder
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
> line 27, in
> from pyflink.fn_execution.beam import beam_coder_impl_fast as
> beam_coder_impl
> File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
> pyflink.fn_execution.beam.beam_coder_impl_fast
> KeyError: '__pyx_vtable__'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (FLINK-28786) Cannot run PyFlink 1.16 on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-28786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reopened FLINK-28786:
--
Assignee: Huang Xingbo
> Cannot run PyFlink 1.16 on MacOS with M1 chip
> -
>
> Key: FLINK-28786
> URL: https://issues.apache.org/jira/browse/FLINK-28786
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Assignee: Huang Xingbo
>Priority: Major
>
> I have tested it with 2 m1 machines. i will reproduce the bug 100%.
> 1.m1 machine
> macos bigsur 11.5.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> 1.m1 machine
> macos monterey 12.1 & jdk8 * & jdk11 & python 3.8 & python 3.9
> reproduce step:
> 1.python -m pip install -r flink-python/dev/dev-requirements.txt
> 2.cd flink-python; python setup.py sdist bdist_wheel; cd
> apache-flink-libraries; python setup.py sdist; cd ..;
> 3.python -m pip install apache-flink-libraries/dist/*.tar.gz
> 4.python -m pip install dist/*.whl
> when run
> [word_count.py|https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/table_api_tutorial/]
> it will cause
> {code:java}
> :219: RuntimeWarning:
> apache_beam.coders.coder_impl.StreamCoderImpl size changed, may indicate
> binary incompatibility. Expected 24 from C header, got 32 from PyObject
> Traceback (most recent call last):
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 129, in
> word_count(known_args.input, known_args.output)
> File "/Users/chucheng/GitLab/pyflink-demo/table/streaming/word_count.py",
> line 49, in word_count
> t_env = TableEnvironment.create(EnvironmentSettings.in_streaming_mode())
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 121, in create
> return TableEnvironment(j_tenv)
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 100, in __init__
> self._open()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1637, in _open
> startup_loopback_server()
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/table/table_environment.py",
> line 1628, in startup_loopback_server
> from pyflink.fn_execution.beam.beam_worker_pool_service import \
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_worker_pool_service.py",
> line 44, in
> from pyflink.fn_execution.beam import beam_sdk_worker_main # noqa #
> pylint: disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py",
> line 21, in
> import pyflink.fn_execution.beam.beam_operations # noqa # pylint:
> disable=unused-import
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_operations.py",
> line 27, in
> from pyflink.fn_execution.state_impl import RemoteKeyedStateBackend,
> RemoteOperatorStateBackend
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/state_impl.py",
> line 33, in
> from pyflink.fn_execution.beam.beam_coders import FlinkCoder
> File
> "/Users/chucheng/venv/lib/python3.8/site-packages/pyflink/fn_execution/beam/beam_coders.py",
> line 27, in
> from pyflink.fn_execution.beam import beam_coder_impl_fast as
> beam_coder_impl
> File "pyflink/fn_execution/beam/beam_coder_impl_fast.pyx", line 1, in init
> pyflink.fn_execution.beam.beam_coder_impl_fast
> KeyError: '__pyx_vtable__'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #21258: [Flink]Support accumulator in source reader.
flinkbot commented on PR #21258: URL: https://github.com/apache/flink/pull/21258#issuecomment-1306739139 ## CI report: * 546e95ec02bd54371b06675ad930f7d07e7afcb8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #21203: [FLINK-21239][Table SQL/API] Upgrade Calcite to 1.28.0
snuyanzin commented on code in PR #21203: URL: https://github.com/apache/flink/pull/21203#discussion_r1016217693 ## flink-table/flink-table-planner/pom.xml: ## @@ -49,6 +49,18 @@ under the License. + Review Comment: also piece of `mvn dependency:tree` with `org.immutables` looks like ``` [INFO] --- maven-dependency-plugin:3.2.0:tree (default-cli) @ flink-table-planner_2.12 --- [INFO] org.apache.flink:flink-table-planner_2.12:jar:1.17-SNAPSHOT ... [INFO] +- org.immutables:value:jar:2.8.8:compile [INFO] +- org.immutables:value-annotations:jar:2.8.8:compile ... ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29387) IntervalJoinITCase.testIntervalJoinSideOutputRightLateData failed with AssertionError
[
https://issues.apache.org/jira/browse/FLINK-29387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630236#comment-17630236
]
Matthias Pohl commented on FLINK-29387:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42908=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798=10229
> IntervalJoinITCase.testIntervalJoinSideOutputRightLateData failed with
> AssertionError
> -
>
> Key: FLINK-29387
> URL: https://issues.apache.org/jira/browse/FLINK-29387
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.17.0
>Reporter: Huang Xingbo
>Priority: Blocker
> Labels: test-stability
>
> {code:java}
> 2022-09-22T04:40:21.9296331Z Sep 22 04:40:21 [ERROR]
> org.apache.flink.test.streaming.runtime.IntervalJoinITCase.testIntervalJoinSideOutputRightLateData
> Time elapsed: 2.46 s <<< FAILURE!
> 2022-09-22T04:40:21.9297487Z Sep 22 04:40:21 java.lang.AssertionError:
> expected:<[(key,2)]> but was:<[]>
> 2022-09-22T04:40:21.9298208Z Sep 22 04:40:21 at
> org.junit.Assert.fail(Assert.java:89)
> 2022-09-22T04:40:21.9298927Z Sep 22 04:40:21 at
> org.junit.Assert.failNotEquals(Assert.java:835)
> 2022-09-22T04:40:21.9299655Z Sep 22 04:40:21 at
> org.junit.Assert.assertEquals(Assert.java:120)
> 2022-09-22T04:40:21.9300403Z Sep 22 04:40:21 at
> org.junit.Assert.assertEquals(Assert.java:146)
> 2022-09-22T04:40:21.9301538Z Sep 22 04:40:21 at
> org.apache.flink.test.streaming.runtime.IntervalJoinITCase.expectInAnyOrder(IntervalJoinITCase.java:521)
> 2022-09-22T04:40:21.9302578Z Sep 22 04:40:21 at
> org.apache.flink.test.streaming.runtime.IntervalJoinITCase.testIntervalJoinSideOutputRightLateData(IntervalJoinITCase.java:280)
> 2022-09-22T04:40:21.9303641Z Sep 22 04:40:21 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-09-22T04:40:21.9304472Z Sep 22 04:40:21 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-09-22T04:40:21.9305371Z Sep 22 04:40:21 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-09-22T04:40:21.9306195Z Sep 22 04:40:21 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-09-22T04:40:21.9307011Z Sep 22 04:40:21 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-09-22T04:40:21.9308077Z Sep 22 04:40:21 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-09-22T04:40:21.9308968Z Sep 22 04:40:21 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-09-22T04:40:21.9309849Z Sep 22 04:40:21 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-09-22T04:40:21.9310704Z Sep 22 04:40:21 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-09-22T04:40:21.9311533Z Sep 22 04:40:21 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-09-22T04:40:21.9312386Z Sep 22 04:40:21 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-09-22T04:40:21.9313231Z Sep 22 04:40:21 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-09-22T04:40:21.9314985Z Sep 22 04:40:21 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-09-22T04:40:21.9315857Z Sep 22 04:40:21 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-09-22T04:40:21.9316633Z Sep 22 04:40:21 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-09-22T04:40:21.9317450Z Sep 22 04:40:21 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-09-22T04:40:21.9318209Z Sep 22 04:40:21 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-09-22T04:40:21.9318949Z Sep 22 04:40:21 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-09-22T04:40:21.9319680Z Sep 22 04:40:21 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-09-22T04:40:21.9320401Z Sep 22 04:40:21 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-09-22T04:40:21.9321130Z Sep 22 04:40:21 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2022-09-22T04:40:21.9321822Z Sep 22 04:40:21 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2022-09-22T04:40:21.9322498Z Sep 22 04:40:21 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2022-09-22T04:40:21.9323248Z Sep 22 04:40:21 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
> 2022-09-22T04:40:21.9324080Z
[GitHub] [flink] hk-lrzy opened a new pull request, #21258: [Flink]Support accumulator in source reader.
hk-lrzy opened a new pull request, #21258: URL: https://github.com/apache/flink/pull/21258 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* Source Reader is mainly for union batch and streaming logic in single interface, it's good point for the developer, but in the `SourceFunction` we can access `runtimeconext` to use accumulator before, now the `SourceReaderContext` have no method for it, this PR is mainly to support it. ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper opened a new pull request, #362: [FLINK-29876] Explicitly throw exception from Table Store sink when unaligned checkpoint is enabled or at least once checkpoint m
tsreaper opened a new pull request, #362: URL: https://github.com/apache/flink-table-store/pull/362 (Cherry-picked from #352) Currently table store sink does not support unaligned checkpoint or at least once checkpoint mode, but no exception is explicitly thrown. We should throw exception so that users can change their configurations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] czy006 commented on pull request #21012: [FLINK-29543] Jar Run Rest Handler Support Flink Configuration
czy006 commented on PR #21012: URL: https://github.com/apache/flink/pull/21012#issuecomment-1306725733 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-23238) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-23238:
--
Priority: Major (was: Not a Priority)
> EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism
> fails on azure
>
>
> Key: FLINK-23238
> URL: https://issues.apache.org/jira/browse/FLINK-23238
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.4, 1.15.2
>Reporter: Xintong Song
>Priority: Major
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19873=logs=baf26b34-3c6a-54e8-f93f-cf269b32f802=6dff16b1-bf54-58f3-23c6-76282f49a185=4490
> {code}
> [ERROR] Tests run: 42, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 261.311 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase
> [ERROR] testTumblingTimeWindowWithKVStateMaxMaxParallelism[statebackend type
> =ROCKSDB_INCREMENTAL](org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase)
> Time elapsed: 79.062 s <<< FAILURE!
> java.lang.AssertionError: Job execution failed.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.doTestTumblingTimeWindowWithKVState(EventTimeWindowCheckpointingITCase.java:434)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism(EventTimeWindowCheckpointingITCase.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
[jira] [Updated] (FLINK-23238) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-23238:
--
Affects Version/s: 1.14.6
> EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism
> fails on azure
>
>
> Key: FLINK-23238
> URL: https://issues.apache.org/jira/browse/FLINK-23238
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.4, 1.15.2, 1.14.6
>Reporter: Xintong Song
>Priority: Major
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19873=logs=baf26b34-3c6a-54e8-f93f-cf269b32f802=6dff16b1-bf54-58f3-23c6-76282f49a185=4490
> {code}
> [ERROR] Tests run: 42, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 261.311 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase
> [ERROR] testTumblingTimeWindowWithKVStateMaxMaxParallelism[statebackend type
> =ROCKSDB_INCREMENTAL](org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase)
> Time elapsed: 79.062 s <<< FAILURE!
> java.lang.AssertionError: Job execution failed.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.doTestTumblingTimeWindowWithKVState(EventTimeWindowCheckpointingITCase.java:434)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism(EventTimeWindowCheckpointingITCase.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>
[jira] [Commented] (FLINK-23238) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630234#comment-17630234
]
Matthias Pohl commented on FLINK-23238:
---
Raising the priority of the issue to Major again.
> EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism
> fails on azure
>
>
> Key: FLINK-23238
> URL: https://issues.apache.org/jira/browse/FLINK-23238
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.4, 1.15.2
>Reporter: Xintong Song
>Priority: Major
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19873=logs=baf26b34-3c6a-54e8-f93f-cf269b32f802=6dff16b1-bf54-58f3-23c6-76282f49a185=4490
> {code}
> [ERROR] Tests run: 42, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 261.311 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase
> [ERROR] testTumblingTimeWindowWithKVStateMaxMaxParallelism[statebackend type
> =ROCKSDB_INCREMENTAL](org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase)
> Time elapsed: 79.062 s <<< FAILURE!
> java.lang.AssertionError: Job execution failed.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.doTestTumblingTimeWindowWithKVState(EventTimeWindowCheckpointingITCase.java:434)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism(EventTimeWindowCheckpointingITCase.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
>
[jira] [Updated] (FLINK-23238) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-23238:
--
Affects Version/s: 1.15.2
> EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism
> fails on azure
>
>
> Key: FLINK-23238
> URL: https://issues.apache.org/jira/browse/FLINK-23238
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.4, 1.15.2
>Reporter: Xintong Song
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19873=logs=baf26b34-3c6a-54e8-f93f-cf269b32f802=6dff16b1-bf54-58f3-23c6-76282f49a185=4490
> {code}
> [ERROR] Tests run: 42, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 261.311 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase
> [ERROR] testTumblingTimeWindowWithKVStateMaxMaxParallelism[statebackend type
> =ROCKSDB_INCREMENTAL](org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase)
> Time elapsed: 79.062 s <<< FAILURE!
> java.lang.AssertionError: Job execution failed.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.doTestTumblingTimeWindowWithKVState(EventTimeWindowCheckpointingITCase.java:434)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism(EventTimeWindowCheckpointingITCase.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>
[jira] [Commented] (FLINK-23238) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism fails on azure
[
https://issues.apache.org/jira/browse/FLINK-23238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630233#comment-17630233
]
Matthias Pohl commented on FLINK-23238:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42909=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=6218
This one failed in {{testTumblingTimeWindowWithKVStateMinMaxParallelism}} but I
put it in here, anyway, because it has the same error in the same test class.
> EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism
> fails on azure
>
>
> Key: FLINK-23238
> URL: https://issues.apache.org/jira/browse/FLINK-23238
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.4
>Reporter: Xintong Song
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=19873=logs=baf26b34-3c6a-54e8-f93f-cf269b32f802=6dff16b1-bf54-58f3-23c6-76282f49a185=4490
> {code}
> [ERROR] Tests run: 42, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 261.311 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase
> [ERROR] testTumblingTimeWindowWithKVStateMaxMaxParallelism[statebackend type
> =ROCKSDB_INCREMENTAL](org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase)
> Time elapsed: 79.062 s <<< FAILURE!
> java.lang.AssertionError: Job execution failed.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.doTestTumblingTimeWindowWithKVState(EventTimeWindowCheckpointingITCase.java:434)
> at
> org.apache.flink.test.checkpointing.EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMaxMaxParallelism(EventTimeWindowCheckpointingITCase.java:350)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
>
[jira] [Commented] (FLINK-29860) Pulsar connector bug when using Hybrid.Builder
[
https://issues.apache.org/jira/browse/FLINK-29860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630232#comment-17630232
]
Zili Chen commented on FLINK-29860:
---
[~syhily] [~noelo] I'd prefer the solution by setting {{InlineElement}} as
{{transient}}.
[~noelo] if you're still willing to submit a patch, please submit one and I'll
assign to you.
> Pulsar connector bug when using Hybrid.Builder
> --
>
> Key: FLINK-29860
> URL: https://issues.apache.org/jira/browse/FLINK-29860
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.6.4
>Reporter: Noel O'Connor
>Assignee: Yufan Sheng
>Priority: Major
> Labels: pull-request-available
>
> When using a HybridSource with a set of pulsar sources submitting a job to a
> flink cluster results in the following error
>
> The program finished with the following exception:
> The implementation of the BlockElement is not serializable. The object
> probably contains or references non serializable fields.
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:164)
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:132)
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:69)
>
> org.apache.flink.connector.base.source.hybrid.HybridSource$HybridSourceBuilder.addSource(HybridSource.java:246)
>
> org.apache.flink.connector.base.source.hybrid.HybridSource$HybridSourceBuilder.addSource(HybridSource.java:233)
>
> org.apache.flink.connector.base.source.hybrid.HybridSource.builder(HybridSource.java:104)
>
> I think this is related to https://issues.apache.org/jira/browse/FLINK-25444
> From a pulsar connector perspective it's simple fixed, just mark the "private
> final InlineElement desc" attribute in
> flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/config/CursorVerification.java
> as transient to avoid the serialisation process.
>
> I've tested this and it seems to solve the issue. I can submit a PR with this
> fix.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29322) Expose savepoint format on Web UI
[ https://issues.apache.org/jira/browse/FLINK-29322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630231#comment-17630231 ] Yu Chen commented on FLINK-29322: - I have already implemented it, if no one has any objection, I can take this ticket. > Expose savepoint format on Web UI > - > > Key: FLINK-29322 > URL: https://issues.apache.org/jira/browse/FLINK-29322 > Project: Flink > Issue Type: New Feature > Components: Runtime / Web Frontend >Reporter: Matyas Orhidi >Assignee: Matyas Orhidi >Priority: Major > Fix For: 1.17.0 > > > Savepoint format is not exposed on the Web UI, thus users should remember how > they triggered it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29805) Table Store sink continuously fails with "Trying to add file which is already added" when snapshot committing is slow
[
https://issues.apache.org/jira/browse/FLINK-29805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caizhi Weng closed FLINK-29805.
---
Resolution: Fixed
master: 0a193a8110aa4716250fed3c4223018ab519c9b1
release-0.2: d41d14a5921aa27ebd903eed45365841093e97ed
> Table Store sink continuously fails with "Trying to add file which is already
> added" when snapshot committing is slow
> -
>
> Key: FLINK-29805
> URL: https://issues.apache.org/jira/browse/FLINK-29805
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Affects Versions: table-store-0.3.0, table-store-0.2.2
>Reporter: Caizhi Weng
>Assignee: Caizhi Weng
>Priority: Major
> Labels: pull-request-available
> Fix For: table-store-0.3.0, table-store-0.2.2
>
>
> Table Store sink continuously fails with "Trying to add file which is already
> added" when snapshot committing is slow.
> This is due to a bug in {{FileStoreCommitImpl#filterCommitted}}. When this
> method finds an identifier, it removes the identifier from a map. However
> different snapshots may have the same identifier (for example an APPEND
> commit and the following COMPACT commit will have the same identifier), so we
> need to use another set to check for identifiers.
> When snapshot committing is fast there is at most 1 identifier to check after
> the job restarts, so nothing happens. However when snapshot committing is
> slow, there will be multiple identifiers to check and some identifiers will
> be mistakenly kept.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-table-store] tsreaper merged pull request #360: [FLINK-29805] Fix incorrect snapshot filter when snapshots are committing too slow
tsreaper merged PR #360: URL: https://github.com/apache/flink-table-store/pull/360 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20184) update hive streaming read and temporal table documents
[ https://issues.apache.org/jira/browse/FLINK-20184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630229#comment-17630229 ] luoyuxia commented on FLINK-20184: -- [~heigebupahei] Thanks for you attention . Left my comments in here in case others may want to know the answer. 1: No, you won't need to use ddl statement in flink if the table has been in Hive. What's more, you can create a hive table in hive via flink sql client. 2: Here [https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/hive/hive_read_write/#temporal-table-join] is the doc for using hive as dimensional table. > update hive streaming read and temporal table documents > --- > > Key: FLINK-20184 > URL: https://issues.apache.org/jira/browse/FLINK-20184 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Documentation >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > The hive streaming read and temporal table document has been out of style, we > need to update it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29752) Modify Flink Table Store connector to trigger full compaction constantly when full changelog is needed
[ https://issues.apache.org/jira/browse/FLINK-29752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29752: --- Labels: pull-request-available (was: ) > Modify Flink Table Store connector to trigger full compaction constantly when > full changelog is needed > -- > > Key: FLINK-29752 > URL: https://issues.apache.org/jira/browse/FLINK-29752 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.3.0 > > > The last step to produce full compaction changelog is to modify Flink Table > Store connector, so that full compaction will be triggered once in a while. > If not, changelog files are not guaranteed to be produced, and the last few > records for a partition may not appear in changelog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] tsreaper opened a new pull request, #361: [FLINK-29752] Modify Flink Table Store connector to trigger full compaction constantly when full changelog is needed
tsreaper opened a new pull request, #361: URL: https://github.com/apache/flink-table-store/pull/361 The last step to produce full compaction changelog is to modify Flink Table Store connector, so that full compaction will be triggered once in a while. If not, changelog files are not guaranteed to be produced, and the last few records for a partition may not appear in changelog. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-29495) PulsarSinkE2ECase hang
[
https://issues.apache.org/jira/browse/FLINK-29495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen closed FLINK-29495.
-
Resolution: Fixed
follow-up to fix 1.15 via b37999514cbbd019b31fb2d9c4ae751a956f6c87
> PulsarSinkE2ECase hang
> --
>
> Key: FLINK-29495
> URL: https://issues.apache.org/jira/browse/FLINK-29495
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.16.0, 1.17.0, 1.15.2
>Reporter: Xingbo Huang
>Assignee: Yufan Sheng
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0, 1.15.3, 1.16.0
>
>
> {code:java}
> 2022-10-02T05:53:56.0611489Z "main" #1 prio=5 os_prio=0 cpu=5171.60ms
> elapsed=9072.82s tid=0x7f9508028000 nid=0x54ef1 waiting on condition
> [0x7f950f994000]
> 2022-10-02T05:53:56.0612041Zjava.lang.Thread.State: TIMED_WAITING
> (parking)
> 2022-10-02T05:53:56.0612475Z at
> jdk.internal.misc.Unsafe.park(java.base@11.0.16.1/Native Method)
> 2022-10-02T05:53:56.0613302Z - parking to wait for <0x87d261f8> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> 2022-10-02T05:53:56.0613959Z at
> java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.16.1/LockSupport.java:234)
> 2022-10-02T05:53:56.0614661Z at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.16.1/AbstractQueuedSynchronizer.java:2123)
> 2022-10-02T05:53:56.0615428Z at
> org.apache.pulsar.common.util.collections.GrowableArrayBlockingQueue.poll(GrowableArrayBlockingQueue.java:203)
> 2022-10-02T05:53:56.0616165Z at
> org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.internalReceive(MultiTopicsConsumerImpl.java:370)
> 2022-10-02T05:53:56.0616807Z at
> org.apache.pulsar.client.impl.ConsumerBase.receive(ConsumerBase.java:198)
> 2022-10-02T05:53:56.0617486Z at
> org.apache.flink.connector.pulsar.testutils.sink.PulsarPartitionDataReader.poll(PulsarPartitionDataReader.java:72)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41526=logs=6e8542d7-de38-5a33-4aca-458d6c87066d=5846934b-7a4f-545b-e5b0-eb4d8bda32e1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-29095) Improve logging in SharedStateRegistry
[ https://issues.apache.org/jira/browse/FLINK-29095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Congxian Qiu resolved FLINK-29095. -- Resolution: Fixed > Improve logging in SharedStateRegistry > --- > > Key: FLINK-29095 > URL: https://issues.apache.org/jira/browse/FLINK-29095 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Jing Ge >Assignee: Yanfei Lei >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0 > > > with the incremental checkpoint, conceptually, state files that are never > used by any checkpoint will be deleted/GC . In practices, state files might > be deleted when they are still somehow required by the failover which will > lead to Flink job fails. > We should add the log for trouble shooting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] tisonkun merged pull request #21250: [FLINK-29495][Connector/Pulsar] Bump the Pulsar to latest 2.10.2
tisonkun merged PR #21250: URL: https://github.com/apache/flink/pull/21250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tisonkun commented on pull request #21250: [FLINK-29495][Connector/Pulsar] Bump the Pulsar to latest 2.10.2
tisonkun commented on PR #21250: URL: https://github.com/apache/flink/pull/21250#issuecomment-1306718736 Merging.. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29095) Improve logging in SharedStateRegistry
[ https://issues.apache.org/jira/browse/FLINK-29095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630226#comment-17630226 ] Congxian Qiu commented on FLINK-29095: -- merged into master 9a4250d248e93f3e87b211df98ce3d3c66aabca0 > Improve logging in SharedStateRegistry > --- > > Key: FLINK-29095 > URL: https://issues.apache.org/jira/browse/FLINK-29095 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Jing Ge >Assignee: Yanfei Lei >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0 > > > with the incremental checkpoint, conceptually, state files that are never > used by any checkpoint will be deleted/GC . In practices, state files might > be deleted when they are still somehow required by the failover which will > lead to Flink job fails. > We should add the log for trouble shooting. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-28083) PulsarSource cannot work with object-reusing DeserializationSchema.
[ https://issues.apache.org/jira/browse/FLINK-28083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-28083. - Resolution: Fixed master via 27d42b2e599d4fafc45698711167810407ea0fa2 > PulsarSource cannot work with object-reusing DeserializationSchema. > --- > > Key: FLINK-28083 > URL: https://issues.apache.org/jira/browse/FLINK-28083 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.14.4, 1.15.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.17.0 > > > This issue is the same as Kafka's > https://issues.apache.org/jira/browse/FLINK-25132 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] klion26 merged pull request #21136: [FLINK-29095][state] Improve logging in SharedStateRegistry
klion26 merged PR #21136: URL: https://github.com/apache/flink/pull/21136 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tisonkun merged pull request #21205: [FLINK-28083][Connector/Pulsar] Object-reusing for Pulsar source
tisonkun merged PR #21205: URL: https://github.com/apache/flink/pull/21205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tisonkun commented on pull request #21205: [FLINK-28083][Connector/Pulsar] Object-reusing for Pulsar source
tisonkun commented on PR #21205: URL: https://github.com/apache/flink/pull/21205#issuecomment-1306717201 Merging... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] klion26 commented on pull request #21136: [FLINK-29095][state] Improve logging in SharedStateRegistry
klion26 commented on PR #21136: URL: https://github.com/apache/flink/pull/21136#issuecomment-1306717133 @fredia thanks for the contribution, merging it now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21257: [FLINK-29879][filesystem] introduce operators for merging files in batch mode.
flinkbot commented on PR #21257: URL: https://github.com/apache/flink/pull/21257#issuecomment-1306713547 ## CI report: * f9ea72353323159ccceb94b9d97322692c2efc70 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29879) Introduce operators for files merging in batch mode
[ https://issues.apache.org/jira/browse/FLINK-29879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29879: --- Labels: pull-request-available (was: ) > Introduce operators for files merging in batch mode > --- > > Key: FLINK-29879 > URL: https://issues.apache.org/jira/browse/FLINK-29879 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Affects Versions: 1.16.0 >Reporter: luoyuxia >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > Similar to streaming mode, we introuce the following four operators: > BatchFileWriter -> BatchCompactCoordinator -> BatchCompactOperator -> > BatchPartitionCommitter -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] luoyuxia opened a new pull request, #21257: [FLINK-29879][filesystem] introduce operators for merging files in batch mode.
luoyuxia opened a new pull request, #21257: URL: https://github.com/apache/flink/pull/21257 ## What is the purpose of the change Introduce operators for merging files in batch mode. The whole pipeline will be `BatchFileWriter ` -> `BatchCompactCoordinator ` -> `BatchCompactOperator ` -> `BatchPartitionCommitter ` ## Brief change log - Introuce `BatchFileWriter` which will write files and emit the written path to downstream - introuce `BatchCompactCoordinator `which will collect all written files and decide what files should be merged to a single file. - Introuce `BatchCompactOperator` which will do the actual compaction for files* - Introuce `BatchPartitionCommitter` which will commit files and partitions ## Verifying this change Added UT for each single operator ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29092) HadoopPathBasedPartFileWriterTest.testWriteFile failed with AssertionError
[
https://issues.apache.org/jira/browse/FLINK-29092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630218#comment-17630218
]
Matthias Pohl commented on FLINK-29092:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42896=logs=4eda0b4a-bd0d-521a-0916-8285b9be9bb5=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9=12322
> HadoopPathBasedPartFileWriterTest.testWriteFile failed with AssertionError
> --
>
> Key: FLINK-29092
> URL: https://issues.apache.org/jira/browse/FLINK-29092
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.16.0
>Reporter: Xingbo Huang
>Assignee: KurtDing
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0
>
> Attachments: image-2022-09-05-17-31-44-813.png
>
>
> {code:java}
> 2022-08-24T02:00:01.1670618Z Aug 24 02:00:01 [ERROR]
> org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriterTest.testWriteFile
> Time elapsed: 2.311 s <<< FAILURE!
> 2022-08-24T02:00:01.1671250Z Aug 24 02:00:01 java.lang.AssertionError:
> 2022-08-24T02:00:01.1671626Z Aug 24 02:00:01
> 2022-08-24T02:00:01.1672001Z Aug 24 02:00:01 Expected size: 1 but was: 2 in:
> 2022-08-24T02:00:01.1673656Z Aug 24 02:00:01
> [DeprecatedRawLocalFileStatus{path=file:/tmp/junit3893779198554813459/junit1595046776902782406/2022-08-24--02;
> isDirectory=true; modification_time=166130640;
> access_time=1661306400396; owner=; group=; permission=rwxrwxrwx;
> isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false},
> 2022-08-24T02:00:01.1676131Z Aug 24 02:00:01
> DeprecatedRawLocalFileStatus{path=file:/tmp/junit3893779198554813459/junit1595046776902782406/2022-08-24--01;
> isDirectory=true; modification_time=166130640;
> access_time=1661306400326; owner=; group=; permission=rwxrwxrwx;
> isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}]
> 2022-08-24T02:00:01.1677339Z Aug 24 02:00:01 at
> org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriterTest.validateResult(HadoopPathBasedPartFileWriterTest.java:107)
> 2022-08-24T02:00:01.1678274Z Aug 24 02:00:01 at
> org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriterTest.testWriteFile(HadoopPathBasedPartFileWriterTest.java:97)
> 2022-08-24T02:00:01.1679017Z Aug 24 02:00:01 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-08-24T02:00:01.1679666Z Aug 24 02:00:01 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-08-24T02:00:01.1680369Z Aug 24 02:00:01 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-08-24T02:00:01.1681019Z Aug 24 02:00:01 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-08-24T02:00:01.1681666Z Aug 24 02:00:01 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-08-24T02:00:01.1682385Z Aug 24 02:00:01 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-08-24T02:00:01.1683094Z Aug 24 02:00:01 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-08-24T02:00:01.1683965Z Aug 24 02:00:01 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-08-24T02:00:01.1684713Z Aug 24 02:00:01 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-08-24T02:00:01.1685956Z Aug 24 02:00:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> 2022-08-24T02:00:01.1687185Z Aug 24 02:00:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> 2022-08-24T02:00:01.1688347Z Aug 24 02:00:01 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2022-08-24T02:00:01.1689324Z Aug 24 02:00:01 at
> java.lang.Thread.run(Thread.java:748)
> 2022-08-24T02:00:01.1690062Z Aug 24 02:00:01
> 2022-08-24T02:00:03.4727706Z Aug 24 02:00:03 Formatting using clusterid:
> testClusterID
> 2022-08-24T02:00:07.9860626Z Aug 24 02:00:07 [INFO] Tests run: 12, Failures:
> 0, Errors: 0, Skipped: 0, Time elapsed: 13.712 s - in
> org.apache.flink.formats.hadoop.bulk.committer.HadoopRenameCommitterHDFSTest
> 2022-08-24T02:00:08.4139747Z Aug 24 02:00:08 [INFO]
> 2022-08-24T02:00:08.4140678Z Aug 24 02:00:08 [INFO] Results:
> 2022-08-24T02:00:08.4141326Z Aug 24 02:00:08 [INFO]
> 2022-08-24T02:00:08.4142008Z Aug 24 02:00:08 [ERROR] Failures:
> 2022-08-24T02:00:08.4144242Z Aug 24 02:00:08 [ERROR]
> HadoopPathBasedPartFileWriterTest.testWriteFile:97->validateResult:107
> 2022-08-24T02:00:08.4145317Z
[GitHub] [flink] luoyuxia commented on pull request #21242: [FLINK-29878][hive] Fix flink-sql-connector hive error
luoyuxia commented on PR #21242: URL: https://github.com/apache/flink/pull/21242#issuecomment-1306706377 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora merged pull request #428: [FLINK-29851] Correctly depend on flink-kubernetes-standalone shaded classifier
gyfora merged PR #428: URL: https://github.com/apache/flink-kubernetes-operator/pull/428 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora merged pull request #430: [FLINK-29851] Remove deprecated fabric8 api usage
gyfora merged PR #430: URL: https://github.com/apache/flink-kubernetes-operator/pull/430 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #430: [FLINK-29851] Remove deprecated fabric8 api usage
gyfora commented on code in PR #430: URL: https://github.com/apache/flink-kubernetes-operator/pull/430#discussion_r1016190856 ## flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/utils/EventUtils.java: ## @@ -107,7 +104,7 @@ public static boolean createOrUpdateEvent( .withNamespace(target.getMetadata().getNamespace()) .endMetadata() .build(); - client.v1().events().inNamespace(target.getMetadata().getNamespace()).create(event); +client.resource(event).createOrReplace(); Review Comment: The one in the event metadata -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #427: [FLINK-29871] Add 1.16 to CI tests
gyfora commented on code in PR #427: URL: https://github.com/apache/flink-kubernetes-operator/pull/427#discussion_r1016189226 ## e2e-tests/data/multi-sessionjob.yaml: ## @@ -74,7 +74,7 @@ metadata: namespace: flink name: session-cluster-1 spec: - image: flink:1.15 + image: flink:1.16 flinkVersion: v1_15 Review Comment: Seems like it. :) will fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-29924) Update official document
[ https://issues.apache.org/jira/browse/FLINK-29924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caizhi Weng reassigned FLINK-29924: --- Assignee: Hang HOU > Update official document > > > Key: FLINK-29924 > URL: https://issues.apache.org/jira/browse/FLINK-29924 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.2.1 > Environment: Flink Table Store v0.2.1 >Reporter: Hang HOU >Assignee: Hang HOU >Priority: Minor > Labels: pull-request-available > > Common missing of a period,in Description of "kafka.bootstrap.servers". > [link|https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/docs/development/configuration/#kafkalogoptions] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29924) Update official document
[ https://issues.apache.org/jira/browse/FLINK-29924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Caizhi Weng closed FLINK-29924. --- Resolution: Fixed master: 5e176d286c87a422a5eaab7a9b3e0a3d1546b45d > Update official document > > > Key: FLINK-29924 > URL: https://issues.apache.org/jira/browse/FLINK-29924 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.2.1 > Environment: Flink Table Store v0.2.1 >Reporter: Hang HOU >Priority: Minor > Labels: pull-request-available > > Common missing of a period,in Description of "kafka.bootstrap.servers". > [link|https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/docs/development/configuration/#kafkalogoptions] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] tsreaper merged pull request #359: [FLINK-29924] Fix missing a period in the description of kafka.bootstrap.servers.
tsreaper merged PR #359: URL: https://github.com/apache/flink-table-store/pull/359 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper opened a new pull request, #360: [FLINK-29805] Fix incorrect snapshot filter when snapshots are committing too slow
tsreaper opened a new pull request, #360: URL: https://github.com/apache/flink-table-store/pull/360 (Cherry-picked from #350) Table Store sink continuously fails with "Trying to add file which is already added" when snapshot committing is slow. This is due to a bug in `FileStoreCommitImpl#filterCommitted`. When this method finds an identifier, it removes the identifier from a map. However different snapshots may have the same identifier (for example an APPEND commit and the following COMPACT commit will have the same identifier), so we need to use another set to check for identifiers. When snapshot committing is fast there is at most 1 identifier to check after the job restarts, so nothing happens. However when snapshot committing is slow, there will be multiple identifiers to check and some identifiers will be mistakenly kept. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] czy006 commented on a diff in pull request #21012: [FLINK-29543] Jar Run Rest Handler Support Flink Configuration
czy006 commented on code in PR #21012:
URL: https://github.com/apache/flink/pull/21012#discussion_r1016183568
##
flink-runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/handlers/JarHandlerParameterTest.java:
##
@@ -69,6 +71,11 @@ enum ProgramArgsParType {
static final String[] PROG_ARGS = new String[] {"--host", "localhost",
"--port", "1234"};
static final int PARALLELISM = 4;
+static final Map FLINK_CONFIGURATION =
+ImmutableMap.of(
+CoreOptions.DEFAULT_PARALLELISM.key(), "2",
Review Comment:
Now I do this with a new validation with JobGraph method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-29842) Change commitIdentifier in Table Store snapshot to long value
[
https://issues.apache.org/jira/browse/FLINK-29842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caizhi Weng closed FLINK-29842.
---
Resolution: Fixed
master: 350627439a70b187f7270278b497841fa3f2c554
release-0.2: bc5646cc62bb341954d6320c7f3ed375195e747f
> Change commitIdentifier in Table Store snapshot to long value
> -
>
> Key: FLINK-29842
> URL: https://issues.apache.org/jira/browse/FLINK-29842
> Project: Flink
> Issue Type: Improvement
> Components: Table Store
>Affects Versions: table-store-0.3.0, table-store-0.2.2
>Reporter: Caizhi Weng
>Assignee: Caizhi Weng
>Priority: Major
> Labels: pull-request-available
> Fix For: table-store-0.3.0, table-store-0.2.2
>
>
> Currently {{commitIdentifier}} in {{Snapshot}} is a {{String}} value. However
> there are many scenarios where we need to compare two identifiers to find out
> which one is newer. For example
> * In FLINK-29840, we need to store the latest modified commit for each
> writer. Only when the latest snapshot is newer than this commit can we safely
> close the writer.
> * In FLINK-29805, we can read the commit identifier of the latest snapshot.
> All identifiers older than that should be filtered out.
> * In FLINK-29752, we need to trigger full compaction once in a few commits.
> We can read the latest commit identifier and compare it with the full
> compaction identifier to check if full compaction is successfully committed.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-table-store] tsreaper merged pull request #358: [FLINK-29842] Change commitIdentifier in Table Store snapshot to long value
tsreaper merged PR #358: URL: https://github.com/apache/flink-table-store/pull/358 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22587) Support aggregations in batch mode with DataStream API
[
https://issues.apache.org/jira/browse/FLINK-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630193#comment-17630193
]
Yun Gao commented on FLINK-22587:
-
Hi, sorry for forgetting to update here, with some more try we found that it
works to use an event-time window that assigns all the records to the same
window [0,
+Inf):[https://github.com/apache/flink-ml/blob/master/flink-ml-core/src/main/java/org/apache/flink/ml/common/datastream/EndOfStreamWindows.java]
And the join works with
source1.join(source2)
.where(a -> a.f0)
.equalTo(b -> b.f0)
.window(new EndOfStreamWindows())
.apply(xx)
for both bounded streaming processing and batch processing. It does not require
the records to have event-time and watermark, since the assignment does not
rely on event-time, and the window will be triggered by the Long.MAX_VALUE
inserted at the end of stream.
But we'll still try to propose a proper fix for this issue. One option is that
we does not force to set a window in this case, if the window is not set, we'll
by default mark it all the records.
> Support aggregations in batch mode with DataStream API
> --
>
> Key: FLINK-22587
> URL: https://issues.apache.org/jira/browse/FLINK-22587
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Etienne Chauchot
>Priority: Major
>
> A pipeline like this *in batch mode* would output no data
> {code:java}
> stream.join(otherStream)
> .where()
> .equalTo()
> .window(GlobalWindows.create())
> .apply()
> {code}
> Indeed the default trigger for GlobalWindow is NeverTrigger which never
> fires. If we set a _EventTimeTrigger_ it will fire with every element as the
> watermark will be set to +INF (batch mode) and will pass the end of the
> global window with each new element. A _ProcessingTimeTrigger_ never fires
> either and all elapsed time or delta based triggers would not be suited for
> batch.
> Same goes for _reduce()_ instead of join().
> So I guess we miss something for batch support with DataStream.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29924) Update official document
[ https://issues.apache.org/jira/browse/FLINK-29924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29924: --- Labels: pull-request-available (was: ) > Update official document > > > Key: FLINK-29924 > URL: https://issues.apache.org/jira/browse/FLINK-29924 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.2.1 > Environment: Flink Table Store v0.2.1 >Reporter: Hang HOU >Priority: Minor > Labels: pull-request-available > > Common missing of a period,in Description of "kafka.bootstrap.servers". > [link|https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/docs/development/configuration/#kafkalogoptions] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] houhang1005 opened a new pull request, #359: [FLINK-29924] Fix missing a period in the description of kafka.bootstrap.servers.
houhang1005 opened a new pull request, #359: URL: https://github.com/apache/flink-table-store/pull/359 Just add a missing period about description of "kafka.bootstrap.servers". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-5463) RocksDB.disposeInternal does not react to interrupts, blocks task cancellation
[
https://issues.apache.org/jira/browse/FLINK-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630174#comment-17630174
]
wangzhewenyi commented on FLINK-5463:
-
We had the same problem。 We use flink version 1.11 and submit the flink task in
standalon mode。{*}An error occurs when the sstable file in the tmp directory of
linux is deleted。{*} *As you can see from the flink thread stack, the
disposeInternal function will be suspended。This results in flink tasks that
cannot be canceled and new tasks that cannot be restarted.* Flink should be
able to handle the error elegantly instead of hanging forever.{*}{*}
*error info* *as follows:*
org.apache.flink.util.FlinkRuntimeException: Error while adding data to RocksDB
at
org.apache.flink.contrib.streaming.state.RocksDBListState.add(RocksDBListState.java:168)
at
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:394)
at
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:161)
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:178)
at
org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:153)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:345)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:191)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:181)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:558)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:530)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.rocksdb.RocksDBException: While open a file for appending:
/tmp/flink-io-b990a9a9-fb96-4d30-bc60-2a7cce923887/job_0cb5b535f8b9a74302233a9a520c828e_op_WindowOperator_d3340fe028bd2a0d8d6de62cea3d11cf__14_20__uuid_6f52bf03-612c-4d13-bd72-76893dfcf709/db/13.sst:
No such file or directory
at org.rocksdb.RocksDB.merge(Native Method)
at org.rocksdb.RocksDB.merge(RocksDB.java:683)
at
org.apache.flink.contrib.streaming.state.RocksDBListState.add(RocksDBListState.java:161)
... 13 more
*The stack as follows：*
"Window(ProcessingTimeSessionWindows(3), ProcessingTimeTrigger,
ProcessWindowFunction$2) (8/12)" #1733 prio=5 os_prio=0 cpu=95.13ms
elapsed=1829.37s tid=0x7f3550015000 nid=0x98a8 runnable
[0x7f33c88e6000]
java.lang.Thread.State: RUNNABLE
at org.rocksdb.RocksDB.disposeInternal(Native Method)
at org.rocksdb.RocksObject.disposeInternal(RocksObject.java:37)
at
org.rocksdb.AbstractImmutableNativeReference.close(AbstractImmutableNativeReference.java:57)
at org.apache.flink.util.IOUtils.closeQuietly(IOUtils.java:279)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.dispose(RocksDBKeyedStateBackend.java:366)
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.lambda$dispose$1(StreamOperatorStateHandler.java:131)
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler$$Lambda$929/0x0017ce31cc40.close(Unknown
Source)
at
org.apache.flink.shaded.guava18.com.google.common.io.Closer.close(Closer.java:214)
at
org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.dispose(StreamOperatorStateHandler.java:133)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.dispose(AbstractStreamOperator.java:298)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.dispose(AbstractUdfStreamOperator.java:114)
at
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.dispose(WindowOperator.java:286)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.disposeAllOperators(StreamTask.java:703)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.cleanUpInvoke(StreamTask.java:635)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:542)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
Locked ownable synchronizers:
- None
> RocksDB.disposeInternal does not react to interrupts, blocks task cancellation
> --
>
> Key: FLINK-5463
> URL:
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #156: [FLINK-29323] Refine Transformer for VectorAssembler
zhipeng93 commented on code in PR #156:
URL: https://github.com/apache/flink-ml/pull/156#discussion_r1016126953
##
docs/content/docs/operators/feature/vectorassembler.md:
##
@@ -26,10 +26,20 @@ under the License.
-->
## Vector Assembler
-
-Vector Assembler combines a given list of input columns into a vector column.
-Types of input columns must be either vector or numerical value.
-
+A Transformer which combines a given list of input columns into a vector
column. Input columns
+would be numerics or vectors whose size is specified by the {@link
#INPUT_SIZES} parameter.
Review Comment:
nit: `Input columns would be numerics` --> `Input Columns could be numerical
or vectors whose sizes are specified by...`
Same for the java docs in python and java source code.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/vectorassembler/VectorAssembler.java:
##
@@ -74,64 +85,109 @@ public Table[] transform(Table... inputs) {
DataStream output =
tEnv.toDataStream(inputs[0])
.flatMap(
-new AssemblerFunc(getInputCols(),
getHandleInvalid()),
+new AssemblerFunction(
+getInputCols(), getHandleInvalid(),
getInputSizes()),
outputTypeInfo);
Table outputTable = tEnv.fromDataStream(output);
return new Table[] {outputTable};
}
-private static class AssemblerFunc implements FlatMapFunction {
+private static class AssemblerFunction implements FlatMapFunction {
private final String[] inputCols;
private final String handleInvalid;
+private final Integer[] inputSizes;
+private final boolean keepInvalid;
-public AssemblerFunc(String[] inputCols, String handleInvalid) {
+public AssemblerFunction(String[] inputCols, String handleInvalid,
Integer[] inputSizes) {
this.inputCols = inputCols;
this.handleInvalid = handleInvalid;
+this.inputSizes = inputSizes;
+keepInvalid = handleInvalid.equals(HasHandleInvalid.KEEP_INVALID);
}
@Override
public void flatMap(Row value, Collector out) {
int nnz = 0;
int vectorSize = 0;
try {
-for (String inputCol : inputCols) {
-Object object = value.getField(inputCol);
-Preconditions.checkNotNull(object, "Input column value
should not be null.");
-if (object instanceof Number) {
-nnz += 1;
-vectorSize += 1;
-} else if (object instanceof SparseVector) {
-nnz += ((SparseVector) object).indices.length;
-vectorSize += ((SparseVector) object).size();
-} else if (object instanceof DenseVector) {
-nnz += ((DenseVector) object).size();
-vectorSize += ((DenseVector) object).size();
+for (int i = 0; i < inputCols.length; ++i) {
Review Comment:
This logic is a bit messy here. How about we split it into several
functions? e.g.,
`assembleNullValues()` and `assembleNotNullValues()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-29873) CannotPlanException raised after "CROSS JOIN UNNEST"
[
https://issues.apache.org/jira/browse/FLINK-29873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630173#comment-17630173
]
Daeyeon Bae commented on FLINK-29873:
-
If it works well on the master branch, does it mean that the 1.15.2 version
doesn't also throw an error?
> CannotPlanException raised after "CROSS JOIN UNNEST"
>
>
> Key: FLINK-29873
> URL: https://issues.apache.org/jira/browse/FLINK-29873
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.2
>Reporter: Daeyeon Bae
>Priority: Major
> Attachments: error-log.txt
>
>
> When I create a VIEW as a result of "CROSS JOIN UNNEST" and then use the
> condition in the WHERE clause of the VIEW, it throws an exception
> "org.apache.calcite.plan.RelOptPlanner$CannotPlanException".
> Why am I getting this exception and how should I handle it the right way?
> The following is the test code in which an error occurs.
> {code:java}
> it should "filter with object_key" in {
> tEnv.executeSql(
> s"""CREATE TABLE s3_put_event (
> | Records ARRAY<
> |ROW<
> | s3 ROW<
> |bucket ROW,
> |object ROW
> | >
> |>
> | >
> |) WITH (
> | 'connector' = 'datagen',
> | 'number-of-rows' = '3',
> | 'rows-per-second' = '1',
> | 'fields.Records.element.s3.bucket.name.length' = '8',
> | 'fields.Records.element.s3.object.key.length' = '15',
> | 'fields.Records.element.s3.object.size.min' = '1',
> | 'fields.Records.element.s3.object.size.max' = '1000'
> |)
> |""".stripMargin
> )
> tEnv.executeSql(
> s"""CREATE TEMPORARY VIEW s3_objects AS
> |SELECT object_key, bucket_name
> |FROM (
> | SELECT
> |r.s3.bucket.name AS bucket_name,
> |r.s3.object.key AS object_key,
> |r.s3.object.size AS object_size
> | FROM s3_put_event
> | CROSS JOIN UNNEST(s3_put_event.Records) AS r(s3)
> |) rs
> |WHERE object_size > 0
> |""".stripMargin
> )
> tEnv.executeSql(
> s"""CREATE TEMPORARY VIEW filtered_s3_objects AS
> |SELECT bucket_name, object_key
> |FROM s3_objects
> |WHERE object_key > ''
> |""".stripMargin)
> val result = tEnv.sqlQuery("SELECT * FROM filtered_s3_objects")
> tEnv.toChangelogStream(result).print()
> env.execute()
> } {code}
> If I remove the condition object_key > '' in the "filtered_s3_objects" VIEW,
> and do it in the "s3_objects" VIEW, no exception is thrown.However, my actual
> query is complicated, so it is not easy to move the condition of the WHERE
> clause like this. It's hard to use especially if I need to separate the
> output stream.
> I have attached the error log.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] Samrat002 commented on pull request #21242: [FLINK-29878][hive] Fix flink-sql-connector hive error
Samrat002 commented on PR #21242: URL: https://github.com/apache/flink/pull/21242#issuecomment-1306616040 ``` Nov 07 19:06:18 java.lang.AssertionError: Create test topic : partition_failure_recovery_EventTime failed, org.apache.kafka.common.errors.TopicExistsException: Topic 'partition_failure_recovery_EventTime' already exists. Nov 07 19:06:18 at org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:207) Nov 07 19:06:18 at org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:97) Nov 07 19:06:18 at org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:217) Nov 07 19:06:18 at org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecovery(KafkaShuffleExactlyOnceITCase.java:158) Nov 07 19:06:18 at org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime(KafkaShuffleExactlyOnceITCase.java:101) Nov 07 19:06:18 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ``` not related test failure ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29923) Hybrid Shuffle may face deadlock when running a task need to execute big size data
[ https://issues.apache.org/jira/browse/FLINK-29923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630171#comment-17630171 ] Weijie Guo commented on FLINK-29923: [~AlexXXX] Thanks for the feedback. If I'm not wrong, the reason for the failure should be insufficient network memory or batch read memory, and this is an expected behavior. After all, pipelined execution requires more resources than all blocking. So now we have to solve the problem that the task thread is stuck. Can you provide more detailed information, such as the thread dump of the stuck subtask. In addition, if it is difficult to describe the problem clearly, you can communicate with me offline via wechat(a644813550) or any other contact ways you want. > Hybrid Shuffle may face deadlock when running a task need to execute big size > data > -- > > Key: FLINK-29923 > URL: https://issues.apache.org/jira/browse/FLINK-29923 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0 >Reporter: AlexHu >Priority: Major > Attachments: 性能差距.png, 死锁2-select.png, 死锁检测.png > > > The flink 1.16 offers hybrid shuffle to combine the superiority of blocking > shuffle and pipeline shuffle. But when I want to test this new feature I face > a problem that it may cause deadlock when it running. > Actually, it will run well at beginning. However, when it runs to a certain > number it may failure for the buffer size and if I set a bigger size it may > running without data execution like the picture. So I want to ask the cause > of this problem and a solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29924) Update official document
[ https://issues.apache.org/jira/browse/FLINK-29924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hang HOU updated FLINK-29924: - Description: Common missing of a period,in Description of "kafka.bootstrap.servers". [link|https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/docs/development/configuration/#kafkalogoptions] was: Common missing of a period. [链接标题|https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/docs/development/configuration/#kafkalogoptions] > Update official document > > > Key: FLINK-29924 > URL: https://issues.apache.org/jira/browse/FLINK-29924 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.2.1 > Environment: Flink Table Store v0.2.1 >Reporter: Hang HOU >Priority: Minor > > Common missing of a period,in Description of "kafka.bootstrap.servers". > [link|https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/docs/development/configuration/#kafkalogoptions] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29924) Update official document
Hang HOU created FLINK-29924: Summary: Update official document Key: FLINK-29924 URL: https://issues.apache.org/jira/browse/FLINK-29924 Project: Flink Issue Type: Improvement Components: Table Store Affects Versions: table-store-0.2.1 Environment: Flink Table Store v0.2.1 Reporter: Hang HOU Common missing of a period. [链接标题|https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/docs/development/configuration/#kafkalogoptions] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29923) Hybrid Shuffle may face deadlock when running a task need to execute big size data
[ https://issues.apache.org/jira/browse/FLINK-29923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630168#comment-17630168 ] Xintong Song commented on FLINK-29923: -- Thanks for reporting this, [~AlexXXX]. [~Weijie Guo], could you take a look at this? > Hybrid Shuffle may face deadlock when running a task need to execute big size > data > -- > > Key: FLINK-29923 > URL: https://issues.apache.org/jira/browse/FLINK-29923 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0 >Reporter: AlexHu >Priority: Major > Attachments: 性能差距.png, 死锁2-select.png, 死锁检测.png > > > The flink 1.16 offers hybrid shuffle to combine the superiority of blocking > shuffle and pipeline shuffle. But when I want to test this new feature I face > a problem that it may cause deadlock when it running. > Actually, it will run well at beginning. However, when it runs to a certain > number it may failure for the buffer size and if I set a bigger size it may > running without data execution like the picture. So I want to ask the cause > of this problem and a solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29923) Hybrid Shuffle may face deadlock when running a task need to execute big size data
[ https://issues.apache.org/jira/browse/FLINK-29923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-29923: - Description: The flink 1.16 offers hybrid shuffle to combine the superiority of blocking shuffle and pipeline shuffle. But when I want to test this new feature I face a problem that it may cause deadlock when it running. Actually, it will run well at beginning. However, when it runs to a certain number it may failure for the buffer size and if I set a bigger size it may running without data execution like the picture. So I want to ask the cause of this problem and a solution. was: The flink 1.11.6 offers hybrid shuffle to combine the superiority of blocking shuffle and pipeline shuffle. But when I want to test this new feature I face a problem that it may cause deadlock when it running. Actually, it will run well at beginning. However, when it runs to a certain number it may failure for the buffer size and if I set a bigger size it may running without data execution like the picture. So I want to ask the cause of this problem and a solution. > Hybrid Shuffle may face deadlock when running a task need to execute big size > data > -- > > Key: FLINK-29923 > URL: https://issues.apache.org/jira/browse/FLINK-29923 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0 >Reporter: AlexHu >Priority: Major > Attachments: 性能差距.png, 死锁2-select.png, 死锁检测.png > > > The flink 1.16 offers hybrid shuffle to combine the superiority of blocking > shuffle and pipeline shuffle. But when I want to test this new feature I face > a problem that it may cause deadlock when it running. > Actually, it will run well at beginning. However, when it runs to a certain > number it may failure for the buffer size and if I set a bigger size it may > running without data execution like the picture. So I want to ask the cause > of this problem and a solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29923) Hybrid Shuffle may face deadlock when running a task need to execute big size data
[ https://issues.apache.org/jira/browse/FLINK-29923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-29923: - Affects Version/s: 1.16.0 (was: 1.11.6) > Hybrid Shuffle may face deadlock when running a task need to execute big size > data > -- > > Key: FLINK-29923 > URL: https://issues.apache.org/jira/browse/FLINK-29923 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0 >Reporter: AlexHu >Priority: Major > Attachments: 性能差距.png, 死锁2-select.png, 死锁检测.png > > > The flink 1.11.6 offers hybrid shuffle to combine the superiority of blocking > shuffle and pipeline shuffle. But when I want to test this new feature I face > a problem that it may cause deadlock when it running. > Actually, it will run well at beginning. However, when it runs to a certain > number it may failure for the buffer size and if I set a bigger size it may > running without data execution like the picture. So I want to ask the cause > of this problem and a solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29923) Hybrid Shuffle may face deadlock when running a task need to execute big size data
AlexHu created FLINK-29923: -- Summary: Hybrid Shuffle may face deadlock when running a task need to execute big size data Key: FLINK-29923 URL: https://issues.apache.org/jira/browse/FLINK-29923 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.11.6 Reporter: AlexHu Attachments: 性能差距.png, 死锁2-select.png, 死锁检测.png The flink 1.11.6 offers hybrid shuffle to combine the superiority of blocking shuffle and pipeline shuffle. But when I want to test this new feature I face a problem that it may cause deadlock when it running. Actually, it will run well at beginning. However, when it runs to a certain number it may failure for the buffer size and if I set a bigger size it may running without data execution like the picture. So I want to ask the cause of this problem and a solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] tsreaper merged pull request #354: [FLINK-29840] Fix bug that old record may overwrite new record in Table Store when snapshot committing is slow
tsreaper merged PR #354: URL: https://github.com/apache/flink-table-store/pull/354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper opened a new pull request, #358: [FLINK-29842] Change commitIdentifier in Table Store snapshot to long value
tsreaper opened a new pull request, #358: URL: https://github.com/apache/flink-table-store/pull/358 (Cherry-picked from #346) Currently `commitIdentifier` in `Snapshot` is a `String` value. However there are many scenarios where we need to compare two identifiers to find out which one is newer. For example * In FLINK-29840, we need to store the latest modified commit for each writer. Only when the latest snapshot is newer than this commit can we safely close the writer. * In FLINK-29805, we can read the commit identifier of the latest snapshot. All identifiers older than that should be filtered out. * In FLINK-29752, we need to trigger full compaction once in a few commits. We can read the latest commit identifier and compare it with the full compaction identifier to check if full compaction is successfully committed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29922) Table store hive catalog support create external table
[ https://issues.apache.org/jira/browse/FLINK-29922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29922: --- Labels: pull-request-available (was: ) > Table store hive catalog support create external table > -- > > Key: FLINK-29922 > URL: https://issues.apache.org/jira/browse/FLINK-29922 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: konwu >Priority: Major > Labels: pull-request-available > > Support create external table for table store hive catalog -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] wxplovecc opened a new pull request, #357: [FLINK-29922] Support create external table for hive catalog
wxplovecc opened a new pull request, #357: URL: https://github.com/apache/flink-table-store/pull/357 support hive catalog to create external table -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] weibozhao commented on a diff in pull request #156: [FLINK-29323] Refine Transformer for VectorAssembler
weibozhao commented on code in PR #156:
URL: https://github.com/apache/flink-ml/pull/156#discussion_r1016098840
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/vectorassembler/VectorAssemblerParams.java:
##
@@ -21,11 +21,44 @@
import org.apache.flink.ml.common.param.HasHandleInvalid;
import org.apache.flink.ml.common.param.HasInputCols;
import org.apache.flink.ml.common.param.HasOutputCol;
+import org.apache.flink.ml.param.IntArrayParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidator;
/**
* Params of {@link VectorAssembler}.
*
* @param The class type of this instance.
*/
public interface VectorAssemblerParams
-extends HasInputCols, HasOutputCol, HasHandleInvalid {}
+extends HasInputCols, HasOutputCol, HasHandleInvalid {
+Param INPUT_SIZES =
+new IntArrayParam(
+"inputSizes",
+"Sizes of the input elements to be assembled.",
+null,
+sizesValidator());
+
+default Integer[] getInputSizes() {
+return get(INPUT_SIZES);
+}
+
+default T setInputSizes(Integer... value) {
+return set(INPUT_SIZES, value);
+}
+
+// Checks the inputSizes parameter.
+static ParamValidator sizesValidator() {
+return inputSizes -> {
+if (inputSizes == null) {
+return false;
+}
+for (Integer ele : inputSizes) {
+if (ele < 0) {
Review Comment:
If size of element equals to 0, return false as `return inputSizes.length !=
0;`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-29922) Table store hive catalog support create external table
konwu created FLINK-29922: - Summary: Table store hive catalog support create external table Key: FLINK-29922 URL: https://issues.apache.org/jira/browse/FLINK-29922 Project: Flink Issue Type: Improvement Components: Table Store Reporter: konwu Support create external table for table store hive catalog -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] liuzhuang2017 commented on pull request #21255: [hotfix][docs] Update the missing parts of this fault_tolerance Chinese documentation.
liuzhuang2017 commented on PR #21255: URL: https://github.com/apache/flink/pull/21255#issuecomment-1306550847 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liuzhuang2017 commented on pull request #21256: [hotfix][flink-runtime] Fix the typo of RecordingChannelStateWriter class.
liuzhuang2017 commented on PR #21256: URL: https://github.com/apache/flink/pull/21256#issuecomment-1306550196 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] yangjf2019 commented on pull request #425: [hotfix][docs] Typo fix in doc pages
yangjf2019 commented on PR #425: URL: https://github.com/apache/flink-kubernetes-operator/pull/425#issuecomment-1306547321 @gyfora I ran the maven build locally before committing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27246) Code of method "processElement(Lorg/apache/flink/streaming/runtime/streamrecord/StreamRecord;)V" of class "HashAggregateWithKeys$9211" grows beyond 64 KB
[
https://issues.apache.org/jira/browse/FLINK-27246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630134#comment-17630134
]
Caizhi Weng commented on FLINK-27246:
-
[~KristoffSC]
Thanks for the clarification. The general approach looks good to me.
Could you please sketch out your intended solution a bit more? Like which class
are you going to modify and how are you going to modify it?
> Code of method
> "processElement(Lorg/apache/flink/streaming/runtime/streamrecord/StreamRecord;)V"
> of class "HashAggregateWithKeys$9211" grows beyond 64 KB
> -
>
> Key: FLINK-27246
> URL: https://issues.apache.org/jira/browse/FLINK-27246
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.14.3
>Reporter: Maciej Bryński
>Priority: Major
> Attachments: endInput_falseFilter9123_split9704.txt
>
>
> I think this bug should get fixed in
> https://issues.apache.org/jira/browse/FLINK-23007
> Unfortunately I spotted it on Flink 1.14.3
> {code}
> java.lang.RuntimeException: Could not instantiate generated class
> 'HashAggregateWithKeys$9211'
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:85)
> ~[flink-table_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.table.runtime.operators.CodeGenOperatorFactory.createStreamOperator(CodeGenOperatorFactory.java:40)
> ~[flink-table_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:81)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:198)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.(RegularOperatorChain.java:63)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:666)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at java.lang.Thread.run(Unknown Source) ~[?:?]
> Caused by: org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:76)
> ~[flink-table_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:102)
> ~[flink-table_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:83)
> ~[flink-table_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> ... 11 more
> Caused by:
> org.apache.flink.shaded.guava30.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.shaded.guava30.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.shaded.guava30.com.google.common.cache.LocalCache.get(LocalCache.java:3962)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.shaded.guava30.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4859)
> ~[flink-dist_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:74)
> ~[flink-table_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:102)
> ~[flink-table_2.12-1.14.3-stream1.jar:1.14.3-stream1]
> at
>
[jira] [Commented] (FLINK-29527) Make unknownFieldsIndices work for single ParquetReader
[ https://issues.apache.org/jira/browse/FLINK-29527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630132#comment-17630132 ] Sun Shun commented on FLINK-29527: -- [~lirui] could you please help take a look at this PR when you are free, thanks > Make unknownFieldsIndices work for single ParquetReader > --- > > Key: FLINK-29527 > URL: https://issues.apache.org/jira/browse/FLINK-29527 > Project: Flink > Issue Type: Bug > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.16.0 >Reporter: Sun Shun >Assignee: Sun Shun >Priority: Major > Labels: pull-request-available > > Currently, from the improvement FLINK-23715, Flink use a collection named > `unknownFieldsIndices` to track the nonexistent fields, and it is kept inside > the `ParquetVectorizedInputFormat`, and applied to all parquet files under > given path. > However, some fields may only be nonexistent in some of the historical > parquet files, while exist in latest ones. And based on > `unknownFieldsIndices`, flink will always skip these fields, even thought > they are existing in the later parquets. > As a result, the value of these fields will become empty when they are > nonexistent in some historical parquet files. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zhyg closed pull request #20311: [hotfix][docs-zh]Fixed syntax error
zhyg closed pull request #20311: [hotfix][docs-zh]Fixed syntax error URL: https://github.com/apache/flink/pull/20311 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] wangyang0918 commented on a diff in pull request #427: [FLINK-29871] Add 1.16 to CI tests
wangyang0918 commented on code in PR #427: URL: https://github.com/apache/flink-kubernetes-operator/pull/427#discussion_r1016076388 ## e2e-tests/data/multi-sessionjob.yaml: ## @@ -74,7 +74,7 @@ metadata: namespace: flink name: session-cluster-1 spec: - image: flink:1.15 + image: flink:1.16 flinkVersion: v1_15 Review Comment: Do we forget to update this line? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liuzhuang2017 commented on pull request #21255: [hotfix][docs] Update the missing parts of this fault_tolerance Chinese documentation.
liuzhuang2017 commented on PR #21255: URL: https://github.com/apache/flink/pull/21255#issuecomment-1306512674 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liuzhuang2017 commented on pull request #21256: [hotfix][flink-runtime] Fix the typo of RecordingChannelStateWriter class.
liuzhuang2017 commented on PR #21256: URL: https://github.com/apache/flink/pull/21256#issuecomment-1306512146 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-29892) flink-conf.yaml does not accept hash (#) in the env.java.opts property
[ https://issues.apache.org/jira/browse/FLINK-29892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song closed FLINK-29892. Resolution: Duplicate > flink-conf.yaml does not accept hash (#) in the env.java.opts property > -- > > Key: FLINK-29892 > URL: https://issues.apache.org/jira/browse/FLINK-29892 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.15.2 >Reporter: Sergio Sainz >Priority: Major > > When adding a string with hash (#) character in env.java.opts in > flink-conf.yaml , the string will be truncated from the # onwards even when > the value is surrounded by single quotes or double quotes. > example: > (in flink-conf.yaml): > env.java.opts: "-Djavax.net.ssl.trustStorePassword=my#pwd" > > the value shown on the flink taskmanagers or job managers is : > env.java.opts: -Djavax.net.ssl.trustStorePassword=my > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-29892) flink-conf.yaml does not accept hash (#) in the env.java.opts property
[ https://issues.apache.org/jira/browse/FLINK-29892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song reopened FLINK-29892: -- > flink-conf.yaml does not accept hash (#) in the env.java.opts property > -- > > Key: FLINK-29892 > URL: https://issues.apache.org/jira/browse/FLINK-29892 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.15.2 >Reporter: Sergio Sainz >Priority: Major > > When adding a string with hash (#) character in env.java.opts in > flink-conf.yaml , the string will be truncated from the # onwards even when > the value is surrounded by single quotes or double quotes. > example: > (in flink-conf.yaml): > env.java.opts: "-Djavax.net.ssl.trustStorePassword=my#pwd" > > the value shown on the flink taskmanagers or job managers is : > env.java.opts: -Djavax.net.ssl.trustStorePassword=my > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29892) flink-conf.yaml does not accept hash (#) in the env.java.opts property
[ https://issues.apache.org/jira/browse/FLINK-29892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song closed FLINK-29892. Resolution: Fixed Let's close this duplicated ticket and keep the discussion in FLINK-15358 > flink-conf.yaml does not accept hash (#) in the env.java.opts property > -- > > Key: FLINK-29892 > URL: https://issues.apache.org/jira/browse/FLINK-29892 > Project: Flink > Issue Type: Bug > Components: Deployment / Kubernetes >Affects Versions: 1.15.2 >Reporter: Sergio Sainz >Priority: Major > > When adding a string with hash (#) character in env.java.opts in > flink-conf.yaml , the string will be truncated from the # onwards even when > the value is surrounded by single quotes or double quotes. > example: > (in flink-conf.yaml): > env.java.opts: "-Djavax.net.ssl.trustStorePassword=my#pwd" > > the value shown on the flink taskmanagers or job managers is : > env.java.opts: -Djavax.net.ssl.trustStorePassword=my > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29890) UDFs classloading from JARs in 1.16 is broken
[ https://issues.apache.org/jira/browse/FLINK-29890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630126#comment-17630126 ] dalongliu commented on FLINK-29890: --- [~afedulov] Thanks, I will see it as soon as possible. > UDFs classloading from JARs in 1.16 is broken > - > > Key: FLINK-29890 > URL: https://issues.apache.org/jira/browse/FLINK-29890 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.16.0 >Reporter: Alexander Fedulov >Priority: Critical > > 1.16 introduced a lot of changes with respect to classloading in the Table > API. The way UDFs could previously be loaded from JARs in 1.15 does not work > in 1.16 anymore - it fails with the ClassNotFound exception when UDFs are > used at runtime. > Here is a repository with a reproducible example: > [https://github.com/afedulov/udfs-flink-1.16/blob/main/src/test/java/com/example/UDFTest.java] > > It works as is (Flink 1.15.2) and fails when switching the dependencies to > 1.16.0. > Here are some of the PRs that might be related to the issue: > [https://github.com/apache/flink/pull/20001] > [https://github.com/apache/flink/pull/19845] > [https://github.com/apache/flink/pull/20211] (fixes a similar issue > introduced after classloading changes in 1.16) > > It is unclear how UDFs can be loaded from JARs in 1.16. > Ideally, this should be covered by tests and described in the documentation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #21256: [hotfix][flink-runtime] Fix the typo of RecordingChannelStateWriter class.
flinkbot commented on PR #21256: URL: https://github.com/apache/flink/pull/21256#issuecomment-1306499192 ## CI report: * 97c7322423aeec681f88de40d8e0ccf76b8f9269 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liuzhuang2017 opened a new pull request, #21256: [hotfix][flink-runtime] Fix the typo of RecordingChannelStateWriter class.
liuzhuang2017 opened a new pull request, #21256: URL: https://github.com/apache/flink/pull/21256 ## What is the purpose of the change - Fix the typo of RecordingChannelStateWriter class. ## Brief change log - Fix the typo of RecordingChannelStateWriter class. ## Verifying this change - This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29592) Add Transformer and Estimator for RobustScaler
[ https://issues.apache.org/jira/browse/FLINK-29592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29592: --- Labels: pull-request-available (was: ) > Add Transformer and Estimator for RobustScaler > -- > > Key: FLINK-29592 > URL: https://issues.apache.org/jira/browse/FLINK-29592 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Reporter: Jiang Xin >Priority: Major > Labels: pull-request-available > Fix For: ml-2.2.0 > > > Add Transformer and Estimator for RobustScaler. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] jiangxin369 opened a new pull request, #172: [FLINK-29592] Add Estimator and Transformer for RobustScaler
jiangxin369 opened a new pull request, #172: URL: https://github.com/apache/flink-ml/pull/172 ## What is the purpose of the change Add Estimator and Transformer for RobustScaler. ## Brief change log - Adds Transformer and Estimator implementation of RobustScaler in Java and Python - Adds examples and documentation of RobustScaler ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs / JavaDocs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21255: [hotfix][docs] Update the missing parts of this fault_tolerance Chinese documentation.
flinkbot commented on PR #21255: URL: https://github.com/apache/flink/pull/21255#issuecomment-1306475668 ## CI report: * c690977c1258074b090f8912c2d188c6997cfaec UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liuzhuang2017 opened a new pull request, #21255: [hotfix][docs] Update the missing parts of this fault_tolerance Chinese documentation.
liuzhuang2017 opened a new pull request, #21255: URL: https://github.com/apache/flink/pull/21255 ## What is the purpose of the change - Update the missing parts of this fault_tolerance Chinese documentation. ## Brief change log - Update the missing parts of this fault_tolerance Chinese documentation. ## Verifying this change - No need to test. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21254: Update docker.md
flinkbot commented on PR #21254: URL: https://github.com/apache/flink/pull/21254#issuecomment-1306451186 ## CI report: * a11193c407491a278b0cfb53c48a1cf53c5ffbf1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] mattfysh opened a new pull request, #21254: Update docker.md
mattfysh opened a new pull request, #21254: URL: https://github.com/apache/flink/pull/21254 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #21203: [FLINK-21239][Table SQL/API] Upgrade Calcite to 1.28.0
snuyanzin commented on code in PR #21203:
URL: https://github.com/apache/flink/pull/21203#discussion_r1015947724
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/AbstractCodeGeneratorCastRule.java:
##
@@ -204,9 +204,10 @@ public String getSessionTimeZoneTerm() {
}
@Override
-public String declareVariable(String type, String variablePrefix) {
+public String declareVariable(String type, String variablePrefix,
String defaultValue) {
Review Comment:
yes that's possible.
However it's not only change in pom, after
https://issues.apache.org/jira/browse/CALCITE-3745 it uses new api for compiler
factory initialization. To use Janino 3.0.11 (current master's version) need to
rollback this (includes Calcite's
`EnumerableInterpretable.java`,`JaninoRexCompiler.java`,
`JaninoRelMetadataProvider.java`,
`JaninoCompiler.java`, after movement to 3.1.9 these classes will be
removed)
the commit switching to Janino 3.0.11 will look like
https://github.com/snuyanzin/flink/commit/0fd02d8ab9ac589b8d1bab9b92740bba193a2bed
and other 3 commits from this PR workarounding Janino's issues could be
dropped then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-29912) jdbc scan.partition.column can specify any type of field
[ https://issues.apache.org/jira/browse/FLINK-29912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] waywtdcc updated FLINK-29912: - Description: scan.partition. column can specify any type of field. At present, scan.partition. column must be a numeric, date, or timestamp column from the table in question. You can specify any type of field, such as string type, which can satisfy all high concurrent read scenarios (was: scan.partition.column can specify any type of field. At present, scan.partition.column must be a numeric, date, or timestamp column from the table in question. You can specify any type of field, which can satisfy all high concurrent read scenarios.) > jdbc scan.partition.column can specify any type of field > > > Key: FLINK-29912 > URL: https://issues.apache.org/jira/browse/FLINK-29912 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC >Affects Versions: 1.16.0 >Reporter: waywtdcc >Priority: Major > Fix For: 1.17.0 > > > scan.partition. column can specify any type of field. At present, > scan.partition. column must be a numeric, date, or timestamp column from the > table in question. You can specify any type of field, such as string type, > which can satisfy all high concurrent read scenarios -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] snuyanzin commented on a diff in pull request #21203: [FLINK-21239][Table SQL/API] Upgrade Calcite to 1.28.0
snuyanzin commented on code in PR #21203: URL: https://github.com/apache/flink/pull/21203#discussion_r1016031451 ## flink-table/flink-table-planner/pom.xml: ## @@ -49,6 +49,18 @@ under the License. + Review Comment: so I guess no relocation and mentioning in NOTICE is required. May be a clarification comment in pom -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
