[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 810e4e349946878d5727b61b5c76da77a2310ea2 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29684) * 76134252d18e14772c18739278af72c9b519e353 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29687) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18137: [FLINK-25287][connector-testing-framework] Refactor interfaces of connector testing framework to support more scenarios
flinkbot edited a comment on pull request #18137: URL: https://github.com/apache/flink/pull/18137#issuecomment-996615034 ## CI report: * bee4ba9ac6648dbdb56ed806ef8643b6c84aeb68 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29675) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
flinkbot edited a comment on pull request #17601: URL: https://github.com/apache/flink/pull/17601#issuecomment-954546978 ## CI report: * a65b12ce87fe15a5474b0a740e9a054e7685941d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29612) * 124c7893234b31fc1a3ce5b3b29aff32e3b7847d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29683) * 7667bd94a9d0af29b443a1b07f6e7077f5b9cce6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29691) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18391: [FLINK-25478][chaneglog] Correct the state register logic of ChangelogStateBackendHandle
Myasuka commented on a change in pull request #18391:
URL: https://github.com/apache/flink/pull/18391#discussion_r787422412
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/ChangelogStateBackendHandle.java
##
@@ -66,16 +71,36 @@ public ChangelogStateBackendHandleImpl(
@Override
public void registerSharedStates(SharedStateRegistry stateRegistry,
long checkpointID) {
+for (KeyedStateHandle keyedStateHandle : materialized) {
+registerState(stateRegistry, checkpointID, MATERIALIZED_FLAG,
keyedStateHandle);
+}
+for (ChangelogStateHandle stateHandle : nonMaterialized) {
+registerState(stateRegistry, checkpointID,
NON_MATERIALIZED_FLAG, stateHandle);
+}
stateRegistry.registerAll(materialized, checkpointID);
stateRegistry.registerAll(nonMaterialized, checkpointID);
}
+private void registerState(
+SharedStateRegistry stateRegistry,
+long checkpointID,
+String prefix,
+KeyedStateHandle keyedStateHandle) {
+stateRegistry.registerReference(
+new SharedStateRegistryKey(
+prefix,
+// here use hash code as registry key identifier.
+new
StateHandleID(String.valueOf(keyedStateHandle.hashCode(,
+new StreamStateHandleWrapper(keyedStateHandle),
+checkpointID);
Review comment:
Thanks for your advice. I think SequenceNumber is better for
materialized/non-materialized part. However, these numbers do not persist and
restart from `INITIAL_SQN` once restored. Do you think we could persist these
numbers and reset the number from last restored?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] LadyForest edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
LadyForest edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1016159041 > I just feel, the testing classes are too complicate, can we simplify them? We only need to test what we need, there is no need to have very strong features. I'm afraid this version has been simplified to the most extent. After all, we need to invoke a job running (whatever job) for the ITCase right? Or to be more reasonable, let a meaningful job (which presents a "compaction") run. Because the helper test factory `TestManagedTableFactory` is located in the `table-common` module (which causes us not to depend on any existing source/sink impl, o.w. we will get a cyclic dependency), we cannot leverage the ready-made source/sink impl to let even a word count job run. So everything needs to be built from scratch. What if we move `TestManagedTableFactory` from the module `table-common` to `table-planner`? That is great to some extent, all the pom issues and "too complicated" test helpers are resolved, but `FactoryUtilTest#testManagedConnector` will fail. And I don't think it's a good idea to maintain two `TestManagedTableFactory` both under `table-common` and `table-planner` modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
flinkbot edited a comment on pull request #17601: URL: https://github.com/apache/flink/pull/17601#issuecomment-954546978 ## CI report: * a65b12ce87fe15a5474b0a740e9a054e7685941d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29612) * 124c7893234b31fc1a3ce5b3b29aff32e3b7847d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29683) * 7667bd94a9d0af29b443a1b07f6e7077f5b9cce6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25650) Document unaligned checkpoints performance limitations (larger records/flat map/timers/...)
[ https://issues.apache.org/jira/browse/FLINK-25650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski closed FLINK-25650. -- Fix Version/s: 1.15.0 1.14.4 Resolution: Fixed merged commit 09cd1ff into apache:master merged as 43b073e8571 into release-1.14 > Document unaligned checkpoints performance limitations (larger records/flat > map/timers/...) > --- > > Key: FLINK-25650 > URL: https://issues.apache.org/jira/browse/FLINK-25650 > Project: Flink > Issue Type: Improvement > Components: Documentation, Runtime / Checkpointing >Affects Versions: 1.14.0 >Reporter: Anton Kalashnikov >Assignee: Anton Kalashnikov >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.14.4 > > > The unaligned checkpoint can be delayed if the current record is consumed too > long(because it is too large or it is the flat map etc.). Which can be pretty > confused. So it makes sense to document this limitation to give the user > understanding of this situation. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] LadyForest commented on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
LadyForest commented on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1016159041 > I just feel, the testing classes are too complicate, can we simplify them? We only need to test what we need, there is no need to have very strong features. I'm afraid this version has been simplified to the most extent. After all, we need to invoke a job running (whatever job) for the ITCase right? Because the helper test factory `TestManagedFactory` is located in the `table-common` module (which causes us not to depend on any existing source/sink impl, o.w. we will get a cyclic dependency), we cannot leverage the ready-made source/sink impl to let even a word count job run. So everything needs to be built from scratch. What if we move `TestManagedFactory` from the module `table-common` to `table-planner`? That is great to some extent, all the pom issues and "too complicated" test helpers are resolved, but `FactoryUtilTest#testManagedConnector` will fail. And I don't think it's a good idea to maintain two `TestManagedFactory` both under `table-common` and `table-planner` modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski merged pull request #18354: [FLINK-25650][docs] Added "Interplay with long-running record process…
pnowojski merged pull request #18354: URL: https://github.com/apache/flink/pull/18354 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25194) Implement an API for duplicating artefacts
[ https://issues.apache.org/jira/browse/FLINK-25194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz reassigned FLINK-25194: Assignee: Dawid Wysakowicz (was: Piotr Nowojski) > Implement an API for duplicating artefacts > -- > > Key: FLINK-25194 > URL: https://issues.apache.org/jira/browse/FLINK-25194 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem, Runtime / Checkpointing >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > We should implement methods that let us duplicate artefacts in a DFS. We can > later on use it for cheaply duplicating shared snapshots artefacts instead of > reuploading them. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-25194) Implement an API for duplicating artefacts
[ https://issues.apache.org/jira/browse/FLINK-25194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz closed FLINK-25194. Resolution: Implemented Implemented in fc1d63579ae5bcd9db207e1a5cd1b6365a87e871 > Implement an API for duplicating artefacts > -- > > Key: FLINK-25194 > URL: https://issues.apache.org/jira/browse/FLINK-25194 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem, Runtime / Checkpointing >Reporter: Dawid Wysakowicz >Assignee: Piotr Nowojski >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > We should implement methods that let us duplicate artefacts in a DFS. We can > later on use it for cheaply duplicating shared snapshots artefacts instead of > reuploading them. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] dawidwys merged pull request #18341: [FLINK-25194] Implement an API for duplicating artefacts
dawidwys merged pull request #18341: URL: https://github.com/apache/flink/pull/18341 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) * 455dfd8ad2146f35d31a602715a9b27a07aa2aba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29678) * cc5eaf1dd5be0414a77c59f619e407c01d083e0c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29685) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24163) PartiallyFinishedSourcesITCase fails due to timeout
[
https://issues.apache.org/jira/browse/FLINK-24163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17478366#comment-17478366
]
Yun Gao edited comment on FLINK-24163 at 1/19/22, 7:18 AM:
---
This seems to be due to different reason.
Hi [~roman] , by binary search it seems with
https://issues.apache.org/jira/browse/FLINK-25395 the running time of
PartiallyFinishedSourcesITCase#test[complex graph SINGLE_SUBTASK, failover:
true, strategy: region] has increased from 2s to about 1 minute, the case is
blocked on restoring state after failover:
{code:java}
"transform-2-keyed (1/4)#1" #1517 prio=5 os_prio=31 tid=0x7f862136a000
nid=0x10423 runnable [0x700011fee000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at
org.apache.flink.core.fs.local.LocalDataInputStream.read(LocalDataInputStream.java:73)
at
org.apache.flink.core.fs.FSDataInputStreamWrapper.read(FSDataInputStreamWrapper.java:60)
at
org.apache.flink.runtime.util.ForwardingInputStream.read(ForwardingInputStream.java:52)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.flink.api.java.typeutils.runtime.DataInputViewStream.read(DataInputViewStream.java:68)
at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:77)
at com.esotericsoftware.kryo.io.Input.readAscii_slow(Input.java:598)
at com.esotericsoftware.kryo.io.Input.readAscii(Input.java:576)
at com.esotericsoftware.kryo.io.Input.readString(Input.java:454)
at
com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:177)
at
com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:166)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:402)
at
org.apache.flink.runtime.state.heap.StateTableByKeyGroupReaders.lambda$createV2PlusReader$0(StateTableByKeyGroupReaders.java:78)
at
org.apache.flink.runtime.state.heap.StateTableByKeyGroupReaders$$Lambda$2196/1169355256.readElement(Unknown
Source)
at
org.apache.flink.runtime.state.KeyGroupPartitioner$PartitioningResultKeyGroupReader.readMappingsInKeyGroup(KeyGroupPartitioner.java:297)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readKeyGroupStateData(HeapRestoreOperation.java:258)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readStateHandleStateData(HeapRestoreOperation.java:220)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.restore(HeapRestoreOperation.java:166)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.restore(HeapRestoreOperation.java:62)
at
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.restoreState(HeapKeyedStateBackendBuilder.java:169)
at
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:106)
at
org.apache.flink.runtime.state.hashmap.HashMapStateBackend.createKeyedStateBackend(HashMapStateBackend.java:137)
{code}
Could you help to have a look~?
The commit before the PR on the master branch is
265a0a0708ae743c63505bb02e0659984a565fbb and the commit right after the PR is
4691b66545010ed812624a259869c7a522663720 .
was (Author: gaoyunhaii):
This seems to be due to different reason.
Hi [~roman] , by binary search it seems with
https://issues.apache.org/jira/browse/FLINK-25395 the running time of
PartiallyFinishedSourcesITCase#test[complex graph SINGLE_SUBTASK, failover:
true, strategy: region] has increased from 2s to about 1 minute, the case is
blocked on restoring state after failover:
{code:java}
"transform-2-keyed (1/4)#1" #1517 prio=5 os_prio=31 tid=0x7f862136a000
nid=0x10423 runnable [0x700011fee000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at
org.apache.flink.core.fs.local.LocalDataInputStream.read(LocalDataInputStream.java:73)
at
org.apache.flink.core.fs.FSDataInputStreamWrapper.read(FSDataInputStreamWrapper.java:60)
at
org.apache.flink.runtime.util.ForwardingInputStream.read(ForwardingInputStream.java:52)
at
[jira] [Commented] (FLINK-24163) PartiallyFinishedSourcesITCase fails due to timeout
[
https://issues.apache.org/jira/browse/FLINK-24163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17478366#comment-17478366
]
Yun Gao commented on FLINK-24163:
-
This seems to be due to different reason.
Hi [~roman] , by binary search it seems with
https://issues.apache.org/jira/browse/FLINK-25395 the running time of
PartiallyFinishedSourcesITCase#test[complex graph SINGLE_SUBTASK, failover:
true, strategy: region] has increased from 2s to about 1 minute, the case is
blocked on restoring state after failover:
{code:java}
"transform-2-keyed (1/4)#1" #1517 prio=5 os_prio=31 tid=0x7f862136a000
nid=0x10423 runnable [0x700011fee000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at
org.apache.flink.core.fs.local.LocalDataInputStream.read(LocalDataInputStream.java:73)
at
org.apache.flink.core.fs.FSDataInputStreamWrapper.read(FSDataInputStreamWrapper.java:60)
at
org.apache.flink.runtime.util.ForwardingInputStream.read(ForwardingInputStream.java:52)
at java.io.DataInputStream.read(DataInputStream.java:149)
at
org.apache.flink.api.java.typeutils.runtime.DataInputViewStream.read(DataInputViewStream.java:68)
at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
at
org.apache.flink.api.java.typeutils.runtime.NoFetchingInput.require(NoFetchingInput.java:77)
at com.esotericsoftware.kryo.io.Input.readAscii_slow(Input.java:598)
at com.esotericsoftware.kryo.io.Input.readAscii(Input.java:576)
at com.esotericsoftware.kryo.io.Input.readString(Input.java:454)
at
com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:177)
at
com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:166)
at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
at
com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
at
com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
at
org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:402)
at
org.apache.flink.runtime.state.heap.StateTableByKeyGroupReaders.lambda$createV2PlusReader$0(StateTableByKeyGroupReaders.java:78)
at
org.apache.flink.runtime.state.heap.StateTableByKeyGroupReaders$$Lambda$2196/1169355256.readElement(Unknown
Source)
at
org.apache.flink.runtime.state.KeyGroupPartitioner$PartitioningResultKeyGroupReader.readMappingsInKeyGroup(KeyGroupPartitioner.java:297)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readKeyGroupStateData(HeapRestoreOperation.java:258)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.readStateHandleStateData(HeapRestoreOperation.java:220)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.restore(HeapRestoreOperation.java:166)
at
org.apache.flink.runtime.state.heap.HeapRestoreOperation.restore(HeapRestoreOperation.java:62)
at
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.restoreState(HeapKeyedStateBackendBuilder.java:169)
at
org.apache.flink.runtime.state.heap.HeapKeyedStateBackendBuilder.build(HeapKeyedStateBackendBuilder.java:106)
at
org.apache.flink.runtime.state.hashmap.HashMapStateBackend.createKeyedStateBackend(HashMapStateBackend.java:137)
{code}
Could you help to have a look~?
> PartiallyFinishedSourcesITCase fails due to timeout
> ---
>
> Key: FLINK-24163
> URL: https://issues.apache.org/jira/browse/FLINK-24163
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.14.0, 1.15.0
>Reporter: Xintong Song
>Assignee: Yun Gao
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23529=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798=10996
> {code}
> Sep 04 04:35:28 [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 155.236 s <<< FAILURE! - in
> org.apache.flink.runtime.operators.lifecycle.PartiallyFinishedSourcesITCase
> Sep 04 04:35:28 [ERROR] test[complex graph ALL_SUBTASKS, failover: false]
> Time elapsed: 65.999 s <<< ERROR!
> Sep 04 04:35:28 java.util.concurrent.TimeoutException: Condition was not met
> in given timeout.
> Sep 04 04:35:28 at
>
[GitHub] [flink] flinkbot edited a comment on pull request #18365: [FLINK-22643][network] Reuse tpc connections between taskmanagers
flinkbot edited a comment on pull request #18365: URL: https://github.com/apache/flink/pull/18365#issuecomment-1013081147 ## CI report: * 22c8ded41179db1141351bb78cebcd77e8d35b2c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29671) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 2dbb1737df0af1db911cf2b2a449016395991bd1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29626) * 810e4e349946878d5727b61b5c76da77a2310ea2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29684) * 76134252d18e14772c18739278af72c9b519e353 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29687) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18303: [FLINK-25085][runtime] Add a scheduled thread pool for periodic tasks in RpcEndpoint
flinkbot edited a comment on pull request #18303: URL: https://github.com/apache/flink/pull/18303#issuecomment-1008005239 ## CI report: * 6024e28f7d315a7d1157f9af33da9b815f81a27a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29646) * ac345a207c5ac67b7dcb8ed34ad939a701c1c834 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29686) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LadyForest commented on a change in pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
LadyForest commented on a change in pull request #18394:
URL: https://github.com/apache/flink/pull/18394#discussion_r787405220
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/ddl/AlterTableCompactOperation.java
##
@@ -18,41 +18,32 @@
package org.apache.flink.table.operations.ddl;
-import org.apache.flink.table.catalog.CatalogPartitionSpec;
import org.apache.flink.table.catalog.ObjectIdentifier;
-import org.apache.flink.table.operations.OperationUtils;
+import org.apache.flink.table.catalog.ResolvedCatalogTable;
+import org.apache.flink.table.operations.CatalogQueryOperation;
-import javax.annotation.Nullable;
-
-import java.util.Collections;
-import java.util.LinkedHashMap;
import java.util.Map;
/** Operation to describe "ALTER TABLE [PARTITION partition_spec] COMPACT"
statement. */
-public class AlterTableCompactOperation extends AlterTableOperation {
+public class AlterTableCompactOperation extends CatalogQueryOperation {
-private final CatalogPartitionSpec partitionSpec;
+private final ResolvedCatalogTable resolvedManagedTable;
+private final Map compactOptions;
public AlterTableCompactOperation(
Review comment:
If so, how can we tell the normal `CatalogQueryOperation` from `ALTER
TABLE COMPACT` during the rel conversion in`QueryOperationConverter` (e.g.
check `isBatch` condition and inject dynamic options)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhuzhengjun01 edited a comment on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
zhuzhengjun01 edited a comment on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-1016148079 @maosuhan hi, flink-protobuf works very well in my job, but i found some corner case that it was not supported `oneof` in proto3. so this will be dealt in future or not ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhengjun01 commented on pull request #14376: [FLINK-18202][PB format] New Format of protobuf
zhuzhengjun01 commented on pull request #14376: URL: https://github.com/apache/flink/pull/14376#issuecomment-1016148079 @maosuhan hi, flink-protobuf works very well in my job, but i fount some corner case that it was not supported `oneof` in proto3. so this will be dealt in future or not ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) * 0fb5e7a2b2f583a36775e1d7347080e834e0d806 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29679) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 2dbb1737df0af1db911cf2b2a449016395991bd1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29626) * 810e4e349946878d5727b61b5c76da77a2310ea2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29684) * 76134252d18e14772c18739278af72c9b519e353 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18303: [FLINK-25085][runtime] Add a scheduled thread pool for periodic tasks in RpcEndpoint
flinkbot edited a comment on pull request #18303: URL: https://github.com/apache/flink/pull/18303#issuecomment-1008005239 ## CI report: * 6024e28f7d315a7d1157f9af33da9b815f81a27a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29646) * ac345a207c5ac67b7dcb8ed34ad939a701c1c834 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * b8a9d9704d289aa32e8d040b7090a6fb97c4907c Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29673) * 99653eb7e622b96ba0bb47836fe3eefe7b71d942 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29677) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25649) Scheduling jobs fails with org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException
[
https://issues.apache.org/jira/browse/FLINK-25649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17478365#comment-17478365
]
Zhu Zhu commented on FLINK-25649:
-
>> Would it make sens to say that we encounter this issue because 2 jobs are
>> waiting for the same slots to be available when slots are available in the
>> cluster?
Probably not. This is not the designed behavior of slot allocation of Flink.
One question I have is that when you say "100% of the jobs working", do you
mean "All tasks of all jobs are in RUNNING state at the same time", or "All
bounded jobs can finish and all unbounded jobs' tasks are in RUNING state"?
And I second Till that the debug logs is very important to find the root cause.
It would be very helpful if you can share it.
> Scheduling jobs fails with
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException
> -
>
> Key: FLINK-25649
> URL: https://issues.apache.org/jira/browse/FLINK-25649
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.13.1
>Reporter: Gil De Grove
>Priority: Major
>
> Following comment from Till on this [SO
> question|https://stackoverflow.com/questions/70683048/scheduling-jobs-fails-with-org-apache-flink-runtime-jobmanager-scheduler-noresou?noredirect=1#comment124980546_70683048]
> h2. *Summary*
> We are currently experiencing a scheduling issue with our flink cluster.
> The symptoms are that some/most/all (it depend, the symptoms are not always
> the same) of our tasks are showed as _SCHEDULED_ but fail after a timeout.
> The jobs are them showed a _RUNNING_
> The failing exception is the following one:
> {{Caused by: java.util.concurrent.CompletionException:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Slot request bulk is not fulfillable! Could not allocate the required slot
> within slot request timeout}}
> After analysis, we assume (we cannot prove it, as there are not that much
> logs for that part of the code) that the failure is due to a deadlock/race
> condition that is happening when several jobs are being submitted at the same
> time to the flink cluster, even though we have enough slots available in the
> cluster.
> We actually have the error with 52 available task slots, and have 12 jobs
> that are not scheduled.
> h2. Additional information
> * Flink version: 1.13.1 commit a7f3192
> * Flink cluster in session mode
> * 2 Job managers using k8s HA mode (resource requests: 2 CPU, 4Gb Ram,
> limits sets on memory to 4Gb)
> * 50 task managers with 2 slots each (resource requests: 2 CPUs, 2GB Ram. No
> limits set).
> * Our Flink cluster is shut down every night, and restarted every morning.
> The error seems to occur when a lot of jobs needs to be scheduled. The jobs
> are configured to restore their state, and we do not see any issues for jobs
> that are being scheduled and run correctly, it seems to really be related to
> a scheduling issue.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] LadyForest commented on a change in pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
LadyForest commented on a change in pull request #18394: URL: https://github.com/apache/flink/pull/18394#discussion_r787401902 ## File path: flink-formats/flink-orc/pom.xml ## @@ -145,6 +145,19 @@ under the License. test test-jar + Review comment: Because `TestManagedFactory` is located in the `table-common` module but tested under the `table-planner` module. To achieve this, `TestManagedFactory` is added to the `org.apache.flink.table.factories.Factory` file under the `table-planner` module, and this leads to all of these pom file changes. O.w. SPI for table discovery will fail, the stack trace looks like ``` org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath at org.apache.flink.table.api.bridge.internal.AbstractStreamTableEnvironmentImpl.lookupExecutor(AbstractStreamTableEnvironmentImpl.java:109) at org.apache.flink.table.api.bridge.scala.internal.StreamTableEnvironmentImpl$.create(StreamTableEnvironmentImpl.scala:320) at org.apache.flink.table.api.bridge.scala.StreamTableEnvironment$.create(StreamTableEnvironment.scala:925) at org.apache.flink.table.planner.runtime.utils.StreamingTestBase.before(StreamingTestBase.scala:54) at org.apache.flink.table.planner.runtime.stream.FsStreamingSinkITCaseBase.before(FsStreamingSinkITCaseBase.scala:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.flink.table.api.TableException: Unexpected error when trying to load service provider for factories. at org.apache.flink.table.factories.FactoryUtil.lambda$discoverFactories$19(FactoryUtil.java:813) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:799) at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:517) at org.apache.flink.table.api.bridge.internal.AbstractStreamTableEnvironmentImpl.lookupExecutor(AbstractStreamTableEnvironmentImpl.java:106) ... 18 more Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.Factory: Provider org.apache.flink.table.factories.TestManagedTableFactory not found at java.util.ServiceLoader.fail(ServiceLoader.java:239) at java.util.ServiceLoader.access$300(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.flink.table.factories.ServiceLoaderUtil.load(ServiceLoaderUtil.java:42) at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:798) ... 20 more ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) * 455dfd8ad2146f35d31a602715a9b27a07aa2aba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29678) * cc5eaf1dd5be0414a77c59f619e407c01d083e0c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29685) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
JingsongLi commented on a change in pull request #18394:
URL: https://github.com/apache/flink/pull/18394#discussion_r787389637
##
File path:
flink-table/flink-table-common/src/test/java/org/apache/flink/table/connector/source/TestManagedTableSink.java
##
@@ -0,0 +1,468 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.connector.source;
+
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.connector.sink.Committer;
+import org.apache.flink.api.connector.sink.GlobalCommitter;
+import org.apache.flink.api.connector.sink.Sink;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.io.SimpleVersionedSerializer;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.core.memory.DataOutputSerializer;
+import org.apache.flink.table.catalog.CatalogPartitionSpec;
+import org.apache.flink.table.catalog.ObjectIdentifier;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.sink.SinkProvider;
+import org.apache.flink.table.connector.sink.abilities.SupportsOverwrite;
+import org.apache.flink.table.connector.sink.abilities.SupportsPartitioning;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.TestManagedTableFactory;
+import org.apache.flink.table.utils.PartitionPathUtils;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicReference;
+
+/** Managed {@link DynamicTableSink} for testing. */
+public class TestManagedTableSink
+implements DynamicTableSink, SupportsOverwrite, SupportsPartitioning {
+
+private final DynamicTableFactory.Context context;
+private final Path basePath;
+
+private LinkedHashMap staticPartitionSpecs = new
LinkedHashMap<>();
+private boolean overwrite = false;
+
+public TestManagedTableSink(DynamicTableFactory.Context context, Path
basePath) {
+this.context = context;
+this.basePath = basePath;
+}
+
+@Override
+public ChangelogMode getChangelogMode(ChangelogMode requestedMode) {
+return ChangelogMode.insertOnly();
+}
+
+@Override
+public SinkRuntimeProvider getSinkRuntimeProvider(Context context) {
+return SinkProvider.of(new
TestManagedSink(this.context.getObjectIdentifier(), basePath));
+}
+
+@Override
+public DynamicTableSink copy() {
+TestManagedTableSink copied = new TestManagedTableSink(context,
basePath);
+copied.overwrite = this.overwrite;
+copied.staticPartitionSpecs = this.staticPartitionSpecs;
+return copied;
+}
+
+@Override
+public String asSummaryString() {
+return "TestManagedSink";
+}
+
+@Override
+public void applyOverwrite(boolean overwrite) {
+this.overwrite = overwrite;
+}
+
+@Override
+public void applyStaticPartition(Map partition) {
+List partitionKeys =
context.getCatalogTable().getPartitionKeys();
+for (String partitionKey : partitionKeys) {
+if (partition.containsKey(partitionKey)) {
+staticPartitionSpecs.put(partitionKey,
partition.get(partitionKey));
+}
+}
+}
+
+/** Managed {@link Sink} for testing compaction. */
+public static class TestManagedSink
+implements Sink {
+
+private final ObjectIdentifier tableIdentifier;
+private
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) * 455dfd8ad2146f35d31a602715a9b27a07aa2aba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29678) * cc5eaf1dd5be0414a77c59f619e407c01d083e0c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] tsreaper commented on a change in pull request #10: [FLINK-25680] Introduce Table Store Flink Sink
tsreaper commented on a change in pull request #10:
URL: https://github.com/apache/flink-table-store/pull/10#discussion_r787383851
##
File path:
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/sink/SinkRecord.java
##
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.connector.sink;
+
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.types.RowKind;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+
+/** A sink records contains key, value and partition, bucket, row kind
information. */
+public class SinkRecord {
+
+private final BinaryRowData partition;
+
+private final int bucket;
+
+private final RowKind rowKind;
+
+private final BinaryRowData key;
+
+private final RowData row;
+
+public SinkRecord(
+BinaryRowData partition, int bucket, RowKind rowKind,
BinaryRowData key, RowData row) {
+checkArgument(partition.getRowKind() == RowKind.INSERT);
+checkArgument(key.getRowKind() == RowKind.INSERT);
+checkArgument(row.getRowKind() == RowKind.INSERT);
Review comment:
Why do we need this check? We have no concern on the row kind of each
`BinaryRowData`. We only need the `rowKind` given in the argument.
##
File path:
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/sink/StoreSinkWriter.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.connector.sink;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.connector.sink.SinkWriter;
+import org.apache.flink.table.data.GenericRowData;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.data.binary.BinaryRowData;
+import org.apache.flink.table.store.file.ValueKind;
+import org.apache.flink.table.store.file.operation.FileStoreWrite;
+import org.apache.flink.table.store.file.utils.RecordWriter;
+import org.apache.flink.types.RowKind;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+
+/** A {@link SinkWriter} for dynamic store. */
+public class StoreSinkWriter implements SinkWriter {
+
+private final FileStoreWrite fileStoreWrite;
+
+private final SinkRecordConverter recordConverter;
+
+private final boolean overwrite;
+
+private final ExecutorService compactExecutor;
+
+private final Map> writers;
+
+public StoreSinkWriter(
+FileStoreWrite fileStoreWrite, SinkRecordConverter
recordConverter, boolean overwrite) {
+this.fileStoreWrite = fileStoreWrite;
+this.recordConverter = recordConverter;
+this.overwrite = overwrite;
+this.compactExecutor = Executors.newSingleThreadScheduledExecutor();
+this.writers = new HashMap<>();
+}
+
+private RecordWriter getWriter(BinaryRowData partition, int bucket) {
+Map buckets = writers.get(partition);
+if (buckets == null) {
+buckets = new HashMap<>();
+writers.put(partition.copy(), buckets);
+}
+return buckets.computeIfAbsent(
+bucket,
+k
[GitHub] [flink] flinkbot edited a comment on pull request #18400: [FLINK-25198][docs] Add doc about name and description of operator
flinkbot edited a comment on pull request #18400: URL: https://github.com/apache/flink/pull/18400#issuecomment-1016057839 ## CI report: * 0ee72a0732026c046d991d959033995c7baf44e1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29680) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25683) wrong result if table transfrom to DataStream then window process in batch mode
[ https://issues.apache.org/jira/browse/FLINK-25683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17478355#comment-17478355 ] Yao Zhang commented on FLINK-25683: --- Hi [~twalthr] , This issue and FLINK-25471 is quite similar. The root cause is BatchExecutionKeyedStateBackend calls notifyKeySelected only if it receives the data with different key. As a result, last elements(right before Task Manager exits) stored in BatchExecutionKeyedStateBackend the will never have a chance to be collected by the downstream. I think it is the root cause. I plan to change AbstractStreamOperator by extending BoundedMultiInput and set the key to a designed value when the input reaches END_OF_DATA. By doing this, in batch mode it will trigger notifyKeySelected and finally all elements will be collected. It might not have any side effect in streaming mode. Both FLINK-25471 and this ticket can be solved by this change. Correct me if I am wrong. Could you please assign this ticket to me? > wrong result if table transfrom to DataStream then window process in batch > mode > --- > > Key: FLINK-25683 > URL: https://issues.apache.org/jira/browse/FLINK-25683 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Runtime >Affects Versions: 1.14.2 > Environment: mac book pro m1 > jdk 8 > scala 2.11 > flink 1.14.2 > idea 2020 >Reporter: zhangzh >Priority: Major > Attachments: TableToDataStreamBatchWindowTest.scala, pom.xml > > > I have 5 line datas, > i first need to transform current data with SQL > then mix current data and historical data which is batch get from hbase > for some special reason the program must run in batch mode > i think the correct result should be like this： > (BOB,1) > (EMA,1) > (DOUG,1) > (ALICE,1) > (CENDI,1) > but the result is : > (EMA,1) > > if i set different parallelism ,the result is different. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] JingsongLi commented on a change in pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
JingsongLi commented on a change in pull request #18394:
URL: https://github.com/apache/flink/pull/18394#discussion_r787387670
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/ManagedTableFactory.java
##
@@ -50,6 +51,13 @@ default String factoryIdentifier() {
/** Notifies the listener that a table drop occurred. */
void onDropTable(Context context, boolean ignoreIfNotExists);
+/**
+ * Notifies the listener that a table compaction occurred.
+ *
+ * @return dynamic options of the file entries under compaction for this
table.
Review comment:
dynamic options for source and sink? there is no file concept in
flink-sql
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/calcite/FlinkRelBuilder.scala
##
@@ -186,6 +195,44 @@ class FlinkRelBuilder(
push(relNode)
this
}
+
+ def compactScan(
+ identifier: ObjectIdentifier,
+ catalogTable: ResolvedCatalogTable,
+ isTemporary: Boolean,
+ compactOptions: util.Map[String, String]): RelBuilder = {
+val flinkContext = context.unwrap(classOf[FlinkContext])
+val config = flinkContext.getTableConfig.getConfiguration
+
+val hints = new util.ArrayList[RelHint]
Review comment:
Can you just use `CatalogSourceTable` from `FlinkCalciteCatalogReader`?
(You can refer to `RelBuilder`) Then you can create a `ToRelContext` for it.
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/ddl/AlterTableCompactOperation.java
##
@@ -18,41 +18,32 @@
package org.apache.flink.table.operations.ddl;
-import org.apache.flink.table.catalog.CatalogPartitionSpec;
import org.apache.flink.table.catalog.ObjectIdentifier;
-import org.apache.flink.table.operations.OperationUtils;
+import org.apache.flink.table.catalog.ResolvedCatalogTable;
+import org.apache.flink.table.operations.CatalogQueryOperation;
-import javax.annotation.Nullable;
-
-import java.util.Collections;
-import java.util.LinkedHashMap;
import java.util.Map;
/** Operation to describe "ALTER TABLE [PARTITION partition_spec] COMPACT"
statement. */
-public class AlterTableCompactOperation extends AlterTableOperation {
+public class AlterTableCompactOperation extends CatalogQueryOperation {
-private final CatalogPartitionSpec partitionSpec;
+private final ResolvedCatalogTable resolvedManagedTable;
+private final Map compactOptions;
public AlterTableCompactOperation(
Review comment:
Maybe we can drop this class, just add `dynamicOptions` to
`CatalogQueryOperation`.
##
File path: flink-formats/flink-orc/pom.xml
##
@@ -145,6 +145,19 @@ under the License.
test
test-jar
+
Review comment:
Why modify orc parquet python sql-client?
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/calcite/FlinkRelBuilder.scala
##
@@ -186,6 +195,44 @@ class FlinkRelBuilder(
push(relNode)
this
}
+
+ def compactScan(
Review comment:
Maybe just `scan(ObjectIdentifier objectIdentifier, Map
dynamicOptions)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-25697) Support HTTPS for Prometheus Push Gateway
[ https://issues.apache.org/jira/browse/FLINK-25697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guowei Ma reassigned FLINK-25697: - Assignee: Mason Chen > Support HTTPS for Prometheus Push Gateway > - > > Key: FLINK-25697 > URL: https://issues.apache.org/jira/browse/FLINK-25697 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Affects Versions: 1.13.5, 1.14.3 >Reporter: Mason Chen >Assignee: Mason Chen >Priority: Major > > Currently prometheus push gateway only supports http endpoints. I am > proposing a configuration similar to the influxdb reporter > (https://issues.apache.org/jira/browse/FLINK-12336) that would allow users to > configure a https endpoint. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] jelly-1203 commented on pull request #18017: [FLINK-25171] Validation of duplicate fields in ddl sql
jelly-1203 commented on pull request #18017: URL: https://github.com/apache/flink/pull/18017#issuecomment-1016122102 anyone? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25614) Let LocalWindowAggregate be chained with upstream
[
https://issues.apache.org/jira/browse/FLINK-25614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17478347#comment-17478347
]
Q Kang commented on FLINK-25614:
[~jingzhang] Sorry for the late response. [~wenlong.lwl] had given the exact
explanation.
There seems to be more impacts than I originally expected, so I agree with you
that we move it to new version.
> Let LocalWindowAggregate be chained with upstream
> -
>
> Key: FLINK-25614
> URL: https://issues.apache.org/jira/browse/FLINK-25614
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.14.2
>Reporter: Q Kang
>Assignee: Q Kang
>Priority: Minor
> Labels: pull-request-available
>
> When enabling two-phase aggregation (local-global) strategy for Window TVF,
> the physical plan is shown as follows:
> {code:java}
> TableSourceScan -> Calc -> WatermarkAssigner -> Calc
> ||
> || [FORWARD]
> ||
> LocalWindowAggregate
> ||
> || [HASH]
> ||
> GlobalWindowAggregate
> ||
> ||
> ...{code}
> We can let the `LocalWindowAggregate` node be chained with upstream operators
> in order to improve efficiency, just like the non-windowing counterpart
> `LocalGroupAggregate`.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] KarmaGYZ commented on pull request #18360: [FLINK-25329][runtime] Support memory execution graph store in session cluster
KarmaGYZ commented on pull request #18360: URL: https://github.com/apache/flink/pull/18360#issuecomment-1016116941 Thanks for the PR, @zjureel . I think it is a valid improvement. Regarding the implementation, I think we can simply introduce a config option "jobstore.flush-to-disk" to `FileExecutionGraphInfoStore` instead of introducing a new one, for deduplication. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25697) Support HTTPS for Prometheus Push Gateway
Mason Chen created FLINK-25697: -- Summary: Support HTTPS for Prometheus Push Gateway Key: FLINK-25697 URL: https://issues.apache.org/jira/browse/FLINK-25697 Project: Flink Issue Type: Improvement Components: Runtime / Metrics Affects Versions: 1.14.3, 1.13.5 Reporter: Mason Chen Currently prometheus push gateway only supports http endpoints. I am proposing a configuration similar to the influxdb reporter (https://issues.apache.org/jira/browse/FLINK-12336) that would allow users to configure a https endpoint. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 2dbb1737df0af1db911cf2b2a449016395991bd1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29626) * 810e4e349946878d5727b61b5c76da77a2310ea2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29684) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
flinkbot edited a comment on pull request #17601: URL: https://github.com/apache/flink/pull/17601#issuecomment-954546978 ## CI report: * a65b12ce87fe15a5474b0a740e9a054e7685941d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29612) * 124c7893234b31fc1a3ce5b3b29aff32e3b7847d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29683) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18388: [FLINK-25530][python][connector/pulsar] Support Pulsar source connector in Python DataStream API.
flinkbot edited a comment on pull request #18388: URL: https://github.com/apache/flink/pull/18388#issuecomment-1015205179 ## CI report: * 2dbb1737df0af1db911cf2b2a449016395991bd1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29626) * 810e4e349946878d5727b61b5c76da77a2310ea2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
flinkbot edited a comment on pull request #17601: URL: https://github.com/apache/flink/pull/17601#issuecomment-954546978 ## CI report: * a65b12ce87fe15a5474b0a740e9a054e7685941d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29612) * 124c7893234b31fc1a3ce5b3b29aff32e3b7847d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a change in pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
ruanhang1993 commented on a change in pull request #17601:
URL: https://github.com/apache/flink/pull/17601#discussion_r787369372
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/KafkaTableTestUtils.java
##
@@ -74,23 +77,40 @@
}
public static void waitingExpectedResults(
-String sinkName, List expected, Duration timeout) throws
InterruptedException {
-long now = System.currentTimeMillis();
-long stop = now + timeout.toMillis();
+String sinkName, List expected, Duration timeout)
+throws InterruptedException, TimeoutException {
Collections.sort(expected);
-while (System.currentTimeMillis() < stop) {
-List actual = TestValuesTableFactory.getResults(sinkName);
-Collections.sort(actual);
-if (expected.equals(actual)) {
-return;
-}
-Thread.sleep(100);
-}
+CommonTestUtils.waitUtil(
+() -> {
+List actual =
TestValuesTableFactory.getResults(sinkName);
+Collections.sort(actual);
+return expected.equals(actual);
+},
+timeout,
+"Can not get the expected result.");
+}
-// timeout, assert again
-List actual = TestValuesTableFactory.getResults(sinkName);
-Collections.sort(actual);
-assertEquals(expected, actual);
+public static void checkReturnEmptyResults(String sinkName, Duration
timeout)
+throws InterruptedException, TimeoutException {
+CommonTestUtils.waitUtil(
+new Supplier() {
Review comment:
Yes, I do not think waiting and checking the empty result is a good way
to test this situation. I have changed the test to write some data after the
job started, and check the new data to make sure we read from latest offsets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-25684) Support enhanced show databases syntax
[
https://issues.apache.org/jira/browse/FLINK-25684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Moses updated FLINK-25684:
--
Description:
Enhanced `show databases` statement like ` show databasesfrom like 'db%' ` has
been supported broadly in many popular SQL engine like Spark SQL/MySQL.
We could use such statement to easily show the databases that we wannted.
h3. SHOW DATABSES [ LIKE regex_pattern ]
Examples:
{code:java}
Flink SQL> create database db1;
[INFO] Execute statement succeed.
Flink SQL> create database db1_1;
[INFO] Execute statement succeed.
Flink SQL> create database pre_db;
[INFO] Execute statement succeed.
Flink SQL> show databases;
+--+
|database name |
+--+
| default_database |
| db1 |
|db1_1 |
| pre_db |
+--+
4 rows in set
Flink SQL> show databases like 'db1';
+---+
| database name |
+---+
| db1 |
+---+
1 row in set
Flink SQL> show databases like 'db%';
+---+
| database name |
+---+
| db1 |
| db1_1 |
+---+
2 rows in set
Flink SQL> show databases like '%db%';
+---+
| database name |
+---+
| db1 |
| db1_1 |
|pre_db |
+---+
3 rows in set
Flink SQL> show databases like '%db';
+---+
| database name |
+---+
|pre_db |
+---+
1 row in set
{code}
was:
Enhanced `show databases` statement like ` show databasesfrom like 'db%' ` has
been supported broadly in many popular SQL engine like Spark SQL/MySQL.
We could use such statement to easily show the databases that we wannted.
h3. SHOW DATABSES [ LIKE regex_pattern ]
> Support enhanced show databases syntax
> --
>
> Key: FLINK-25684
> URL: https://issues.apache.org/jira/browse/FLINK-25684
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Moses
>Priority: Major
> Labels: pull-request-available
>
> Enhanced `show databases` statement like ` show databasesfrom like 'db%' `
> has been supported broadly in many popular SQL engine like Spark SQL/MySQL.
> We could use such statement to easily show the databases that we wannted.
> h3. SHOW DATABSES [ LIKE regex_pattern ]
> Examples:
> {code:java}
> Flink SQL> create database db1;
> [INFO] Execute statement succeed.
> Flink SQL> create database db1_1;
> [INFO] Execute statement succeed.
> Flink SQL> create database pre_db;
> [INFO] Execute statement succeed.
> Flink SQL> show databases;
> +--+
> |database name |
> +--+
> | default_database |
> | db1 |
> |db1_1 |
> | pre_db |
> +--+
> 4 rows in set
> Flink SQL> show databases like 'db1';
> +---+
> | database name |
> +---+
> | db1 |
> +---+
> 1 row in set
> Flink SQL> show databases like 'db%';
> +---+
> | database name |
> +---+
> | db1 |
> | db1_1 |
> +---+
> 2 rows in set
> Flink SQL> show databases like '%db%';
> +---+
> | database name |
> +---+
> | db1 |
> | db1_1 |
> |pre_db |
> +---+
> 3 rows in set
> Flink SQL> show databases like '%db';
> +---+
> | database name |
> +---+
> |pre_db |
> +---+
> 1 row in set
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25696) Introduce MetadataPublisher interface to SinkWriter
Jingsong Lee created FLINK-25696:
Summary: Introduce MetadataPublisher interface to SinkWriter
Key: FLINK-25696
URL: https://issues.apache.org/jira/browse/FLINK-25696
Project: Flink
Issue Type: New Feature
Components: API / DataStream, Table Store
Reporter: Jingsong Lee
Assignee: Jingsong Lee
Fix For: 1.15.0
In Table Store, we want to get the offsets of kafka writer, only the offset
returned by the callback inside the KafkaWriter is accurate, so we need this
callback mechanism.
This ticket wants to add a interface MetadataPublisher:
{code:java}
public interface MetadataPublisher {
void subscribe(Consumer consumer);
} {code}
SinkWriter can implement this interface, so that table store can subscribe
metadata from SinkWriter.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] wanglijie95 commented on pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
wanglijie95 commented on pull request #18102: URL: https://github.com/apache/flink/pull/18102#issuecomment-1016104566 @zhuzhurk Thanks for review. I've addressed all the comments , please help to review it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
flinkbot edited a comment on pull request #18102: URL: https://github.com/apache/flink/pull/18102#issuecomment-993332110 ## CI report: * 6ca8ad8ab7817ed28cf8619b36d7a8a21f1db987 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29506) * 2182c3801f333d4b4827b177bb05ec04c648ea87 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29682) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
flinkbot edited a comment on pull request #18102: URL: https://github.com/apache/flink/pull/18102#issuecomment-993332110 ## CI report: * 6ca8ad8ab7817ed28cf8619b36d7a8a21f1db987 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29506) * 2182c3801f333d4b4827b177bb05ec04c648ea87 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wsry commented on pull request #18365: [FLINK-22643][network] Reuse tpc connections between taskmanagers
wsry commented on pull request #18365: URL: https://github.com/apache/flink/pull/18365#issuecomment-1016097587 I will help to merge after all tests pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17720: [FLINK-24210][serializer] Return the correct serialized length for wi…
flinkbot edited a comment on pull request #17720: URL: https://github.com/apache/flink/pull/17720#issuecomment-963141306 ## CI report: * Unknown: [CANCELED](TBD) * cec9ccbee64a29f43eb24f42dfd8db48e0d9e08a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29681) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17720: [FLINK-24210][serializer] Return the correct serialized length for wi…
flinkbot edited a comment on pull request #17720: URL: https://github.com/apache/flink/pull/17720#issuecomment-963141306 ## CI report: * Unknown: [CANCELED](TBD) * cec9ccbee64a29f43eb24f42dfd8db48e0d9e08a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ljz2051 commented on pull request #17720: [FLINK-24210][serializer] Return the correct serialized length for wi…
ljz2051 commented on pull request #17720: URL: https://github.com/apache/flink/pull/17720#issuecomment-1016075421 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
wanglijie95 commented on a change in pull request #18102:
URL: https://github.com/apache/flink/pull/18102#discussion_r787338920
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/failover/flip1/partitionrelease/RegionPartitionGroupReleaseStrategy.java
##
@@ -144,6 +149,54 @@ private PipelinedRegionExecutionView
getPipelinedRegionExecutionViewForVertex(
return releasablePartitionGroups;
}
+@VisibleForTesting
+public boolean isRegionOfVertexFinished(final ExecutionVertexID
executionVertexId) {
+final PipelinedRegionExecutionView regionExecutionView =
+getPipelinedRegionExecutionViewForVertex(executionVertexId);
+return regionExecutionView.isFinished();
+}
+
+@Override
+public void notifySchedulingTopologyUpdated(
+SchedulingTopology schedulingTopology, List
newlyAddedVertices) {
+
+final List newlyAddedRegionGroup =
new ArrayList<>();
Review comment:
OK, I will change the method accoridng your suggesstion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a change in pull request #17601: [FLINK-24697][flink-connectors-kafka] add auto.offset.reset configuration for group-offsets startup mode
ruanhang1993 commented on a change in pull request #17601:
URL: https://github.com/apache/flink/pull/17601#discussion_r787338430
##
File path:
flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicTableFactoryTest.java
##
@@ -679,106 +729,165 @@ private void verifyEncoderSubject(
@Test
public void testSourceTableWithTopicAndTopicPattern() {
-thrown.expect(ValidationException.class);
-thrown.expect(
-containsCause(
-new ValidationException(
-"Option 'topic' and 'topic-pattern' shouldn't
be set together.")));
-
-final Map modifiedOptions =
-getModifiedOptions(
-getBasicSourceOptions(),
-options -> {
-options.put("topic", TOPICS);
-options.put("topic-pattern", TOPIC_REGEX);
-});
-
-createTableSource(SCHEMA, modifiedOptions);
+assertThatThrownBy(
+() -> {
+final Map modifiedOptions =
+getModifiedOptions(
+getBasicSourceOptions(),
+options -> {
+options.put("topic", TOPICS);
+options.put("topic-pattern",
TOPIC_REGEX);
+});
+
+createTableSource(SCHEMA, modifiedOptions);
+})
+.isInstanceOf(ValidationException.class)
+.satisfies(
+throwable ->
+assertThatChainOfCauses(throwable)
+.anySatisfy(
+cause ->
+assertThat(cause)
+.isInstanceOf(
+
ValidationException.class)
+.hasMessage(
+
"Option 'topic' and 'topic-pattern' shouldn't be set together.")));
Review comment:
`FlinkAssertions.anyCauseMatches()` uses `hasMessageContaining` to check
the error message. We need to check with `hasMessage` here.
I will add a new method `containsCause` to `FlinkAssertions` to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
wanglijie95 commented on a change in pull request #18102:
URL: https://github.com/apache/flink/pull/18102#discussion_r787337783
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adapter/DefaultExecutionTopology.java
##
@@ -159,50 +162,79 @@ public static DefaultExecutionTopology fromExecutionGraph(
topologicallySortedJobVertices)
.getAllPipelinedRegions();
-ExecutionGraphIndex executionGraphIndex =
-computeExecutionGraphIndex(
-executionGraph.getAllExecutionVertices(),
-logicalPipelinedRegions,
-edgeManager);
-
-IndexedPipelinedRegions indexedPipelinedRegions =
-computePipelinedRegions(
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion.keySet(),
-
executionGraphIndex.sortedExecutionVerticesInPipelinedRegion::get,
-executionGraphIndex.executionVerticesById::get,
-executionGraphIndex.resultPartitionsById::get);
-
-ensureCoLocatedVerticesInSameRegion(
-indexedPipelinedRegions.pipelinedRegions, executionGraph);
-
-return new DefaultExecutionTopology(
-executionGraphIndex.executionVerticesById,
-executionGraphIndex.executionVerticesList,
-executionGraphIndex.resultPartitionsById,
-indexedPipelinedRegions.pipelinedRegionsByVertex,
-indexedPipelinedRegions.pipelinedRegions,
-edgeManager);
-}
-
-private static ExecutionGraphIndex computeExecutionGraphIndex(
-Iterable executionVertices,
-Iterable logicalPipelinedRegions,
-EdgeManager edgeManager) {
-Map executionVerticesById =
new HashMap<>();
-List executionVerticesList = new ArrayList<>();
-Map
resultPartitionsById =
-new HashMap<>();
-Map>
-sortedExecutionVerticesInPipelinedRegion = new
IdentityHashMap<>();
-
-Map
logicalPipelinedRegionByJobVertexId =
+Map
logicalPipelinedRegionsByJobVertexId =
new HashMap<>();
for (DefaultLogicalPipelinedRegion logicalPipelinedRegion :
logicalPipelinedRegions) {
for (LogicalVertex vertex : logicalPipelinedRegion.getVertices()) {
-logicalPipelinedRegionByJobVertexId.put(vertex.getId(),
logicalPipelinedRegion);
+logicalPipelinedRegionsByJobVertexId.put(vertex.getId(),
logicalPipelinedRegion);
}
}
+return logicalPipelinedRegionsByJobVertexId;
+}
+
+public void notifyExecutionGraphUpdated(
+final DefaultExecutionGraph executionGraph,
+final List newJobVertices) {
+
+checkNotNull(executionGraph, "execution graph can not be null");
+
+final Set newVertexSet =
+newJobVertices.stream()
+.map(ExecutionJobVertex::getJobVertexId)
+.collect(Collectors.toSet());
+
+// any PIPELINED input should be from within this new set so that
existing pipelined regions
+// will not change
+newJobVertices.stream()
+.map(ExecutionJobVertex::getJobVertex)
+.flatMap(v -> v.getInputs().stream())
+.map(JobEdge::getSource)
+.filter(r -> r.getResultType().isPipelined())
+.map(IntermediateDataSet::getProducer)
+.map(JobVertex::getID)
+.forEach(id -> checkState(newVertexSet.contains(id)));
+
+final Iterable newAddedExecutionVertices =
+newJobVertices.stream()
+.flatMap(jobVertex ->
Stream.of(jobVertex.getTaskVertices()))
+.collect(Collectors.toList());
+
+updateExecutionGraphIndex(newAddedExecutionVertices);
+
+updatePipelinedRegions(newAddedExecutionVertices);
+
+ensureCoLocatedVerticesInSameRegion(pipelinedRegions, executionGraph);
+}
+
+public static DefaultExecutionTopology fromExecutionGraph(
+DefaultExecutionGraph executionGraph) {
+checkNotNull(executionGraph, "execution graph can not be null");
+
+EdgeManager edgeManager = executionGraph.getEdgeManager();
+
+Iterable executionVertices =
executionGraph.getAllExecutionVertices();
+
+DefaultExecutionTopology schedulingTopology =
+new DefaultExecutionTopology(
+() ->
+IterableUtils.toStream(executionVertices)
+.map(ExecutionVertex::getID)
+.collect(Collectors.toList()),
+edgeManager,
+
[jira] [Updated] (FLINK-25695) TemporalJoin cause state leak in some cases
[ https://issues.apache.org/jira/browse/FLINK-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lyn Zhang updated FLINK-25695: -- Priority: Critical (was: Major) > TemporalJoin cause state leak in some cases > --- > > Key: FLINK-25695 > URL: https://issues.apache.org/jira/browse/FLINK-25695 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.14.3 >Reporter: Lyn Zhang >Priority: Critical > Labels: pull-request-available > > Last year, I reported the similar bug of TemporalJoin cause state leak. > Detail: FLINK-21833 > Recently, I found the fix code can reduce the the leak size but can not > resolve it completely. > The code of line 213 cause it and the right fix is to invoke cleanUp() method. > In FLINK-21833, we discussed when the code is running on line 213, that means > Left State, Right State, registeredTimerState is empty, actually the Left > State and Right State value(MapState) is empty but the key is still be in > state, So invoke state.clear() is necessary. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25695) TemporalJoin cause state leak in some cases
[ https://issues.apache.org/jira/browse/FLINK-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lyn Zhang updated FLINK-25695: -- Priority: Major (was: Critical) > TemporalJoin cause state leak in some cases > --- > > Key: FLINK-25695 > URL: https://issues.apache.org/jira/browse/FLINK-25695 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.14.3 >Reporter: Lyn Zhang >Priority: Major > Labels: pull-request-available > > Last year, I reported the similar bug of TemporalJoin cause state leak. > Detail: FLINK-21833 > Recently, I found the fix code can reduce the the leak size but can not > resolve it completely. > The code of line 213 cause it and the right fix is to invoke cleanUp() method. > In FLINK-21833, we discussed when the code is running on line 213, that means > Left State, Right State, registeredTimerState is empty, actually the Left > State and Right State value(MapState) is empty but the key is still be in > state, So invoke state.clear() is necessary. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wanglijie95 commented on a change in pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
wanglijie95 commented on a change in pull request #18102:
URL: https://github.com/apache/flink/pull/18102#discussion_r787335458
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adapter/DefaultResultPartition.java
##
@@ -54,16 +55,32 @@
IntermediateDataSetID intermediateDataSetId,
ResultPartitionType partitionType,
Supplier resultPartitionStateSupplier,
-ConsumerVertexGroup consumerVertexGroup,
+Supplier consumerVertexGroupSupplier,
Supplier>
consumerPartitionGroupSupplier) {
this.resultPartitionId = checkNotNull(partitionId);
this.intermediateDataSetId = checkNotNull(intermediateDataSetId);
this.partitionType = checkNotNull(partitionType);
this.resultPartitionStateSupplier =
checkNotNull(resultPartitionStateSupplier);
-this.consumerVertexGroup = consumerVertexGroup;
+this.consumerVertexGroupSupplier = consumerVertexGroupSupplier;
Review comment:
OK, I will add a hotfix commit to add `checkNotNull` for
`Supplier/Function`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
wanglijie95 commented on a change in pull request #18102:
URL: https://github.com/apache/flink/pull/18102#discussion_r787335115
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adapter/DefaultResultPartition.java
##
@@ -28,12 +28,13 @@
import org.apache.flink.runtime.scheduler.strategy.SchedulingResultPartition;
import java.util.List;
+import java.util.Optional;
import java.util.function.Supplier;
import static org.apache.flink.util.Preconditions.checkNotNull;
/** Default implementation of {@link SchedulingResultPartition}. */
-class DefaultResultPartition implements SchedulingResultPartition {
+public class DefaultResultPartition implements SchedulingResultPartition {
Review comment:
It's not needed, I will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18102: [FLINK-25033][runtime] Let some scheduler components updatable
wanglijie95 commented on a change in pull request #18102: URL: https://github.com/apache/flink/pull/18102#discussion_r787334289 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraph.java ## @@ -59,8 +59,10 @@ import org.apache.flink.runtime.jobgraph.JobVertex; import org.apache.flink.runtime.jobgraph.JobVertexID; import org.apache.flink.runtime.jobgraph.tasks.CheckpointCoordinatorConfiguration; +import org.apache.flink.runtime.jobmanager.scheduler.SlotSharingGroup; Review comment: I have separated the "Support calculate network memory for dynamic graph" and "Support register operator coordinators for new initialized ExecutionJobVertex" into two new tickets: FLINK-25668 and FLINK-25669 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18400: [FLINK-25198][docs] Add doc about name and description of operator
flinkbot edited a comment on pull request #18400: URL: https://github.com/apache/flink/pull/18400#issuecomment-1016057839 ## CI report: * 0ee72a0732026c046d991d959033995c7baf44e1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29680) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18400: [FLINK-25198][docs] Add doc about name and description of operator
flinkbot commented on pull request #18400: URL: https://github.com/apache/flink/pull/18400#issuecomment-1016057839 ## CI report: * 0ee72a0732026c046d991d959033995c7baf44e1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18400: [FLINK-25198][docs] Add doc about name and description of operator
flinkbot commented on pull request #18400: URL: https://github.com/apache/flink/pull/18400#issuecomment-1016057431 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0ee72a0732026c046d991d959033995c7baf44e1 (Wed Jan 19 03:59:29 UTC 2022) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-25198).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25198) add document about how to debug with the name and description
[ https://issues.apache.org/jira/browse/FLINK-25198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25198: --- Labels: pull-request-available (was: ) > add document about how to debug with the name and description > -- > > Key: FLINK-25198 > URL: https://issues.apache.org/jira/browse/FLINK-25198 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Wenlong Lyu >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > the doc could in the debugging section > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/ops/debugging/debugging_event_time/ > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wenlong88 opened a new pull request #18400: [FLINK-25198][docs] Add doc about name and description of operator
wenlong88 opened a new pull request #18400: URL: https://github.com/apache/flink/pull/18400 ## What is the purpose of the change this is part of This PR is part of https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+job , aims at to add doc about name and description. ## Brief change log add docs about name and description in the section of operators. ## Verifying this change This change is a trivial work without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) * 0fb5e7a2b2f583a36775e1d7347080e834e0d806 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29679) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) * 0fb5e7a2b2f583a36775e1d7347080e834e0d806 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) * 455dfd8ad2146f35d31a602715a9b27a07aa2aba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29678) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * 51b23df97a9e47ce6ab8f4ca321cce5c04f729bd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29642) * b8a9d9704d289aa32e8d040b7090a6fb97c4907c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29673) * 99653eb7e622b96ba0bb47836fe3eefe7b71d942 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29677) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) * 0fb5e7a2b2f583a36775e1d7347080e834e0d806 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) * 455dfd8ad2146f35d31a602715a9b27a07aa2aba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * 51b23df97a9e47ce6ab8f4ca321cce5c04f729bd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29642) * b8a9d9704d289aa32e8d040b7090a6fb97c4907c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29673) * 99653eb7e622b96ba0bb47836fe3eefe7b71d942 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * 51b23df97a9e47ce6ab8f4ca321cce5c04f729bd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29642) * b8a9d9704d289aa32e8d040b7090a6fb97c4907c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) * 455dfd8ad2146f35d31a602715a9b27a07aa2aba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) * 0fb5e7a2b2f583a36775e1d7347080e834e0d806 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * 51b23df97a9e47ce6ab8f4ca321cce5c04f729bd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29642) * b8a9d9704d289aa32e8d040b7090a6fb97c4907c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29673) * 99653eb7e622b96ba0bb47836fe3eefe7b71d942 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18391: [FLINK-25478][chaneglog] Correct the state register logic of ChangelogStateBackendHandle
Myasuka commented on a change in pull request #18391:
URL: https://github.com/apache/flink/pull/18391#discussion_r787321825
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/state/SharedStateRegistryTest.java
##
@@ -98,6 +106,77 @@ public void testUnregisterWithUnexistedKey() {
sharedStateRegistry.unregisterUnusedState(Long.MAX_VALUE);
}
+@Test
+public void testRegisterChangelogStateBackendHandles() throws
InterruptedException {
+SharedStateRegistry sharedStateRegistry = new
SharedStateRegistryImpl();
+long materializationId1 = 1L;
+IncrementalStateHandleWrapper incrementalTMStateHandle1 =
+createDummyIncrementalStateHandle(materializationId1);
+
+IncrementalStateHandleWrapper incrementalJMStateHandle11 =
incrementalTMStateHandle1.copy();
+ChangelogStateHandleWrapper changelogStateHandle1 =
createDummyChangelogStateHandle(1, 2);
+long checkpointId1 = 41;
+ChangelogStateBackendHandle.ChangelogStateBackendHandleImpl
changelogStateBackendHandle1 =
+new
ChangelogStateBackendHandle.ChangelogStateBackendHandleImpl(
+Collections.singletonList(incrementalJMStateHandle11),
+Collections.singletonList(changelogStateHandle1),
+incrementalTMStateHandle1.getKeyGroupRange());
+changelogStateBackendHandle1.registerSharedStates(sharedStateRegistry,
checkpointId1);
+sharedStateRegistry.checkpointCompleted(checkpointId1);
+sharedStateRegistry.unregisterUnusedState(checkpointId1);
+
+IncrementalStateHandleWrapper incrementalJMStateHandle12 =
incrementalTMStateHandle1.copy();
Review comment:
The `copy()` method is due to I still call
`stateRegistry.registerAll(materialized, checkpointID)` and the problem in
FLINK-25479 caused this.
As we still need to call the materialized CompositeStateHandle to register
themselves, I have to add the copy method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * 51b23df97a9e47ce6ab8f4ca321cce5c04f729bd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29642) * b8a9d9704d289aa32e8d040b7090a6fb97c4907c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) * 0fb5e7a2b2f583a36775e1d7347080e834e0d806 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) * 455dfd8ad2146f35d31a602715a9b27a07aa2aba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * 51b23df97a9e47ce6ab8f4ca321cce5c04f729bd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29642) * b8a9d9704d289aa32e8d040b7090a6fb97c4907c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29673) * 99653eb7e622b96ba0bb47836fe3eefe7b71d942 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17720: [FLINK-24210][serializer] Return the correct serialized length for wi…
flinkbot edited a comment on pull request #17720: URL: https://github.com/apache/flink/pull/17720#issuecomment-963141306 ## CI report: * 0c8d9c69ed2cf13ebdb1dfb6e0b44ca31ad4e706 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26152) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18398: [FLINK-25695][flink-table] Fix state leak in temporalRowTimeJoinOper…
flinkbot edited a comment on pull request #18398: URL: https://github.com/apache/flink/pull/18398#issuecomment-1016025272 ## CI report: * be961c279e602761b7dd727664bff6a46f78ec0d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29674) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18399: [FLINK-25118][streaming] Add vertex topology index prefix in vertex name
flinkbot edited a comment on pull request #18399: URL: https://github.com/apache/flink/pull/18399#issuecomment-1016036536 ## CI report: * 1b59e0775a0fc8288e915036f8ba7a658069881d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29676) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18119: [FLINK-24947] Support hostNetwork for native K8s integration on session mode
flinkbot edited a comment on pull request #18119: URL: https://github.com/apache/flink/pull/18119#issuecomment-994734000 ## CI report: * 51b23df97a9e47ce6ab8f4ca321cce5c04f729bd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29642) * b8a9d9704d289aa32e8d040b7090a6fb97c4907c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29673) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-25494) Duplicate element serializer during DefaultOperatorStateBackendSnapshotStrategy#syncPrepareResources
[
https://issues.apache.org/jira/browse/FLINK-25494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang closed FLINK-25494.
Resolution: Invalid
> Duplicate element serializer during
> DefaultOperatorStateBackendSnapshotStrategy#syncPrepareResources
>
>
> Key: FLINK-25494
> URL: https://issues.apache.org/jira/browse/FLINK-25494
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0, 1.13.6, 1.14.4
>
>
> Currently, during
> DefaultOperatorStateBackendSnapshotStrategy#syncPrepareResources, it will
> copy the array list serializer via PartitionableListState#deepCopy. However,
> it just initialize another ArrayListSerializer and not duplicate the internal
> state serializer:
>
> See "{{{}internalListCopySerializer{}}}":
>
> {code:java}
> private PartitionableListState(
> RegisteredOperatorStateBackendMetaInfo stateMetaInfo, ArrayList
> internalList) {
> this.stateMetaInfo = Preconditions.checkNotNull(stateMetaInfo);
> this.internalList = Preconditions.checkNotNull(internalList);
> this.internalListCopySerializer =
> new
> ArrayListSerializer<>(stateMetaInfo.getPartitionStateSerializer());
> } {code}
>
> This would cause unexpected problem with the usage of kryo serializer.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25695) TemporalJoin cause state leak in some cases
[ https://issues.apache.org/jira/browse/FLINK-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lyn Zhang updated FLINK-25695: -- Description: Last year, I reported the similar bug of TemporalJoin cause state leak. Detail: FLINK-21833 Recently, I found the fix code can reduce the the leak size but can not resolve it completely. The code of line 213 cause it and the right fix is to invoke cleanUp() method. In FLINK-21833, we discussed when the code is running on line 213, that means Left State, Right State, registeredTimerState is empty, actually the Left State and Right State value(MapState) is empty but the key is still be in state, So invoke state.clear() is necessary. was: Last year, I reported the similar bug of TemporalJoin cause state leak. Detail: FLINK-21833 Recently, I found the fix code can reduce the the leak size but can not resolve it completely. The code of line 213 cause it and the right fix is to invoke cleanUp() method. In FLINK-21833, we discussed that if the code is running on line 213, That means Left State, Right State, registeredTimerState is empty, actually the Left State and Right State value(MapState) is empty but the key is still be in state, So invoke state.clear() is necessary. > TemporalJoin cause state leak in some cases > --- > > Key: FLINK-25695 > URL: https://issues.apache.org/jira/browse/FLINK-25695 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.14.3 >Reporter: Lyn Zhang >Priority: Major > Labels: pull-request-available > > Last year, I reported the similar bug of TemporalJoin cause state leak. > Detail: FLINK-21833 > Recently, I found the fix code can reduce the the leak size but can not > resolve it completely. > The code of line 213 cause it and the right fix is to invoke cleanUp() method. > In FLINK-21833, we discussed when the code is running on line 213, that means > Left State, Right State, registeredTimerState is empty, actually the Left > State and Right State value(MapState) is empty but the key is still be in > state, So invoke state.clear() is necessary. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] Myasuka commented on a change in pull request #18387: [FLINK-25494][state] Duplicate element serializer during operator state sync snapshot phase
Myasuka commented on a change in pull request #18387:
URL: https://github.com/apache/flink/pull/18387#discussion_r787316200
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/PartitionableListState.java
##
@@ -45,23 +47,25 @@
private final ArrayListSerializer internalListCopySerializer;
PartitionableListState(RegisteredOperatorStateBackendMetaInfo
stateMetaInfo) {
-this(stateMetaInfo, new ArrayList());
+this(stateMetaInfo, new ArrayList<>(),
stateMetaInfo.getPartitionStateSerializer());
+}
+
+// constructor for deep copy.
+private PartitionableListState(PartitionableListState toCopy) {
+this(
+toCopy.stateMetaInfo.deepCopy(),
+toCopy.internalListCopySerializer.copy(toCopy.internalList),
+
toCopy.stateMetaInfo.getPartitionStateSerializer().duplicate());
Review comment:
You are right, I will close the original ticket. Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Myasuka closed pull request #18387: [FLINK-25494][state] Duplicate element serializer during operator state sync snapshot phase
Myasuka closed pull request #18387: URL: https://github.com/apache/flink/pull/18387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25695) TemporalJoin cause state leak in some cases
[ https://issues.apache.org/jira/browse/FLINK-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lyn Zhang updated FLINK-25695: -- Description: Last year, I reported the similar bug of TemporalJoin cause state leak. Detail: FLINK-21833 Recently, I found the fix code can reduce the the leak size but can not resolve it completely. The code of line 213 cause it and the right fix is to invoke cleanUp() method. In FLINK-21833, we discussed when the code is running on line 213, that means Left State, Right State, registeredTimerState is empty, actually the Left State and Right State value(MapState) is empty but the key is still be in state, So invoke state.clear() is necessary. was: Last year, I reported the similar bug of TemporalJoin cause state leak. Detail: FLINK-21833 Recently, I found the fix code can reduce the the leak size but can not resolve it completely. The code of line 213 cause it and the right fix is to invoke cleanUp() method. In FLINK-21833, we discussed when the code is running on line 213, that means Left State, Right State, registeredTimerState is empty, actually the Left State and Right State value(MapState) is empty but the key is still be in state, So invoke state.clear() is necessary. > TemporalJoin cause state leak in some cases > --- > > Key: FLINK-25695 > URL: https://issues.apache.org/jira/browse/FLINK-25695 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.14.3 >Reporter: Lyn Zhang >Priority: Major > Labels: pull-request-available > > Last year, I reported the similar bug of TemporalJoin cause state leak. > Detail: FLINK-21833 > Recently, I found the fix code can reduce the the leak size but can not > resolve it completely. > The code of line 213 cause it and the right fix is to invoke cleanUp() method. > In FLINK-21833, we discussed when the code is running on line 213, that means > Left State, Right State, registeredTimerState is empty, actually the Left > State and Right State value(MapState) is empty but the key is still be in > state, So invoke state.clear() is necessary. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25695) TemporalJoin cause state leak in some cases
[ https://issues.apache.org/jira/browse/FLINK-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25695: --- Labels: pull-request-available (was: ) > TemporalJoin cause state leak in some cases > --- > > Key: FLINK-25695 > URL: https://issues.apache.org/jira/browse/FLINK-25695 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.14.3 >Reporter: Lyn Zhang >Priority: Major > Labels: pull-request-available > > Last year, I reported the similar bug of TemporalJoin cause state leak. > Detail: FLINK-21833 > Recently, I found the fix code can reduce the the leak size but can not > resolve it completely. > The code of line 213 cause it and the right fix is to invoke cleanUp() method. > In FLINK-21833, we discussed that if the code is running on line 213, That > means Left State, Right State, registeredTimerState is empty, actually the > Left State and Right State value(MapState) is empty but the key is still be > in state, So invoke state.clear() is necessary. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18399: [FLINK-25118][streaming] Add vertex topology index prefix in vertex name
flinkbot commented on pull request #18399: URL: https://github.com/apache/flink/pull/18399#issuecomment-1016036536 ## CI report: * 1b59e0775a0fc8288e915036f8ba7a658069881d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18394: [FLINK-25520][Table SQL/API] Implement "ALTER TABLE ... COMPACT" SQL
flinkbot edited a comment on pull request #18394: URL: https://github.com/apache/flink/pull/18394#issuecomment-1015323011 ## CI report: * 0720a8c1fc29b0ce8d953de0014a30066bf866ce Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29659) * b8d38cab5e0afaf20f07d9aa707729f19962e2b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=29672) * 455dfd8ad2146f35d31a602715a9b27a07aa2aba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
