[GitHub] [flink] flinkbot edited a comment on issue #11540: [FLINK-16099][docs-zh] Translate "HiveCatalog" page of "Hive Integration" into…
flinkbot edited a comment on issue #11540: [FLINK-16099][docs-zh] Translate "HiveCatalog" page of "Hive Integration" into… URL: https://github.com/apache/flink/pull/11540#issuecomment-604804466 ## CI report: * 07b38deaa1461a92bbc537c40a66915875aee98f Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156949994) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6860) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11559: [FLINK-16858][table] Expose partitioned by grammar
flinkbot edited a comment on issue #11559: [FLINK-16858][table] Expose partitioned by grammar URL: https://github.com/apache/flink/pull/11559#issuecomment-605804330 ## CI report: * 648bfa7907e6d9b4a2640c577ba181b3f212da74 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156476002) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6817) * c22dcf150bd567c611c764ab45b7e28542f42e6a Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156958058) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6863) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16874) Respect the dynamic options when calculating memory options in taskmanager.sh
[
https://issues.apache.org/jira/browse/FLINK-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071479#comment-17071479
]
Xintong Song commented on FLINK-16874:
--
Thanks for reporting this issue, [~karmagyz].
I would not consider this as a bug, because the memory configurations are read
before {{TaskManagerRunner}} is launched. However, I agree with you that it
would be good to support memory configurations with dynamic properties for
standalone clusters. IIUC, this does not require changes. The only thing we
need to do is to forward the user arguments to {{BashJavaUtils}}.
> Respect the dynamic options when calculating memory options in taskmanager.sh
> -
>
> Key: FLINK-16874
> URL: https://issues.apache.org/jira/browse/FLINK-16874
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.10.0
>Reporter: Yangze Guo
>Priority: Major
> Fix For: 1.10.1, 1.11.0
>
>
> Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options
> to the TaskManagerRunner. However, in FLINK-13983, we calculate the
> memory-related configuration only according to the FLINK_CONF_DIR. We then
> append the calculation result as dynamic options to the TM, the user-defined
> dynamic options would be overridden and ignored.
> The BashJavaUtils is already support loading dynamic options, we just need to
> pass it in bash script.
> cc [~xintongsong] [~azagrebin]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16874) Respect the dynamic options when calculating memory options in taskmanager.sh
[ https://issues.apache.org/jira/browse/FLINK-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yangze Guo updated FLINK-16874: --- Priority: Critical (was: Blocker) > Respect the dynamic options when calculating memory options in taskmanager.sh > - > > Key: FLINK-16874 > URL: https://issues.apache.org/jira/browse/FLINK-16874 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Yangze Guo >Priority: Critical > Fix For: 1.10.1, 1.11.0 > > > Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options > to the TaskManagerRunner. However, in FLINK-13983, we calculate the > memory-related configuration only according to the FLINK_CONF_DIR. We then > append the calculation result as dynamic options to the TM, the user-defined > dynamic options would be overridden and ignored. > The BashJavaUtils is already support loading dynamic options, we just need to > pass it in bash script. > cc [~xintongsong] [~azagrebin] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16874) Respect the dynamic options when calculating memory options in taskmanager.sh
[ https://issues.apache.org/jira/browse/FLINK-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-16874: - Priority: Major (was: Critical) > Respect the dynamic options when calculating memory options in taskmanager.sh > - > > Key: FLINK-16874 > URL: https://issues.apache.org/jira/browse/FLINK-16874 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Yangze Guo >Priority: Major > Fix For: 1.10.1, 1.11.0 > > > Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options > to the TaskManagerRunner. However, in FLINK-13983, we calculate the > memory-related configuration only according to the FLINK_CONF_DIR. We then > append the calculation result as dynamic options to the TM, the user-defined > dynamic options would be overridden and ignored. > The BashJavaUtils is already support loading dynamic options, we just need to > pass it in bash script. > cc [~xintongsong] [~azagrebin] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16874) Respect the dynamic options when calculating memory options in taskmanager.sh
[ https://issues.apache.org/jira/browse/FLINK-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yangze Guo updated FLINK-16874: --- Issue Type: Improvement (was: Bug) > Respect the dynamic options when calculating memory options in taskmanager.sh > - > > Key: FLINK-16874 > URL: https://issues.apache.org/jira/browse/FLINK-16874 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Yangze Guo >Priority: Blocker > Fix For: 1.10.1, 1.11.0 > > > Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options > to the TaskManagerRunner. However, in FLINK-13983, we calculate the > memory-related configuration only according to the FLINK_CONF_DIR. We then > append the calculation result as dynamic options to the TM, the user-defined > dynamic options would be overridden and ignored. > The BashJavaUtils is already support loading dynamic options, we just need to > pass it in bash script. > cc [~xintongsong] [~azagrebin] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16874) Respect the dynamic options when calculating memory options in taskmanager.sh
[ https://issues.apache.org/jira/browse/FLINK-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071475#comment-17071475 ] Yangze Guo commented on FLINK-16874: I'd like to work on it. Could someone assign this to me? > Respect the dynamic options when calculating memory options in taskmanager.sh > - > > Key: FLINK-16874 > URL: https://issues.apache.org/jira/browse/FLINK-16874 > Project: Flink > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Yangze Guo >Priority: Blocker > Fix For: 1.10.1, 1.11.0 > > > Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options > to the TaskManagerRunner. However, in FLINK-13983, we calculate the > memory-related configuration only according to the FLINK_CONF_DIR. We then > append the calculation result as dynamic options to the TM, the user-defined > dynamic options would be overridden and ignored. > The BashJavaUtils is already support loading dynamic options, we just need to > pass it in bash script. > cc [~xintongsong] [~azagrebin] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11559: [FLINK-16858][table] Expose partitioned by grammar
flinkbot edited a comment on issue #11559: [FLINK-16858][table] Expose partitioned by grammar URL: https://github.com/apache/flink/pull/11559#issuecomment-605804330 ## CI report: * 648bfa7907e6d9b4a2640c577ba181b3f212da74 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156476002) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6817) * c22dcf150bd567c611c764ab45b7e28542f42e6a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321#issuecomment-595208662 ## CI report: * 7cebc64ea4a49f0f1d246889cb71b730b4a170eb Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/156952304) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6862) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16876) Make TtlTimeProvider configurable when creating keyed state backend
[ https://issues.apache.org/jira/browse/FLINK-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071473#comment-17071473 ] Jark Wu commented on FLINK-16876: - +1 to do this. This can also expose the state TTL ability to the operator harness tests. > Make TtlTimeProvider configurable when creating keyed state backend > --- > > Key: FLINK-16876 > URL: https://issues.apache.org/jira/browse/FLINK-16876 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.10.0 >Reporter: Yun Tang >Priority: Major > Fix For: 1.11.0 > > > Currently, we would always use TtlTimeProvider.DEFAULT to create keyed state > backend. This is somehow acceptable since we only support processing time for > TTL now. However, this would make UT tests which would verify TTL logic not > so convenient like FLINK-16581. > I propose to let TtlTimeProvider configurable when creating keyed state > backend to not block other feature development. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571#issuecomment-606377768 ## CI report: * 91507d011419afe848d1a90ae0027cd2511817e5 UNKNOWN * fd838b6268e0233589afed81f4f1fd9a0ec77bf9 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156948705) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6859) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400645765
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java
##
@@ -153,13 +153,17 @@ protected boolean copyFromSerializerToTargetChannel(int
targetChannel) throws IO
return pruneTriggered;
}
- public void broadcastEvent(AbstractEvent event) throws IOException {
+ public void broadcastEvent(AbstractEvent event, boolean
isPriorityEvent) throws IOException {
try (BufferConsumer eventBufferConsumer =
EventSerializer.toBufferConsumer(event)) {
for (int targetChannel = 0; targetChannel <
numberOfChannels; targetChannel++) {
tryFinishCurrentBufferBuilder(targetChannel);
- // Retain the buffer so that it can be recycled
by each channel of targetPartition
-
targetPartition.addBufferConsumer(eventBufferConsumer.copy(), targetChannel);
+ if (isPriorityEvent) {
+ targetPartition.addPriorityEvent(event,
targetChannel, eventBufferConsumer);
Review comment:
I am a bit torn about passing both event and buffer in the interface method.
It seems not very clean on my side.
I guess we can make it happen to guarantee that `CheckpointBarrier` should
be inserted in head for unaligned option in any cases. Then it would make
things easy.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321#issuecomment-595208662 ## CI report: * 7cebc64ea4a49f0f1d246889cb71b730b4a170eb Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156952304) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6862) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11540: [FLINK-16099][docs-zh] Translate "HiveCatalog" page of "Hive Integration" into…
flinkbot edited a comment on issue #11540: [FLINK-16099][docs-zh] Translate "HiveCatalog" page of "Hive Integration" into… URL: https://github.com/apache/flink/pull/11540#issuecomment-604804466 ## CI report: * 07b38deaa1461a92bbc537c40a66915875aee98f Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156949994) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6860) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571#issuecomment-606377768 ## CI report: * 91507d011419afe848d1a90ae0027cd2511817e5 UNKNOWN * fd838b6268e0233589afed81f4f1fd9a0ec77bf9 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156948705) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6859) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400643335
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/netty/CreditBasedSequenceNumberingViewReader.java
##
@@ -210,4 +214,12 @@ public String toString() {
", isRegisteredAsAvailable=" + isRegisteredAsAvailable +
'}';
}
+
+ @Override
+ public EnqueueAction priorityEvent(AbstractEvent event, BufferConsumer
eventBufferConsumer) {
+ // make sure netty consumer knows that it can immediately
process next buffer
+ notifyDataAvailable();
Review comment:
I think it is not necessary to call `notifyDataAvailable` here. The event
actually has not been added into queue yet, so the netty thread can not fetch
data immediately, even I am not sure it might bring mistake here. And after
adding event into queue inside `PipelinedSubpartition`, the
`notifyDataAvailable` would be called in normal way.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400642380
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -83,24 +86,45 @@
/** The total number of bytes (both data and event buffers). */
private long totalNumberOfBytes;
+ /** The collection of buffers which are spanned over by checkpoint
barrier and needs to be persisted for snapshot. */
+ private final List inflightBufferSnapshot = new ArrayList<>();
+
//
PipelinedSubpartition(int index, ResultPartition parent) {
super(index, parent);
}
@Override
- public boolean add(BufferConsumer bufferConsumer) {
- return add(bufferConsumer, false);
+ public boolean add(BufferConsumer bufferConsumer) throws IOException {
+ return add(bufferConsumer, false, false);
}
@Override
public void finish() throws IOException {
-
add(EventSerializer.toBufferConsumer(EndOfPartitionEvent.INSTANCE), true);
+
add(EventSerializer.toBufferConsumer(EndOfPartitionEvent.INSTANCE), true,
false);
LOG.debug("{}: Finished {}.", parent.getOwningTaskName(), this);
}
- private boolean add(BufferConsumer bufferConsumer, boolean finish) {
+ @Override
+ public EnqueueAction priorityEvent(AbstractEvent event, BufferConsumer
eventBufferConsumer) throws IOException {
+ if (readView != null) {
+ switch (readView.priorityEvent(event,
eventBufferConsumer)) {
Review comment:
I think we can guarantee to insert the event in head for any cases. Refer to
the comments in `LocalBufferPool` method. Then wen can remove the return value
from `#priorityEvent`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400637920
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PriorityEventListener.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.network.buffer.BufferConsumer;
+
+import java.io.IOException;
+
+/**
+ * Listens to priority events (e.g., checkpoint barriers) of subpartitions.
The listener can completely process the
+ * event without the buffer queue or let the subpartition enqueue the event in
the front or back of the buffer queue.
+ */
+public interface PriorityEventListener {
+ /**
+* Allows the listener to react to a priority event independent of the
outgoing buffer queue.
+*
+* @return the action to be performed by the caller.
+*/
+ EnqueueAction priorityEvent(AbstractEvent event, BufferConsumer
eventBufferConsumer) throws IOException;
Review comment:
I guess this return value actually not be used in practice now. Do we need
to introduce it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400641076
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/LocalInputChannel.java
##
@@ -270,4 +277,18 @@ public int unsynchronizedGetNumberOfQueuedBuffers() {
public String toString() {
return "LocalInputChannel [" + partitionId + "]";
}
+
+ @Override
+ public EnqueueAction priorityEvent(AbstractEvent event, BufferConsumer
eventBufferConsumer) throws IOException {
+ if (inputGate.bufferReceivedListener == null) {
Review comment:
I think this path should never happen in practice. Although we can not make
`bufferReceivedListener` as final in form, we should still guarantee the
listener is ready before requesting partition from local channel.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321#issuecomment-595208662 ## CI report: * 2a005b9c7d287b338a5a4c0795816d61e8f09312 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/151951525) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5970) * 7cebc64ea4a49f0f1d246889cb71b730b4a170eb Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156952304) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6862) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400639370
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java
##
@@ -153,13 +153,17 @@ protected boolean copyFromSerializerToTargetChannel(int
targetChannel) throws IO
return pruneTriggered;
}
- public void broadcastEvent(AbstractEvent event) throws IOException {
+ public void broadcastEvent(AbstractEvent event, boolean
isPriorityEvent) throws IOException {
try (BufferConsumer eventBufferConsumer =
EventSerializer.toBufferConsumer(event)) {
for (int targetChannel = 0; targetChannel <
numberOfChannels; targetChannel++) {
tryFinishCurrentBufferBuilder(targetChannel);
- // Retain the buffer so that it can be recycled
by each channel of targetPartition
-
targetPartition.addBufferConsumer(eventBufferConsumer.copy(), targetChannel);
+ if (isPriorityEvent) {
+ targetPartition.addPriorityEvent(event,
targetChannel, eventBufferConsumer);
Review comment:
`eventBufferConsumer` -> `eventBufferConsumer.copy()`, to keep the
consistency with below action, and also avoid calling `copy` multiple times in
`ResultSubpartition#priorityEvent`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400638771
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -83,24 +86,45 @@
/** The total number of bytes (both data and event buffers). */
private long totalNumberOfBytes;
+ /** The collection of buffers which are spanned over by checkpoint
barrier and needs to be persisted for snapshot. */
+ private final List inflightBufferSnapshot = new ArrayList<>();
+
//
PipelinedSubpartition(int index, ResultPartition parent) {
super(index, parent);
}
@Override
- public boolean add(BufferConsumer bufferConsumer) {
- return add(bufferConsumer, false);
+ public boolean add(BufferConsumer bufferConsumer) throws IOException {
+ return add(bufferConsumer, false, false);
}
@Override
public void finish() throws IOException {
-
add(EventSerializer.toBufferConsumer(EndOfPartitionEvent.INSTANCE), true);
+
add(EventSerializer.toBufferConsumer(EndOfPartitionEvent.INSTANCE), true,
false);
LOG.debug("{}: Finished {}.", parent.getOwningTaskName(), this);
}
- private boolean add(BufferConsumer bufferConsumer, boolean finish) {
+ @Override
+ public EnqueueAction priorityEvent(AbstractEvent event, BufferConsumer
eventBufferConsumer) throws IOException {
+ if (readView != null) {
+ switch (readView.priorityEvent(event,
eventBufferConsumer)) {
+ case ENQUEUE_AS_FIRST_BUFFER:
+ add(eventBufferConsumer.copy(), false,
true);
+ break;
+ case ENQUEUE_AS_LAST_BUFFER:
+ add(eventBufferConsumer.copy(), false,
false);
+ break;
+ }
+ } else {
+ buffers.add(eventBufferConsumer.copy());
Review comment:
If the current view is null, maybe it is also worth inserting the barrier
into head, because the view might be established soon after inserting. Or are
there any considerations for making the difference based on the view status?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400637920
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PriorityEventListener.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.network.buffer.BufferConsumer;
+
+import java.io.IOException;
+
+/**
+ * Listens to priority events (e.g., checkpoint barriers) of subpartitions.
The listener can completely process the
+ * event without the buffer queue or let the subpartition enqueue the event in
the front or back of the buffer queue.
+ */
+public interface PriorityEventListener {
+ /**
+* Allows the listener to react to a priority event independent of the
outgoing buffer queue.
+*
+* @return the action to be performed by the caller.
+*/
+ EnqueueAction priorityEvent(AbstractEvent event, BufferConsumer
eventBufferConsumer) throws IOException;
Review comment:
I guess this return value actually not be used in practice now. Do we need
to introduce it?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400636648
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/ResultPartitionWriter.java
##
@@ -67,6 +68,13 @@
*/
boolean addBufferConsumer(BufferConsumer bufferConsumer, int
subpartitionIndex) throws IOException;
+ default void addPriorityEvent(
+ AbstractEvent event,
+ int subpartitionIndex,
+ BufferConsumer eventBufferConsumer) throws IOException {
+ getSubpartition(subpartitionIndex).priorityEvent(event,
eventBufferConsumer);
Review comment:
We create another path for handling priority event, which might be
inconsistent with previous handling.
In normal `ResultPartition#addBufferConsumer`, we would also check the
`checkInProduceState` before adding into subpartition. Now we bypass this
process which might bring trouble for code maintenance.
Are there any blockers for reuse the previous code path?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi commented on a change in pull request #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink
JingsongLi commented on a change in pull request #11474: FLINK-10114: Add ORC
BulkWriter support for StreamingFileSink
URL: https://github.com/apache/flink/pull/11474#discussion_r400632781
##
File path:
flink-formats/flink-orc-compress/src/main/java/org/apache/flink/formats/orc/vectorizer/Vectorizer.java
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.orc.vectorizer;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch;
+
+import java.io.IOException;
+import java.io.Serializable;
+
+/**
+ * Implementors of this interface provide the logic to transform their data to
{@link VectorizedRowBatch}.
+ *
+ * @param The type of the element
+ */
+@PublicEvolving
+public interface Vectorizer extends Serializable {
+
+ /**
+* Creates a VectorizedRowBatch containing an array of ColumnVectors
+* from the provided element.
+*
+* @param element The input element
+* @return The VectorizedRowBatch containing the ColumnVectors of the
input element
+* @throws IOException if there is an error while transforming the
input.
+*/
+ VectorizedRowBatch vectorize(T element) throws IOException;
Review comment:
This interface looks like one element one batch? Test class
`RecordVectorizer` is this behavior. This behavior is against the ORC design.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi commented on a change in pull request #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink
JingsongLi commented on a change in pull request #11474: FLINK-10114: Add ORC
BulkWriter support for StreamingFileSink
URL: https://github.com/apache/flink/pull/11474#discussion_r400634673
##
File path:
flink-formats/flink-orc-compress/src/test/java/org/apache/flink/formats/orc/writers/OrcBulkWriterTest.java
##
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.orc.writers;
+
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.formats.orc.data.Record;
+import org.apache.flink.formats.orc.vectorizer.RecordVectorizer;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink;
+import org.apache.flink.streaming.api.operators.StreamSink;
+import org.apache.flink.streaming.util.OneInputStreamOperatorTestHarness;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.orc.CompressionKind;
+import org.apache.orc.OrcFile;
+import org.apache.orc.Reader;
+import org.apache.orc.RecordReader;
+import org.apache.orc.TypeDescription;
+import org.apache.orc.storage.ql.exec.vector.BytesColumnVector;
+import org.apache.orc.storage.ql.exec.vector.LongColumnVector;
+import org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch;
+import org.junit.ClassRule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertSame;
+import static org.junit.Assert.assertTrue;
+
+/**
+ * Unit test for the ORC BulkWriter implementation.
+ */
+public class OrcBulkWriterTest {
+
+ @ClassRule
+ public static final TemporaryFolder TEMPORARY_FOLDER = new
TemporaryFolder();
+
+ private final String schema = "struct<_col0:string,_col1:int>";
+ private final List input = Arrays.asList(
+ new Record("Shiv", 44), new Record("Jesse", 23), new
Record("Walt", 50));
+
+ @Test
+ public void testOrcBulkWriter() throws Exception {
+ File outDir = TEMPORARY_FOLDER.newFolder();
+ OrcBulkWriterFactory writer = new
OrcBulkWriterFactory<>(
+ new RecordVectorizer(schema),
TypeDescription.fromString(schema));
+
+ StreamingFileSink sink = StreamingFileSink
+ .forBulkFormat(new Path(outDir.toURI()), writer)
+ .withBucketCheckInterval(1)
+ .build();
+
+ try (OneInputStreamOperatorTestHarness
testHarness = new OneInputStreamOperatorTestHarness<>(
+ new StreamSink<>(sink), 1, 1, 0)) {
+
+ testHarness.setup();
+ testHarness.open();
+
+ int time = 0;
+ for (final Record record : input) {
+ testHarness.processElement(record, ++time);
+ }
+
+ testHarness.snapshot(1, ++time);
+ testHarness.notifyOfCompletedCheckpoint(1);
+
+ validate(outDir);
+ }
+ }
+
+ private void validate(File files) throws IOException {
+ File[] buckets = files.listFiles();
+
+ assertNotNull(buckets);
+ assertEquals(1, buckets.length);
+
+ final File[] partFiles = buckets[0].listFiles();
+
+ assertNotNull(partFiles);
+ assertEquals(1, partFiles.length);
+
+ for (File partFile : partFiles) {
+ assertTrue(partFile.length() > 0);
+
+ OrcFile.ReaderOptions readerOptions =
OrcFile.readerOptions(new Configuration());
Review comment:
Maybe you can implement a `TestOrcInputFormat` for verify.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service,
[GitHub] [flink] JingsongLi commented on a change in pull request #11474: FLINK-10114: Add ORC BulkWriter support for StreamingFileSink
JingsongLi commented on a change in pull request #11474: FLINK-10114: Add ORC
BulkWriter support for StreamingFileSink
URL: https://github.com/apache/flink/pull/11474#discussion_r400635006
##
File path:
flink-formats/flink-orc-compress/src/main/java/org/apache/flink/formats/orc/writers/PhysicalWriterImpl.java
##
@@ -0,0 +1,368 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.formats.orc.writers;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.core.fs.FSDataOutputStream;
+
+import com.google.protobuf25.CodedOutputStream;
+import org.apache.orc.CompressionCodec;
+import org.apache.orc.CompressionKind;
+import org.apache.orc.OrcFile;
+import org.apache.orc.OrcProto;
+import org.apache.orc.PhysicalWriter;
+import org.apache.orc.impl.HadoopShims;
+import org.apache.orc.impl.OrcCodecPool;
+import org.apache.orc.impl.OutStream;
+import org.apache.orc.impl.StreamName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+
+import static org.apache.orc.impl.WriterImpl.getEstimatedBufferSize;
+
+/**
+ * A slightly customised clone of {@link org.apache.orc.impl.PhysicalFsWriter}.
Review comment:
We need add a `NOTE`, every ORC version upgrade, maybe need update this file
to bugfix.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11564: [FLINK-16864] Add IdleTime metric for task.
flinkbot edited a comment on issue #11564: [FLINK-16864] Add IdleTime metric for task. URL: https://github.com/apache/flink/pull/11564#issuecomment-605967084 ## CI report: * 0542c5dc966c56853433d487858711cbba1d9eb3 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156942800) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6855) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11540: [FLINK-16099][docs-zh] Translate "HiveCatalog" page of "Hive Integration" into…
flinkbot edited a comment on issue #11540: [FLINK-16099][docs-zh] Translate "HiveCatalog" page of "Hive Integration" into… URL: https://github.com/apache/flink/pull/11540#issuecomment-604804466 ## CI report: * 31c00a13b2bb7f15b4fa594b5c9148f971769e86 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155736840) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6715) * 07b38deaa1461a92bbc537c40a66915875aee98f Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156949994) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6860) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571#issuecomment-606377768 ## CI report: * 91507d011419afe848d1a90ae0027cd2511817e5 UNKNOWN * fd838b6268e0233589afed81f4f1fd9a0ec77bf9 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156948705) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6859) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest
flinkbot edited a comment on issue #11321: [FLINK-15585] [table] Improve function identifier string in plan digest URL: https://github.com/apache/flink/pull/11321#issuecomment-595208662 ## CI report: * 2a005b9c7d287b338a5a4c0795816d61e8f09312 Travis: [FAILURE](https://travis-ci.com/flink-ci/flink/builds/151951525) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=5970) * 7cebc64ea4a49f0f1d246889cb71b730b4a170eb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16793) Add jobName to log4j ConversionPattern
[
https://issues.apache.org/jira/browse/FLINK-16793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071449#comment-17071449
]
jinhai commented on FLINK-16793:
[~chesnay]
Modify the log4j property files and pass in the additional job.name property
does not work on TM/JM.
> Add jobName to log4j ConversionPattern
> --
>
> Key: FLINK-16793
> URL: https://issues.apache.org/jira/browse/FLINK-16793
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Affects Versions: 1.10.0
>Reporter: jinhai
>Priority: Major
> Fix For: 1.11.0
>
>
> Add jobName to log4j, so we can search log by jobName on ELK.
> We can submit a job by add shell parameter：-yDjob.name=flink-test.
> And add ${job.name} to log4j pattern:
> log4j.appender.sls.layout.ConversionPattern=%d\{-MM-dd HH:mm:ss,SSS}
> ${job.name} %-5p %-60c %x - %m%n
> Then we can seach log by jobName on ELK or Graylog
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16876) Make TtlTimeProvider configurable when creating keyed state backend
[ https://issues.apache.org/jira/browse/FLINK-16876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071450#comment-17071450 ] Yun Tang commented on FLINK-16876: -- [~azagrebin], what do you think of this? Please assign to me if you agree. > Make TtlTimeProvider configurable when creating keyed state backend > --- > > Key: FLINK-16876 > URL: https://issues.apache.org/jira/browse/FLINK-16876 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.10.0 >Reporter: Yun Tang >Priority: Major > Fix For: 1.11.0 > > > Currently, we would always use TtlTimeProvider.DEFAULT to create keyed state > backend. This is somehow acceptable since we only support processing time for > TTL now. However, this would make UT tests which would verify TTL logic not > so convenient like FLINK-16581. > I propose to let TtlTimeProvider configurable when creating keyed state > backend to not block other feature development. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] docete commented on a change in pull request #11493: [FLINK-16160][table-common] Support computed column, proctime and watermark in Schema descriptor
docete commented on a change in pull request #11493:
[FLINK-16160][table-common] Support computed column, proctime and watermark in
Schema descriptor
URL: https://github.com/apache/flink/pull/11493#discussion_r400634261
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/descriptors/Schema.java
##
@@ -183,11 +250,26 @@ public Schema proctime() {
return this;
}
+ /**
+* Specifies the computed column as a processing-time attribute.
+*
+* E.g. proctime("pt")
+*/
+ public Schema proctime(String proctimeAttribute) {
+ addField(proctimeAttribute, "TIMESTAMP(3)", "PROCTIME()");
+ tableSchema.get(proctimeAttribute).put(SCHEMA_PROCTIME, "true");
Review comment:
The old `field(...).proctime()` api use this to identify a proctime field.
We add this for back-compatibility. And we also use this property to
deserialize/serialize TableSchema in DescriptorProperties.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (FLINK-16876) Make TtlTimeProvider configurable when creating keyed state backend
Yun Tang created FLINK-16876: Summary: Make TtlTimeProvider configurable when creating keyed state backend Key: FLINK-16876 URL: https://issues.apache.org/jira/browse/FLINK-16876 Project: Flink Issue Type: Improvement Components: Runtime / State Backends Affects Versions: 1.10.0 Reporter: Yun Tang Fix For: 1.11.0 Currently, we would always use TtlTimeProvider.DEFAULT to create keyed state backend. This is somehow acceptable since we only support processing time for TTL now. However, this would make UT tests which would verify TTL logic not so convenient like FLINK-16581. I propose to let TtlTimeProvider configurable when creating keyed state backend to not block other feature development. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16875) Tracking replace QueryConfig and TableConfig implementation with ConfigOption
jinhai created FLINK-16875:
--
Summary: Tracking replace QueryConfig and TableConfig
implementation with ConfigOption
Key: FLINK-16875
URL: https://issues.apache.org/jira/browse/FLINK-16875
Project: Flink
Issue Type: Improvement
Components: Table SQL / API
Reporter: jinhai
Tracking replace QueryConfig and TableConfig implementation with ConfigOption.
First remove QueryConfig integrated into TableConfig. issue:
https://issues.apache.org/jira/browse/FLINK-13691
Second consider removing {{TableConfig}} and fully rely on a
Configuration-based object with {{ConfigOptions. issue:
https://issues.apache.org/jira/browse/FLINK-16835}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic CheckpointBarrierHandler for unaligned checkpoint
zhijiangW commented on a change in pull request #11507: [FLINK-16587] Add basic
CheckpointBarrierHandler for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11507#discussion_r400632538
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/serialization/SpillingAdaptiveSpanningRecordDeserializer.java
##
@@ -557,6 +579,28 @@ private void addNextChunkFromMemorySegment(MemorySegment
segment, int offset, in
}
}
+ @Nullable
+ MemorySegment copyToTargetSegment() {
+ // for the case of only partial length, no data
+ final int position = lengthBuffer.position();
+ if (position > 0) {
+ MemorySegment segment =
MemorySegmentFactory.allocateUnpooledSegment(lengthBuffer.remaining());
+ segment.put(0, lengthBuffer,
lengthBuffer.remaining());
+ lengthBuffer.position(position);
+ return segment;
+ }
+
+ // for the case of full length, partial data in buffer
+ if (recordLength != -1) {
+ // In the PoC we skip the case of large record
which size exceeds THRESHOLD_FOR_SPILLING,
Review comment:
I guess we also need to consider this case in MVP.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] docete commented on a change in pull request #11493: [FLINK-16160][table-common] Support computed column, proctime and watermark in Schema descriptor
docete commented on a change in pull request #11493:
[FLINK-16160][table-common] Support computed column, proctime and watermark in
Schema descriptor
URL: https://github.com/apache/flink/pull/11493#discussion_r400632587
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/descriptors/Schema.java
##
@@ -85,11 +111,25 @@ public Schema schema(TableSchema schema) {
* @param fieldType the type information of the field
*/
public Schema field(String fieldName, DataType fieldType) {
- addField(fieldName,
fieldType.getLogicalType().asSerializableString());
- return this;
+ return field(fieldName, fieldType, null);
}
+ /**
+* Adds a field with the field name and the data type. Required.
+* This method can be called multiple times. The call order of this
method defines
+* also the order of the fields in a row.
+*
+* @param fieldName the field name
+* @param fieldType the type information of the field
+* @param fieldExpr Computed column expression, it should be a
SQL-style expression whose
+* identifiers should be all quoted and expanded.
+*/
+ public Schema field(String fieldName, DataType fieldType, String
fieldExpr) {
Review comment:
The basic of quoting expression and deriving data type is SqlNode in DDL
code base. I'm afraid it's hard achieve that in descriptor API. Any ideas?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (FLINK-16857) Support partition prune by getPartitions of source
[ https://issues.apache.org/jira/browse/FLINK-16857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee resolved FLINK-16857. -- Resolution: Fixed master: e789bb7165d32b2501f87a071f0f17022d82ac95 > Support partition prune by getPartitions of source > -- > > Key: FLINK-16857 > URL: https://issues.apache.org/jira/browse/FLINK-16857 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Now if a PartitionableTableSource implement the getPartitions, the partition > pruner by planner still go to catalog instead of using this method. > We need use getPartitions when the PartitionableTableSource implement it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on issue #11560: [FLINK-16857][table-planner-blink] Support partition prune by getPartitions of source
JingsongLi commented on issue #11560: [FLINK-16857][table-planner-blink] Support partition prune by getPartitions of source URL: https://github.com/apache/flink/pull/11560#issuecomment-606384898 Merged #11560 to Master. Thanks @leonardBang for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] JingsongLi merged pull request #11560: [FLINK-16857][table-planner-blink] Support partition prune by getPartitions of source
JingsongLi merged pull request #11560: [FLINK-16857][table-planner-blink] Support partition prune by getPartitions of source URL: https://github.com/apache/flink/pull/11560 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] docete commented on a change in pull request #11493: [FLINK-16160][table-common] Support computed column, proctime and watermark in Schema descriptor
docete commented on a change in pull request #11493: [FLINK-16160][table-common] Support computed column, proctime and watermark in Schema descriptor URL: https://github.com/apache/flink/pull/11493#discussion_r400628945 ## File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/descriptors/CustomConnectorDescriptor.java ## @@ -16,18 +16,16 @@ * limitations under the License. */ -package org.apache.flink.table.descriptors.python; +package org.apache.flink.table.descriptors; -import org.apache.flink.annotation.Internal; -import org.apache.flink.table.descriptors.ConnectorDescriptor; -import org.apache.flink.table.descriptors.DescriptorProperties; +import org.apache.flink.annotation.PublicEvolving; import java.util.Map; /** * Describes a custom connector to an other system. */ -@Internal +@PublicEvolving Review comment: There are many user-defined table sources (and related factory). I think we should provide a common Connector Descriptor. Or user must define an extra descriptor for their table source. That's tedious and not user friendly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-14257) Integrate csv to FileSystemTableFactory
[ https://issues.apache.org/jira/browse/FLINK-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-14257: Assignee: Leonard Xu > Integrate csv to FileSystemTableFactory > --- > > Key: FLINK-14257 > URL: https://issues.apache.org/jira/browse/FLINK-14257 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Reporter: Jingsong Lee >Assignee: Leonard Xu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14267) Introduce Row Csv Encoder
[ https://issues.apache.org/jira/browse/FLINK-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-14267: Assignee: Leonard Xu > Introduce Row Csv Encoder > - > > Key: FLINK-14267 > URL: https://issues.apache.org/jira/browse/FLINK-14267 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Reporter: Jingsong Lee >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Now, we have an old CSV, but that is not standard CSV support. we should > support the RFC-compliant CSV format for table/sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14267) Introduce Row Csv Encoder
[ https://issues.apache.org/jira/browse/FLINK-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-14267: - Fix Version/s: 1.11.0 > Introduce Row Csv Encoder > - > > Key: FLINK-14267 > URL: https://issues.apache.org/jira/browse/FLINK-14267 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Reporter: Jingsong Lee >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Now, we have an old CSV, but that is not standard CSV support. we should > support the RFC-compliant CSV format for table/sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14257) Integrate csv to FileSystemTableFactory
[ https://issues.apache.org/jira/browse/FLINK-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-14257: - Fix Version/s: 1.11.0 > Integrate csv to FileSystemTableFactory > --- > > Key: FLINK-14257 > URL: https://issues.apache.org/jira/browse/FLINK-14257 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Reporter: Jingsong Lee >Assignee: Leonard Xu >Priority: Major > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-14267) Introduce Row Csv Encoder
[ https://issues.apache.org/jira/browse/FLINK-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-14267: Assignee: (was: Jingsong Lee) > Introduce Row Csv Encoder > - > > Key: FLINK-14267 > URL: https://issues.apache.org/jira/browse/FLINK-14267 > Project: Flink > Issue Type: Sub-task > Components: Connectors / FileSystem >Reporter: Jingsong Lee >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Now, we have an old CSV, but that is not standard CSV support. we should > support the RFC-compliant CSV format for table/sql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on issue #9884: [FLINK-14266][table] Introduce RowCsvInputFormat to new CSV module
JingsongLi commented on issue #9884: [FLINK-14266][table] Introduce RowCsvInputFormat to new CSV module URL: https://github.com/apache/flink/pull/9884#issuecomment-606383941 Hi @twalthr , do you have other concerns? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16791) Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar
[
https://issues.apache.org/jira/browse/FLINK-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071439#comment-17071439
]
yanxiaobin commented on FLINK-16791:
ok, thanks.
> Could not deploy Yarn job cluster when using flink-s3-fs-hadoop-1.10.0.jar
> ---
>
> Key: FLINK-16791
> URL: https://issues.apache.org/jira/browse/FLINK-16791
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.10.0
>Reporter: yanxiaobin
>Priority: Major
>
> Since aws s3 is needed, so copy the flink-s3-fs-hadoop-1.10.0.jar JAR file to
> the {{lib}} directory of Flink distribution. But when I submit a single flink
> job on yarn mode, I found the following problems：
>
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.yarn.proto.YarnServiceProtos$GetClusterNodesRequestProto
> cannot be cast to com.google.protobuf.Message
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:225)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy11.getClusterNodes(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterNodes(ApplicationClientProtocolPBClientImpl.java:303)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy12.getClusterNodes(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNodeReports(YarnClientImpl.java:564)
> at
> org.apache.flink.yarn.YarnClientYarnClusterInformationRetriever.getMaxVcores(YarnClientYarnClusterInformationRetriever.java:43)
> at
> org.apache.flink.yarn.YarnClusterDescriptor.isReadyForDeployment(YarnClusterDescriptor.java:278)
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:444)
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:390)
> ... 24 more
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11564: [FLINK-16864] Add IdleTime metric for task.
flinkbot edited a comment on issue #11564: [FLINK-16864] Add IdleTime metric for task. URL: https://github.com/apache/flink/pull/11564#issuecomment-605967084 ## CI report: * 0542c5dc966c56853433d487858711cbba1d9eb3 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156942800) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6855) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11540: [FLINK-16099][docs-zh] Translate "HiveCatalog" page of "Hive Integration" into…
flinkbot edited a comment on issue #11540: [FLINK-16099][docs-zh] Translate "HiveCatalog" page of "Hive Integration" into… URL: https://github.com/apache/flink/pull/11540#issuecomment-604804466 ## CI report: * 31c00a13b2bb7f15b4fa594b5c9148f971769e86 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/155736840) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6715) * 07b38deaa1461a92bbc537c40a66915875aee98f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
flinkbot edited a comment on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571#issuecomment-606377768 ## CI report: * 91507d011419afe848d1a90ae0027cd2511817e5 UNKNOWN * fd838b6268e0233589afed81f4f1fd9a0ec77bf9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11534: [FLINK-16537][network] Implement ResultPartition state recovery for unaligned checkpoint
zhijiangW commented on a change in pull request #11534: [FLINK-16537][network]
Implement ResultPartition state recovery for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11534#discussion_r400625877
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -434,6 +434,13 @@ protected void beforeInvoke() throws Exception {
// so that we avoid race conditions in the case that
initializeState()
// registers a timer, that fires before the open() is
called.
operatorChain.initializeStateAndOpenOperators(createStreamTaskStateInitializer());
+
+ ResultPartitionWriter[] writers =
getEnvironment().getAllWriters();
+ if (writers != null) {
+ for (ResultPartitionWriter writer : writers) {
+ writer.initializeState();
+ }
+ }
Review comment:
The reader would also be used for input side for recovery in another PR.
Have you considered when to close it in your PR? The `ChannelStateReader` is a
task-level component and passed into `Environment` to be used anywhere in
theory. Input and output sides are the users for reader ATM, but they should
not be aware whether there are other users. So they should not close it after
using . The proper way might consider its lifecycle together with task's
lifecycle. WDYT?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi commented on a change in pull request #11481: [FLINK-13691][Table]Remove deprecated query config
JingsongLi commented on a change in pull request #11481:
[FLINK-13691][Table]Remove deprecated query config
URL: https://github.com/apache/flink/pull/11481#discussion_r400624159
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/PlannerConfig.java
##
@@ -20,13 +20,14 @@
import org.apache.flink.annotation.PublicEvolving;
+import java.io.Serializable;
import java.util.Optional;
/**
* The {@link PlannerConfig} holds parameters to configure the behavior of
queries.
*/
@PublicEvolving
-public interface PlannerConfig {
+public interface PlannerConfig extends Serializable {
Review comment:
Why do this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] JingsongLi commented on a change in pull request #11481: [FLINK-13691][Table]Remove deprecated query config
JingsongLi commented on a change in pull request #11481:
[FLINK-13691][Table]Remove deprecated query config
URL: https://github.com/apache/flink/pull/11481#discussion_r400624268
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/TableConfig.java
##
@@ -60,7 +61,7 @@
* @see OptimizerConfigOptions
*/
@PublicEvolving
-public class TableConfig {
+public class TableConfig implements Serializable {
Review comment:
Why need `Serializable`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-16874) Respect the dynamic options when calculating memory options in taskmanager.sh
[ https://issues.apache.org/jira/browse/FLINK-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yangze Guo updated FLINK-16874: --- Issue Type: Bug (was: Improvement) > Respect the dynamic options when calculating memory options in taskmanager.sh > - > > Key: FLINK-16874 > URL: https://issues.apache.org/jira/browse/FLINK-16874 > Project: Flink > Issue Type: Bug >Affects Versions: 1.10.0 >Reporter: Yangze Guo >Priority: Blocker > Fix For: 1.10.1, 1.11.0 > > > Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options > to the TaskManagerRunner. However, in FLINK-13983, we calculate the > memory-related configuration only according to the FLINK_CONF_DIR. We then > append the calculation result as dynamic options to the TM, the user-defined > dynamic options would be overridden and ignored. > The BashJavaUtils is already support loading dynamic options, we just need to > pass it in bash script. > cc [~xintongsong] [~azagrebin] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16874) Respect the dynamic options when calculating memory options in taskmanager.sh
[ https://issues.apache.org/jira/browse/FLINK-16874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yangze Guo updated FLINK-16874: --- Description: Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options to the TaskManagerRunner. However, in FLINK-13983, we calculate the memory-related configuration only according to the FLINK_CONF_DIR. We then append the calculation result as dynamic options to the TM, the user-defined dynamic options would be overridden and ignored. The BashJavaUtils is already support loading dynamic options, we just need to pass it in bash script. cc [~xintongsong] [~azagrebin] was: Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options to the TaskManagerRunner. However, in FLINK-13983, we calculate the memory-related configuration only according to the FLINK_CONF_DIR. We then append the calculation result as dynamic options to the TM, the user-defined dynamic options would be overridden and ignored. The BashJavaUtils is already support loading dynamic options, we just need to pass it in bash script. > Respect the dynamic options when calculating memory options in taskmanager.sh > - > > Key: FLINK-16874 > URL: https://issues.apache.org/jira/browse/FLINK-16874 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.10.0 >Reporter: Yangze Guo >Priority: Blocker > Fix For: 1.10.1, 1.11.0 > > > Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options > to the TaskManagerRunner. However, in FLINK-13983, we calculate the > memory-related configuration only according to the FLINK_CONF_DIR. We then > append the calculation result as dynamic options to the TM, the user-defined > dynamic options would be overridden and ignored. > The BashJavaUtils is already support loading dynamic options, we just need to > pass it in bash script. > cc [~xintongsong] [~azagrebin] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-16874) Respect the dynamic options when calculating memory options in taskmanager.sh
Yangze Guo created FLINK-16874: -- Summary: Respect the dynamic options when calculating memory options in taskmanager.sh Key: FLINK-16874 URL: https://issues.apache.org/jira/browse/FLINK-16874 Project: Flink Issue Type: Improvement Affects Versions: 1.10.0 Reporter: Yangze Guo Fix For: 1.10.1, 1.11.0 Since FLINK-9821, the taskmanager.sh will pass user-defined dynamic options to the TaskManagerRunner. However, in FLINK-13983, we calculate the memory-related configuration only according to the FLINK_CONF_DIR. We then append the calculation result as dynamic options to the TM, the user-defined dynamic options would be overridden and ignored. The BashJavaUtils is already support loading dynamic options, we just need to pass it in bash script. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #11534: [FLINK-16537][network] Implement ResultPartition state recovery for unaligned checkpoint
zhijiangW commented on a change in pull request #11534: [FLINK-16537][network] Implement ResultPartition state recovery for unaligned checkpoint URL: https://github.com/apache/flink/pull/11534#discussion_r400623300 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java ## @@ -107,6 +108,8 @@ @Nullable protected final BufferCompressor bufferCompressor; + private final ChannelStateReader channelStateReader; + Review comment: Replied above. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11534: [FLINK-16537][network] Implement ResultPartition state recovery for unaligned checkpoint
zhijiangW commented on a change in pull request #11534: [FLINK-16537][network]
Implement ResultPartition state recovery for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11534#discussion_r400623114
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -89,6 +91,25 @@
super(index, parent);
}
+ @Override
+ public void initializeState() throws IOException, InterruptedException {
+ ReadResult readResult = ReadResult.HAS_MORE_DATA;
+ while (readResult == ReadResult.HAS_MORE_DATA) {
+ BufferBuilder bufferBuilder =
parent.getBufferPool().requestBufferBuilderBlocking();
+ BufferConsumer bufferConsumer =
bufferBuilder.createBufferConsumer();
+ readResult =
parent.getChannelStateReader().readOutputData(subpartitionInfo, bufferBuilder);
+
+ // check whether there are some states data filled in
this time
+ bufferConsumer.update();
Review comment:
> I guess update() was added to read the value that was written in
ChannelStateReader?
Not really. The `update()` is used for checking whether there are any data
written in above `readOutputData` call. And it is actually used together with
below `bufferConsumer.getWrittenBytes()`. There are actually two implicit
limitations here:
- The `ReadResult` from `ChannelStateReader` only indicates the future
situation, not indicates whether the current call actually reads data or not,
especially for the first call. So we have to judge whether the passed
`BufferBuilder` has written any data or not.
- The cached position is only updated after calling `BufferConsumer#build()`
atm. In order to check the written position before constructing the slice
buffer, we have to call `update()` explicitly before calling
`bufferConsumer.getWrittenBytes()`. I ever tried to break this rule to also add
the `update()` inside `BufferConsumer#getWrittenBytes()` and
`BufferConsumer#isFinished()`. But it would bring many unit tests failure and
break previous design of `BufferConsumer` which might bring additional
discussions. So I introduce another separate `update()` from `BufferConsumer`
which can be used by demand.
> Wouldn't creation of bufferConsumer after readOutputData() have the same
effect?
Actually not. The property usage is to to create `BufferConsumer` firstly,
and then written data into `BufferBuilder`. Otherwise the delay creation of
`BufferConsumer` can not see the data written before creation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-13691) Remove deprecated query config
[
https://issues.apache.org/jira/browse/FLINK-13691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071425#comment-17071425
]
Jingsong Lee commented on FLINK-13691:
--
[~csbliss] See FLINK-16835. We can create an issue for tracking replacing
idleStateRetentionTime implementation with {{ConfigOption}}.
> Remove deprecated query config
> --
>
> Key: FLINK-13691
> URL: https://issues.apache.org/jira/browse/FLINK-13691
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: Timo Walther
>Assignee: jinhai
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-03-24-13-57-05-958.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In Flink 1.9 the query config was marked deprecated and integrated into
> TableConfig. We should remove it from the API to reduce complexity and
> confusion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
flinkbot commented on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571#issuecomment-606377768 ## CI report: * 91507d011419afe848d1a90ae0027cd2511817e5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-16873) Document the suggested memory size for statefun docker machine
Yu Li created FLINK-16873: - Summary: Document the suggested memory size for statefun docker machine Key: FLINK-16873 URL: https://issues.apache.org/jira/browse/FLINK-16873 Project: Flink Issue Type: Improvement Components: Documentation, Stateful Functions Affects Versions: statefun-2.0 Reporter: Yu Li Fix For: statefun-2.1 When testing the ride-sharing example for stateful functions 2.0.0 release candidate, I found it will stably fail on my local Mac, with 2GB as the default memory setting in Docker Desktop. After tuning the memory up to 4GB, the issue disappeared. According to this experience, I suggest we explicitly document the recommended memory size for stateful functions docker machine in README. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on a change in pull request #11534: [FLINK-16537][network] Implement ResultPartition state recovery for unaligned checkpoint
zhijiangW commented on a change in pull request #11534: [FLINK-16537][network]
Implement ResultPartition state recovery for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11534#discussion_r400619759
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -89,6 +91,25 @@
super(index, parent);
}
+ @Override
+ public void initializeState() throws IOException, InterruptedException {
+ ReadResult readResult = ReadResult.HAS_MORE_DATA;
+ while (readResult == ReadResult.HAS_MORE_DATA) {
+ BufferBuilder bufferBuilder =
parent.getBufferPool().requestBufferBuilderBlocking();
+ BufferConsumer bufferConsumer =
bufferBuilder.createBufferConsumer();
+ readResult =
parent.getChannelStateReader().readOutputData(subpartitionInfo, bufferBuilder);
Review comment:
I think it is not so necessary to decouple `ResultPartition` with
`ResultSubpartition`, because the `ResultSubpartition` is derived from
`ResultPartition` and it can get everything from parent by design. In the
constructor of `ResultSubpartition`, `ResultPartition` is already passed as
argument and defined as protected variable to be referenced directly by any
kind of subpartitions. So we do not destroy the existing rules or add any new
rules. Also for tests, the separate `PipelinedSubpartitionTest` still needs to
construct parent `ResultPartition` in practice.
But your comment makes me think of another possible option, to pass
`ChannelStateReader` into `ResultPartitionWriter#initializeState`, then we do
not need to pass it into constructor to touch many components.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
flinkbot commented on issue #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571#issuecomment-606374448 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 91507d011419afe848d1a90ae0027cd2511817e5 (Tue Mar 31 03:18:43 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-16710) Log Upload blocks Main Thread in TaskExecutor
[
https://issues.apache.org/jira/browse/FLINK-16710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-16710:
---
Labels: pull-request-available (was: )
> Log Upload blocks Main Thread in TaskExecutor
> -
>
> Key: FLINK-16710
> URL: https://issues.apache.org/jira/browse/FLINK-16710
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Gary Yao
>Assignee: wangsan
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Uploading logs to the BlobServer blocks the TaskExecutor's main thread. We
> should introduce an IO thread pool that carries out file system accesses
> (listing files in a directory, checking if file exists, uploading files).
> Affected RPCs:
> * {{TaskExecutor#requestLogList(Time)}}
> * {{TaskExecutor#requestFileUploadByName(String, Time)}}
> * {{TaskExecutor#requestFileUploadByType(FileType, Time)}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] jrthe42 opened a new pull request #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor
jrthe42 opened a new pull request #11571: [FLINK-16710][runtime] Log Upload blocks Main Thread in TaskExecutor URL: https://github.com/apache/flink/pull/11571 ## What is the purpose of the change TaskExecutors upload logs to BlobServer in main thread, which may cause main thread blocked. This pr intraduce a separate IO thread pool to carries out file system accesses (listing files in a directory, checking if file exists, uploading files). ## Brief change log - *TaskExecutors upload logs to BlobServer using a separate IO thread pool* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11534: [FLINK-16537][network] Implement ResultPartition state recovery for unaligned checkpoint
zhijiangW commented on a change in pull request #11534: [FLINK-16537][network]
Implement ResultPartition state recovery for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11534#discussion_r400614992
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -89,6 +91,25 @@
super(index, parent);
}
+ @Override
+ public void initializeState() throws IOException, InterruptedException {
+ ReadResult readResult = ReadResult.HAS_MORE_DATA;
Review comment:
> we can get rid of this variable using for loop
I guess you mean the way via `while(true)` and break the loop inside
`while`. I also tried that way in first version, but it seems not very
suggested by intelliJ.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] zhijiangW commented on a change in pull request #11534: [FLINK-16537][network] Implement ResultPartition state recovery for unaligned checkpoint
zhijiangW commented on a change in pull request #11534: [FLINK-16537][network]
Implement ResultPartition state recovery for unaligned checkpoint
URL: https://github.com/apache/flink/pull/11534#discussion_r400614992
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedSubpartition.java
##
@@ -89,6 +91,25 @@
super(index, parent);
}
+ @Override
+ public void initializeState() throws IOException, InterruptedException {
+ ReadResult readResult = ReadResult.HAS_MORE_DATA;
Review comment:
> we can get rid of this variable using for loop
I guess you mean the way via `while(true)` and break the loop inside
`while`. I also tried that way in first version, but it seems not very
suggested by intelliJ.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-16865) 【Flink Kafka Connector】Restore from Savepoint，if add new Kafka topic，Flink will consume the new topic from earlist，it may cause duplicate data sink
[
https://issues.apache.org/jira/browse/FLINK-16865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071418#comment-17071418
]
zhisheng commented on FLINK-16865:
--
hi, [~wind_ljy] [~kangzai]
This is how I solve the problem for the time being
{code:java}
ExecutionConfig.GlobalJobParameters globalJobParameters =
getRuntimeContext().getExecutionConfig().getGlobalJobParameters();
String xxx = globalJobParameters.toMap().get("xxx");
if ("group".equals(xxx)) {
restoredState.put(partition, KafkaTopicPartitionStateSentinel.GROUP_OFFSET);
} else if ("latest".equals(xxx)) {
restoredState.put(partition, KafkaTopicPartitionStateSentinel.LATEST_OFFSET);
} else {
restoredState.put(partition,
KafkaTopicPartitionStateSentinel.EARLIEST_OFFSET);
}
{code}
If user want to consume the topic from latest, then he can set the config(eg:
xxx) in his job.
{code:java}
Configuration configuration = new Configuration();
configuration.setBoolean("xxx", "latest");
env.getConfig().setGlobalJobParameters(configuration);
{code}
this config can not affect the logic of the original Flink code.
I'm not sure whether this is a good way or not. I'm willing to hear your
opinions.
> 【Flink Kafka Connector】Restore from Savepoint，if add new Kafka topic，Flink
> will consume the new topic from earlist，it may cause duplicate data sink
> ---
>
> Key: FLINK-16865
> URL: https://issues.apache.org/jira/browse/FLINK-16865
> Project: Flink
> Issue Type: Bug
>Reporter: zhisheng
>Priority: Major
> Attachments: image-2020-03-30-19-57-42-451.png
>
>
> h3. 【Flink Kafka Connector】
> If the job adds another Kafka topic when the job start from Savepoint, it
> will start to consume from the earlist of that topic, it may cause duplicate
> data sink.
> I found that the configuration is already written in the code
> FlinkKafkaConsumerBase#open(), maybe it can be Configurable.
>
> !image-2020-03-30-19-57-42-451.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11564: [FLINK-16864] Add IdleTime metric for task.
flinkbot edited a comment on issue #11564: [FLINK-16864] Add IdleTime metric for task. URL: https://github.com/apache/flink/pull/11564#issuecomment-605967084 ## CI report: * 623248594914ebf2dab46965390f72ba7f3c1346 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/156553465) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6832) * 0542c5dc966c56853433d487858711cbba1d9eb3 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156942800) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6855) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-16673) Support Counter, Gauge, Meter, Distribution metric type for Python UDTF
[ https://issues.apache.org/jira/browse/FLINK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-16673. --- Resolution: Resolved Merged to master via ee1e83ceb176cd5700b07f7a26e23b082d8f38fd > Support Counter, Gauge, Meter, Distribution metric type for Python UDTF > --- > > Key: FLINK-16673 > URL: https://issues.apache.org/jira/browse/FLINK-16673 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Support Counter, Gauge, Meter, Distribution metric type for Python UDTF -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #11564: [FLINK-16864] Add IdleTime metric for task.
flinkbot edited a comment on issue #11564: [FLINK-16864] Add IdleTime metric for task. URL: https://github.com/apache/flink/pull/11564#issuecomment-605967084 ## CI report: * 623248594914ebf2dab46965390f72ba7f3c1346 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/156553465) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6832) * 0542c5dc966c56853433d487858711cbba1d9eb3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dianfu merged pull request #11566: [FLINK-16673][python] Support metrics for Python UDTF
dianfu merged pull request #11566: [FLINK-16673][python] Support metrics for Python UDTF URL: https://github.com/apache/flink/pull/11566 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-16796) Fix The Bug of Python UDTF in SQL Query
[ https://issues.apache.org/jira/browse/FLINK-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hequn Cheng closed FLINK-16796. --- Resolution: Fixed > Fix The Bug of Python UDTF in SQL Query > --- > > Key: FLINK-16796 > URL: https://issues.apache.org/jira/browse/FLINK-16796 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When executes Python UDTF in sql query, it will cause some problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16796) Fix The Bug of Python UDTF in SQL Query
[ https://issues.apache.org/jira/browse/FLINK-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071408#comment-17071408 ] Hequn Cheng commented on FLINK-16796: - Fixed in 1.11.0 via 52315e79b439eda4f6cd7836051c1791382a4a7d > Fix The Bug of Python UDTF in SQL Query > --- > > Key: FLINK-16796 > URL: https://issues.apache.org/jira/browse/FLINK-16796 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > > When executes Python UDTF in sql query, it will cause some problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] hequn8128 merged pull request #11521: [FLINK-16796][python] Fix The Bug of Python UDTF in SQL Query
hequn8128 merged pull request #11521: [FLINK-16796][python] Fix The Bug of Python UDTF in SQL Query URL: https://github.com/apache/flink/pull/11521 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (FLINK-16796) Fix The Bug of Python UDTF in SQL Query
[ https://issues.apache.org/jira/browse/FLINK-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hequn Cheng reassigned FLINK-16796: --- Assignee: Huang Xingbo > Fix The Bug of Python UDTF in SQL Query > --- > > Key: FLINK-16796 > URL: https://issues.apache.org/jira/browse/FLINK-16796 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 10m > Remaining Estimate: 0h > > When executes Python UDTF in sql query, it will cause some problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15064) Remove XmlOutput util class in blink planner
[
https://issues.apache.org/jira/browse/FLINK-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-15064:
Fix Version/s: 1.11.0
> Remove XmlOutput util class in blink planner
>
>
> Key: FLINK-15064
> URL: https://issues.apache.org/jira/browse/FLINK-15064
> Project: Flink
> Issue Type: Task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.11.0
>
>
> {{org.apache.flink.table.planner.utils.XmlOutput}} is introduced in blink
> planner to fix a redundant line break for each CDATA section. This has been
> fixed in Calcite via 1745f752561be04ae34d1fa08593c2d3ba4470e8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15064) Remove XmlOutput util class in blink planner
[
https://issues.apache.org/jira/browse/FLINK-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-15064:
Parent: (was: FLINK-14338)
Issue Type: Task (was: Sub-task)
> Remove XmlOutput util class in blink planner
>
>
> Key: FLINK-15064
> URL: https://issues.apache.org/jira/browse/FLINK-15064
> Project: Flink
> Issue Type: Task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
>
> {{org.apache.flink.table.planner.utils.XmlOutput}} is introduced in blink
> planner to fix a redundant line break for each CDATA section. This has been
> fixed in Calcite via 1745f752561be04ae34d1fa08593c2d3ba4470e8.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-16847) Support timestamp types in vectorized Python UDF
[ https://issues.apache.org/jira/browse/FLINK-16847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hequn Cheng closed FLINK-16847. --- Resolution: Resolved > Support timestamp types in vectorized Python UDF > > > Key: FLINK-16847 > URL: https://issues.apache.org/jira/browse/FLINK-16847 > Project: Flink > Issue Type: Task > Components: API / Python >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16847) Support timestamp types in vectorized Python UDF
[ https://issues.apache.org/jira/browse/FLINK-16847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071406#comment-17071406 ] Hequn Cheng commented on FLINK-16847: - Resolved in 1.11.0 via 851a830536c7811df7f207bc47dca851be17f56b > Support timestamp types in vectorized Python UDF > > > Key: FLINK-16847 > URL: https://issues.apache.org/jira/browse/FLINK-16847 > Project: Flink > Issue Type: Task > Components: API / Python >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15210) Move java files in flink-sql-parser module package org.apache.calcite.sql to org.apache.flink.sql.parser.type
[ https://issues.apache.org/jira/browse/FLINK-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-15210. --- Fix Version/s: 1.11.0 Resolution: Fixed This has been fixed in FLINK-14338. > Move java files in flink-sql-parser module package org.apache.calcite.sql to > org.apache.flink.sql.parser.type > - > > Key: FLINK-15210 > URL: https://issues.apache.org/jira/browse/FLINK-15210 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.10.0 >Reporter: Danny Chen >Priority: Major > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-15282) Remove ParameterScope in blink planner
[
https://issues.apache.org/jira/browse/FLINK-15282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-15282.
---
Fix Version/s: 1.11.0
Resolution: Fixed
This has been fixed in FLINK-14338.
> Remove ParameterScope in blink planner
> --
>
> Key: FLINK-15282
> URL: https://issues.apache.org/jira/browse/FLINK-15282
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.11.0
>
>
> Remove {{org.apache.calcite.sql.validate.ParameterScope}} class in blink
> planner. This class is introduced because of CALCITE-3476.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-14338) Upgrade Calcite version to 1.22 for Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu resolved FLINK-14338. - Fix Version/s: 1.11.0 Assignee: Danny Chen Resolution: Fixed [FLINK-14338][table-planner][table-planner-blink] Update all kinds of left plan changes - Fixed in master (1.11.0): fabff4f38a56b29400b644d370e99cb8a398e8bb [FLINK-14338][table-planner-blink] Update files due to CALCITE-1824 - Fixed in master (1.11.0): 5fef3c1272e5be6ca20ed83f87c147a2d18379b0 [FLINK-14338][table-planner][table-planner-blink] Update files due to CALCITE-3763 - Fixed in master (1.11.0): 8a6877d14eb8f15c9f5124e400f2f3e351f42c13 [FLINK-14338][table-planner][table-planner-blink] Update files due to builtin TUMBLE operator name changes to $Tumble - Fixed in master (1.11.0): 333885a5014bdd09bfe8ea6decfcd2c785c7603c [FLINK-14338][table-planner-blink] Plan verify changes from DIGEST to EXPLAIN: change plan for values node - Fixed in master (1.11.0): d5f45c3539835f8c2dda4007eb5f69091a9913d1 [FLINK-14338][table-planner-blink] Plan verify changes from DIGEST to EXPLAIN: change plan for expand node - Fixed in master (1.11.0): c41887df9e92572ef4c9b3a0884b1df7e01d7299 [FLINK-14338][table-planner-blink] Plan verify changes from DIGEST to EXPLAIN - Fixed in master (1.11.0): 06489d1982e83e4890f69cfa3b492def38bb3dfd [FLINK-14338][table-planner][table-planner-blink] Remove redundant code copy because the bugs are already fixed in Calcite - Fixed in master (1.11.0): 10c8392b9041d636f194b80492afc492eae806b7 [FLINK-14338][table-planner][table-planner-blink] Tweak implementations due to API change - Fixed in master (1.11.0): 7dad5c47c92db305dc39f58acb1f8cf88b6eaed1 [FLINK-14338][table-planner][table-planner-blink] Remove usage of TableScanRule and use new TableScanFactory extension - Fixed in master (1.11.0): ac972cd78da4ef1b972dbd2b9afc39e774a2f6fe [FLINK-14338][table-planner][table-planner-blink] Implements new RelMetadataQuery extension - Fixed in master (1.11.0): 07dec981ccb368f64a86181ffba7b48b6409621c [FLINK-14338][sql-parser] Bump sql parser Calcite dependency to 1.22.0 - Fixed in master (1.11.0): 78499853bff8eccc2b174a883a7cfdc4125eb499 [FLINK-14338][flink-table] Update flink table POM and NOTICE file Calcite dependency to 1.22.0 - Fixed in master (1.11.0): e9315155fb8ae916bc4f0bd3173c249698d6a588 > Upgrade Calcite version to 1.22 for Flink SQL > - > > Key: FLINK-14338 > URL: https://issues.apache.org/jira/browse/FLINK-14338 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.10.0 >Reporter: Danny Chen >Assignee: Danny Chen >Priority: Major > Fix For: 1.11.0 > > > Since FLINK-14092 has been resolved, now we can prepare to upgrade to Calcite > 1.22. > There are some fix code introduced by 1.21 upgrade: > * Move java files in flink-sql-parser module package org.apache.calcite.sql > to org.apache.flink.sql.parser.type, since CALCITE-3360 has been resolved > * Remove SqlFunction from flink-table-planner and flink-table-planner-blink > since CALCITE-3360 has been resolved -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #11340: [Flink 14338] Upgrade Calcite version to 1.22 for Flink SQL
wuchong closed pull request #11340: [Flink 14338] Upgrade Calcite version to 1.22 for Flink SQL URL: https://github.com/apache/flink/pull/11340 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on issue #11340: [Flink 14338] Upgrade Calcite version to 1.22 for Flink SQL
wuchong commented on issue #11340: [Flink 14338] Upgrade Calcite version to 1.22 for Flink SQL URL: https://github.com/apache/flink/pull/11340#issuecomment-606354511 Thanks @danny0405 for the great effort, and thanks all for the reviewing. I will merge this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] PatrickRen commented on issue #11425: [FLINK-16125][kafka] Remove unnecessary zookeeper.connect property validation
PatrickRen commented on issue #11425: [FLINK-16125][kafka] Remove unnecessary zookeeper.connect property validation URL: https://github.com/apache/flink/pull/11425#issuecomment-606352442 Hi Becket @becketqin and Jark @wuchong , can you help to review the code? I think it's in good shape now. Travis and Azure tests have passed. Thank you~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhengcanbin commented on issue #11569: [FLINK-16737][k8s] Remove KubernetesUtils#getContentFromFile
zhengcanbin commented on issue #11569: [FLINK-16737][k8s] Remove KubernetesUtils#getContentFromFile URL: https://github.com/apache/flink/pull/11569#issuecomment-606350461 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] hequn8128 merged pull request #11556: [FLINK-16847][python] Support timestamp types in vectorized Python UDF
hequn8128 merged pull request #11556: [FLINK-16847][python] Support timestamp types in vectorized Python UDF URL: https://github.com/apache/flink/pull/11556 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-16870) OrcTableSource throws NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071371#comment-17071371
]
Kurt Young commented on FLINK-16870:
I think this is the same with FLINK-16646, could you check whether the fix
could solve your issue?
> OrcTableSource throws NullPointerException
> --
>
> Key: FLINK-16870
> URL: https://issues.apache.org/jira/browse/FLINK-16870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ORC
>Affects Versions: 1.10.0
> Environment: flink 1.10
>Reporter: Nikola
>Priority: Critical
> Attachments: flink-1.10-minimal-example.txt,
> flink-1.10-orc-exception.log
>
>
> I am trying to read some ORC data from HDFS as given the example here:
> [https://ci.apache.org/projects/flink/flink-docs-release-1.10/api/java/org/apache/flink/orc/OrcTableSource.html]
> When I try to do this, the job crashes with NullPointerException:
>
> {code:java}
> Caused by: java.lang.NullPointerException at
> org.apache.flink.orc.shim.OrcShimV200.computeProjectionMask(OrcShimV200.java:188)
> at
> org.apache.flink.orc.shim.OrcShimV200.createRecordReader(OrcShimV200.java:120)
> {code}
>
> I have attached a minimal version of code which can reproduce the issue. The
> same piece of code (and more complex) runs fine on flink 1.8.2
> I have tried to look what is causing it and it seems that the
> NullPointerException happens on this line:
> [https://github.com/apache/flink/blob/release-1.10/flink-formats/flink-orc/src/main/java/org/apache/flink/orc/shim/OrcShimV200.java#L188
> ]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-16728) Taskmanager dies after job got stuck and canceling fails
[ https://issues.apache.org/jira/browse/FLINK-16728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17071367#comment-17071367 ] Leonid Ilyevsky commented on FLINK-16728: - Hi [~zhuzh], thanks for the explanation. Of course, that job had its own problem and it got stuck. However, in my specific scenario I would really prefer somewhat different behavior. Here is what happened. I knew this job had a problem, and I tried to cancel it; I did not want automatic recovery in this case. I actually managed to cancel it, but with it it brought down two taskmanagers (out of five) where this job was running. Those taskmanagers contained other jobs, and suddenly there were not enough available slots to run all the jobs. So maybe it is possible to optionally provide behavior like this: if the job is being canceled, even in case of timeout, just cancel the job and clean up all resources associated with it, but keep taskmanager up. What do you think? > Taskmanager dies after job got stuck and canceling fails > > > Key: FLINK-16728 > URL: https://issues.apache.org/jira/browse/FLINK-16728 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.10.0 >Reporter: Leonid Ilyevsky >Priority: Major > Attachments: taskmanager.log.20200323.gz > > > At some point I noticed that a few jobs got stuck (they basically stopped > processing the messages, I could detect this watching the expected output), > so I tried to cancel them. > The cancel operation failed, complaining that the job got stuck at > StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.run(StreamTaskActionExecutor.java:86) > and then the whole taskmanager shut down. > See the attached log. > This is actually happening practically every day in our staging environment > where we are testing Flink 1.10.0. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] faaronzheng commented on a change in pull request #11356: [FLINK-15000] [web] WebUI metrics is very slow in large parallelism
faaronzheng commented on a change in pull request #11356: [FLINK-15000] [web]
WebUI metrics is very slow in large parallelism
URL: https://github.com/apache/flink/pull/11356#discussion_r399729007
##
File path:
flink-runtime-web/web-dashboard/src/app/pages/job/overview/chart/job-overview-drawer-chart.component.ts
##
@@ -64,15 +71,35 @@ export class JobOverviewDrawerChartComponent implements
OnInit, OnDestroy {
closeMetric(metric: string) {
this.listOfSelectedMetric = this.listOfSelectedMetric.filter(item => item
!== metric);
this.jobService.metricsCacheMap.set(this.cacheMetricKey,
this.listOfSelectedMetric);
+this.showList = [metric,...this.showList];
this.updateUnselectedMetricList();
}
updateUnselectedMetricList() {
this.listOfUnselectedMetric = this.listOfMetricName.filter(item =>
this.listOfSelectedMetric.indexOf(item) === -1);
+this.showList = this.showList.filter(item =>
this.listOfSelectedMetric.indexOf(item) === -1);
+this.optionList = this.listOfUnselectedMetric;
}
constructor(private metricsService: MetricsService, private jobService:
JobService, private cdr: ChangeDetectorRef) {}
+ nzOnSearch(val: string){
+this.showListPageNum=0;
Review comment:
Hi @vthinkxie
thanks, the code is already formatted by pre-commit hooks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
flinkbot edited a comment on issue #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints URL: https://github.com/apache/flink/pull/11515#issuecomment-604128992 ## CI report: * 22e12ad32ac71e45f7c40fd08ad0cc4498664613 UNKNOWN * da674cafad1c62dde92588b94b8d44ab699b8280 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156902747) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6851) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11491: [FLINK-16513][checkpointing] Unaligned checkpoints: checkpoint metadata
flinkbot edited a comment on issue #11491: [FLINK-16513][checkpointing] Unaligned checkpoints: checkpoint metadata URL: https://github.com/apache/flink/pull/11491#issuecomment-602805203 ## CI report: * b752ebcec0c6cfba091790353b76ed71597ed335 UNKNOWN * 5045204ca09a49c8f392eed817183d8e8a3fba55 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156902689) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6850) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol commented on issue #11303: [FLINK-16245] Decoupling user classloader from context classloader.
zentol commented on issue #11303: [FLINK-16245] Decoupling user classloader from context classloader. URL: https://github.com/apache/flink/pull/11303#issuecomment-606292140 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
flinkbot edited a comment on issue #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints URL: https://github.com/apache/flink/pull/11515#issuecomment-604128992 ## CI report: * 22e12ad32ac71e45f7c40fd08ad0cc4498664613 UNKNOWN * da674cafad1c62dde92588b94b8d44ab699b8280 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156902747) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6851) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11518: [FLINK-16555] Block Enum as key type
flinkbot edited a comment on issue #11518: [FLINK-16555] Block Enum as key type URL: https://github.com/apache/flink/pull/11518#issuecomment-604351209 ## CI report: * 53d3a2d0b005747e588c7ad552bb1d29ff2eee24 Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/156885405) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6849) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints
flinkbot edited a comment on issue #11515: [FLINK-16744][task] implement channel state persistence for unaligned checkpoints URL: https://github.com/apache/flink/pull/11515#issuecomment-604128992 ## CI report: * 22e12ad32ac71e45f7c40fd08ad0cc4498664613 UNKNOWN * 85c5d0044d8c8c53abc57e4e24383505462a48ae Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/156864511) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6847) * da674cafad1c62dde92588b94b8d44ab699b8280 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156902747) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6851) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #11491: [FLINK-16513][checkpointing] Unaligned checkpoints: checkpoint metadata
flinkbot edited a comment on issue #11491: [FLINK-16513][checkpointing] Unaligned checkpoints: checkpoint metadata URL: https://github.com/apache/flink/pull/11491#issuecomment-602805203 ## CI report: * b7eb8c4e86489fb9d9f8c455b21d7e3adb3d49f2 Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/155668534) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6697) * b752ebcec0c6cfba091790353b76ed71597ed335 UNKNOWN * 5045204ca09a49c8f392eed817183d8e8a3fba55 Travis: [PENDING](https://travis-ci.com/github/flink-ci/flink/builds/156902689) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=6850) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
