[GitHub] [flink] liming30 commented on pull request #13109: [FLINK-18808][runtime/metrics] Include side outputs in numRecordsOut metric.
liming30 commented on pull request #13109: URL: https://github.com/apache/flink/pull/13109#issuecomment-687535459 Hi, @pnowojski I have rebase the master and reorganized the commit. > Could you restructure first three commits in the following way ([following our coding style](https://flink.apache.org/contributing/code-style-and-quality-pull-requests.html#separate-refactoring-cleanup-and-independent-changes)): > > 1. my `[hotfix][task] Move output and collector helper classes out of OperatorChain` commit > The first commit has been merged into the master, so it is removed from this pr. > 2. your introduction of `SelectedOutputsCollector` - a hotfix/optimisation of the pre-existing code > 3. your functional changes to the metrics counting (current 1st and part of the current 3rd commit) > 4. your `[hotfix] wrapping single RecordWriterOutput with RecordWriterCountingOutput` > Only two commits are retained. The first commit is the introduction of `SelectedOutputsCollector`, and the second commit is a modification of the metrics counting function. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
flinkbot edited a comment on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-672766836 ## CI report: * 313b80f4474455d7013b0852929b5a8458f391a1 UNKNOWN * ece578a9fdbaee4d815de501187d92a729790c9b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6174) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6196) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
flinkbot edited a comment on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-672766836 ## CI report: * 313b80f4474455d7013b0852929b5a8458f391a1 UNKNOWN * ece578a9fdbaee4d815de501187d92a729790c9b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6196) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6174) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] miklosgergely commented on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
miklosgergely commented on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-687446541 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13217: [FLINK-16866] Make job submission non-blocking
flinkbot edited a comment on pull request #13217: URL: https://github.com/apache/flink/pull/13217#issuecomment-678285884 ## CI report: * 3655fcea1966bfbcb85c86d6a159c354f20d6cc7 UNKNOWN * 4d4d6690d8ad282f71e5f3d890e28421aef17766 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6223) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18815) AbstractCloseableRegistryTest.testClose unstable
[
https://issues.apache.org/jira/browse/FLINK-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190870#comment-17190870
]
Kezhu Wang commented on FLINK-18815:
[~trohrmann] Your are right, I made mistake. After revise, I found
{{AbstractCloseableRegistryTest.testClose}} itself is faulty for
{{SafetyNetCloseableRegistry}}. The failed assertion could happens before
{{innerCloseable.close}}. This could be easily verified by place endless loop
before {{innerCloseable.close}}.
I think we can fix this with following changes:
1. Join {{CloseableReaperThread}} before assertion for
{{SafetyNetCloseableRegistryTest}}. This should fix recently failed cases.
2. Replace {{CloseableReaperThread.interrupt}} with
{{CloseableReaperThread.stop}}, eg. set {{CloseableReaperThread.running}} to
false without interrupting {{CloseableReaperThread}}. Replace
{{CloseableReaperThread.referenceQueue.remove()}} with
{{CloseableReaperThread.referenceQueue.remove(long timeout)}}. This prevent
blocking {{Closeable.close}} from interruption, though I think it is not a must
since {{Closeable.close}} are not allowed to throw {{InterruptedException}} in
spec. It is a friendly improvement for registered {{Closeable}}.
> AbstractCloseableRegistryTest.testClose unstable
>
>
> Key: FLINK-18815
> URL: https://issues.apache.org/jira/browse/FLINK-18815
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Affects Versions: 1.10.1, 1.12.0, 1.11.1
>Reporter: Robert Metzger
>Assignee: Kezhu Wang
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.10.2, 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5164=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=05b74a19-4ee4-5036-c46f-ada307df6cf0
> {code}
> [ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.509
> s <<< FAILURE! - in org.apache.flink.core.fs.SafetyNetCloseableRegistryTest
> [ERROR] testClose(org.apache.flink.core.fs.SafetyNetCloseableRegistryTest)
> Time elapsed: 1.15 s <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:<-1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.flink.core.fs.AbstractCloseableRegistryTest.testClose(AbstractCloseableRegistryTest.java:93)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13109: [FLINK-18808][runtime/metrics] Include side outputs in numRecordsOut metric.
flinkbot edited a comment on pull request #13109: URL: https://github.com/apache/flink/pull/13109#issuecomment-671381707 ## CI report: * b43e9eda0e94696b0563922ecb37b202bca47af5 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6222) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
flinkbot edited a comment on pull request #13181: URL: https://github.com/apache/flink/pull/13181#issuecomment-675091412 ## CI report: * e7553689356882de1ffe606400d1255d1d757bc4 UNKNOWN * db1607ead33ae1b60634c1162ba8247a9957f0f4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6219) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13315: [FLINK-19070][hive] Hive connector should throw a meaningful exception if user reads/writes ACID tables
flinkbot edited a comment on pull request #13315: URL: https://github.com/apache/flink/pull/13315#issuecomment-686398823 ## CI report: * 2ed647fc736a248d536d6f5422e0a50a7119045f UNKNOWN * 6f796f529a1dc3aed87785eaeb251725c4dd17c2 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6221) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuxiaoshang removed a comment on pull request #13329: [hot fix]fix typo in the java doc
zhuxiaoshang removed a comment on pull request #13329: URL: https://github.com/apache/flink/pull/13329#issuecomment-687272049 @libenchao @wuchong This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuxiaoshang commented on pull request #13329: [hot fix]fix typo in the java doc
zhuxiaoshang commented on pull request #13329: URL: https://github.com/apache/flink/pull/13329#issuecomment-687272049 @libenchao @wuchong This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * 273cc6bf445b4a44daca0e98d49aa0745a36a534 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6225) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * 67942d0ea538985ec25cecf7cefdc1bec0c1b53b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5833) * 273cc6bf445b4a44daca0e98d49aa0745a36a534 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] asfgit closed pull request #13241: [hotfix] Fix a typo in the watermark docs
asfgit closed pull request #13241: URL: https://github.com/apache/flink/pull/13241 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #13270: [hotfix] [javadocs] fix typo in TaskManagerServices
rmetzger commented on pull request #13270: URL: https://github.com/apache/flink/pull/13270#issuecomment-687234365 I agree, this is not a typo. Closing ticket... Please reopen if you disagree. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger closed pull request #13218: remove unnecessary integer boxing
rmetzger closed pull request #13218: URL: https://github.com/apache/flink/pull/13218 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #13218: remove unnecessary integer boxing
rmetzger commented on pull request #13218: URL: https://github.com/apache/flink/pull/13218#issuecomment-687234116 Closing ticket... Please reopen if you disagree. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] asfgit closed pull request #13324: [hotfix][docs] Fix ExecutionEnvironment.scala doc error
asfgit closed pull request #13324: URL: https://github.com/apache/flink/pull/13324 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] asfgit closed pull request #13329: [hot fix]fix typo in the java doc
asfgit closed pull request #13329: URL: https://github.com/apache/flink/pull/13329 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] asfgit closed pull request #12005: [hotfix][runtime] Remove never thrown IOException in ContinuousFileMnitoringFunction
asfgit closed pull request #12005: URL: https://github.com/apache/flink/pull/12005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13216: [FLINK-18999][table-planner-blink][hive] Temporary generic table does…
flinkbot edited a comment on pull request #13216: URL: https://github.com/apache/flink/pull/13216#issuecomment-678268420 ## CI report: * 167e78ad9d9f8fdb79ce5b2483feed7f120456a3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6215) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #13320: [FLINK-19035] Remove fold from DataStream API
gaoyunhaii commented on a change in pull request #13320:
URL: https://github.com/apache/flink/pull/13320#discussion_r483691287
##
File path: docs/dev/stream/state/queryable_state.md
##
@@ -155,7 +150,7 @@ descriptor.setQueryable("query-name"); // queryable state
name
This variant has no limitations as to which type of state can be made
queryable. This means that this can be used for
-any `ValueState`, `ReduceState`, `ListState`, `MapState`, `AggregatingState`,
and the currently deprecated `FoldingState`.
+any `ValueState`, `ReduceState`, `ListState`, `MapState`, and
`AggregatingState`.
Review comment:
This and the next one should also be removed from the Chinese version.
But I think I could help remove it separately after this PR.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-10407) Reactive container mode
[ https://issues.apache.org/jira/browse/FLINK-10407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190775#comment-17190775 ] Antonio Verardi commented on FLINK-10407: - Hi [~trohrmann], I have seen Reactive Scaling in the list for version 1.12 [https://cwiki.apache.org/confluence/display/FLINK/1.12+Release] Is that likely to happen? This ticket haven't seen much progress lately and I don't see the version label, that's why I am asking :P > Reactive container mode > --- > > Key: FLINK-10407 > URL: https://issues.apache.org/jira/browse/FLINK-10407 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Affects Versions: 1.7.0 >Reporter: Till Rohrmann >Priority: Major > > The reactive container mode is a new operation mode where a Flink cluster > will react to newly available resources (e.g. started by an external service) > and make use of them by rescaling the existing job. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19089) Improvement of ClosableBlockingQueue, replace Lock with ReadWriteLock, replace signalAll with signal
[ https://issues.apache.org/jira/browse/FLINK-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dugenkui updated FLINK-19089: - Description: 1. Replace ReentrantLock with ReentrantReadWriteLock to improve concurrency; 2. Use signal instead of signalAll to reduce the thread scheduling. was: 1. Replace ReentrantLock with ReentrantReadWriteLock to improve concurrency; 2. Use signal instead of signalAll to reduce the thread scheduling. > Improvement of ClosableBlockingQueue, replace Lock with ReadWriteLock, > replace signalAll with signal > > > Key: FLINK-19089 > URL: https://issues.apache.org/jira/browse/FLINK-19089 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: dugenkui >Priority: Major > Labels: pull-request-available > > 1. Replace ReentrantLock with ReentrantReadWriteLock to improve concurrency; > 2. Use signal instead of signalAll to reduce the thread scheduling. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13330: [FLINK-19004][table-planner-blink] Use FIRST_VALUE when re-write non-…
flinkbot edited a comment on pull request #13330: URL: https://github.com/apache/flink/pull/13330#issuecomment-687094698 ## CI report: * dda51e888c3f58e83346118f03ad55075e990eb8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6216) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13217: [FLINK-16866] Make job submission non-blocking
flinkbot edited a comment on pull request #13217: URL: https://github.com/apache/flink/pull/13217#issuecomment-678285884 ## CI report: * 3655fcea1966bfbcb85c86d6a159c354f20d6cc7 UNKNOWN * 8036016c752bce433dc65d1c08695377c917836f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6162) * 4d4d6690d8ad282f71e5f3d890e28421aef17766 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6223) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
tillrohrmann commented on a change in pull request #13181:
URL: https://github.com/apache/flink/pull/13181#discussion_r483678060
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequestBulkCheckerImpl.java
##
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Default implementation of {@link PhysicalSlotRequestBulkChecker}.
+ */
+public class PhysicalSlotRequestBulkCheckerImpl implements
PhysicalSlotRequestBulkChecker {
+
+ private ComponentMainThreadExecutor componentMainThreadExecutor;
Review comment:
I'd initialize this field with a `DummyComponentMainThreadExecutor` to
ensure that `start` is called before any other action is called on it.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SharedSlot.java
##
@@ -38,6 +38,26 @@
import java.util.function.Consumer;
import java.util.function.Function;
+/**
+ * Shared slot implementation for the {@link
SlotSharingExecutionSlotAllocator}.
+ *
+ * The shared slots are owned and tracked by {@link
SlotSharingExecutionSlotAllocator}.
+ * The shared slot represents a collection of {@link SingleLogicalSlot}
requests which share one physical slot.
+ * The shared slot is created by the {@link SlotSharingExecutionSlotAllocator}
from the physical slot request.
+ * Afterwards, {@link SlotSharingExecutionSlotAllocator} requests logical
slots from the underlying physical slot
+ * for execution which share it.
Review comment:
```suggestion
* for {@link Execution executions} which share it.
```
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotRequestBulkCheckerImpl.java
##
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import org.apache.flink.util.clock.Clock;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static
[GitHub] [flink] flinkbot edited a comment on pull request #13102: [FLINK-18515][Kinesis] Adding FanOutRecordPublisher for Kinesis EFO support
flinkbot edited a comment on pull request #13102: URL: https://github.com/apache/flink/pull/13102#issuecomment-671238228 ## CI report: * d82d0c6582050aa4364b4c7dda2c3053f3c2d756 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6211) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19089) Improvement of ClosableBlockingQueue, replace Lock with ReadWriteLock, replace signalAll with signal
[ https://issues.apache.org/jira/browse/FLINK-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dugenkui updated FLINK-19089: - Summary: Improvement of ClosableBlockingQueue, replace Lock with ReadWriteLock, replace signalAll with signal (was: In ClosableBlockingQueue, Replace ReentrantLock with ReentrantReadWriteLock ) > Improvement of ClosableBlockingQueue, replace Lock with ReadWriteLock, > replace signalAll with signal > > > Key: FLINK-19089 > URL: https://issues.apache.org/jira/browse/FLINK-19089 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: dugenkui >Priority: Major > Labels: pull-request-available > > 1. Replace ReentrantLock with ReentrantReadWriteLock to improve concurrency; > 2. Use signal instead of signalAll to reduce the thread scheduling. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19089) In ClosableBlockingQueue, Replace ReentrantLock with ReentrantReadWriteLock
[ https://issues.apache.org/jira/browse/FLINK-19089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dugenkui updated FLINK-19089: - Summary: In ClosableBlockingQueue, Replace ReentrantLock with ReentrantReadWriteLock (was: Replace ReentrantLock with ReentrantReadWriteLock in ClosableBlockingQueue) > In ClosableBlockingQueue, Replace ReentrantLock with ReentrantReadWriteLock > > > Key: FLINK-19089 > URL: https://issues.apache.org/jira/browse/FLINK-19089 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: dugenkui >Priority: Major > Labels: pull-request-available > > 1. Replace ReentrantLock with ReentrantReadWriteLock to improve concurrency; > 2. Use signal instead of signalAll to reduce the thread scheduling. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13295: [FLINK-18598][python][docs] Add instructions for asynchronous execute in PyFlink doc.
flinkbot edited a comment on pull request #13295: URL: https://github.com/apache/flink/pull/13295#issuecomment-684586420 ## CI report: * 00804638850e8cc6779ebb3331ebb42be69f352b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6220) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
flinkbot edited a comment on pull request #13306: URL: https://github.com/apache/flink/pull/13306#issuecomment-685717187 ## CI report: * a7f4abab05ad875bf6acef6510fcec6df7d2dc6c UNKNOWN * c7628fc4b97f68c44d7f5e9cf316814c9f73db18 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6212) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #13324: [hotfix][docs] Fix ExecutionEnvironment.scala doc error
rmetzger commented on pull request #13324: URL: https://github.com/apache/flink/pull/13324#issuecomment-687204426 Thanks for review. Merging ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #13241: [hotfix] Fix a typo in the watermark docs
rmetzger commented on pull request #13241: URL: https://github.com/apache/flink/pull/13241#issuecomment-687204523 Thanks for review. Merging ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #13329: [hot fix]fix typo in the java doc
rmetzger commented on pull request #13329: URL: https://github.com/apache/flink/pull/13329#issuecomment-687204212 Thanks for review. Merging ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #12005: [hotfix][runtime] Remove never thrown IOException in ContinuousFileMnitoringFunction
rmetzger commented on pull request #12005: URL: https://github.com/apache/flink/pull/12005#issuecomment-687204617 Thanks for review. Merging ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13217: [FLINK-16866] Make job submission non-blocking
flinkbot edited a comment on pull request #13217: URL: https://github.com/apache/flink/pull/13217#issuecomment-678285884 ## CI report: * 3655fcea1966bfbcb85c86d6a159c354f20d6cc7 UNKNOWN * 8036016c752bce433dc65d1c08695377c917836f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6162) * 4d4d6690d8ad282f71e5f3d890e28421aef17766 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19142) Investigate slot hijacking from preceding pipelined regions after failover
[ https://issues.apache.org/jira/browse/FLINK-19142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-19142: -- Fix Version/s: 1.12.0 > Investigate slot hijacking from preceding pipelined regions after failover > -- > > Key: FLINK-19142 > URL: https://issues.apache.org/jira/browse/FLINK-19142 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.12.0 >Reporter: Andrey Zagrebin >Priority: Major > Fix For: 1.12.0 > > > The ticket originates from [this PR > discussion|https://github.com/apache/flink/pull/13181#discussion_r481087221]. > The previous AllocationIDs are used by > PreviousAllocationSlotSelectionStrategy to schedule subtasks into the slot > where they were previously executed before a failover. If the previous slot > (AllocationID) is not available, we do not want subtasks to take previous > slots (AllocationIDs) of other subtasks. > The MergingSharedSlotProfileRetriever gets all previous AllocationIDs of the > bulk from SlotSharingExecutionSlotAllocator but only from the current bulk. > The previous AllocationIDs of other bulks stay unknown. Therefore, the > current bulk can potentially hijack the previous slots from the preceding > bulks. On the other hand the previous AllocationIDs of other tasks should be > taken if the other tasks are not going to run at the same time, e.g. not > enough resources after failover or other bulks are done. > One way to do it may be to give to MergingSharedSlotProfileRetriever all > previous AllocationIDs of bulks which are going to run at the same time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] azagrebin commented on a change in pull request #13284: [FLINK-17016][runtime] Enable pipelined region scheduling
azagrebin commented on a change in pull request #13284:
URL: https://github.com/apache/flink/pull/13284#discussion_r483646625
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/ExecutionSlotAllocatorFactory.java
##
@@ -19,11 +19,39 @@
package org.apache.flink.runtime.scheduler;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.jobmanager.scheduler.CoLocationGroupDesc;
+import org.apache.flink.runtime.jobmanager.scheduler.SlotSharingGroup;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.scheduler.strategy.SchedulingTopology;
+
+import java.util.Set;
+import java.util.function.Function;
+import java.util.function.Supplier;
+
/**
* Interface for {@link ExecutionSlotAllocator} factories.
*/
+@FunctionalInterface
public interface ExecutionSlotAllocatorFactory {
- ExecutionSlotAllocator createInstance(PreferredLocationsRetriever
preferredLocationsRetriever);
-
+ /**
+* Instantiates the {@link ExecutionSlotAllocator}.
+*
+* @param preferredLocationsRetriever to retrieve preferred locations
for each execution
+* @param resourceProfileRetriever to retrieve required {@link
ResourceProfile} for each execution
+* @param priorAllocationIdRetriever to retrieve prior {@link
AllocationID} for each execution
+* @param topology scheduling topology of the job
+* @param logicalSlotSharingGroupSupplier to supply all slot sharing
groups in the job
+* @param coLocationGroupSupplier to supply all co-location groups in
the job
+* @return The instantiated slot allocator
+*/
+ ExecutionSlotAllocator createInstance(
+ PreferredLocationsRetriever preferredLocationsRetriever,
+ Function
resourceProfileRetriever,
+ Function
priorAllocationIdRetriever,
+ SchedulingTopology topology,
+ Supplier> logicalSlotSharingGroupSupplier,
+ Supplier> coLocationGroupSupplier);
Review comment:
but they are invoked immediately in `SlotSharingExecutionSlotAllocator`
and only once, right?
this is probably not problem with `ExecutionSlotAllocationContext`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13284: [FLINK-17016][runtime] Enable pipelined region scheduling
azagrebin commented on a change in pull request #13284:
URL: https://github.com/apache/flink/pull/13284#discussion_r483645607
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobGraph.java
##
@@ -300,6 +305,32 @@ public int getNumberOfVertices() {
return this.taskVertices.size();
}
+ public Set getSlotSharingGroups() {
+ final Set slotSharingGroups = new HashSet<>();
+ for (JobVertex vertex : getVertices()) {
+ final SlotSharingGroup slotSharingGroup =
vertex.getSlotSharingGroup();
+ checkNotNull(slotSharingGroup);
+
+ slotSharingGroups.add(slotSharingGroup);
+ }
+ return Collections.unmodifiableSet(slotSharingGroups);
+ }
+
+ public Set getCoLocationGroupDescriptors() {
+ final Set coLocationGroups = new HashSet<>();
+ for (JobVertex vertex : getVertices()) {
+ CoLocationGroup coLocationGroup =
vertex.getCoLocationGroup();
+ if (coLocationGroup != null) {
+ coLocationGroups.add(coLocationGroup);
+ }
+ }
+ final Set coLocationGroupDescs =
coLocationGroups
+ .stream()
+ .map(CoLocationGroupDesc::from)
+ .collect(Collectors.toSet());
Review comment:
Then maybe we substitute `Set` with
`Map`?
```
final Set coLocationGroupDescs = getVertices()
.stream()
.map(vertex -> CoLocationGroupDesc.from(vertex.getCoLocationGroup()))
.collect(Collectors.toMap(CoLocationGroupDesc::getId, d -> d))
.values()
.stream()
.collect(Collectors.toSet());
```
It can be also in a for-loop, but it is nit anyways so I think it is ok as
it is.
Maybe, at least a comment why we do it like this: `because equals ...`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17554) Add release hooks for user code class loader
[
https://issues.apache.org/jira/browse/FLINK-17554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190743#comment-17190743
]
Robert Metzger commented on FLINK-17554:
I confirmed that the leak is gone when using
{{AwsSdkMetrics.unregisterMetricAdminMBean()}} and fixing the
{{FileAgeManager}}.
I reported the issue here:
https://github.com/awslabs/amazon-kinesis-producer/issues/308 The code has also
been published as a pull request.
I will now finalize the pull request.
> Add release hooks for user code class loader
>
>
> Key: FLINK-17554
> URL: https://issues.apache.org/jira/browse/FLINK-17554
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Till Rohrmann
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Release hooks for the user code class loader which are run just before the
> user code class loader is released would allow clean up static references to
> classes of the user code class loader. This is important because these static
> references could prevent the user code classes from being garbage collected
> and eventually causing metaspace OOMs.
> Hence I suggest to extend the {{RuntimeContext}} with an additional method
> {{registerUserCodeClassLoaderReleaseHook(Runnable releaseHook)}} which allows
> the user code to register a release hook for the user code class loader.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18858) Kinesis Flink SQL Connector
[
https://issues.apache.org/jira/browse/FLINK-18858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190744#comment-17190744
]
Timo Walther commented on FLINK-18858:
--
Feel free to also ping me if you need any help. SQL connectors are based on
regular DataStream API SourceFunctions. I think the Kinesis connector can copy
much code from the Kafka SQL connector. A rough overview how the SQL connectors
work can be found here:
https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/sourceSinks.html
> Kinesis Flink SQL Connector
> ---
>
> Key: FLINK-18858
> URL: https://issues.apache.org/jira/browse/FLINK-18858
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis, Table SQL / Ecosystem
>Reporter: Waldemar Hummer
>Priority: Major
>
> Hi all,
> as far as I can see in the [list of
> connectors|https://github.com/apache/flink/tree/master/flink-connectors], we
> have a
> {{[flink-connector-kinesis|https://github.com/apache/flink/tree/master/flink-connectors/flink-connector-kinesis]}}
> for *programmatic access* to Kinesis streams, but there does not yet seem to
> exist a *Kinesis SQL connector* (something like
> {{flink-sql-connector-kinesis}}, analogous to {{flink-sql-connector-kafka}}).
> Our use case would be to enable SQL queries with direct access to Kinesis
> sources (and potentially sinks), to enable something like the following Flink
> SQL queries:
> {code:java}
> $ bin/sql-client.sh embedded
> ...
> Flink SQL> CREATE TABLE Orders(`user` string, amount int, rowtime TIME) WITH
> ('connector' = 'kinesis', ...);
> ...
> Flink SQL> SELECT * FROM Orders ...;
> ...{code}
>
> I was wondering if this is something that has been considered, or is already
> actively being worked on? If one of you can provide some guidance, we may be
> able to work on a PoC implementation to add this functionality.
>
> (Wasn't able to find an existing issue in the backlog - if this is a
> duplicate, then please let me know as well.)
> Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13189: [FLINK-18661][Kinesis] Stream consumer Registration/Deregistration
flinkbot edited a comment on pull request #13189: URL: https://github.com/apache/flink/pull/13189#issuecomment-675350675 ## CI report: * 17ab6b978f786d321781a589867fdd331731 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6209) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13102: [FLINK-18515][Kinesis] Adding FanOutRecordPublisher for Kinesis EFO support
flinkbot edited a comment on pull request #13102: URL: https://github.com/apache/flink/pull/13102#issuecomment-671238228 ## CI report: * 2d0fb75c7d741e191192c6647ffeb5a44b155e6f Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6208) * d82d0c6582050aa4364b4c7dda2c3053f3c2d756 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6211) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13284: [FLINK-17016][runtime] Enable pipelined region scheduling
azagrebin commented on a change in pull request #13284:
URL: https://github.com/apache/flink/pull/13284#discussion_r480166064
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobGraph.java
##
@@ -300,6 +305,32 @@ public int getNumberOfVertices() {
return this.taskVertices.size();
}
+ public Set getSlotSharingGroups() {
+ final Set slotSharingGroups = new HashSet<>();
+ for (JobVertex vertex : getVertices()) {
+ final SlotSharingGroup slotSharingGroup =
vertex.getSlotSharingGroup();
+ checkNotNull(slotSharingGroup);
+
+ slotSharingGroups.add(slotSharingGroup);
+ }
+ return Collections.unmodifiableSet(slotSharingGroups);
+ }
+
+ public Set getCoLocationGroupDescriptors() {
+ final Set coLocationGroups = new HashSet<>();
+ for (JobVertex vertex : getVertices()) {
+ CoLocationGroup coLocationGroup =
vertex.getCoLocationGroup();
+ if (coLocationGroup != null) {
+ coLocationGroups.add(coLocationGroup);
+ }
+ }
+ final Set coLocationGroupDescs =
coLocationGroups
+ .stream()
+ .map(CoLocationGroupDesc::from)
+ .collect(Collectors.toSet());
Review comment:
```suggestion
final Set coLocationGroupDescs = new
HashSet<>();
for (JobVertex vertex : getVertices()) {
CoLocationGroup coLocationGroup =
vertex.getCoLocationGroup();
if (coLocationGroup != null) {
CoLocationGroupDesc coLocationGroupDesc =
CoLocationGroupDesc.from(coLocationGroup)
coLocationGroupDescs.add(coLocationGroupDesc);
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] Tartarus0zm commented on a change in pull request #13319: [FLINK-19022][runtime]Register the TerminationFuture of ResourceManager and Dispatcher with DispatcherResourceManagerComponen
Tartarus0zm commented on a change in pull request #13319:
URL: https://github.com/apache/flink/pull/13319#discussion_r483637646
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/DispatcherResourceManagerComponent.java
##
@@ -88,6 +95,18 @@
private void registerShutDownFuture() {
FutureUtils.forward(dispatcherRunner.getShutDownFuture(),
shutDownFuture);
+ BiConsumer terminateAction = (ignored,
throwable) -> {
+ if (throwable != null) {
+ shutDownFuture.completeExceptionally(throwable);
+ } else {
+
shutDownFuture.complete(ApplicationStatus.SUCCEEDED);
+ }
+ if (isRunning.get()) {
+ fatalErrorHandler.onFatalError(throwable);
+ }
+ };
+
dispatcherRunner.getTerminationFuture().whenComplete(terminateAction);
Review comment:
But how we get the Dispatcher's TerminationFuture in
DispatcherResourceManagerComponent?
Please give a hint, thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-19141) Flink Job Submitted on Yarn Does not Work
[
https://issues.apache.org/jira/browse/FLINK-19141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190727#comment-17190727
]
Huang Xiao commented on FLINK-19141:
Hi, [~yunhui].
According to the log, it seems there's no enough slots to run this job.
For streaming jobs, the default slot request timeout is 300s. If the scheduler
cannot get the needed slot after 300s, it will throw this exception.
You can try to decrease the memory of tm or add the number of slots per tm to
increase the total number of slots in cluster.
Hope this can solve your problem :)
> Flink Job Submitted on Yarn Does not Work
> -
>
> Key: FLINK-19141
> URL: https://issues.apache.org/jira/browse/FLINK-19141
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.11.1
>Reporter: Yunhui
>Priority: Major
>
> I first launch a cluster on yarn.
> {code:java}
> $flink_path/bin/yarn-session.sh \
> -qu dev \
> -d -nm flink_cluster_1.11 \
> -jm 8192 \
> -tm 12288 \
> -s 2 \
> -D taskmanager.memory.framework.off-heap.size=2048m \
> -D taskmanager.memory.managed.size=0{code}
> Then I submit my job with the following command
> {code:java}
> $flink_path/bin/flink run \
> -d -m $host_port \
> -c MyMainClass \
> my-jar.jar{code}
> It take a long time to schedule. And it ended with the following Exception.
> But it works for flink-1.10.1
> {code:java}
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Could not allocate the required slot within slot request timeout. Please make
> sure that the cluster has enough resources.at
> org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]at
> org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]at
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> ~[?:1.8.0_77]at
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> ~[?:1.8.0_77]at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]at
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]at
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> ~[?:1.8.0_77]at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> ~[?:1.8.0_77]at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> ~[?:1.8.0_77]at
> org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
> ~[flink-dist_2.11-1.11.1.jar:1.11.1]at
>
[GitHub] [flink] flinkbot edited a comment on pull request #13315: [FLINK-19070][hive] Hive connector should throw a meaningful exception if user reads/writes ACID tables
flinkbot edited a comment on pull request #13315: URL: https://github.com/apache/flink/pull/13315#issuecomment-686398823 ## CI report: * 2e0da845caf6355b898bdcff29e3b7ac56b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6154) * 2ed647fc736a248d536d6f5422e0a50a7119045f UNKNOWN * 6f796f529a1dc3aed87785eaeb251725c4dd17c2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6221) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13295: [FLINK-18598][python][docs] Add instructions for asynchronous execute in PyFlink doc.
flinkbot edited a comment on pull request #13295: URL: https://github.com/apache/flink/pull/13295#issuecomment-684586420 ## CI report: * 38a9a93bc6f82f321b94390f74c19f89aa7d7eca Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6217) * 00804638850e8cc6779ebb3331ebb42be69f352b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6220) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13109: [FLINK-18808][runtime/metrics] Include side outputs in numRecordsOut metric.
flinkbot edited a comment on pull request #13109: URL: https://github.com/apache/flink/pull/13109#issuecomment-671381707 ## CI report: * 3a7e71c54243a063412da38da4be35c4f3b2d175 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6109) * b43e9eda0e94696b0563922ecb37b202bca47af5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6222) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] haveanote commented on pull request #13324: [hotfix][docs] Fix ExecutionEnvironment.scala doc error
haveanote commented on pull request #13324: URL: https://github.com/apache/flink/pull/13324#issuecomment-687155612 @XComp ,not at all ,i'm happy to do it This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13315: [FLINK-19070][hive] Hive connector should throw a meaningful exception if user reads/writes ACID tables
flinkbot edited a comment on pull request #13315: URL: https://github.com/apache/flink/pull/13315#issuecomment-686398823 ## CI report: * 2e0da845caf6355b898bdcff29e3b7ac56b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6154) * 2ed647fc736a248d536d6f5422e0a50a7119045f UNKNOWN * 6f796f529a1dc3aed87785eaeb251725c4dd17c2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13295: [FLINK-18598][python][docs] Add instructions for asynchronous execute in PyFlink doc.
flinkbot edited a comment on pull request #13295: URL: https://github.com/apache/flink/pull/13295#issuecomment-684586420 ## CI report: * f67e466f192eaecfc167467334d844a571123afe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6125) * 38a9a93bc6f82f321b94390f74c19f89aa7d7eca Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6217) * 00804638850e8cc6779ebb3331ebb42be69f352b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6220) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
flinkbot edited a comment on pull request #13181: URL: https://github.com/apache/flink/pull/13181#issuecomment-675091412 ## CI report: * e7553689356882de1ffe606400d1255d1d757bc4 UNKNOWN * d34648bab5e9a642c97db6e98f80f8059500c145 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6202) * db1607ead33ae1b60634c1162ba8247a9957f0f4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6219) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13109: [FLINK-18808][runtime/metrics] Include side outputs in numRecordsOut metric.
flinkbot edited a comment on pull request #13109: URL: https://github.com/apache/flink/pull/13109#issuecomment-671381707 ## CI report: * 3a7e71c54243a063412da38da4be35c4f3b2d175 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6109) * b43e9eda0e94696b0563922ecb37b202bca47af5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13315: [FLINK-19070][hive] Hive connector should throw a meaningful exception if user reads/writes ACID tables
flinkbot edited a comment on pull request #13315: URL: https://github.com/apache/flink/pull/13315#issuecomment-686398823 ## CI report: * 2e0da845caf6355b898bdcff29e3b7ac56b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6154) * 2ed647fc736a248d536d6f5422e0a50a7119045f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13316: [FLINK-14422][runtime] Expose network memory usage to TaskManagerDetailsHandler's endpoint
flinkbot edited a comment on pull request #13316: URL: https://github.com/apache/flink/pull/13316#issuecomment-686398901 ## CI report: * e37cb771c66ed8cab48e0b7abd53132fb15dfca3 UNKNOWN * 194b07dfb332eca1b8d99d0fe1425e117eff9b33 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6206) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13295: [FLINK-18598][python][docs] Add instructions for asynchronous execute in PyFlink doc.
flinkbot edited a comment on pull request #13295: URL: https://github.com/apache/flink/pull/13295#issuecomment-684586420 ## CI report: * f67e466f192eaecfc167467334d844a571123afe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6125) * 38a9a93bc6f82f321b94390f74c19f89aa7d7eca Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6217) * 00804638850e8cc6779ebb3331ebb42be69f352b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache commented on a change in pull request #13315: [FLINK-19070][hive] Hive connector should throw a meaningful exception if user reads/writes ACID tables
lirui-apache commented on a change in pull request #13315:
URL: https://github.com/apache/flink/pull/13315#discussion_r483604044
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/HiveTableSink.java
##
@@ -328,4 +330,16 @@ public void setStaticPartition(Map
partitionSpec) {
public void setOverwrite(boolean overwrite) {
this.overwrite = overwrite;
}
+
+ private void checkAcidTable() {
+ if (catalogTable != null && catalogTable.getOptions() != null) {
+ String tableIsTransactional =
catalogTable.getOptions().get("transactional");
+ if (tableIsTransactional == null) {
+ tableIsTransactional =
catalogTable.getOptions().get("transactional".toUpperCase());
+ }
+ if (tableIsTransactional != null &&
tableIsTransactional.equalsIgnoreCase("true")) {
+ throw new TableException(String.format("Cannot
write on the ACID table %s.", identifier.asSummaryString()));
Review comment:
```suggestion
throw new
FlinkHiveException(String.format("Writing ACID table %s is not supported",
identifier.asSummaryString()));
```
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/HiveTableSink.java
##
@@ -330,6 +332,18 @@ public void setOverwrite(boolean overwrite) {
this.overwrite = overwrite;
}
+ private void checkAcidTable() {
+ if (catalogTable != null && catalogTable.getOptions() != null) {
+ String tableIsTransactional =
catalogTable.getOptions().get("transactional");
+ if (tableIsTransactional == null) {
+ tableIsTransactional =
catalogTable.getOptions().get("transactional".toUpperCase());
+ }
+ if (tableIsTransactional != null &&
tableIsTransactional.equalsIgnoreCase("true")) {
Review comment:
Let's extract these into a util method
##
File path:
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/TableEnvHiveConnectorITCase.java
##
@@ -574,6 +575,43 @@ public void testStreamCompressTextTable() throws Exception
{
testCompressTextTable(false);
}
+ private void testTransactionalTable(boolean batch) {
+ TableEnvironment tableEnv = batch ?
+ getTableEnvWithHiveCatalog() :
+ getStreamTableEnvWithHiveCatalog();
+ tableEnv.executeSql("create database db1");
+ try {
+ tableEnv.executeSql("create table db1.src (x string,y
string)");
+ hiveShell.execute("create table db1.dest (x string,y
string) clustered by (x) into 3 buckets stored as orc tblproperties
('transactional'='true')");
+ HiveTestUtils.createTextTableInserter(hiveShell, "db1",
"src")
+ .addRow(new Object[]{"a", "b"})
+ .addRow(new Object[]{"c", "d"})
+ .commit();
Review comment:
No need to insert data
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/HiveTableSource.java
##
@@ -556,4 +558,16 @@ public String explainSource() {
public boolean isAsyncEnabled() {
return false;
}
+
+ private void checkAcidTable() {
+ if (catalogTable != null && catalogTable.getOptions() != null) {
+ String tableIsTransactional =
catalogTable.getOptions().get("transactional");
+ if (tableIsTransactional == null) {
+ tableIsTransactional =
catalogTable.getOptions().get("transactional".toUpperCase());
+ }
+ if (tableIsTransactional != null &&
tableIsTransactional.equalsIgnoreCase("true")) {
+ throw new TableException(String.format("Cannot
read on the ACID table %s.", tablePath));
Review comment:
```suggestion
throw new
FlinkHiveException(String.format("Reading ACID table %s is not supported",
tablePath));
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13274: [FLINK-19061][hive] HiveCatalog fails to get partition column stats i…
flinkbot edited a comment on pull request #13274: URL: https://github.com/apache/flink/pull/13274#issuecomment-682368291 ## CI report: * 6d99f26131d335a4548c7eb7d5c64611d3de8094 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6205) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13329: [hot fix]fix typo in the java doc
flinkbot edited a comment on pull request #13329: URL: https://github.com/apache/flink/pull/13329#issuecomment-687023379 ## CI report: * 54964991f6b3ec7b12f2cb045d29a2a33dad1011 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6207) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #13320: [FLINK-19035] Remove fold from DataStream API
twalthr commented on a change in pull request #13320:
URL: https://github.com/apache/flink/pull/13320#discussion_r483589525
##
File path:
flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/AllWindowTranslationTest.scala
##
@@ -91,54 +91,10 @@ class AllWindowTranslationTest {
fail("exception was not thrown")
}
- /**
-* .fold() does not support [[RichFoldFunction]], since the reduce function
is used internally
-* in a [[org.apache.flink.api.common.state.FoldingState]].
-*/
- @Test(expected = classOf[UnsupportedOperationException])
- def testFoldWithRichFolderFails() {
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val source = env.fromElements(("hello", 1), ("hello", 2))
-
-env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime)
-
-source
- .windowAll(SlidingEventTimeWindows.of(Time.seconds(1),
Time.milliseconds(100)))
- .fold(("", 0), new RichFoldFunction[(String, Int), (String, Int)] {
-override def fold(accumulator: (String, Int), value: (String, Int)) =
null
- })
-
-fail("exception was not thrown")
- }
-
//
// merging window precondition
//
- @Test
- def testSessionWithFoldFails() {
-// verify that fold does not work with merging windows
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-
-val windowedStream = env.fromElements("Hello", "Ciao")
- .windowAll(EventTimeSessionWindows.withGap(Time.seconds(5)))
-
-try
- windowedStream.fold("", new FoldFunction[String, String]() {
-@throws[Exception]
-def fold(accumulator: String, value: String): String = accumulator
- })
-
-catch {
- case _: UnsupportedOperationException =>
-// expected
-// use a catch to ensure that the exception is thrown by the fold
Review comment:
same for java
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #13320: [FLINK-19035] Remove fold from DataStream API
twalthr commented on a change in pull request #13320:
URL: https://github.com/apache/flink/pull/13320#discussion_r483589434
##
File path:
flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/WindowTranslationTest.scala
##
@@ -92,56 +93,10 @@ class WindowTranslationTest {
fail("exception was not thrown")
}
- /**
-* .fold() does not support [[RichFoldFunction]], since the reduce function
is used internally
-* in a [[org.apache.flink.api.common.state.FoldingState]].
-*/
- @Test(expected = classOf[UnsupportedOperationException])
- def testFoldWithRichFolderFails() {
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-val source = env.fromElements(("hello", 1), ("hello", 2))
-
-env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime)
-
-source
- .keyBy(0)
- .window(SlidingEventTimeWindows.of(Time.seconds(1),
Time.milliseconds(100)))
- .fold(("", 0), new RichFoldFunction[(String, Int), (String, Int)] {
-override def fold(accumulator: (String, Int), value: (String, Int)) =
null
- })
-
-fail("exception was not thrown")
- }
-
// --
// merging window checks
// --
- @Test
- def testSessionWithFoldFails() {
-// verify that fold does not work with merging windows
-val env = StreamExecutionEnvironment.getExecutionEnvironment
-
-val windowedStream = env.fromElements("Hello", "Ciao")
- .keyBy(x => x)
- .window(EventTimeSessionWindows.withGap(Time.seconds(5)))
-
-try
- windowedStream.fold("", new FoldFunction[String, String]() {
-@throws[Exception]
-def fold(accumulator: String, value: String): String = accumulator
- })
-
-catch {
- case _: UnsupportedOperationException =>
-// expected
-// use a catch to ensure that the exception is thrown by the fold
Review comment:
same for java
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13327: [FLINK-19134][python] Add BasicArrayTypeInfo and coder for PrimitiveArrayTypeInfo for Python DataStream API.
flinkbot edited a comment on pull request #13327: URL: https://github.com/apache/flink/pull/13327#issuecomment-687006937 ## CI report: * 352380e8ab8941541fbc1773bcd29820bcf646c3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6204) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #13320: [FLINK-19035] Remove fold from DataStream API
twalthr commented on a change in pull request #13320:
URL: https://github.com/apache/flink/pull/13320#discussion_r483579510
##
File path: flink-streaming-scala/pom.xml
##
@@ -262,6 +262,22 @@ under the License.
org.apache.flink.streaming.api.scala.DataStream#iterate\$default\$3()
org.apache.flink.streaming.api.scala.DataStream#assignTimestamps(org.apache.flink.streaming.api.functions.TimestampExtractor)
org.apache.flink.streaming.api.scala.ConnectedStreams#keyBy(scala.Function1,scala.Function1,org.apache.flink.api.common.typeinfo.TypeInformation,org.apache.flink.api.common.typeinfo.TypeInformation)
+
+
org.apache.flink.streaming.api.scala.AllWindowedStream#apply(java.lang.Object,org.apache.flink.api.common.functions.FoldFunction,org.apache.flink.streaming.api.scala.function.AllWindowFunction,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.AllWindowedStream#apply(java.lang.Object,scala.Function2,scala.Function3,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.AllWindowedStream#fold(java.lang.Object,scala.Function2,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.AllWindowedStream#fold(java.lang.Object,org.apache.flink.api.common.functions.FoldFunction,org.apache.flink.streaming.api.scala.function.AllWindowFunction,org.apache.flink.api.common.typeinfo.TypeInformation,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.AllWindowedStream#fold(java.lang.Object,scala.Function2,scala.Function3,org.apache.flink.api.common.typeinfo.TypeInformation,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.KeyedStream#fold(java.lang.Object,org.apache.flink.api.common.functions.FoldFunction,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.KeyedStream#fold(java.lang.Object,scala.Function2,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.WindowedStream#apply(java.lang.Object,org.apache.flink.api.common.functions.FoldFunction,org.apache.flink.streaming.api.scala.function.WindowFunction,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.WindowedStream#apply(java.lang.Object,scala.Function2,scala.Function4,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.WindowedStream#fold(java.lang.Object,scala.Function2,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.WindowedStream#fold(java.lang.Object,org.apache.flink.api.common.functions.FoldFunction,org.apache.flink.streaming.api.scala.function.WindowFunction,org.apache.flink.api.common.typeinfo.TypeInformation,org.apache.flink.api.common.typeinfo.TypeInformation)
+
org.apache.flink.streaming.api.scala.WindowedStream#fold(java.lang.Object,scala.Function2,scala.Function4,org.apache.flink.api.common.typeinfo.TypeInformation,org.apache.flink.api.common.typeinfo.TypeInformation)
+
Review comment:
nit: remove empty line and add a comment to this section that fold was
dropped in 1.12
##
File path:
flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/DataStreamTest.scala
##
@@ -35,10 +35,13 @@ import org.apache.flink.streaming.runtime.partitioner._
import org.apache.flink.test.util.AbstractTestBase
import org.apache.flink.util.Collector
+import org.hamcrest.CoreMatchers.equalTo
import org.junit.Assert._
import org.junit.rules.ExpectedException
import org.junit.{Rule, Test}
+import java.lang
+
class DataStreamTest extends AbstractTestBase {
Review comment:
update `testResource()` in this class
##
File path:
flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/AllWindowTranslationTest.scala
##
@@ -91,54 +91,10 @@ class AllWindowTranslationTest {
fail("exception was not thrown")
}
- /**
-* .fold() does not support [[RichFoldFunction]], since the reduce function
is used internally
-* in a
[GitHub] [flink] flinkbot edited a comment on pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
flinkbot edited a comment on pull request #13181: URL: https://github.com/apache/flink/pull/13181#issuecomment-675091412 ## CI report: * e7553689356882de1ffe606400d1255d1d757bc4 UNKNOWN * d34648bab5e9a642c97db6e98f80f8059500c145 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6202) * db1607ead33ae1b60634c1162ba8247a9957f0f4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13326: [FLINK-19133] Open custom partitioners in KafkaSerializationSchemaWrapper
flinkbot edited a comment on pull request #13326: URL: https://github.com/apache/flink/pull/13326#issuecomment-686943261 ## CI report: * e8a32ce376b09df4766fd6ddd439d8f634210ad7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6190) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6203) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13321: [FLINK-14870] Remove nullable assumption of task slot sharing group
flinkbot edited a comment on pull request #13321: URL: https://github.com/apache/flink/pull/13321#issuecomment-686567896 ## CI report: * d79f152fa91b8bc555af6fb2a00a8d62b184be5a UNKNOWN * bcd04d78e47e1c3a9b0cb0165d625f424dea4951 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6201) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13295: [FLINK-18598][python][docs] Add instructions for asynchronous execute in PyFlink doc.
flinkbot edited a comment on pull request #13295: URL: https://github.com/apache/flink/pull/13295#issuecomment-684586420 ## CI report: * f67e466f192eaecfc167467334d844a571123afe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6125) * 38a9a93bc6f82f321b94390f74c19f89aa7d7eca Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6217) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13314: [FLINK-18913][docs][python] Add a "TableEnvironment" document under the "Python API" -> "Table API User's Guide" section
flinkbot edited a comment on pull request #13314: URL: https://github.com/apache/flink/pull/13314#issuecomment-686373715 ## CI report: * 7566c3c966f7771fc6cb14970c9ccf318c6ba5ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6210) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
azagrebin commented on pull request #13181: URL: https://github.com/apache/flink/pull/13181#issuecomment-687108594 Thanks for the review @tillrohrmann and @zhuzhurk I have addressed comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19036) Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-19036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-19036. --- Resolution: Fixed master: 5cce7652f5afa31c4bd25c2819aadbd58d4c8ce3 > Translate page 'Application Profiling & Debugging' of 'Debugging & > Monitoring' into Chinese > --- > > Key: FLINK-19036 > URL: https://issues.apache.org/jira/browse/FLINK-19036 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.11.0, 1.11.1 >Reporter: Roc Marshal >Assignee: Roc Marshal >Priority: Major > Labels: pull-request-available, translation-zh > > The markdown file location: flink/docs/monitoring/application_profiling.zh.md > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/application_profiling.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19036) Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-19036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-19036: Fix Version/s: 1.12.0 > Translate page 'Application Profiling & Debugging' of 'Debugging & > Monitoring' into Chinese > --- > > Key: FLINK-19036 > URL: https://issues.apache.org/jira/browse/FLINK-19036 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.11.0, 1.11.1 >Reporter: Roc Marshal >Assignee: Roc Marshal >Priority: Major > Labels: pull-request-available, translation-zh > Fix For: 1.12.0 > > > The markdown file location: flink/docs/monitoring/application_profiling.zh.md > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/application_profiling.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #13235: [FLINK-19036][docs-zh] Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
dianfu closed pull request #13235: URL: https://github.com/apache/flink/pull/13235 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
azagrebin commented on a change in pull request #13181: URL: https://github.com/apache/flink/pull/13181#discussion_r483577436 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SlotSharingExecutionSlotAllocator.java ## @@ -108,27 +122,30 @@ SharedSlotProfileRetriever sharedSlotProfileRetriever = sharedSlotProfileRetrieverFactory .createFromBulk(new HashSet<>(executionVertexIds)); Review comment: Indeed, this can result in suboptimal behaviour. I created an issue to investigate this: [FLINK-19142](https://issues.apache.org/jira/browse/FLINK-19142). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19142) Investigate slot hijacking from preceding pipelined regions after failover
[ https://issues.apache.org/jira/browse/FLINK-19142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Zagrebin updated FLINK-19142: Affects Version/s: 1.12.0 > Investigate slot hijacking from preceding pipelined regions after failover > -- > > Key: FLINK-19142 > URL: https://issues.apache.org/jira/browse/FLINK-19142 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.12.0 >Reporter: Andrey Zagrebin >Priority: Major > > The ticket originates from [this PR > discussion|https://github.com/apache/flink/pull/13181#discussion_r481087221]. > The previous AllocationIDs are used by > PreviousAllocationSlotSelectionStrategy to schedule subtasks into the slot > where they were previously executed before a failover. If the previous slot > (AllocationID) is not available, we do not want subtasks to take previous > slots (AllocationIDs) of other subtasks. > The MergingSharedSlotProfileRetriever gets all previous AllocationIDs of the > bulk from SlotSharingExecutionSlotAllocator but only from the current bulk. > The previous AllocationIDs of other bulks stay unknown. Therefore, the > current bulk can potentially hijack the previous slots from the preceding > bulks. On the other hand the previous AllocationIDs of other tasks should be > taken if the other tasks are not going to run at the same time, e.g. not > enough resources after failover or other bulks are done. > One way to do it may be to give to MergingSharedSlotProfileRetriever all > previous AllocationIDs of bulks which are going to run at the same time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19142) Investigate slot hijacking from preceding pipelined regions after failover
[ https://issues.apache.org/jira/browse/FLINK-19142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Zagrebin updated FLINK-19142: Component/s: Runtime / Coordination > Investigate slot hijacking from preceding pipelined regions after failover > -- > > Key: FLINK-19142 > URL: https://issues.apache.org/jira/browse/FLINK-19142 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.12.0 >Reporter: Andrey Zagrebin >Priority: Major > > The ticket originates from [this PR > discussion|https://github.com/apache/flink/pull/13181#discussion_r481087221]. > The previous AllocationIDs are used by > PreviousAllocationSlotSelectionStrategy to schedule subtasks into the slot > where they were previously executed before a failover. If the previous slot > (AllocationID) is not available, we do not want subtasks to take previous > slots (AllocationIDs) of other subtasks. > The MergingSharedSlotProfileRetriever gets all previous AllocationIDs of the > bulk from SlotSharingExecutionSlotAllocator but only from the current bulk. > The previous AllocationIDs of other bulks stay unknown. Therefore, the > current bulk can potentially hijack the previous slots from the preceding > bulks. On the other hand the previous AllocationIDs of other tasks should be > taken if the other tasks are not going to run at the same time, e.g. not > enough resources after failover or other bulks are done. > One way to do it may be to give to MergingSharedSlotProfileRetriever all > previous AllocationIDs of bulks which are going to run at the same time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19142) Investigate slot hijacking from preceding pipelined regions after failover
Andrey Zagrebin created FLINK-19142: --- Summary: Investigate slot hijacking from preceding pipelined regions after failover Key: FLINK-19142 URL: https://issues.apache.org/jira/browse/FLINK-19142 Project: Flink Issue Type: Improvement Reporter: Andrey Zagrebin The ticket originates from [this PR discussion|https://github.com/apache/flink/pull/13181#discussion_r481087221]. The previous AllocationIDs are used by PreviousAllocationSlotSelectionStrategy to schedule subtasks into the slot where they were previously executed before a failover. If the previous slot (AllocationID) is not available, we do not want subtasks to take previous slots (AllocationIDs) of other subtasks. The MergingSharedSlotProfileRetriever gets all previous AllocationIDs of the bulk from SlotSharingExecutionSlotAllocator but only from the current bulk. The previous AllocationIDs of other bulks stay unknown. Therefore, the current bulk can potentially hijack the previous slots from the preceding bulks. On the other hand the previous AllocationIDs of other tasks should be taken if the other tasks are not going to run at the same time, e.g. not enough resources after failover or other bulks are done. One way to do it may be to give to MergingSharedSlotProfileRetriever all previous AllocationIDs of bulks which are going to run at the same time. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13330: [FLINK-19004][table-planner-blink] Use FIRST_VALUE when re-write non-…
flinkbot edited a comment on pull request #13330: URL: https://github.com/apache/flink/pull/13330#issuecomment-687094698 ## CI report: * dda51e888c3f58e83346118f03ad55075e990eb8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6216) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13295: [FLINK-18598][python][docs] Add instructions for asynchronous execute in PyFlink doc.
flinkbot edited a comment on pull request #13295: URL: https://github.com/apache/flink/pull/13295#issuecomment-684586420 ## CI report: * f67e466f192eaecfc167467334d844a571123afe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6125) * 38a9a93bc6f82f321b94390f74c19f89aa7d7eca UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13284: [FLINK-17016][runtime] Enable pipelined region scheduling
flinkbot edited a comment on pull request #13284: URL: https://github.com/apache/flink/pull/13284#issuecomment-683683000 ## CI report: * c89d0141f4a854bc7deb4512a2b2007fccfb13fd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6200) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13216: [FLINK-18999][table-planner-blink][hive] Temporary generic table does…
flinkbot edited a comment on pull request #13216: URL: https://github.com/apache/flink/pull/13216#issuecomment-678268420 ## CI report: * 31eef5a5d1f413435c3a3ccd9764c185b66c1c08 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5970) * 167e78ad9d9f8fdb79ce5b2483feed7f120456a3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6215) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13218: remove unnecessary integer boxing
XComp commented on pull request #13218: URL: https://github.com/apache/flink/pull/13218#issuecomment-687098584 Thanks for looking into this. But the issue was fixed already in c34c7e41. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18815) AbstractCloseableRegistryTest.testClose unstable
[
https://issues.apache.org/jira/browse/FLINK-18815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190687#comment-17190687
]
Till Rohrmann commented on FLINK-18815:
---
Thanks for the analysis [~kezhuw]. I am not totally convinced that what you
describe is the problem we are seeing here. The problem with your explanation
is that {{SafetyNetCloseableRegistry.close()}} should explicitly close all
registered closeables which are still contained in
{{AbstractCloseableRegistry.closeableToRef}}. Hence, it should not matter
whether the reaper thread still have some phantom references enqueued in its
{{referenceQueue}}. Once {{close()}} is called, no further {{Closeables}}
should be able to register at the registry.
The only way I could see this problem happening is after we have removed the
closeable from {{closeableRegistry}} in
{{PhantomDelegatingCloseableRef.close()}} we interrupt the
{{innerCloseable.close()}} call
(https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/fs/SafetyNetCloseableRegistry.java#L183).
However, in the test implementation we are using {{TestStream}} as
{{Closeable}} and the {{close()}} implementation is not blocking.
> AbstractCloseableRegistryTest.testClose unstable
>
>
> Key: FLINK-18815
> URL: https://issues.apache.org/jira/browse/FLINK-18815
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Affects Versions: 1.10.1, 1.12.0, 1.11.1
>Reporter: Robert Metzger
>Assignee: Kezhu Wang
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.10.2, 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=5164=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=05b74a19-4ee4-5036-c46f-ada307df6cf0
> {code}
> [ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.509
> s <<< FAILURE! - in org.apache.flink.core.fs.SafetyNetCloseableRegistryTest
> [ERROR] testClose(org.apache.flink.core.fs.SafetyNetCloseableRegistryTest)
> Time elapsed: 1.15 s <<< FAILURE!
> java.lang.AssertionError: expected:<0> but was:<-1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.flink.core.fs.AbstractCloseableRegistryTest.testClose(AbstractCloseableRegistryTest.java:93)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19127) Provide a replacement of StreamExecutionEnvironment.createRemoteEnvironment for TableEnvironment
[ https://issues.apache.org/jira/browse/FLINK-19127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther updated FLINK-19127: - Description: Connecting to a remote cluster from the unified TableEnvironment is neither tested nor documented. Since StreamExecutionEnvironment is not necessary anymore, users should be able to do the same in TableEnvironment easily. This is in particular useful for interactive sessions that run in an IDE, shell, or notebook. (was: Connecting to a remote cluster from the unified TableEnvironment is neither tested nor documented. Since StreamExecutionEnvironment is not necessary anymore, users should be able to do the same in TableEnvironment easily. This is in particular useful for interactive sessions that run in an IDE.) > Provide a replacement of StreamExecutionEnvironment.createRemoteEnvironment > for TableEnvironment > > > Key: FLINK-19127 > URL: https://issues.apache.org/jira/browse/FLINK-19127 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Timo Walther >Priority: Major > > Connecting to a remote cluster from the unified TableEnvironment is neither > tested nor documented. Since StreamExecutionEnvironment is not necessary > anymore, users should be able to do the same in TableEnvironment easily. This > is in particular useful for interactive sessions that run in an IDE, shell, > or notebook. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] shuiqiangchen commented on pull request #13295: [FLINK-18598][python][docs] Add instructions for asynchronous execute in PyFlink doc.
shuiqiangchen commented on pull request #13295: URL: https://github.com/apache/flink/pull/13295#issuecomment-687096968 @dianfu Highly appreciate for your suggestions, I have updated the PR, please have a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13241: [hotfix] Fix a typo in the watermark docs
XComp commented on pull request #13241: URL: https://github.com/apache/flink/pull/13241#issuecomment-687095970 FYI: The Chinese version of this file is already translated and, therefore, does not need to be changed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13330: [FLINK-19004][table-planner-blink] Use FIRST_VALUE when re-write non-…
flinkbot commented on pull request #13330: URL: https://github.com/apache/flink/pull/13330#issuecomment-687094698 ## CI report: * dda51e888c3f58e83346118f03ad55075e990eb8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13216: [FLINK-18999][table-planner-blink][hive] Temporary generic table does…
flinkbot edited a comment on pull request #13216: URL: https://github.com/apache/flink/pull/13216#issuecomment-678268420 ## CI report: * 31eef5a5d1f413435c3a3ccd9764c185b66c1c08 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5970) * 167e78ad9d9f8fdb79ce5b2483feed7f120456a3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
flinkbot edited a comment on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-672766836 ## CI report: * 313b80f4474455d7013b0852929b5a8458f391a1 UNKNOWN * ece578a9fdbaee4d815de501187d92a729790c9b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6196) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6174) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp edited a comment on pull request #13270: [hotfix] [javadocs] fix typo in TaskManagerServices
XComp edited a comment on pull request #13270: URL: https://github.com/apache/flink/pull/13270#issuecomment-687094033 Thanks for your contribution. Not sure whether this fix is necessary since the `iff` is also perfectly fine here, though (see [Wikipedia](https://en.wikipedia.org/wiki/If_and_only_if)). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp commented on pull request #13270: [hotfix] [javadocs] fix typo in TaskManagerServices
XComp commented on pull request #13270: URL: https://github.com/apache/flink/pull/13270#issuecomment-687094033 Not sure whether this fix is necessary since the `iff` is also perfectly fine here (see [Wikipedia](https://en.wikipedia.org/wiki/If_and_only_if)). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu commented on a change in pull request #13292: [FLINK-19097][python] Support add_jars() for Python StreamExecutionEnvironment.
dianfu commented on a change in pull request #13292:
URL: https://github.com/apache/flink/pull/13292#discussion_r483563989
##
File path: flink-python/pyflink/datastream/stream_execution_environment.py
##
@@ -553,6 +553,33 @@ def set_python_executable(self, python_exec: str):
.getEnvironmentConfig(self._j_stream_execution_environment)
env_config.setString(jvm.PythonOptions.PYTHON_EXECUTABLE.key(),
python_exec)
+def add_jars(self, jars_path: str):
+"""
+Adds a list of jar files that will be uploaded to the cluster and
referenced by the job.
+
+:param jars_path: Path of jars that delimited by ';'.
+"""
+jvm = get_gateway().jvm
+jars_key =
jvm.org.apache.flink.configuration.PipelineOptions.JARS.key()
+add_jars_to_context_class_loader(jars_path.split(";"))
+env_config = jvm.org.apache.flink.python.util.PythonConfigUtil \
+.getEnvironmentConfig(self._j_stream_execution_environment)
+env_config.setString(jars_key, jars_path)
+
+def add_classpaths(self, classpaths: str):
Review comment:
ditto
##
File path: flink-python/pyflink/datastream/stream_execution_environment.py
##
@@ -553,6 +553,33 @@ def set_python_executable(self, python_exec: str):
.getEnvironmentConfig(self._j_stream_execution_environment)
env_config.setString(jvm.PythonOptions.PYTHON_EXECUTABLE.key(),
python_exec)
+def add_jars(self, jars_path: str):
Review comment:
Do you think it makes sense to change the interface a bit as following?
```suggestion
def add_jars(self, *jars: str):
```
Besides, we should also consider cases where users may call add_jars
multiple times.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-19135) (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
[
https://issues.apache.org/jira/browse/FLINK-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-19135:
-
Affects Version/s: 1.12.0
1.10.0
1.11.0
> (Stream)ExecutionEnvironment.execute() should not throw ExecutionException
> --
>
> Key: FLINK-19135
> URL: https://issues.apache.org/jira/browse/FLINK-19135
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet, API / DataStream
>Affects Versions: 1.10.0, 1.11.0, 1.12.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
>
> In FLINK-14850 we changed the {{execute()}} method to be basically
> {code}
> final JobClient jobClient = executeAsync(...);
> return jobClient.getJobExecutionResult(userClassloader).get();
> {code}
> Unfortunately, this means that {{execute()}} now throws an
> {{ExecutionException}} instead of a {{ProgramInvocationException}} or
> {{JobExecutionException}} as before. The {{ExecutionException}} is wrapping
> the other exceptions that we were throwing before.
> We didn't notice this in tests because most tests use
> {{Test(Stream)Environment}} which overrides the {{execute()}} method and so
> doesn't go through the {{PipelineExecutor}} logic or the normal code path of
> delegating to {{executeAsync()}}.
> We should fix this to go back to the old behaviour.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13330: [FLINK-19004][table-planner-blink] Use FIRST_VALUE when re-write non-…
flinkbot commented on pull request #13330: URL: https://github.com/apache/flink/pull/13330#issuecomment-687087968 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit dda51e888c3f58e83346118f03ad55075e990eb8 (Fri Sep 04 11:30:27 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-19004).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
flinkbot edited a comment on pull request #13306: URL: https://github.com/apache/flink/pull/13306#issuecomment-685717187 ## CI report: * a7f4abab05ad875bf6acef6510fcec6df7d2dc6c UNKNOWN * 8048b8f9ded32203cb0ca0761414ef7186a68e5f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6132) * c7628fc4b97f68c44d7f5e9cf316814c9f73db18 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6212) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19004) Fail to call Hive percentile function together with distinct aggregate call
[
https://issues.apache.org/jira/browse/FLINK-19004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19004:
---
Labels: pull-request-available (was: )
> Fail to call Hive percentile function together with distinct aggregate call
> ---
>
> Key: FLINK-19004
> URL: https://issues.apache.org/jira/browse/FLINK-19004
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / Planner
>Reporter: Rui Li
>Priority: Major
> Labels: pull-request-available
>
> The following test case would fail:
> {code}
> @Test
> public void test() throws Exception {
> TableEnvironment tableEnv = getTableEnvWithHiveCatalog();
> tableEnv.unloadModule("core");
> tableEnv.loadModule("hive", new HiveModule());
> tableEnv.loadModule("core", CoreModule.INSTANCE);
> tableEnv.executeSql("create table src(x int,y int)");
> tableEnv.executeSql("select count(distinct
> y),`percentile`(y,`array`(0.5,0.99)) from src group by x").collect();
> }
> {code}
> The error is:
> {noformat}
> org.apache.flink.table.api.TableException: Cannot generate a valid execution
> plan for the given query:
> FlinkLogicalLegacySink(name=[collect], fields=[EXPR$0, EXPR$1])
> +- FlinkLogicalCalc(select=[EXPR$0, EXPR$1])
>+- FlinkLogicalAggregate(group=[{0}], EXPR$0=[COUNT($1) FILTER $3],
> EXPR$1=[MIN($2) FILTER $4])
> +- FlinkLogicalCalc(select=[x, y, EXPR$1, =(CASE(=($e, 0:BIGINT),
> 0:BIGINT, 1:BIGINT), 0) AS $g_0, =(CASE(=($e, 0:BIGINT), 0:BIGINT, 1:BIGINT),
> 1) AS $g_1])
> +- FlinkLogicalAggregate(group=[{0, 1, 3}], EXPR$1=[percentile($4,
> $2)])
> +- FlinkLogicalExpand(projects=[x, y, $f2, $e, y_0])
>+- FlinkLogicalCalc(select=[x, y, array(0.5:DECIMAL(2, 1),
> 0.99:DECIMAL(3, 2)) AS $f2])
> +- FlinkLogicalLegacyTableSourceScan(table=[[test-catalog,
> default, src, source: [HiveTableSource(x, y) TablePath: default.src,
> PartitionPruned: false, PartitionNums: null]]], fields=[x, y])
> Min aggregate function does not support type: ''ARRAY''.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] lirui-apache opened a new pull request #13330: [FLINK-19004][table-planner-blink] Use FIRST_VALUE when re-write non-…
lirui-apache opened a new pull request #13330: URL: https://github.com/apache/flink/pull/13330 …distinct agg call in FlinkAggregateExpandDistinctAggregatesRule ## What is the purpose of the change Fix the issue that Hive `percentile` UDAF cannot be used together with distinct agg call ## Brief change log - Use `FIRST_VALUE` to rewrite non-distinct agg call in `FlinkAggregateExpandDistinctAggregatesRule::rewriteUsingGroupingSets` - Support `ARRAY` type for `FIRST_VALUE` - Add test ## Verifying this change Existing and added test case ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? NA This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18984) Add tutorial documentation for Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hequn Cheng closed FLINK-18984. --- Resolution: Resolved > Add tutorial documentation for Python DataStream API > > > Key: FLINK-18984 > URL: https://issues.apache.org/jira/browse/FLINK-18984 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18984) Add tutorial documentation for Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190683#comment-17190683 ] Hequn Cheng commented on FLINK-18984: - Resolved in 1.12.0 via 5b18a7496d210eb2174bd0c7e02c67c4acf67ca9 > Add tutorial documentation for Python DataStream API > > > Key: FLINK-18984 > URL: https://issues.apache.org/jira/browse/FLINK-18984 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
