[GitHub] [flink] flinkbot commented on pull request #13744: [FLINK-19766][table-runtime] Introduce File streaming compaction operators
flinkbot commented on pull request #13744: URL: https://github.com/apache/flink/pull/13744#issuecomment-714369975 ## CI report: * 73d36420f9e131efb1ba70053b5ef315f79afd68 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13745: [FLINK-19767] Add AbstractSlotPoolFactory
flinkbot commented on pull request #13745: URL: https://github.com/apache/flink/pull/13745#issuecomment-714370772 ## CI report: * c6f0ef39c65468a7f613e31bac530b148d36a9d4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #13644: [FLINK-19542][k8s]Implement LeaderElectionService and LeaderRetrievalService based on Kubernetes API
wangyang0918 commented on a change in pull request #13644:
URL: https://github.com/apache/flink/pull/13644#discussion_r510031269
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/highavailability/AbstractHaServices.java
##
@@ -0,0 +1,209 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.highavailability;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.blob.BlobStore;
+import org.apache.flink.runtime.blob.BlobStoreService;
+import org.apache.flink.runtime.leaderelection.LeaderElectionService;
+import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalService;
+import org.apache.flink.util.ExceptionUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.concurrent.Executor;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Abstract high availability services based on distributed system(e.g.
Zookeeper, Kubernetes).
+ */
+public abstract class AbstractHaServices implements HighAvailabilityServices {
Review comment:
This is an interesting solution. Unlike `LeaderElectionService`, we
could not get too much benefit from this. So I will leave this refactor in the
future when we want to touch the `ZookeeperHaService` again. For example, get
rid of the `lock-and-release` and use a K8s similar transactional operation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13653: [FLINK-17528][table] Remove RowData#get() and ArrayData#get() and use FieldGetter and ElementGetter instead
flinkbot edited a comment on pull request #13653: URL: https://github.com/apache/flink/pull/13653#issuecomment-709337488 ## CI report: * 0c876d6befc487412629e6a1e8883fa5fbeb31e6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8067) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8080) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8071) * 7d5ed615ba647d30c4b6183b4ef6ed04bc8add75 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8092) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhijiangW commented on a change in pull request #13595: [FLINK-19582][network] Introduce sort-merge based blocking shuffle to Flink
zhijiangW commented on a change in pull request #13595:
URL: https://github.com/apache/flink/pull/13595#discussion_r510058532
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
##
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.disk.FileChannelManager;
+import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
+import org.apache.flink.runtime.io.network.api.serialization.EventSerializer;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.BufferCompressor;
+import org.apache.flink.runtime.io.network.buffer.BufferPool;
+import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
+import org.apache.flink.util.function.SupplierWithException;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.runtime.io.network.buffer.Buffer.DataType;
+import static
org.apache.flink.runtime.io.network.partition.SortBuffer.BufferWithChannel;
+import static org.apache.flink.util.Preconditions.checkElementIndex;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * {@link SortMergeResultPartition} appends records and events to {@link
SortBuffer} and after the {@link SortBuffer}
+ * is full, all data in the {@link SortBuffer} will be copied and spilled to a
{@link PartitionedFile} in subpartition
+ * index order sequentially. Large records that can not be appended to an
empty {@link SortBuffer} will be spilled to
+ * the {@link PartitionedFile} separately.
+ */
+@NotThreadSafe
+public class SortMergeResultPartition extends ResultPartition {
+
+ private final Object lock = new Object();
+
+ /** All active readers which are consuming data from this result
partition now. */
+ @GuardedBy("lock")
+ private final Set readers = new
HashSet<>();
+
+ /** {@link PartitionedFile} produced by this result partition. */
+ @GuardedBy("lock")
+ private PartitionedFile resultFile;
+
+ /** Used to generate random file channel ID. */
+ private final FileChannelManager channelManager;
+
+ /** Number of data buffers (excluding events) written for each
subpartition. */
+ private final int[] numDataBuffers;
+
+ /** A piece of unmanaged memory for data writing. */
+ private final MemorySegment writeBuffer;
+
+ /** Size of network buffer and write buffer. */
+ private final int networkBufferSize;
+
+ /** Current {@link SortBuffer} to append records to. */
+ private SortBuffer currentSortBuffer;
+
+ /** File writer for this result partition. */
+ private PartitionedFileWriter fileWriter;
+
+ public SortMergeResultPartition(
+ String owningTaskName,
+ int partitionIndex,
+ ResultPartitionID partitionId,
+ ResultPartitionType partitionType,
+ int numSubpartitions,
+ int numTargetKeyGroups,
+ int networkBufferSize,
+ ResultPartitionManager partitionManager,
+ FileChannelManager channelManager,
Review comment:
nit: might be better to pass `Path` instead of big component
`channelManager` as constructor argument.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact
[GitHub] [flink] flinkbot edited a comment on pull request #13726: [BP-1.11][FLINK-19557] Trigger LeaderRetrievalListener notification upon ZooKeeper reconnection in ZooKeeperLeaderRetrievalService
flinkbot edited a comment on pull request #13726: URL: https://github.com/apache/flink/pull/13726#issuecomment-713567666 ## CI report: * f0ac074feed577550c7026b190db031561a25f77 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8087) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13746: [FLINK-19675][python] Fix PythonCalcExpandProjectRule to handle cases when the calc node contains WHERE clause, composite fields access and
flinkbot commented on pull request #13746: URL: https://github.com/apache/flink/pull/13746#issuecomment-714469108 ## CI report: * 9becc81983d757b76614749a96a14a94f14aa7d6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19770) mvn clean verify - testConfigurePythonExecution failing
Juha Mynttinen created FLINK-19770: -- Summary: mvn clean verify - testConfigurePythonExecution failing Key: FLINK-19770 URL: https://issues.apache.org/jira/browse/FLINK-19770 Project: Flink Issue Type: Bug Affects Versions: 1.11.2 Reporter: Juha Mynttinen The PR [https://github.com/apache/flink/pull/13322] lately added the test method testConfigurePythonExecution in org.apache.flink.client.cli.PythonProgramOptionsTest. "mvn clean verify" fails for me in testConfigurePythonExecution: ... INFO] Running org.apache.flink.client.cli.PythonProgramOptionsTest [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.433 s <<< FAILURE! - in org.apache.flink.client.cli.PythonProgramOptionsTest [ERROR] testConfigurePythonExecution(org.apache.flink.client.cli.PythonProgramOptionsTest) Time elapsed: 0.019 s <<< ERROR! java.nio.file.NoSuchFileException: target/dummy-job-jar at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) at java.base/java.nio.file.Files.readAttributes(Files.java:1763) at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) at java.base/java.nio.file.Files.walkFileTree(Files.java:2716) at java.base/java.nio.file.Files.walkFileTree(Files.java:2796) at org.apache.flink.client.cli.PythonProgramOptionsTest.testConfigurePythonExecution(PythonProgramOptionsTest.java:131) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) ... [ERROR] Errors: [ERROR]
[GitHub] [flink] TsReaper opened a new pull request #13748: [FLINK-19769][streaming] Reuse StreamRecord in SourceOutputWithWatermarks#collect
TsReaper opened a new pull request #13748: URL: https://github.com/apache/flink/pull/13748 ## What is the purpose of the change `SourceOutputWithWatermarks#collect` always creates a new `StreamRecord` object which can be reused quite easily. We should reuse the `StreamRecord` for optimization. ## Brief change log - Reuse StreamRecord in `SourceOutputWithWatermarks#collect` ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): yes - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #13732: [FLINK-19275][connector-kafka] Support reading and writing Kafka meta… …data
twalthr commented on a change in pull request #13732:
URL: https://github.com/apache/flink/pull/13732#discussion_r510140104
##
File path: flink-core/src/main/java/org/apache/flink/types/Row.java
##
@@ -274,4 +278,119 @@ public static Row join(Row first, Row... remainings) {
return joinedRow;
}
+
+ /**
+* Compares two {@link Row}s for deep equality. This method supports
all conversion classes of the
+* table ecosystem.
+*
+* The current implementation of {@link Row#equals(Object)} is not
able to compare all deeply
+* nested row structures that might be created in the table ecosystem.
For example, it does not
+* support comparing arrays stored in the values of a map. We might
update the {@link #equals(Object)}
+* with this implementation in future versions.
+*/
+ public static boolean deepEquals(Row row, Object other) {
+ return deepEqualsInternal(row, other);
+ }
+
+ /**
+* Compares two {@link List}s of {@link Row} for deep equality. This
method supports all conversion
+* classes of the table ecosystem.
+*
+* The current implementation of {@link Row#equals(Object)} is not
able to compare all deeply
+* nested row structures that might be created in the table ecosystem.
For example, it does not
+* support comparing arrays stored in the values of a map. We might
update the {@link #equals(Object)}
+* with this implementation in future versions.
+*/
+ public static boolean deepEquals(List l1, List l2) {
+ return deepEqualsInternal(l1, l2);
+ }
+
+ private static boolean deepEqualsInternal(Object o1, Object o2) {
+ if (o1 == o2) {
+ return true;
+ } else if (o1 == null || o2 == null) {
+ return false;
+ } else if (o1 instanceof Row && o2 instanceof Row) {
+ return deepEqualsRow((Row) o1, (Row) o2);
+ } else if (o1 instanceof Object[] && o2 instanceof Object[]) {
+ return deepEqualsArray((Object[]) o1, (Object[]) o2);
+ } else if (o1 instanceof Map && o2 instanceof Map) {
+ return deepEqualsMap((Map) o1, (Map) o2);
+ } else if (o1 instanceof List && o2 instanceof List) {
+ return deepEqualsList((List) o1, (List) o2);
+ }
+ return Objects.deepEquals(o1, o2);
+ }
+
+ private static boolean deepEqualsRow(Row row1, Row row2) {
+ if (row1.getKind() != row2.getKind()) {
+ return false;
+ }
+ if (row1.getArity() != row2.getArity()) {
+ return false;
+ }
+ for (int pos = 0; pos < row1.getArity(); pos++) {
+ final Object f1 = row1.getField(pos);
+ final Object f2 = row2.getField(pos);
+ if (!deepEqualsInternal(f1, f2)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private static boolean deepEqualsArray(Object[] a1, Object[] a2) {
+ if (a1.getClass() != a2.getClass()) {
+ return false;
+ }
+ if (a1.length != a2.length) {
+ return false;
+ }
+ for (int pos = 0; pos < a1.length; pos++) {
+ final Object e1 = a1[pos];
+ final Object e2 = a2[pos];
+ if (!deepEqualsInternal(e1, e2)) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ private static boolean deepEqualsMap(Map m1, Map m2)
{
+ // copied from HashMap.equals but with deepEquals comparision
+ if (m1.size() != m2.size()) {
+ return false;
+ }
+ try {
+ for (Map.Entry e : m1.entrySet()) {
+ K key = e.getKey();
+ V value = e.getValue();
+ if (value == null) {
+ if (!(m2.get(key) == null &&
m2.containsKey(key))) {
+ return false;
+ }
+ } else {
+ if (!deepEqualsInternal(value,
m2.get(key))) {
+ return false;
+ }
+ }
+ }
+ } catch (ClassCastException | NullPointerException unused) {
+ return
[GitHub] [flink] XComp commented on pull request #13747: [FLINK-19662][runtime] Exposing the JobManager's memory configuration through /jobmanager/config
XComp commented on pull request #13747: URL: https://github.com/apache/flink/pull/13747#issuecomment-714475833 @tillrohrmann Feel free to have a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19662) Create or extend REST endpoint to expose memory configuration
[ https://issues.apache.org/jira/browse/FLINK-19662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19662: --- Labels: pull-request-available (was: ) > Create or extend REST endpoint to expose memory configuration > - > > Key: FLINK-19662 > URL: https://issues.apache.org/jira/browse/FLINK-19662 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Reporter: Matthias >Assignee: Matthias >Priority: Major > Labels: pull-request-available > > We need to expose the memory configuration finally used within the > JobManager. The effective configuration is calculated differently depending > on the type of cluster (legacy vs containerized). The goal is to have the > configuration being calculated only in one place. Therefore, we want to > forward the information through environment variables similar to the approach > already done for the TaskManager. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rkhachatryan commented on a change in pull request #13648: [FLINK-19632] Introduce a new ResultPartitionType for Approximate Local Recovery
rkhachatryan commented on a change in pull request #13648:
URL: https://github.com/apache/flink/pull/13648#discussion_r510009258
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/PipelinedApproximateSubpartition.java
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import
org.apache.flink.runtime.io.network.buffer.BufferConsumerWithPartialRecordLength;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * A pipelined in-memory only subpartition, which allows to reconnecting after
failure.
+ * Only one view is allowed at a time to read teh subpartition.
+ */
+public class PipelinedApproximateSubpartition extends PipelinedSubpartition {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(PipelinedApproximateSubpartition.class);
+
+ private boolean isPartialBufferCleanupRequired = false;
Review comment:
`@GuardedBy("buffers")` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-15328) Change the page of taskmanager's metric according to flip-49
[ https://issues.apache.org/jira/browse/FLINK-15328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias closed FLINK-15328. Resolution: Duplicate FLINK-19764 was created to cover the new things discussed. I'm going to close this issue. The UI work will be covered in the other ticket. > Change the page of taskmanager's metric according to flip-49 > - > > Key: FLINK-15328 > URL: https://issues.apache.org/jira/browse/FLINK-15328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / REST, Runtime / Web Frontend >Reporter: lining >Priority: Major > Attachments: image-2019-12-19-18-49-42-122.png > > > * according to resource configuration in flip-49 and memory metric, as the > below picture shows: > !image-2019-12-19-18-49-42-122.png|width=521,height=219! > * Status.JVM.Memory.Heap.Used as the usge of Flink Heap > * Status.JVM.Memory.Direct.MemoryUsed - (shuffle total) as the usage of > Flink offHeap > * shuffle used as the usage of shuffle > * managed used as the usage of shuffle > * Status.JVM.Memory.NonHeap.Used as the usage of overhead -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19768) The shell "./yarn-session.sh " not use log4j-session.properties , it use log4j.properties
YUJIANBO created FLINK-19768: Summary: The shell "./yarn-session.sh " not use log4j-session.properties , it use log4j.properties Key: FLINK-19768 URL: https://issues.apache.org/jira/browse/FLINK-19768 Project: Flink Issue Type: Bug Components: Client / Job Submission Affects Versions: 1.11.2 Reporter: YUJIANBO The shell "./yarn-session.sh " not use log4j-session.properties , it use log4j.properties My Flink Job UI shows the $internal.yarn.log-config-file is "/usr/local/flink-1.11.2/conf/log4j.properties",is it a bug? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #13729: [FLINK-19644][hive] Support read specific partition of Hive table in temporal join
JingsongLi commented on a change in pull request #13729:
URL: https://github.com/apache/flink/pull/13729#discussion_r510029011
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/HiveLookupFunction.java
##
@@ -47,29 +59,46 @@
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Properties;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
+import static
org.apache.flink.connectors.hive.HiveTableFactory.LOOKUP_JOIN_CACHE_TTL;
+import static
org.apache.flink.connectors.hive.HiveTableFactory.LOOKUP_JOIN_PARTITION;
+import static
org.apache.flink.table.catalog.hive.util.HivePartitionUtils.getPartitionByPartitionSpecs;
+import static
org.apache.flink.table.catalog.hive.util.HivePartitionUtils.getTableProps;
+import static
org.apache.flink.table.catalog.hive.util.HivePartitionUtils.toHiveTablePartition;
+import static
org.apache.flink.table.catalog.hive.util.HivePartitionUtils.validateAndParsePartitionSpecs;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
/**
- * Lookup table function for filesystem connector tables.
+ * Lookup table function for Hive connector tables.
*/
-public class FileSystemLookupFunction extends
TableFunction {
+public class HiveLookupFunction extends
TableFunction {
Review comment:
But we can let Filesystem supports this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #13729: [FLINK-19644][hive] Support read specific partition of Hive table in temporal join
JingsongLi commented on a change in pull request #13729:
URL: https://github.com/apache/flink/pull/13729#discussion_r510028759
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/HiveLookupFunction.java
##
@@ -134,9 +188,48 @@ public void eval(Object... values) {
}
}
- @VisibleForTesting
- public Duration getCacheTTL() {
- return cacheTTL;
+ private HiveTableInputFormat getHiveTableInputFormat() {
Review comment:
Can we add a `copy(Partitions)` method to `HiveTableInputFormat`? Then
we don't need move so many logicals.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13744: [FLINK-19766][table-runtime] Introduce File streaming compaction operators
flinkbot edited a comment on pull request #13744: URL: https://github.com/apache/flink/pull/13744#issuecomment-714369975 ## CI report: * 73d36420f9e131efb1ba70053b5ef315f79afd68 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8094) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhijiangW commented on a change in pull request #13595: [FLINK-19582][network] Introduce sort-merge based blocking shuffle to Flink
zhijiangW commented on a change in pull request #13595:
URL: https://github.com/apache/flink/pull/13595#discussion_r510053495
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
##
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.disk.FileChannelManager;
+import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
+import org.apache.flink.runtime.io.network.api.serialization.EventSerializer;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.BufferCompressor;
+import org.apache.flink.runtime.io.network.buffer.BufferPool;
+import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
+import org.apache.flink.util.function.SupplierWithException;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.runtime.io.network.buffer.Buffer.DataType;
+import static
org.apache.flink.runtime.io.network.partition.SortBuffer.BufferWithChannel;
+import static org.apache.flink.util.Preconditions.checkElementIndex;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * {@link SortMergeResultPartition} appends records and events to {@link
SortBuffer} and after the {@link SortBuffer}
+ * is full, all data in the {@link SortBuffer} will be copied and spilled to a
{@link PartitionedFile} in subpartition
+ * index order sequentially. Large records that can not be appended to an
empty {@link SortBuffer} will be spilled to
+ * the {@link PartitionedFile} separately.
+ */
+@NotThreadSafe
+public class SortMergeResultPartition extends ResultPartition {
+
+ private final Object lock = new Object();
+
+ /** All active readers which are consuming data from this result
partition now. */
+ @GuardedBy("lock")
+ private final Set readers = new
HashSet<>();
+
+ /** {@link PartitionedFile} produced by this result partition. */
+ @GuardedBy("lock")
+ private PartitionedFile resultFile;
+
+ /** Used to generate random file channel ID. */
+ private final FileChannelManager channelManager;
+
+ /** Number of data buffers (excluding events) written for each
subpartition. */
+ private final int[] numDataBuffers;
+
+ /** A piece of unmanaged memory for data writing. */
+ private final MemorySegment writeBuffer;
+
+ /** Size of network buffer and write buffer. */
+ private final int networkBufferSize;
+
+ /** Current {@link SortBuffer} to append records to. */
+ private SortBuffer currentSortBuffer;
+
+ /** File writer for this result partition. */
+ private PartitionedFileWriter fileWriter;
+
+ public SortMergeResultPartition(
+ String owningTaskName,
+ int partitionIndex,
+ ResultPartitionID partitionId,
+ ResultPartitionType partitionType,
+ int numSubpartitions,
+ int numTargetKeyGroups,
+ int networkBufferSize,
+ ResultPartitionManager partitionManager,
+ FileChannelManager channelManager,
+ @Nullable BufferCompressor bufferCompressor,
+ SupplierWithException
bufferPoolFactory) {
+
+ super(
+ owningTaskName,
+ partitionIndex,
+ partitionId,
+ partitionType,
+ numSubpartitions,
+ numTargetKeyGroups,
+
[GitHub] [flink] flinkbot edited a comment on pull request #13745: [FLINK-19767] Add AbstractSlotPoolFactory
flinkbot edited a comment on pull request #13745: URL: https://github.com/apache/flink/pull/13745#issuecomment-714370772 ## CI report: * c6f0ef39c65468a7f613e31bac530b148d36a9d4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8095) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13742: [FLINK-19626][table-planner-blink] Introduce multi-input operator construction algorithm
flinkbot edited a comment on pull request #13742: URL: https://github.com/apache/flink/pull/13742#issuecomment-714296461 ## CI report: * ddbf38d29507b98186d2b1a22716e7fb9593f5db Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8091) * 3c50597400946ab21e7dba24673a0b5df30eaf2b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13653: [FLINK-17528][table] Remove RowData#get() and ArrayData#get() and use FieldGetter and ElementGetter instead
flinkbot edited a comment on pull request #13653: URL: https://github.com/apache/flink/pull/13653#issuecomment-709337488 ## CI report: * 0c876d6befc487412629e6a1e8883fa5fbeb31e6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8067) Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8080) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8071) * 7d5ed615ba647d30c4b6183b4ef6ed04bc8add75 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8092) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19596) Do not recover CompletedCheckpointStore on each failover
[
https://issues.apache.org/jira/browse/FLINK-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Liao closed FLINK-19596.
--
Resolution: Fixed
Fixed in https://issues.apache.org/jira/browse/FLINK-19401
> Do not recover CompletedCheckpointStore on each failover
>
>
> Key: FLINK-19596
> URL: https://issues.apache.org/jira/browse/FLINK-19596
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.2
>Reporter: Jiayi Liao
>Priority: Critical
> Fix For: 1.12.0
>
>
> {{completedCheckpointStore.recover()}} in
> {{restoreLatestCheckpointedStateInternal}} could be a bottleneck on failover
> because the {{CompletedCheckpointStore}} needs to load HDFS files to
> instantialize the {{CompleteCheckpoint}} instances.
> The impact is significant in our case below:
> * Jobs with high parallelism (no shuffle) which transfer data from Kafka to
> other filesystems.
> * If a machine goes down, several containers and tens of tasks are affected,
> which means the {{completedCheckpointStore.recover()}} would be called tens
> of times since the tasks are not in a failover region.
> And I notice there is a "TODO" in the source codes:
> {code:java}
> // Recover the checkpoints, TODO this could be done only when there is a new
> leader, not on each recovery
> completedCheckpointStore.recover();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangxlong commented on pull request #13672: [FLINK-19660][docs][python] Translate page "table_environment.zh.md" into Chinese
wangxlong commented on pull request #13672: URL: https://github.com/apache/flink/pull/13672#issuecomment-714450587 Hi @dianfu, could you help cc in your free time, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13300: [FLINK-19077][table-runtime] Import process time temporal join operator.
flinkbot edited a comment on pull request #13300: URL: https://github.com/apache/flink/pull/13300#issuecomment-684957567 ## CI report: * 831c8b47f5ac23fabf85fc6e1e868a39e057e540 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7810) * 26d7664c3772665425813a493ae546208dfae2bb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhijiangW commented on a change in pull request #13595: [FLINK-19582][network] Introduce sort-merge based blocking shuffle to Flink
zhijiangW commented on a change in pull request #13595:
URL: https://github.com/apache/flink/pull/13595#discussion_r510014166
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
##
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.disk.FileChannelManager;
+import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
+import org.apache.flink.runtime.io.network.api.serialization.EventSerializer;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.BufferCompressor;
+import org.apache.flink.runtime.io.network.buffer.BufferPool;
+import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
+import org.apache.flink.util.function.SupplierWithException;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.runtime.io.network.buffer.Buffer.DataType;
+import static
org.apache.flink.runtime.io.network.partition.SortBuffer.BufferWithChannel;
+import static org.apache.flink.util.Preconditions.checkElementIndex;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * {@link SortMergeResultPartition} appends records and events to {@link
SortBuffer} and after the {@link SortBuffer}
+ * is full, all data in the {@link SortBuffer} will be copied and spilled to a
{@link PartitionedFile} in subpartition
+ * index order sequentially. Large records that can not be appended to an
empty {@link SortBuffer} will be spilled to
+ * the {@link PartitionedFile} separately.
+ */
+@NotThreadSafe
+public class SortMergeResultPartition extends ResultPartition {
+
+ private final Object lock = new Object();
+
+ /** All active readers which are consuming data from this result
partition now. */
+ @GuardedBy("lock")
+ private final Set readers = new
HashSet<>();
+
+ /** {@link PartitionedFile} produced by this result partition. */
+ @GuardedBy("lock")
+ private PartitionedFile resultFile;
+
+ /** Used to generate random file channel ID. */
+ private final FileChannelManager channelManager;
+
+ /** Number of data buffers (excluding events) written for each
subpartition. */
+ private final int[] numDataBuffers;
+
+ /** A piece of unmanaged memory for data writing. */
+ private final MemorySegment writeBuffer;
+
+ /** Size of network buffer and write buffer. */
+ private final int networkBufferSize;
+
+ /** Current {@link SortBuffer} to append records to. */
+ private SortBuffer currentSortBuffer;
+
+ /** File writer for this result partition. */
+ private PartitionedFileWriter fileWriter;
+
+ public SortMergeResultPartition(
+ String owningTaskName,
+ int partitionIndex,
+ ResultPartitionID partitionId,
+ ResultPartitionType partitionType,
+ int numSubpartitions,
+ int numTargetKeyGroups,
+ int networkBufferSize,
+ ResultPartitionManager partitionManager,
+ FileChannelManager channelManager,
+ @Nullable BufferCompressor bufferCompressor,
+ SupplierWithException
bufferPoolFactory) {
+
+ super(
+ owningTaskName,
+ partitionIndex,
+ partitionId,
+ partitionType,
+ numSubpartitions,
+ numTargetKeyGroups,
+
[GitHub] [flink] flinkbot edited a comment on pull request #13300: [FLINK-19077][table-runtime] Import process time temporal join operator.
flinkbot edited a comment on pull request #13300: URL: https://github.com/apache/flink/pull/13300#issuecomment-684957567 ## CI report: * 26d7664c3772665425813a493ae546208dfae2bb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8090) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19765) flink SqlUseCatalog.getCatalogName is not unified with SqlCreateCatalog and SqlDropCatalog
[
https://issues.apache.org/jira/browse/FLINK-19765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218900#comment-17218900
]
Jark Wu commented on FLINK-19765:
-
Yes, we can improve them to just return a String.
Btw, {{SqlUseCatalog}} ... classes are not public API, be careful to use them.
It is suggested to use {{UseCatalogOperation}} instead which is located in
{{flink-table-api-java}}.
> flink SqlUseCatalog.getCatalogName is not unified with SqlCreateCatalog and
> SqlDropCatalog
> --
>
> Key: FLINK-19765
> URL: https://issues.apache.org/jira/browse/FLINK-19765
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: jackylau
>Priority: Major
> Fix For: 1.12.0
>
>
> when i develop flink ranger plugin at operation level, i find this method not
> unified.
> And SqlToOperationConverter.convert needs has the good order for user to find
> code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13653: [FLINK-17528][table] Remove RowData#get() and ArrayData#get() and use FieldGetter and ElementGetter instead
flinkbot edited a comment on pull request #13653: URL: https://github.com/apache/flink/pull/13653#issuecomment-709337488 ## CI report: * 0c876d6befc487412629e6a1e8883fa5fbeb31e6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8067) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8080) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8071) * 7d5ed615ba647d30c4b6183b4ef6ed04bc8add75 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #13729: [FLINK-19644][hive] Support read specific partition of Hive table in temporal join
JingsongLi commented on a change in pull request #13729:
URL: https://github.com/apache/flink/pull/13729#discussion_r510029692
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/connectors/hive/read/HiveLookupFunction.java
##
@@ -47,29 +59,46 @@
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Properties;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
+import static
org.apache.flink.connectors.hive.HiveTableFactory.LOOKUP_JOIN_CACHE_TTL;
+import static
org.apache.flink.connectors.hive.HiveTableFactory.LOOKUP_JOIN_PARTITION;
+import static
org.apache.flink.table.catalog.hive.util.HivePartitionUtils.getPartitionByPartitionSpecs;
+import static
org.apache.flink.table.catalog.hive.util.HivePartitionUtils.getTableProps;
+import static
org.apache.flink.table.catalog.hive.util.HivePartitionUtils.toHiveTablePartition;
+import static
org.apache.flink.table.catalog.hive.util.HivePartitionUtils.validateAndParsePartitionSpecs;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
/**
- * Lookup table function for filesystem connector tables.
+ * Lookup table function for Hive connector tables.
*/
-public class FileSystemLookupFunction extends
TableFunction {
+public class HiveLookupFunction extends
TableFunction {
Review comment:
I think we should a better code design, to make the components more
decoupled.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #13595: [FLINK-19582][network] Introduce sort-merge based blocking shuffle to Flink
gaoyunhaii commented on a change in pull request #13595:
URL: https://github.com/apache/flink/pull/13595#discussion_r510037094
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartitionFactory.java
##
@@ -136,26 +144,41 @@ public ResultPartition create(
partition = pipelinedPartition;
}
- else if (type == ResultPartitionType.BLOCKING || type ==
ResultPartitionType.BLOCKING_PERSISTENT) {
- final BoundedBlockingResultPartition blockingPartition
= new BoundedBlockingResultPartition(
- taskNameWithSubtaskAndId,
- partitionIndex,
- id,
- type,
- subpartitions,
- maxParallelism,
- partitionManager,
- bufferCompressor,
- bufferPoolFactory);
-
- initializeBoundedBlockingPartitions(
- subpartitions,
- blockingPartition,
- blockingSubpartitionType,
- networkBufferSize,
- channelManager);
-
- partition = blockingPartition;
+ else if (isBlockingShuffle(type)) {
+ if (isSortMergeBlockingShuffle(type,
numberOfSubpartitions)) {
+ partition = new SortMergeResultPartition(
+ taskNameWithSubtaskAndId,
+ partitionIndex,
+ id,
+ type,
+ subpartitions.length,
+ maxParallelism,
+ networkBufferSize,
+ partitionManager,
+ channelManager,
+ bufferCompressor,
+ bufferPoolFactory);
+ } else {
+ final BoundedBlockingResultPartition
blockingPartition = new BoundedBlockingResultPartition(
Review comment:
directly write as `partition = xxx` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhijiangW commented on a change in pull request #13595: [FLINK-19582][network] Introduce sort-merge based blocking shuffle to Flink
zhijiangW commented on a change in pull request #13595:
URL: https://github.com/apache/flink/pull/13595#discussion_r510055491
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java
##
@@ -0,0 +1,360 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.partition;
+
+import org.apache.flink.core.memory.MemorySegment;
+import org.apache.flink.core.memory.MemorySegmentFactory;
+import org.apache.flink.runtime.event.AbstractEvent;
+import org.apache.flink.runtime.io.disk.FileChannelManager;
+import org.apache.flink.runtime.io.network.api.EndOfPartitionEvent;
+import org.apache.flink.runtime.io.network.api.serialization.EventSerializer;
+import org.apache.flink.runtime.io.network.buffer.Buffer;
+import org.apache.flink.runtime.io.network.buffer.BufferCompressor;
+import org.apache.flink.runtime.io.network.buffer.BufferPool;
+import org.apache.flink.runtime.io.network.buffer.NetworkBuffer;
+import org.apache.flink.util.function.SupplierWithException;
+
+import javax.annotation.Nullable;
+import javax.annotation.concurrent.GuardedBy;
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+
+import static org.apache.flink.runtime.io.network.buffer.Buffer.DataType;
+import static
org.apache.flink.runtime.io.network.partition.SortBuffer.BufferWithChannel;
+import static org.apache.flink.util.Preconditions.checkElementIndex;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * {@link SortMergeResultPartition} appends records and events to {@link
SortBuffer} and after the {@link SortBuffer}
+ * is full, all data in the {@link SortBuffer} will be copied and spilled to a
{@link PartitionedFile} in subpartition
+ * index order sequentially. Large records that can not be appended to an
empty {@link SortBuffer} will be spilled to
+ * the {@link PartitionedFile} separately.
+ */
+@NotThreadSafe
+public class SortMergeResultPartition extends ResultPartition {
+
+ private final Object lock = new Object();
+
+ /** All active readers which are consuming data from this result
partition now. */
+ @GuardedBy("lock")
+ private final Set readers = new
HashSet<>();
+
+ /** {@link PartitionedFile} produced by this result partition. */
+ @GuardedBy("lock")
+ private PartitionedFile resultFile;
+
+ /** Used to generate random file channel ID. */
+ private final FileChannelManager channelManager;
+
+ /** Number of data buffers (excluding events) written for each
subpartition. */
+ private final int[] numDataBuffers;
+
+ /** A piece of unmanaged memory for data writing. */
+ private final MemorySegment writeBuffer;
+
+ /** Size of network buffer and write buffer. */
+ private final int networkBufferSize;
+
+ /** Current {@link SortBuffer} to append records to. */
+ private SortBuffer currentSortBuffer;
+
+ /** File writer for this result partition. */
+ private PartitionedFileWriter fileWriter;
+
+ public SortMergeResultPartition(
+ String owningTaskName,
+ int partitionIndex,
+ ResultPartitionID partitionId,
+ ResultPartitionType partitionType,
+ int numSubpartitions,
+ int numTargetKeyGroups,
+ int networkBufferSize,
+ ResultPartitionManager partitionManager,
+ FileChannelManager channelManager,
+ @Nullable BufferCompressor bufferCompressor,
+ SupplierWithException
bufferPoolFactory) {
+
+ super(
+ owningTaskName,
+ partitionIndex,
+ partitionId,
+ partitionType,
+ numSubpartitions,
+ numTargetKeyGroups,
+
[GitHub] [flink] SteNicholas removed a comment on pull request #13637: [FLINK-19494][DataStream API] Adjust StreamExecutionEnvironment.generateSequence() to new API Sources
SteNicholas removed a comment on pull request #13637: URL: https://github.com/apache/flink/pull/13637#issuecomment-712550577 @StephanEwen ,thanks for your detailed explanation. I have already followed the comments you mentioned. @aljoscha , what do you think about the name of `fromSequence()` method? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas commented on pull request #13637: [FLINK-19494][DataStream API] Adjust StreamExecutionEnvironment.generateSequence() to new API Sources
SteNicholas commented on pull request #13637: URL: https://github.com/apache/flink/pull/13637#issuecomment-714432932 @StephanEwen ,thanks for your detailed explanation. I have followed the comments you mentioned to update the JavaDoc. @aljoscha , what do you think about the naming of `fromSequence()` method? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13703: [FLINK-19696] Add runtime batch committer operators for the new sink API
flinkbot edited a comment on pull request #13703: URL: https://github.com/apache/flink/pull/13703#issuecomment-712821004 ## CI report: * 58ee54d456da57f99b6124c787b7cbaeedcfb261 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8081) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19757) TimeStampData can cause time inconsistent problem
[
https://issues.apache.org/jira/browse/FLINK-19757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218955#comment-17218955
]
Jark Wu commented on FLINK-19757:
-
We stores the epoch seconds since {{1970-01-01 00:00:00}} for TIMESTAMP type,
you can also think it as of epoch sconds since {{1970-01-01 00:00:00Z}} in UTC
time zone.
What's the problem in {{WatermarkGenerator/StreamExecCalc }} ??
> TimeStampData can cause time inconsistent problem
> -
>
> Key: FLINK-19757
> URL: https://issues.apache.org/jira/browse/FLINK-19757
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
>Reporter: xiaogang zhou
>Priority: Major
> Labels: pull-request-available
>
> when we check jdk LocalDateTime code，we find that
>
> {code:java}
> // code placeholder
> public static LocalDateTime ofEpochSecond(long epochSecond, int nanoOfSecond,
> ZoneOffset offset) {
> Objects.requireNonNull(offset, "offset");
> NANO_OF_SECOND.checkValidValue(nanoOfSecond);
> long localSecond = epochSecond + offset.getTotalSeconds(); // overflow
> caught later
> long localEpochDay = Math.floorDiv(localSecond, SECONDS_PER_DAY);
> int secsOfDay = (int)Math.floorMod(localSecond, SECONDS_PER_DAY);
> LocalDate date = LocalDate.ofEpochDay(localEpochDay);
> LocalTime time = LocalTime.ofNanoOfDay(secsOfDay * NANOS_PER_SECOND +
> nanoOfSecond);
> return new LocalDateTime(date, time);
> }
> {code}
>
> offset.getTotalSeconds() they add the offset, but in the TimeStampData
> toLocalDateTime, we don't add a offset.
>
> I'd like to add a TimeZone.getDefault().getRawOffset() in the
> toLocalDateTime()
> and minus a TimeZone.getDefault().getRawOffset() in the
> fromLocalDateTime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19757) TimeStampData can cause time inconsistent problem
[
https://issues.apache.org/jira/browse/FLINK-19757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218976#comment-17218976
]
Jark Wu edited comment on FLINK-19757 at 10/22/20, 12:25 PM:
-
As I said above, your problem is because of the wrong return type of PROCTIME()
function. We should fix PROCTIME() return type to return a {{TIMESTAMP WITH
LOCAL TIME ZONE}} . This is not a trivial work, because that means we need to
support {{TIMESTAMP WITH LOCAL TIME ZONE}} type as time attribute for
time-based operations (e.g. windows).
The code generation is correct.
was (Author: jark):
As I said above, the your problem is because of the wrong return type of
PROCTIME() function. We should fix PROCTIME() return type to return a
{{TIMESTAMP WITH LOCAL TIME ZONE}}. The code generation is correct.
> TimeStampData can cause time inconsistent problem
> -
>
> Key: FLINK-19757
> URL: https://issues.apache.org/jira/browse/FLINK-19757
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
>Reporter: xiaogang zhou
>Priority: Major
> Labels: pull-request-available
>
> when we check jdk LocalDateTime code，we find that
>
> {code:java}
> // code placeholder
> public static LocalDateTime ofEpochSecond(long epochSecond, int nanoOfSecond,
> ZoneOffset offset) {
> Objects.requireNonNull(offset, "offset");
> NANO_OF_SECOND.checkValidValue(nanoOfSecond);
> long localSecond = epochSecond + offset.getTotalSeconds(); // overflow
> caught later
> long localEpochDay = Math.floorDiv(localSecond, SECONDS_PER_DAY);
> int secsOfDay = (int)Math.floorMod(localSecond, SECONDS_PER_DAY);
> LocalDate date = LocalDate.ofEpochDay(localEpochDay);
> LocalTime time = LocalTime.ofNanoOfDay(secsOfDay * NANOS_PER_SECOND +
> nanoOfSecond);
> return new LocalDateTime(date, time);
> }
> {code}
>
> offset.getTotalSeconds() they add the offset, but in the TimeStampData
> toLocalDateTime, we don't add a offset.
>
> I'd like to add a TimeZone.getDefault().getRawOffset() in the
> toLocalDateTime()
> and minus a TimeZone.getDefault().getRawOffset() in the
> fromLocalDateTime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-19757) TimeStampData can cause time inconsistent problem
[
https://issues.apache.org/jira/browse/FLINK-19757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218976#comment-17218976
]
Jark Wu edited comment on FLINK-19757 at 10/22/20, 12:26 PM:
-
As I said above, your problem is because of the wrong return type of PROCTIME()
function. We should fix PROCTIME() return type to return a {{TIMESTAMP WITH
LOCAL TIME ZONE}} . This is not a trivial work, because that means we need to
support {{TIMESTAMP WITH LOCAL TIME ZONE}} type as time attribute for
time-based operations (e.g. windows).
Once the return type is fixed, the generated code will be correct.
was (Author: jark):
As I said above, your problem is because of the wrong return type of PROCTIME()
function. We should fix PROCTIME() return type to return a {{TIMESTAMP WITH
LOCAL TIME ZONE}} . This is not a trivial work, because that means we need to
support {{TIMESTAMP WITH LOCAL TIME ZONE}} type as time attribute for
time-based operations (e.g. windows).
The code generation is correct.
> TimeStampData can cause time inconsistent problem
> -
>
> Key: FLINK-19757
> URL: https://issues.apache.org/jira/browse/FLINK-19757
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Affects Versions: 1.11.1
>Reporter: xiaogang zhou
>Priority: Major
> Labels: pull-request-available
>
> when we check jdk LocalDateTime code，we find that
>
> {code:java}
> // code placeholder
> public static LocalDateTime ofEpochSecond(long epochSecond, int nanoOfSecond,
> ZoneOffset offset) {
> Objects.requireNonNull(offset, "offset");
> NANO_OF_SECOND.checkValidValue(nanoOfSecond);
> long localSecond = epochSecond + offset.getTotalSeconds(); // overflow
> caught later
> long localEpochDay = Math.floorDiv(localSecond, SECONDS_PER_DAY);
> int secsOfDay = (int)Math.floorMod(localSecond, SECONDS_PER_DAY);
> LocalDate date = LocalDate.ofEpochDay(localEpochDay);
> LocalTime time = LocalTime.ofNanoOfDay(secsOfDay * NANOS_PER_SECOND +
> nanoOfSecond);
> return new LocalDateTime(date, time);
> }
> {code}
>
> offset.getTotalSeconds() they add the offset, but in the TimeStampData
> toLocalDateTime, we don't add a offset.
>
> I'd like to add a TimeZone.getDefault().getRawOffset() in the
> toLocalDateTime()
> and minus a TimeZone.getDefault().getRawOffset() in the
> fromLocalDateTime
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] TsReaper commented on pull request #13636: [FLINK-19641][hive] Optimize parallelism calculating of HiveTableSource by checking file number
TsReaper commented on pull request #13636: URL: https://github.com/apache/flink/pull/13636#issuecomment-714471424 > Thanks for contribution, we should not rely on `StatsSetupConst.NUM_FILES`, because in Hive metastore, these stats information is just a hint, and they may not correct. > > We can get file numbers from file system directly. You should make a performance testing for that. Indeed. I've tested the performance of listing the table file directory and its impact is neglectable (at least for TPCDS benchmark). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19770) mvn clean verify - testConfigurePythonExecution failing
[ https://issues.apache.org/jira/browse/FLINK-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juha Mynttinen updated FLINK-19770: --- Component/s: Build System > mvn clean verify - testConfigurePythonExecution failing > --- > > Key: FLINK-19770 > URL: https://issues.apache.org/jira/browse/FLINK-19770 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.11.2 >Reporter: Juha Mynttinen >Priority: Major > > The PR [https://github.com/apache/flink/pull/13322] lately added the test > method testConfigurePythonExecution in > org.apache.flink.client.cli.PythonProgramOptionsTest. > > "mvn clean verify" fails for me in testConfigurePythonExecution: > > ... > INFO] Running org.apache.flink.client.cli.PythonProgramOptionsTest > [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.433 > s <<< FAILURE! - in org.apache.flink.client.cli.PythonProgramOptionsTest > [ERROR] > testConfigurePythonExecution(org.apache.flink.client.cli.PythonProgramOptionsTest) > Time elapsed: 0.019 s <<< ERROR! > java.nio.file.NoSuchFileException: target/dummy-job-jar > at > java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) > at > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) > at > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) > at > java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) > at > java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) > at > java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) > at java.base/java.nio.file.Files.readAttributes(Files.java:1763) > at > java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) > at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) > at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2716) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2796) > at > org.apache.flink.client.cli.PythonProgramOptionsTest.testConfigurePythonExecution(PythonProgramOptionsTest.java:131) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runners.Suite.runChild(Suite.java:128) > at org.junit.runners.Suite.runChild(Suite.java:27) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75) > at >
[GitHub] [flink] wangxlong closed pull request #10356: [FLINK-14902][Connector-JDBC]JDBCTableSource Supports AsyncLookupFunction
wangxlong closed pull request #10356: URL: https://github.com/apache/flink/pull/10356 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19644) Support read specific partition of Hive table in temporal join
[ https://issues.apache.org/jira/browse/FLINK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219439#comment-17219439 ] Leonard Xu edited comment on FLINK-19644 at 10/23/20, 4:39 AM: --- Thanks [~lirui] [~lzljs3620320] [~jark] for sharing your opinions, the option for hive temporal table may influence with hive streaming read options, So looks like we need a unified design for all hive table options I'll prepare a design doc and discuss further was (Author: leonard xu): How about the Option? read.partition-mode = 'max / all' read.partition-order= 'partition-time/ create-time/ partition-name' read.reload-interval= '1 h' > Support read specific partition of Hive table in temporal join > -- > > Key: FLINK-19644 > URL: https://issues.apache.org/jira/browse/FLINK-19644 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / Ecosystem >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > It's a common case to use hive partitioned table as dimension table. > Currently Hive table only supports load all data, It will be helpful if we > can support read user specific partition in temporal table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13696: [FLINK-19726][table] Implement new providers for blink planner
flinkbot edited a comment on pull request #13696: URL: https://github.com/apache/flink/pull/13696#issuecomment-712617122 ## CI report: * f2e5f3867663c020ec292722b63d5788b43fd3b2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8096) * d1440ea836055d51bb808033dbc2e28c11a1599a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8146) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13755: [FLINK-19700][k8s] Make Kubernetes Client in KubernetesResourceManagerDriver use io executor
flinkbot edited a comment on pull request #13755: URL: https://github.com/apache/flink/pull/13755#issuecomment-714883306 ## CI report: * e31a8a6b6e3c8628503b4534c2bdff345365c1e0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8142) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13755: [FLINK-19700][k8s] Make Kubernetes Client in KubernetesResourceManagerDriver use io executor
xintongsong commented on a change in pull request #13755:
URL: https://github.com/apache/flink/pull/13755#discussion_r510580616
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesResourceManagerDriver.java
##
@@ -85,10 +86,20 @@
public KubernetesResourceManagerDriver(
Configuration flinkConfig,
- FlinkKubeClient kubeClient,
KubernetesResourceManagerDriverConfiguration
configuration) {
super(flinkConfig, GlobalConfiguration.loadConfiguration());
+ this.clusterId =
Preconditions.checkNotNull(configuration.getClusterId());
+ this.podCreationRetryInterval =
Preconditions.checkNotNull(configuration.getPodCreationRetryInterval());
+ this.kubeClient =
KubeClientFactory.fromConfiguration(flinkConfig, getIoExecutor());
Review comment:
I'm afraid this would not work.
`AbstractResourceManagerDriver#ioExecutor` is `null` until the driver is
initialized. The constructor is executed before the initialization, thus
`getIoExecutor` should throw `IllegalStateException`.
I think this also exposes the shortcoming for using different constructors
for production and testing. The `getIoExecutor` in the constructor is not
covered by any test.
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/KubeClientFactory.java
##
@@ -71,7 +74,8 @@ public static FlinkKubeClient fromConfiguration(Configuration
flinkConfig) {
final KubernetesClient client = new
DefaultKubernetesClient(config);
- return new Fabric8FlinkKubeClient(flinkConfig, client,
KubeClientFactory::createThreadPoolForAsyncIO);
+ return new Fabric8FlinkKubeClient(flinkConfig, client,
ioExecutor == null ?
+ KubeClientFactory::createThreadPoolForAsyncIO :
() -> ioExecutor);
Review comment:
Might be better to derive `ioExecutor` in
`fromConfiguration(Configuration flinkConfig)`.
Benefits are:
- Avoid passing null value arguments.
- Improves readability that `fromConfiguration(Configuration flinkConfig)`
itself shows which executor will be used when the user does not specify one.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/entrypoint/component/DefaultDispatcherResourceManagerComponentFactory.java
##
@@ -69,7 +69,7 @@
import java.util.ArrayList;
import java.util.Collection;
import java.util.concurrent.CompletableFuture;
-import java.util.concurrent.Executor;
+import java.util.concurrent.ExecutorService;
Review comment:
I'm not sure about replacing `Executor` with `ExecutorService` for all
these places.
Looking into `Fabric8FlinkKubeClient`, I think the only reason
`FlinkKubeClient` uses `ExecutorService` rather than `Executor` is that, it
will shutdown and release the thread pool in `close`. That is because the
client uses a dedicated thread pool, and if it does not shutdown the thread
pool, other components won't do that either.
Since now the client uses a shared IO executor, it should no longer shutdown
the executor on close. Otherwise, all the other components using that IO
executor would be affected. That means we can replace `ExecutorService` with
`Executor` for `FlinkKubeClient`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13678: [FLINK-19586] Add stream committer operators for new Sink API
flinkbot edited a comment on pull request #13678: URL: https://github.com/apache/flink/pull/13678#issuecomment-711439155 ## CI report: * 463b5c8ed21f93caaeb7b938aa9e72abb35619b2 UNKNOWN * 2ba5bda5c5a15c370c231e1cedfa03fd0bc0b41b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8076) * 9d7e355abedbffb48ef6fa865a3606148d2207d2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13736: [FLINK-19654][python][e2e] Reduce pyflink e2e test parallelism
flinkbot edited a comment on pull request #13736: URL: https://github.com/apache/flink/pull/13736#issuecomment-714183394 ## CI report: * 1e2610d18f4402536abaa663581e178f78c3f743 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8135) * e59eeba8f9eafdaf6874d96ffe934ae8cfe55ea3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19215) "Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test" failed with "Dispatcher REST endpoint has not started within a timeout of 20 sec"
[
https://issues.apache.org/jira/browse/FLINK-19215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219462#comment-17219462
]
Robert Metzger commented on FLINK-19215:
Backported to 1.11 in: 0c01563a220a35db266322a83951aa0ede47cdc8
> "Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test" failed
> with "Dispatcher REST endpoint has not started within a timeout of 20 sec"
>
>
> Key: FLINK-19215
> URL: https://issues.apache.org/jira/browse/FLINK-19215
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.11.0
>Reporter: Dian Fu
>Assignee: Robert Metzger
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6476=logs=08866332-78f7-59e4-4f7e-49a56faa3179=3e8647c1-5a28-5917-dd93-bf78594ea994
> {code}
> 2020-09-13T21:26:23.3646770Z Running 'Resuming Savepoint (rocks, scale down,
> rocks timers) end-to-end test'
> 2020-09-13T21:26:23.3647852Z
> ==
> 2020-09-13T21:26:23.3689605Z TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-23367497881
> 2020-09-13T21:26:23.7122791Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
> 2020-09-13T21:26:23.9988115Z Starting cluster.
> 2020-09-13T21:26:27.3702750Z Starting standalonesession daemon on host
> fv-az655.
> 2020-09-13T21:26:35.1213853Z Starting taskexecutor daemon on host fv-az655.
> 2020-09-13T21:26:35.2756714Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:36.4111928Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:37.5358508Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:38.7156039Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:39.8602294Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:41.0399056Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:42.1680966Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:43.2520250Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:44.3833552Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:45.5204296Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:46.6730448Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:47.8274365Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:49.0147447Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:51.5463623Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:52.7366058Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:53.8867521Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:55.0469025Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:56.1901349Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:57.3124935Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:58.4596457Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:59.4828675Z Dispatcher REST endpoint has not started within
> a timeout of 20 sec
> 2020-09-13T21:26:59.4831446Z [FAIL] Test script contains errors.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13746: [FLINK-19675][python] Fix PythonCalcExpandProjectRule to handle cases when the calc node contains WHERE clause, composite fields acce
flinkbot edited a comment on pull request #13746: URL: https://github.com/apache/flink/pull/13746#issuecomment-714469108 ## CI report: * 9becc81983d757b76614749a96a14a94f14aa7d6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8136) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8105) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13736: [FLINK-19654][python][e2e] Reduce pyflink e2e test parallelism
flinkbot edited a comment on pull request #13736: URL: https://github.com/apache/flink/pull/13736#issuecomment-714183394 ## CI report: * 1e2610d18f4402536abaa663581e178f78c3f743 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8135) * e59eeba8f9eafdaf6874d96ffe934ae8cfe55ea3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8147) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shouweikun commented on a change in pull request #13716: [FLINK-19706][table-runtime] Add WARN logs when hive table partition …
shouweikun commented on a change in pull request #13716:
URL: https://github.com/apache/flink/pull/13716#discussion_r510623126
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/MetastoreCommitPolicy.java
##
@@ -45,6 +45,9 @@ public void setMetastore(TableMetaStore metaStore) {
public void commit(Context context) throws Exception {
LinkedHashMap partitionSpec =
context.partitionSpec();
metaStore.createOrAlterPartition(partitionSpec,
context.partitionPath());
+ if (metaStore.getPartition(partitionSpec).isPresent()) {
Review comment:
Sure, code has been changed following ur suggestion
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rmetzger opened a new pull request #13757: [FLINK-18117][e2e] Add debugging information for hadoop startup
rmetzger opened a new pull request #13757: URL: https://github.com/apache/flink/pull/13757 ## What is the purpose of the change This adds some non-invasive environment debugging to understand why the namenode can not allocate the http port inside a docker container. I was not able to reproduce the failing e2e test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18117) "Kerberized YARN per-job on Docker test" fails with "Could not start hadoop cluster."
[
https://issues.apache.org/jira/browse/FLINK-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-18117:
---
Labels: pull-request-available test-stability (was: test-stability)
> "Kerberized YARN per-job on Docker test" fails with "Could not start hadoop
> cluster."
> -
>
> Key: FLINK-18117
> URL: https://issues.apache.org/jira/browse/FLINK-18117
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2683=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-04T06:03:53.2844296Z Creating slave1 ... [32mdone[0m
> 2020-06-04T06:03:53.4981251Z [1BWaiting for hadoop cluster to come up. We
> have been trying for 0 seconds, retrying ...
> 2020-06-04T06:03:58.5980181Z Waiting for hadoop cluster to come up. We have
> been trying for 5 seconds, retrying ...
> 2020-06-04T06:04:03.6997087Z Waiting for hadoop cluster to come up. We have
> been trying for 10 seconds, retrying ...
> 2020-06-04T06:04:08.7910791Z Waiting for hadoop cluster to come up. We have
> been trying for 15 seconds, retrying ...
> 2020-06-04T06:04:13.8921621Z Waiting for hadoop cluster to come up. We have
> been trying for 20 seconds, retrying ...
> 2020-06-04T06:04:18.9648844Z Waiting for hadoop cluster to come up. We have
> been trying for 25 seconds, retrying ...
> 2020-06-04T06:04:24.0381851Z Waiting for hadoop cluster to come up. We have
> been trying for 31 seconds, retrying ...
> 2020-06-04T06:04:29.1220264Z Waiting for hadoop cluster to come up. We have
> been trying for 36 seconds, retrying ...
> 2020-06-04T06:04:34.1882187Z Waiting for hadoop cluster to come up. We have
> been trying for 41 seconds, retrying ...
> 2020-06-04T06:04:39.2784948Z Waiting for hadoop cluster to come up. We have
> been trying for 46 seconds, retrying ...
> 2020-06-04T06:04:44.3843337Z Waiting for hadoop cluster to come up. We have
> been trying for 51 seconds, retrying ...
> 2020-06-04T06:04:49.4703561Z Waiting for hadoop cluster to come up. We have
> been trying for 56 seconds, retrying ...
> 2020-06-04T06:04:54.5463207Z Waiting for hadoop cluster to come up. We have
> been trying for 61 seconds, retrying ...
> 2020-06-04T06:04:59.6650405Z Waiting for hadoop cluster to come up. We have
> been trying for 66 seconds, retrying ...
> 2020-06-04T06:05:04.7500168Z Waiting for hadoop cluster to come up. We have
> been trying for 71 seconds, retrying ...
> 2020-06-04T06:05:09.8177904Z Waiting for hadoop cluster to come up. We have
> been trying for 76 seconds, retrying ...
> 2020-06-04T06:05:14.9751297Z Waiting for hadoop cluster to come up. We have
> been trying for 81 seconds, retrying ...
> 2020-06-04T06:05:20.0336417Z Waiting for hadoop cluster to come up. We have
> been trying for 87 seconds, retrying ...
> 2020-06-04T06:05:25.1627704Z Waiting for hadoop cluster to come up. We have
> been trying for 92 seconds, retrying ...
> 2020-06-04T06:05:30.2583315Z Waiting for hadoop cluster to come up. We have
> been trying for 97 seconds, retrying ...
> 2020-06-04T06:05:35.3283678Z Waiting for hadoop cluster to come up. We have
> been trying for 102 seconds, retrying ...
> 2020-06-04T06:05:40.4184029Z Waiting for hadoop cluster to come up. We have
> been trying for 107 seconds, retrying ...
> 2020-06-04T06:05:45.5388372Z Waiting for hadoop cluster to come up. We have
> been trying for 112 seconds, retrying ...
> 2020-06-04T06:05:50.6155334Z Waiting for hadoop cluster to come up. We have
> been trying for 117 seconds, retrying ...
> 2020-06-04T06:05:55.7225186Z Command: start_hadoop_cluster failed. Retrying...
> 2020-06-04T06:05:55.7237999Z Starting Hadoop cluster
> 2020-06-04T06:05:56.5188293Z kdc is up-to-date
> 2020-06-04T06:05:56.5292716Z master is up-to-date
> 2020-06-04T06:05:56.5301735Z slave2 is up-to-date
> 2020-06-04T06:05:56.5306179Z slave1 is up-to-date
> 2020-06-04T06:05:56.6800566Z Waiting for hadoop cluster to come up. We have
> been trying for 0 seconds, retrying ...
> 2020-06-04T06:06:01.7668291Z Waiting for hadoop cluster to come up. We have
> been trying for 5 seconds, retrying ...
> 2020-06-04T06:06:06.8620265Z Waiting for hadoop cluster to come up. We have
> been trying for 10 seconds, retrying ...
> 2020-06-04T06:06:11.9753596Z Waiting for hadoop cluster to come up. We have
> been trying for 15 seconds, retrying ...
> 2020-06-04T06:06:17.0402846Z Waiting for hadoop cluster to come up. We have
> been trying for 21
[GitHub] [flink] flinkbot commented on pull request #13757: [FLINK-18117][e2e] Add debugging information for hadoop startup
flinkbot commented on pull request #13757: URL: https://github.com/apache/flink/pull/13757#issuecomment-714929264 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b2cc9638a9d919e924a4a67b8d15e68e4dad530d (Fri Oct 23 05:50:17 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13716: [FLINK-19706][table-runtime] Add WARN logs when hive table partition …
flinkbot edited a comment on pull request #13716: URL: https://github.com/apache/flink/pull/13716#issuecomment-713352648 ## CI report: * 6fa76fb97c2c0307557da240746f375838e98136 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7985) * 9d64eae3bb456919ea24d094894c18ad69db49fd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13678: [FLINK-19586] Add stream committer operators for new Sink API
flinkbot edited a comment on pull request #13678: URL: https://github.com/apache/flink/pull/13678#issuecomment-711439155 ## CI report: * 463b5c8ed21f93caaeb7b938aa9e72abb35619b2 UNKNOWN * 2ba5bda5c5a15c370c231e1cedfa03fd0bc0b41b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8076) * 9d7e355abedbffb48ef6fa865a3606148d2207d2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8148) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13753: [FLINK-19765][flink-table-planner] refactor SqlToOperationConverter a…
flinkbot edited a comment on pull request #13753: URL: https://github.com/apache/flink/pull/13753#issuecomment-714862730 ## CI report: * b85f48049fd05f5c6da1cb91c6dfb4c6d201100c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8138) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19770) PythonProgramOptionsTest requires package phase before execution
[ https://issues.apache.org/jira/browse/FLINK-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219404#comment-17219404 ] Shuiqiang Chen commented on FLINK-19770: [~juha.mynttinen] Thank you for reporting this issue, we should make these test cases as integration tests that would be executed after the package building phase completed as [~chesnay] mentioned. I would like to fix it. > PythonProgramOptionsTest requires package phase before execution > > > Key: FLINK-19770 > URL: https://issues.apache.org/jira/browse/FLINK-19770 > Project: Flink > Issue Type: Bug > Components: API / Python, Tests >Affects Versions: 1.11.2 >Reporter: Juha Mynttinen >Priority: Minor > Labels: starter > Fix For: 1.12.0, 1.11.3 > > > The PR [https://github.com/apache/flink/pull/13322] lately added the test > method testConfigurePythonExecution in > org.apache.flink.client.cli.PythonProgramOptionsTest. > > "mvn clean verify" fails for me in testConfigurePythonExecution: > > ... > INFO] Running org.apache.flink.client.cli.PythonProgramOptionsTest > [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.433 > s <<< FAILURE! - in org.apache.flink.client.cli.PythonProgramOptionsTest > [ERROR] > testConfigurePythonExecution(org.apache.flink.client.cli.PythonProgramOptionsTest) > Time elapsed: 0.019 s <<< ERROR! > java.nio.file.NoSuchFileException: target/dummy-job-jar > at > java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) > at > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) > at > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) > at > java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) > at > java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) > at > java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) > at java.base/java.nio.file.Files.readAttributes(Files.java:1763) > at > java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) > at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) > at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2716) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2796) > at > org.apache.flink.client.cli.PythonProgramOptionsTest.testConfigurePythonExecution(PythonProgramOptionsTest.java:131) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runners.Suite.runChild(Suite.java:128) > at org.junit.runners.Suite.runChild(Suite.java:27) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) > at >
[jira] [Closed] (FLINK-19660) Translate page 'table_environment.zh.md' into Chinese
[ https://issues.apache.org/jira/browse/FLINK-19660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-19660. --- Resolution: Fixed Merged to master via 1f8627be60a3e5b4fd2512f1f9446a2cf139f93d > Translate page 'table_environment.zh.md' into Chinese > - > > Key: FLINK-19660 > URL: https://issues.apache.org/jira/browse/FLINK-19660 > Project: Flink > Issue Type: Improvement > Components: API / Python, chinese-translation, Documentation >Affects Versions: 1.12.0 >Reporter: hailong wang >Assignee: hailong wang >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0 > > > The page url is > [TableEnvironment|https://ci.apache.org/projects/flink/flink-docs-master/zh/dev/python/table-api-users-guide/table_environment.html] > The url is located in > /dev/python/table-api-users-guide/table_environment.zh.md > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13746: [FLINK-19675][python] Fix PythonCalcExpandProjectRule to handle cases when the calc node contains WHERE clause, composite fields acce
flinkbot edited a comment on pull request #13746: URL: https://github.com/apache/flink/pull/13746#issuecomment-714469108 ## CI report: * 9becc81983d757b76614749a96a14a94f14aa7d6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8105) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8136) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13736: [FLINK-19654][python][e2e] Reduce pyflink e2e test parallelism
flinkbot edited a comment on pull request #13736: URL: https://github.com/apache/flink/pull/13736#issuecomment-714183394 ## CI report: * 429e63595a174e68e44a339d2dbccca57f07d6f5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8123) * 1e2610d18f4402536abaa663581e178f78c3f743 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liuyongvs opened a new pull request #13753: [FLINK-19765][flink-table-planner] refactor SqlToOperationConverter a…
liuyongvs opened a new pull request #13753: URL: https://github.com/apache/flink/pull/13753 …nd unified the SqlUseCatalog.getCatalogName() with SqlCreateCatalog and SqlDropCatalog. ## What is the purpose of the change *flink SqlUseCatalog.getCatalogName is not unified with SqlCreateCatalog and SqlDropCatalog* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no ) - The S3 file system connector: (no ) ## Documentation - Does this pull request introduce a new feature? ( no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu closed pull request #13672: [FLINK-19660][docs][python] Translate page "table_environment.zh.md" into Chinese
dianfu closed pull request #13672: URL: https://github.com/apache/flink/pull/13672 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19765) flink SqlUseCatalog.getCatalogName is not unified with SqlCreateCatalog and SqlDropCatalog
[ https://issues.apache.org/jira/browse/FLINK-19765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19765: --- Labels: pull-request-available (was: ) > flink SqlUseCatalog.getCatalogName is not unified with SqlCreateCatalog and > SqlDropCatalog > -- > > Key: FLINK-19765 > URL: https://issues.apache.org/jira/browse/FLINK-19765 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.11.0 >Reporter: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > when i develop flink ranger plugin at operation level, i find this method not > unified. > And SqlToOperationConverter.convert needs has the good order for user to find > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-19775) SystemProcessingTimeServiceTest.testImmediateShutdown is instable
Dian Fu created FLINK-19775:
---
Summary: SystemProcessingTimeServiceTest.testImmediateShutdown is
instable
Key: FLINK-19775
URL: https://issues.apache.org/jira/browse/FLINK-19775
Project: Flink
Issue Type: Bug
Components: API / DataStream
Affects Versions: 1.11.0
Reporter: Dian Fu
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8131=logs=d89de3df-4600-5585-dadc-9bbc9a5e661c=66b5c59a-0094-561d-0e44-b149dfdd586d
{code}
2020-10-22T21:12:54.9462382Z [ERROR]
testImmediateShutdown(org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest)
Time elapsed: 0.009 s <<< ERROR!
2020-10-22T21:12:54.9463024Z java.lang.InterruptedException
2020-10-22T21:12:54.9463331Zat java.lang.Object.wait(Native Method)
2020-10-22T21:12:54.9463766Zat java.lang.Object.wait(Object.java:502)
2020-10-22T21:12:54.9464140Zat
org.apache.flink.core.testutils.OneShotLatch.await(OneShotLatch.java:63)
2020-10-22T21:12:54.9466014Zat
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest.testImmediateShutdown(SystemProcessingTimeServiceTest.java:154)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19644) Support read specific partition of Hive table in temporal join
[
https://issues.apache.org/jira/browse/FLINK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219412#comment-17219412
]
Rui Li commented on FLINK-19644:
[~Leonard Xu] Thanks for driving this efforts! Regarding loading the latest
partition, how would we decide which partition is the "{{max_partition}}"? In
streaming read/write, we have already introduced concepts like "partition-time"
and "create-time", I wonder whether it's possible to reuse those concepts here,
so that we have fewer new stuff for users to understand.
> Support read specific partition of Hive table in temporal join
> --
>
> Key: FLINK-19644
> URL: https://issues.apache.org/jira/browse/FLINK-19644
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Hive, Table SQL / Ecosystem
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> It's a common case to use hive partitioned table as dimension table.
> Currently Hive table only supports load all data, It will be helpful if we
> can support read user specific partition in temporal table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19644) Support read specific partition of Hive table in temporal join
[
https://issues.apache.org/jira/browse/FLINK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219415#comment-17219415
]
Jark Wu commented on FLINK-19644:
-
We can give {{max_partition}} a clear definition which is a strategy to get the
first partition values in the alphabetical order.
[~lirui], how can we reuse "partition-time" and "create-time" here? What option
do you want to introduce?
[~lzljs3620320], sorry, I don't get what you mean. Could you elaborate it a bit
more?
> Support read specific partition of Hive table in temporal join
> --
>
> Key: FLINK-19644
> URL: https://issues.apache.org/jira/browse/FLINK-19644
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Hive, Table SQL / Ecosystem
>Reporter: Leonard Xu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> It's a common case to use hive partitioned table as dimension table.
> Currently Hive table only supports load all data, It will be helpful if we
> can support read user specific partition in temporal table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #13636: [FLINK-19641][hive] Optimize parallelism calculating of HiveTableSource by checking file number
JingsongLi commented on a change in pull request #13636:
URL: https://github.com/apache/flink/pull/13636#discussion_r510558674
##
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/util/HiveStatsUtil.java
##
@@ -69,7 +69,7 @@
public class HiveStatsUtil {
private static final Logger LOG =
LoggerFactory.getLogger(HiveStatsUtil.class);
- private static final int DEFAULT_UNKNOWN_STATS_VALUE = -1;
+ public static final int DEFAULT_UNKNOWN_STATS_VALUE = -1;
Review comment:
Revert
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-19755) flink cep docs exist looping match when coming to 17, 14 using AFTER MATCH SKIP TO LAST A
[
https://issues.apache.org/jira/browse/FLINK-19755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219422#comment-17219422
]
jackylau commented on FLINK-19755:
--
hi [~jark] [~dianfu], i modify the data to solve the looping match, could you
please review it ,thanks?
> flink cep docs exist looping match when coming to 17, 14 using AFTER MATCH
> SKIP TO LAST A
> --
>
> Key: FLINK-19755
> URL: https://issues.apache.org/jira/browse/FLINK-19755
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.11.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> {code:java}
> symbol tax price rowtime
> = === =
> XYZ 1 7 2018-09-17 10:00:01
> XYZ 2 9 2018-09-17 10:00:02
> XYZ 1 10 2018-09-17 10:00:03
> XYZ 2 5 2018-09-17 10:00:04
> XYZ 2 17 2018-09-17 10:00:05
> XYZ 2 14 2018-09-17 10:00:06
> SELECT *
> FROM Ticker
> MATCH_RECOGNIZE(
> PARTITION BY symbol
> ORDER BY rowtime
> MEASURES
> SUM(A.price) AS sumPrice,
> FIRST(rowtime) AS startTime,
> LAST(rowtime) AS endTime
> ONE ROW PER MATCH
> [AFTER MATCH STRATEGY]
> PATTERN (A+ C)
> DEFINE
> A AS SUM(A.price) < 30
> )
>
> {code}
> {code:java}
> AFTER MATCH SKIP TO LAST A
> symbol sumPricestartTime endTime
> == = =
> XYZ 26 2018-09-17 10:00:01 2018-09-17 10:00:04
> XYZ 15 2018-09-17 10:00:03 2018-09-17 10:00:05
> XYZ 22 2018-09-17 10:00:04 2018-09-17 10:00:06
> XYZ 17 2018-09-17 10:00:05 2018-09-17 10:00:06
> Again, the first result matched against the rows #1, #2, #3, #4.Compared to
> the previous strategy, the next match includes only row #3 (mapped to A)
> again for the next matching.
> Therefore, the second result matched against the rows #3, #4, #5.
> The third result matched against the rows #4, #5, #6.
> The last result matched against the rows #5, #6.{code}
> h5. i think it will exist looping match when coming to 17, 14 using "AFTER
> MATCH SKIP TO LAST A "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19755) flink cep docs exist looping match when coming to 17, 14 using AFTER MATCH SKIP TO LAST A
[
https://issues.apache.org/jira/browse/FLINK-19755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jackylau updated FLINK-19755:
-
Attachment: 7CA31459-B125-4cea-8C75-3D101C27A9F9.png
0F2917C8-6C35-43a9-BDD1-FD506906257E.png
> flink cep docs exist looping match when coming to 17, 14 using AFTER MATCH
> SKIP TO LAST A
> --
>
> Key: FLINK-19755
> URL: https://issues.apache.org/jira/browse/FLINK-19755
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.11.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Attachments: 0F2917C8-6C35-43a9-BDD1-FD506906257E.png,
> 7CA31459-B125-4cea-8C75-3D101C27A9F9.png
>
>
> {code:java}
> symbol tax price rowtime
> = === =
> XYZ 1 7 2018-09-17 10:00:01
> XYZ 2 9 2018-09-17 10:00:02
> XYZ 1 10 2018-09-17 10:00:03
> XYZ 2 5 2018-09-17 10:00:04
> XYZ 2 17 2018-09-17 10:00:05
> XYZ 2 14 2018-09-17 10:00:06
> SELECT *
> FROM Ticker
> MATCH_RECOGNIZE(
> PARTITION BY symbol
> ORDER BY rowtime
> MEASURES
> SUM(A.price) AS sumPrice,
> FIRST(rowtime) AS startTime,
> LAST(rowtime) AS endTime
> ONE ROW PER MATCH
> [AFTER MATCH STRATEGY]
> PATTERN (A+ C)
> DEFINE
> A AS SUM(A.price) < 30
> )
>
> {code}
> {code:java}
> AFTER MATCH SKIP TO LAST A
> symbol sumPricestartTime endTime
> == = =
> XYZ 26 2018-09-17 10:00:01 2018-09-17 10:00:04
> XYZ 15 2018-09-17 10:00:03 2018-09-17 10:00:05
> XYZ 22 2018-09-17 10:00:04 2018-09-17 10:00:06
> XYZ 17 2018-09-17 10:00:05 2018-09-17 10:00:06
> Again, the first result matched against the rows #1, #2, #3, #4.Compared to
> the previous strategy, the next match includes only row #3 (mapped to A)
> again for the next matching.
> Therefore, the second result matched against the rows #3, #4, #5.
> The third result matched against the rows #4, #5, #6.
> The last result matched against the rows #5, #6.{code}
> h5. i think it will exist looping match when coming to 17, 14 using "AFTER
> MATCH SKIP TO LAST A "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12647: [FLINK-18044][Connectors/Common] Add the subtask index information to the SourceReaderContext.
flinkbot edited a comment on pull request #12647: URL: https://github.com/apache/flink/pull/12647#issuecomment-643898574 ## CI report: * dde8225437b4208d565a74e9a78c9398eea11025 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8102) * ca6be53b824e950ad93e3412967126bba7fb309b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8139) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19700) Make Kubernetes Client in KubernetesResourceManagerDriver use io executor
[
https://issues.apache.org/jira/browse/FLINK-19700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-19700:
---
Labels: pull-request-available (was: )
> Make Kubernetes Client in KubernetesResourceManagerDriver use io executor
> -
>
> Key: FLINK-19700
> URL: https://issues.apache.org/jira/browse/FLINK-19700
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Reporter: Yang Wang
>Assignee: Nicholas Jiang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Currently, the Kubernetes Client in {{KubernetesResourceManagerDriver}} is
> using a dedicated thread pool. After FLINK-19037 and FLINK-18722, we could
> get the io executor in {{KubernetesResourceManagerDriver}} and eliminate the
> redundant thread pool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] SteNicholas opened a new pull request #13755: [FLINK-19700][k8s] Make Kubernetes Client in KubernetesResourceManagerDriver use io executor
SteNicholas opened a new pull request #13755: URL: https://github.com/apache/flink/pull/13755 ## What is the purpose of the change *Currently, the Kubernetes Client in `KubernetesResourceManagerDriver` is using a dedicated thread pool through calling `KubeClientFactory::createThreadPoolForAsyncIO` to create fixed thread pool. `KubernetesResourceManagerDriver` could get the io executor and eliminate the redundant thread pool. `KubeClientFactory` uses `KubeClientFactory::createThreadPoolForAsyncIO` by default.* ## Brief change log - *`KubernetesResourceManagerDriver` adds constructor with creating `FlinkKubeClient ` through `KubeClientFactory#fromConfiguration(flinkConfig, ioExecutor)` to reuse the io executor of `AbstractResourceManagerDriver`.* ## Verifying this change - *`TestingResourceManagerDriver` updates `initialize()` with `ExecutorService` type parameter to verify the `AbstractResourceManagerDriver` whether to work.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas commented on pull request #13755: [FLINK-19700][k8s] Make Kubernetes Client in KubernetesResourceManagerDriver use io executor
SteNicholas commented on pull request #13755: URL: https://github.com/apache/flink/pull/13755#issuecomment-714875554 @xintongsong @wangyang0918, could you please help to review the pull request if you are available? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19644) Support read specific partition of Hive table in temporal join
[ https://issues.apache.org/jira/browse/FLINK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219021#comment-17219021 ] Leonard Xu edited comment on FLINK-19644 at 10/23/20, 3:13 AM: --- According to my Investigation, a major user case is to load latest hive partition table as dimension table , and the hive table also updated by batch pipeline per day. So, I'd like to use following case to specify the partition: // case 1 : always reload specific partition 'lookup.join.partition' = 'pt_year=2020;pt_month=09;pt_day=15', // case 2: load latest partition on time partition key, use specific partition on other partition key: 'lookup.join.partition' = 'pt_area=china;pt_day=max_partition()', // case 3: always reload latest specific partition: 'lookup.join.partition' = 'pt_month=max_partition();pt_day=max_partition()' [~lirui] [~lzljs3620320] You're familiar with Hive ecosystem, Do you have any insight ? was (Author: leonard xu): According to my Investigation, a major user case is to load latest hive partition table as dimension table , and the hive table also updated by batch pipeline per day. So, I'd like to use following case to specify the partition: // case 1 : always reload specific partition 'lookup.join.partition' = 'pt_year=2020;pt_month=09;pt_day=15', // case 2: load latest partition on time partition key, use specific partition on other partition key: 'lookup.join.partition' = 'pt_area=china;pt_day=max_partition()', // case 3: always reload latest specific partition: 'lookup.join.partition' = 'pt_month=max_partition();pt_day=max_partition' [~lirui] [~lzljs3620320] You're familiar with Hive ecosystem, Do you have any insight ? > Support read specific partition of Hive table in temporal join > -- > > Key: FLINK-19644 > URL: https://issues.apache.org/jira/browse/FLINK-19644 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / Ecosystem >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > It's a common case to use hive partitioned table as dimension table. > Currently Hive table only supports load all data, It will be helpful if we > can support read user specific partition in temporal table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13754: [FLINK-19755][cep][docs] fix flink cep docs of looping matching bug.
flinkbot edited a comment on pull request #13754: URL: https://github.com/apache/flink/pull/13754#issuecomment-714876169 ## CI report: * 3717c92c917638c1dbde99081f6aa890d723f39e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8141) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13755: [FLINK-19700][k8s] Make Kubernetes Client in KubernetesResourceManagerDriver use io executor
flinkbot commented on pull request #13755: URL: https://github.com/apache/flink/pull/13755#issuecomment-714883306 ## CI report: * e31a8a6b6e3c8628503b4534c2bdff345365c1e0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19644) Support read specific partition of Hive table in temporal join
[ https://issues.apache.org/jira/browse/FLINK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219439#comment-17219439 ] Leonard Xu edited comment on FLINK-19644 at 10/23/20, 3:32 AM: --- How about the Option? "read.partition=latest / all " "read.reload-interval=1 h" was (Author: leonard xu): How about the Option? "read.partition=latest-partition / all " "read.reload-interval=1 h" > Support read specific partition of Hive table in temporal join > -- > > Key: FLINK-19644 > URL: https://issues.apache.org/jira/browse/FLINK-19644 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / Ecosystem >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > It's a common case to use hive partitioned table as dimension table. > Currently Hive table only supports load all data, It will be helpful if we > can support read user specific partition in temporal table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19770) PythonProgramOptionsTest requires package phase before execution
[ https://issues.apache.org/jira/browse/FLINK-19770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19770: --- Labels: pull-request-available starter (was: starter) > PythonProgramOptionsTest requires package phase before execution > > > Key: FLINK-19770 > URL: https://issues.apache.org/jira/browse/FLINK-19770 > Project: Flink > Issue Type: Bug > Components: API / Python, Tests >Affects Versions: 1.11.2 >Reporter: Juha Mynttinen >Priority: Minor > Labels: pull-request-available, starter > Fix For: 1.12.0, 1.11.3 > > > The PR [https://github.com/apache/flink/pull/13322] lately added the test > method testConfigurePythonExecution in > org.apache.flink.client.cli.PythonProgramOptionsTest. > > "mvn clean verify" fails for me in testConfigurePythonExecution: > > ... > INFO] Running org.apache.flink.client.cli.PythonProgramOptionsTest > [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.433 > s <<< FAILURE! - in org.apache.flink.client.cli.PythonProgramOptionsTest > [ERROR] > testConfigurePythonExecution(org.apache.flink.client.cli.PythonProgramOptionsTest) > Time elapsed: 0.019 s <<< ERROR! > java.nio.file.NoSuchFileException: target/dummy-job-jar > at > java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) > at > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) > at > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) > at > java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) > at > java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) > at > java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) > at java.base/java.nio.file.Files.readAttributes(Files.java:1763) > at > java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) > at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) > at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2716) > at java.base/java.nio.file.Files.walkFileTree(Files.java:2796) > at > org.apache.flink.client.cli.PythonProgramOptionsTest.testConfigurePythonExecution(PythonProgramOptionsTest.java:131) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runners.Suite.runChild(Suite.java:128) > at org.junit.runners.Suite.runChild(Suite.java:27) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) > at > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) > at >
[GitHub] [flink] shuiqiangchen opened a new pull request #13756: [FLINK-19770][python][test] Changed the PythonProgramOptionTest to be an ITCase.
shuiqiangchen opened a new pull request #13756: URL: https://github.com/apache/flink/pull/13756 ## What is the purpose of the change Fixed the error that PythonProgramOptionsTest requires package phase before execution. ## Brief change log Changed the PythonProgramOptionsTest to be an ITCase and the path of dummy job jar from a relative path to absolute path. ## Verifying this change This change has test case covered by PythonProgramOptionTest. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] danny0405 commented on pull request #12919: [FLINK-16048][avro] Support read/write confluent schema registry avro…
danny0405 commented on pull request #12919: URL: https://github.com/apache/flink/pull/12919#issuecomment-714889392 > @danny0405 @dawidwys > Any reasons all the fields read and written by this format has prefix 'record_' ? (I'm using flink sql for this client) > I found responsible code probably here but still have problem with this solution: > https://github.com/apache/flink/blob/de87a2debde8546e6741390a81f43c032521c3c0/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSchemaConverter.java#L365 It's because of the current strategy to infer the Avro schema is convert from the `CREATE TABLE` DDL, and there is no way to get the record name here. So we put a constant `record` as a prefix. The record write out all have explicit field name, but the type should be compatible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13756: [FLINK-19770][python][test] Changed the PythonProgramOptionTest to be an ITCase.
flinkbot commented on pull request #13756: URL: https://github.com/apache/flink/pull/13756#issuecomment-714890476 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d994782b0d8d72c5a59379643d6e588a02b333ad (Fri Oct 23 03:45:22 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-19770).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13676: [FLINK-19326][cep] Allow explicitly configuring time behaviour on CEP PatternStream
flinkbot edited a comment on pull request #13676: URL: https://github.com/apache/flink/pull/13676#issuecomment-711145212 ## CI report: * 80e2c4e0a1fff42e9c2d2798ed7884a148870ee5 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7940) * 7c8f33c27cb6cdfa1682585c9611251d8dcad5a7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19706) Add WARN logs when hive table partition has existed before commit in `MetastoreCommitPolicy`
[
https://issues.apache.org/jira/browse/FLINK-19706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee updated FLINK-19706:
-
Fix Version/s: 1.12.0
> Add WARN logs when hive table partition has existed before commit in
> `MetastoreCommitPolicy`
> ---
>
> Key: FLINK-19706
> URL: https://issues.apache.org/jira/browse/FLINK-19706
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, Connectors / Hive, Table SQL /
> Runtime
>Reporter: Lsw_aka_laplace
>Assignee: Lsw_aka_laplace
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Attachments: image-2020-10-19-16-47-39-354.png,
> image-2020-10-19-16-57-02-661.png, image-2020-10-19-17-00-27-255.png,
> image-2020-10-19-17-03-21-558.png, image-2020-10-19-18-16-35-083.png
>
>
> dfHi all,
> Recently we have been devoted to using Hive Streaming Writing to
> accelerate our data-sync of Data Warehouse based on Hive, and eventually we
> made it.
> For producing purpose, a lot of metrics/logs/measures were added in
> order to help us analyze running info or fix some unexpected problems. Among
> these mentioned above, we found that Checking Repeated Partition Commit is
> the most important one. So here, we are willing to make a contribution of
> introducing this backwards to Community.
> If this proposal is meaning, I am happy to introduce my design and
> implementation.
>
> Looking forward to ANY opinion~
>
>
> UPDATE
>
> Our user（using our own platform to build his own Flink job）raised some
> Requests. One of the requests is that once the parition is commited, the data
> in this partitio is regarded as frozen or completed. [Commiting partition]
> seem like a gurantee(but we all know it is hard to be a promise) in some way
> which tells us this partition is completed. Certainly, we make a lot of
> measures try to achieve that [partition-commit means completed]. So if a
> partition is committed twice or more times, for us, there must be sth wrong
> or our measures are insufficent. On the other hand, it also inform us to do
> sth to make up to avoid data-loss or data-incompletion.
>
> So first of all, it is important to let us or help us know that certain
> partition is committed repeatedly. So that we can do the following things ASAP
> 1. analyze the reason or the cause
> 2. do some trade-off operations
> 3. improve our code/measures
>
> — Design and Implementation---
> There are basically two ways, both of them have been used in prod-env
> Approach1
> Add measures in CommitPolicy and be called before partition commit
> !image-2020-10-19-16-47-39-354.png|width=576,height=235!
> //{color:#ffab00}Newly posted, see here{color}
> !image-2020-10-19-18-16-35-083.png|width=725,height=313!
> 1.1 As the pic shows, add `checkPartitionExists` and implement it in
> sub-class
> !image-2020-10-19-17-03-21-558.png|width=1203,height=88!
> 1.2 call checkPartitionExists before partition commit
> ---
> Approach2
> Build a bounded cache of committed partitions and check it everytime before
> partition commit
> (actually this cache supposed to be a operator state)
> !image-2020-10-19-16-57-02-661.png|width=1298,height=57!
> 2.1 build a cache
> !image-2020-10-19-17-00-27-255.png|width=1235,height=116!
> 2.2 check before commit
>
>
> — UPDATE —
> After discussed with [~lzljs3620320], `Repeated partition check` seems a
> little misleading in semantics, so only some WARN logs will be added in
> `MetastoreCommitPolicy` in aware of repeated commit
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #13716: [FLINK-19706][table-runtime] Add WARN logs when hive table partition …
JingsongLi commented on a change in pull request #13716:
URL: https://github.com/apache/flink/pull/13716#discussion_r510583777
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/MetastoreCommitPolicy.java
##
@@ -45,6 +45,9 @@ public void setMetastore(TableMetaStore metaStore) {
public void commit(Context context) throws Exception {
LinkedHashMap partitionSpec =
context.partitionSpec();
metaStore.createOrAlterPartition(partitionSpec,
context.partitionPath());
+ if (metaStore.getPartition(partitionSpec).isPresent()) {
Review comment:
NIT: print previous path too, you can do this:
```
metaStore.getPartition(partitionSpec).ifPresent(path ->
LOG.warn("The partition {} has existed before
current commit," +
" the path is {}, this
partition will be altered instead of being created",
partitionSpec, path));
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13676: [FLINK-19326][cep] Allow explicitly configuring time behaviour on CEP PatternStream
flinkbot edited a comment on pull request #13676: URL: https://github.com/apache/flink/pull/13676#issuecomment-711145212 ## CI report: * 80e2c4e0a1fff42e9c2d2798ed7884a148870ee5 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7940) * 7c8f33c27cb6cdfa1682585c9611251d8dcad5a7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8143) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13696: [FLINK-19726][table] Implement new providers for blink planner
flinkbot edited a comment on pull request #13696: URL: https://github.com/apache/flink/pull/13696#issuecomment-712617122 ## CI report: * f2e5f3867663c020ec292722b63d5788b43fd3b2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8096) * d1440ea836055d51bb808033dbc2e28c11a1599a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19765) flink SqlUseCatalog.getCatalogName is not unified with SqlCreateCatalog and SqlDropCatalog
[ https://issues.apache.org/jira/browse/FLINK-19765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219405#comment-17219405 ] jackylau commented on FLINK-19765: -- hi [~jark] yeap. thanks. could you review it when the ci passed. > flink SqlUseCatalog.getCatalogName is not unified with SqlCreateCatalog and > SqlDropCatalog > -- > > Key: FLINK-19765 > URL: https://issues.apache.org/jira/browse/FLINK-19765 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.11.0 >Reporter: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > when i develop flink ranger plugin at operation level, i find this method not > unified. > And SqlToOperationConverter.convert needs has the good order for user to find > code. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13753: [FLINK-19765][flink-table-planner] refactor SqlToOperationConverter a…
flinkbot commented on pull request #13753: URL: https://github.com/apache/flink/pull/13753#issuecomment-714857845 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b85f48049fd05f5c6da1cb91c6dfb4c6d201100c (Fri Oct 23 01:54:34 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-19765).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19201) PyFlink e2e tests is instable and failed with "Connection broken: OSError"
[
https://issues.apache.org/jira/browse/FLINK-19201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219406#comment-17219406
]
Dian Fu commented on FLINK-19201:
-
Another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8118=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3
> PyFlink e2e tests is instable and failed with "Connection broken: OSError"
> --
>
> Key: FLINK-19201
> URL: https://issues.apache.org/jira/browse/FLINK-19201
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6452=logs=ff2e2ea5-07e3-5521-7b04-a4fc3ad765e9=6945d9e3-ebef-5993-0c44-838d8ad079c0]
> {code}
> 2020-09-10T21:37:42.9988117Z install conda ... [SUCCESS]
> 2020-09-10T21:37:43.0018449Z install miniconda... [SUCCESS]
> 2020-09-10T21:37:43.0082244Z installing python environment...
> 2020-09-10T21:37:43.0100408Z installing python3.5...
> 2020-09-10T21:37:58.7214400Z install python3.5... [SUCCESS]
> 2020-09-10T21:37:58.7253792Z installing python3.6...
> 2020-09-10T21:38:06.5855143Z install python3.6... [SUCCESS]
> 2020-09-10T21:38:06.5903358Z installing python3.7...
> 2020-09-10T21:38:11.5444706Z
> 2020-09-10T21:38:11.5484852Z ('Connection broken: OSError("(104,
> \'ECONNRESET\')")', OSError("(104, 'ECONNRESET')"))
> 2020-09-10T21:38:11.5513130Z
> 2020-09-10T21:38:11.8044086Z conda install 3.7 failed.You can
> retry to exec the script.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19201) PyFlink e2e tests is instable and failed with "Connection broken: OSError"
[
https://issues.apache.org/jira/browse/FLINK-19201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219407#comment-17219407
]
Dian Fu commented on FLINK-19201:
-
cc [~hxbks2ks]
> PyFlink e2e tests is instable and failed with "Connection broken: OSError"
> --
>
> Key: FLINK-19201
> URL: https://issues.apache.org/jira/browse/FLINK-19201
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6452=logs=ff2e2ea5-07e3-5521-7b04-a4fc3ad765e9=6945d9e3-ebef-5993-0c44-838d8ad079c0]
> {code}
> 2020-09-10T21:37:42.9988117Z install conda ... [SUCCESS]
> 2020-09-10T21:37:43.0018449Z install miniconda... [SUCCESS]
> 2020-09-10T21:37:43.0082244Z installing python environment...
> 2020-09-10T21:37:43.0100408Z installing python3.5...
> 2020-09-10T21:37:58.7214400Z install python3.5... [SUCCESS]
> 2020-09-10T21:37:58.7253792Z installing python3.6...
> 2020-09-10T21:38:06.5855143Z install python3.6... [SUCCESS]
> 2020-09-10T21:38:06.5903358Z installing python3.7...
> 2020-09-10T21:38:11.5444706Z
> 2020-09-10T21:38:11.5484852Z ('Connection broken: OSError("(104,
> \'ECONNRESET\')")', OSError("(104, 'ECONNRESET')"))
> 2020-09-10T21:38:11.5513130Z
> 2020-09-10T21:38:11.8044086Z conda install 3.7 failed.You can
> retry to exec the script.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19215) "Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test" failed with "Dispatcher REST endpoint has not started within a timeout of 20 sec"
[
https://issues.apache.org/jira/browse/FLINK-19215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219409#comment-17219409
]
Dian Fu commented on FLINK-19215:
-
Instance on 1.11:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8131=logs=6caf31d6-847a-526e-9624-468e053467d6=7d4f7375-52df-5ce0-457f-b2ffbb2289a4
[~rmetzger] What about also backport this fix to 1.11?
> "Resuming Savepoint (rocks, scale down, rocks timers) end-to-end test" failed
> with "Dispatcher REST endpoint has not started within a timeout of 20 sec"
>
>
> Key: FLINK-19215
> URL: https://issues.apache.org/jira/browse/FLINK-19215
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.11.0
>Reporter: Dian Fu
>Assignee: Robert Metzger
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6476=logs=08866332-78f7-59e4-4f7e-49a56faa3179=3e8647c1-5a28-5917-dd93-bf78594ea994
> {code}
> 2020-09-13T21:26:23.3646770Z Running 'Resuming Savepoint (rocks, scale down,
> rocks timers) end-to-end test'
> 2020-09-13T21:26:23.3647852Z
> ==
> 2020-09-13T21:26:23.3689605Z TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-23367497881
> 2020-09-13T21:26:23.7122791Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
> 2020-09-13T21:26:23.9988115Z Starting cluster.
> 2020-09-13T21:26:27.3702750Z Starting standalonesession daemon on host
> fv-az655.
> 2020-09-13T21:26:35.1213853Z Starting taskexecutor daemon on host fv-az655.
> 2020-09-13T21:26:35.2756714Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:36.4111928Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:37.5358508Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:38.7156039Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:39.8602294Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:41.0399056Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:42.1680966Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:43.2520250Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:44.3833552Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:45.5204296Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:46.6730448Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:47.8274365Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:49.0147447Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:51.5463623Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:52.7366058Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:53.8867521Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:55.0469025Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:56.1901349Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:57.3124935Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:58.4596457Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-09-13T21:26:59.4828675Z Dispatcher REST endpoint has not started within
> a timeout of 20 sec
> 2020-09-13T21:26:59.4831446Z [FAIL] Test script contains errors.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19201) PyFlink e2e tests is instable and failed with "Connection broken: OSError"
[
https://issues.apache.org/jira/browse/FLINK-19201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219410#comment-17219410
]
Huang Xingbo commented on FLINK-19201:
--
[~dian.fu] Thanks a lot for reporting. I will increase the number of retries to
avoid network problems.
> PyFlink e2e tests is instable and failed with "Connection broken: OSError"
> --
>
> Key: FLINK-19201
> URL: https://issues.apache.org/jira/browse/FLINK-19201
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6452=logs=ff2e2ea5-07e3-5521-7b04-a4fc3ad765e9=6945d9e3-ebef-5993-0c44-838d8ad079c0]
> {code}
> 2020-09-10T21:37:42.9988117Z install conda ... [SUCCESS]
> 2020-09-10T21:37:43.0018449Z install miniconda... [SUCCESS]
> 2020-09-10T21:37:43.0082244Z installing python environment...
> 2020-09-10T21:37:43.0100408Z installing python3.5...
> 2020-09-10T21:37:58.7214400Z install python3.5... [SUCCESS]
> 2020-09-10T21:37:58.7253792Z installing python3.6...
> 2020-09-10T21:38:06.5855143Z install python3.6... [SUCCESS]
> 2020-09-10T21:38:06.5903358Z installing python3.7...
> 2020-09-10T21:38:11.5444706Z
> 2020-09-10T21:38:11.5484852Z ('Connection broken: OSError("(104,
> \'ECONNRESET\')")', OSError("(104, 'ECONNRESET')"))
> 2020-09-10T21:38:11.5513130Z
> 2020-09-10T21:38:11.8044086Z conda install 3.7 failed.You can
> retry to exec the script.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19775) SystemProcessingTimeServiceTest.testImmediateShutdown is instable
[
https://issues.apache.org/jira/browse/FLINK-19775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-19775:
Labels: test-stability (was: )
> SystemProcessingTimeServiceTest.testImmediateShutdown is instable
> -
>
> Key: FLINK-19775
> URL: https://issues.apache.org/jira/browse/FLINK-19775
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8131=logs=d89de3df-4600-5585-dadc-9bbc9a5e661c=66b5c59a-0094-561d-0e44-b149dfdd586d
> {code}
> 2020-10-22T21:12:54.9462382Z [ERROR]
> testImmediateShutdown(org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest)
> Time elapsed: 0.009 s <<< ERROR!
> 2020-10-22T21:12:54.9463024Z java.lang.InterruptedException
> 2020-10-22T21:12:54.9463331Z at java.lang.Object.wait(Native Method)
> 2020-10-22T21:12:54.9463766Z at java.lang.Object.wait(Object.java:502)
> 2020-10-22T21:12:54.9464140Z at
> org.apache.flink.core.testutils.OneShotLatch.await(OneShotLatch.java:63)
> 2020-10-22T21:12:54.9466014Z at
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest.testImmediateShutdown(SystemProcessingTimeServiceTest.java:154)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13753: [FLINK-19765][flink-table-planner] refactor SqlToOperationConverter a…
flinkbot commented on pull request #13753: URL: https://github.com/apache/flink/pull/13753#issuecomment-714862730 ## CI report: * b85f48049fd05f5c6da1cb91c6dfb4c6d201100c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13746: [FLINK-19675][python] Fix PythonCalcExpandProjectRule to handle cases when the calc node contains WHERE clause, composite fields acce
flinkbot edited a comment on pull request #13746: URL: https://github.com/apache/flink/pull/13746#issuecomment-714469108 ## CI report: * 9becc81983d757b76614749a96a14a94f14aa7d6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8136) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8105) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13736: [FLINK-19654][python][e2e] Reduce pyflink e2e test parallelism
flinkbot edited a comment on pull request #13736: URL: https://github.com/apache/flink/pull/13736#issuecomment-714183394 ## CI report: * 429e63595a174e68e44a339d2dbccca57f07d6f5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8123) * 1e2610d18f4402536abaa663581e178f78c3f743 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8135) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #13732: [FLINK-19275][connector-kafka] Support reading and writing Kafka meta… …data
wuchong commented on a change in pull request #13732:
URL: https://github.com/apache/flink/pull/13732#discussion_r510556244
##
File path: flink-core/src/main/java/org/apache/flink/types/Row.java
##
@@ -274,4 +278,119 @@ public static Row join(Row first, Row... remainings) {
return joinedRow;
}
+
+ /**
+* Compares two {@link Row}s for deep equality. This method supports
all conversion classes of the
+* table ecosystem.
+*
+* The current implementation of {@link Row#equals(Object)} is not
able to compare all deeply
+* nested row structures that might be created in the table ecosystem.
For example, it does not
+* support comparing arrays stored in the values of a map. We might
update the {@link #equals(Object)}
Review comment:
Got.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-19644) Support read specific partition of Hive table in temporal join
[ https://issues.apache.org/jira/browse/FLINK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219414#comment-17219414 ] Jingsong Lee commented on FLINK-19644: -- I second [~lirui], it is better to reuse old options. And I don't see the difference between lookup and streaming source when reading max_partition. > Support read specific partition of Hive table in temporal join > -- > > Key: FLINK-19644 > URL: https://issues.apache.org/jira/browse/FLINK-19644 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Table SQL / Ecosystem >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > It's a common case to use hive partitioned table as dimension table. > Currently Hive table only supports load all data, It will be helpful if we > can support read user specific partition in temporal table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-19769) Reuse StreamRecord in SourceOutputWithWatermarks#collect
[
https://issues.apache.org/jira/browse/FLINK-19769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee reassigned FLINK-19769:
Assignee: Caizhi Weng
> Reuse StreamRecord in SourceOutputWithWatermarks#collect
>
>
> Key: FLINK-19769
> URL: https://issues.apache.org/jira/browse/FLINK-19769
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Caizhi Weng
>Assignee: Caizhi Weng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> {{SourceOutputWithWatermarks#collect}} always creates a new {{StreamRecord}}
> object which can be reused quite easily. We should reuse the {{StreamRecord}}
> for optimization.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
