[GitHub] [flink] flinkbot edited a comment on pull request #13750: [FLINK-19394][docs-zh] Translate the 'Monitoring Checkpointing' page of 'Debugging & Monitoring' into Chinese

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13750:
URL: https://github.com/apache/flink/pull/13750#issuecomment-714567487


   
   ## CI report:
   
   * 1ec411ddcfd58a0ab5f22c7ebe5433439b217d27 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8119)
 
   * 275378338d8c0968a73489764de84a0cb2096e2b Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8502)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13741: [FLINK-19680][checkpointing] Announce timeoutable CheckpointBarriers

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13741:
URL: https://github.com/apache/flink/pull/13741#issuecomment-714295796


   
   ## CI report:
   
   * fd9a7059d4b4fa7c0760f2c82731436fe1d1312a Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8428)
 
   * ab71efa489d12df19be9eeb6cc814ee60206598c UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13644: [FLINK-19542][k8s]Implement LeaderElectionService and LeaderRetrievalService based on Kubernetes API

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13644:
URL: https://github.com/apache/flink/pull/13644#issuecomment-708933599


   
   ## CI report:
   
   * e60cdbfeb10a33d4df62453679cd309fc39ee198 UNKNOWN
   * 37efc6f14c55e2bd261b70924c9525c6539447f8 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8420)
 
   * 7f34f6f31596832223b1cf2010bed1c47f7502ea UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13605: [FLINK-19599][table] Introduce Filesystem format factories to integrate new FileSource to table

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13605:
URL: https://github.com/apache/flink/pull/13605#issuecomment-707521684


   
   ## CI report:
   
   * a9df8a1384eb6306656b4fd952edd4be5d7a857d UNKNOWN
   * fa0dc913f56aa2567c30073c044f688f9ed74fee Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8485)
 
   * 96468e31f7614ef314c655f97a63fcabe83505a8 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8499)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13458:
URL: https://github.com/apache/flink/pull/13458#issuecomment-697041219


   
   ## CI report:
   
   * 7311b0d12d19a645391ea0359a9aa6318806363b UNKNOWN
   * c416edd7f8696bcf2ed6b77d0238ae7312282a51 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8484)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] aljoscha commented on a change in pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


aljoscha commented on a change in pull request #13825:
URL: https://github.com/apache/flink/pull/13825#discussion_r513432535



##
File path: 
flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * A {@link TypeSerializer} that delegates to an underlying {@link 
SimpleVersionedSerializer}.
+ */
+@Internal
+public class SimpleVersionedSerializerTypeSerializerProxy extends 
TypeSerializer implements Serializable {
+
+   private final SerializableSupplier> 
serializerSupplier;
+   private transient SimpleVersionedSerializer cachedSerializer;
+
+   public SimpleVersionedSerializerTypeSerializerProxy(
+   SerializableSupplier> 
serializerSupplier) {
+   this.serializerSupplier = serializerSupplier;
+   }
+
+   @Override
+   public boolean isImmutableType() {
+   return false;
+   }
+
+   @Override
+   public TypeSerializer duplicate() {
+   final byte[] serializedSerializer;
+   try {
+   serializedSerializer = 
InstantiationUtil.serializeObject(serializerSupplier);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not serialize 
SimpleVersionedSerializer.", e);
+   }
+   try {
+   return new 
SimpleVersionedSerializerTypeSerializerProxy<>(
+   InstantiationUtil.deserializeObject(
+   serializedSerializer,
+   
serializerSupplier.getClass().getClassLoader()));
+   } catch (ClassNotFoundException | IOException e) {
+   throw new RuntimeException("Could not duplicate 
SimpleVersionedSerializer.", e);
+   }
+   }
+
+   @Override
+   public T createInstance() {
+   return null;
+   }
+
+   @Override
+   public T copy(T from) {
+   SimpleVersionedSerializer serializer = getSerializer();
+   try {
+   byte[] serializedFrom = serializer.serialize(from);
+   return serializer.deserialize(
+   serializer.getVersion(),
+   serializedFrom);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not copy element.", 
e);
+   }
+   }
+
+   @Override
+   public T copy(T from, T reuse) {
+   // the reuse is optional, we can just ignore it
+   return copy(from);
+   }
+
+   @Override
+   public int getLength() {
+   return -1;
+   }
+
+   @Override
+   public void serialize(T record, DataOutputView target) throws 
IOException {
+   SimpleVersionedSerializer serializer = getSerializer();
+   target.writeInt(serializer.getVersion());

Review comment:
   No, I don't want to do that right now because this is not meant as a 
general `TypeSerializer` that users should use for state. I'm expressly 
forbidding that by throwing an exception.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] aljoscha commented on pull request #13073: [FLINK-18820] SourceOperator should send MAX_WATERMARK to downstream operator when closed

2020-10-28 Thread GitBox


aljoscha commented on pull request #13073:
URL: https://github.com/apache/flink/pull/13073#issuecomment-717923105


   Yes, what @dawidwys sounds correct! Please go for it.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] SteNicholas commented on a change in pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


SteNicholas commented on a change in pull request #13825:
URL: https://github.com/apache/flink/pull/13825#discussion_r513426072



##
File path: 
flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * A {@link TypeSerializer} that delegates to an underlying {@link 
SimpleVersionedSerializer}.
+ */
+@Internal
+public class SimpleVersionedSerializerTypeSerializerProxy extends 
TypeSerializer implements Serializable {
+
+   private final SerializableSupplier> 
serializerSupplier;
+   private transient SimpleVersionedSerializer cachedSerializer;
+
+   public SimpleVersionedSerializerTypeSerializerProxy(
+   SerializableSupplier> 
serializerSupplier) {
+   this.serializerSupplier = serializerSupplier;
+   }
+
+   @Override
+   public boolean isImmutableType() {
+   return false;
+   }
+
+   @Override
+   public TypeSerializer duplicate() {
+   final byte[] serializedSerializer;
+   try {
+   serializedSerializer = 
InstantiationUtil.serializeObject(serializerSupplier);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not serialize 
SimpleVersionedSerializer.", e);
+   }
+   try {
+   return new 
SimpleVersionedSerializerTypeSerializerProxy<>(
+   InstantiationUtil.deserializeObject(
+   serializedSerializer,
+   
serializerSupplier.getClass().getClassLoader()));
+   } catch (ClassNotFoundException | IOException e) {
+   throw new RuntimeException("Could not duplicate 
SimpleVersionedSerializer.", e);
+   }
+   }
+
+   @Override
+   public T createInstance() {
+   return null;
+   }
+
+   @Override
+   public T copy(T from) {
+   SimpleVersionedSerializer serializer = getSerializer();
+   try {
+   byte[] serializedFrom = serializer.serialize(from);
+   return serializer.deserialize(
+   serializer.getVersion(),
+   serializedFrom);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not copy element.", 
e);
+   }
+   }
+
+   @Override
+   public T copy(T from, T reuse) {
+   // the reuse is optional, we can just ignore it
+   return copy(from);
+   }
+
+   @Override
+   public int getLength() {
+   return -1;
+   }
+
+   @Override
+   public void serialize(T record, DataOutputView target) throws 
IOException {
+   SimpleVersionedSerializer serializer = getSerializer();
+   target.writeInt(serializer.getVersion());

Review comment:
   > @SteNicholas What do you mean by that?
   
   I just want to ask whether `TypeSerializerSnapshot ` should be handled 
similarly with `TypeSerializer`.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] aljoscha commented on a change in pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


aljoscha commented on a change in pull request #13825:
URL: https://github.com/apache/flink/pull/13825#discussion_r513422586



##
File path: 
flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * A {@link TypeSerializer} that delegates to an underlying {@link 
SimpleVersionedSerializer}.
+ */
+@Internal
+public class SimpleVersionedSerializerTypeSerializerProxy extends 
TypeSerializer implements Serializable {
+
+   private final SerializableSupplier> 
serializerSupplier;
+   private transient SimpleVersionedSerializer cachedSerializer;
+
+   public SimpleVersionedSerializerTypeSerializerProxy(
+   SerializableSupplier> 
serializerSupplier) {
+   this.serializerSupplier = serializerSupplier;
+   }
+
+   @Override
+   public boolean isImmutableType() {
+   return false;
+   }
+
+   @Override
+   public TypeSerializer duplicate() {
+   final byte[] serializedSerializer;
+   try {
+   serializedSerializer = 
InstantiationUtil.serializeObject(serializerSupplier);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not serialize 
SimpleVersionedSerializer.", e);
+   }
+   try {
+   return new 
SimpleVersionedSerializerTypeSerializerProxy<>(
+   InstantiationUtil.deserializeObject(
+   serializedSerializer,
+   
serializerSupplier.getClass().getClassLoader()));
+   } catch (ClassNotFoundException | IOException e) {
+   throw new RuntimeException("Could not duplicate 
SimpleVersionedSerializer.", e);
+   }
+   }
+
+   @Override
+   public T createInstance() {
+   return null;
+   }
+
+   @Override
+   public T copy(T from) {
+   SimpleVersionedSerializer serializer = getSerializer();
+   try {
+   byte[] serializedFrom = serializer.serialize(from);
+   return serializer.deserialize(
+   serializer.getVersion(),
+   serializedFrom);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not copy element.", 
e);
+   }
+   }
+
+   @Override
+   public T copy(T from, T reuse) {
+   // the reuse is optional, we can just ignore it
+   return copy(from);
+   }
+
+   @Override
+   public int getLength() {
+   return -1;
+   }
+
+   @Override
+   public void serialize(T record, DataOutputView target) throws 
IOException {
+   SimpleVersionedSerializer serializer = getSerializer();
+   target.writeInt(serializer.getVersion());

Review comment:
   @SteNicholas What do you mean by that?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] dawidwys commented on a change in pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


dawidwys commented on a change in pull request #13825:
URL: https://github.com/apache/flink/pull/13825#discussion_r513419702



##
File path: 
flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * A {@link TypeSerializer} that delegates to an underlying {@link 
SimpleVersionedSerializer}.
+ */
+@Internal
+public class SimpleVersionedSerializerTypeSerializerProxy extends 
TypeSerializer implements Serializable {
+
+   private final SerializableSupplier> 
serializerSupplier;
+   private transient SimpleVersionedSerializer cachedSerializer;
+
+   public SimpleVersionedSerializerTypeSerializerProxy(
+   SerializableSupplier> 
serializerSupplier) {
+   this.serializerSupplier = serializerSupplier;
+   }
+
+   @Override
+   public boolean isImmutableType() {
+   return false;
+   }
+
+   @Override
+   public TypeSerializer duplicate() {
+   final byte[] serializedSerializer;
+   try {
+   serializedSerializer = 
InstantiationUtil.serializeObject(serializerSupplier);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not serialize 
SimpleVersionedSerializer.", e);
+   }
+   try {
+   return new 
SimpleVersionedSerializerTypeSerializerProxy<>(
+   InstantiationUtil.deserializeObject(
+   serializedSerializer,
+   
serializerSupplier.getClass().getClassLoader()));
+   } catch (ClassNotFoundException | IOException e) {
+   throw new RuntimeException("Could not duplicate 
SimpleVersionedSerializer.", e);
+   }
+   }
+
+   @Override
+   public T createInstance() {
+   return null;
+   }
+
+   @Override
+   public T copy(T from) {
+   SimpleVersionedSerializer serializer = getSerializer();
+   try {
+   byte[] serializedFrom = serializer.serialize(from);
+   return serializer.deserialize(
+   serializer.getVersion(),
+   serializedFrom);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not copy element.", 
e);
+   }
+   }
+
+   @Override
+   public T copy(T from, T reuse) {
+   // the reuse is optional, we can just ignore it
+   return copy(from);
+   }
+
+   @Override
+   public int getLength() {
+   return -1;
+   }
+
+   @Override
+   public void serialize(T record, DataOutputView target) throws 
IOException {
+   SimpleVersionedSerializer serializer = getSerializer();
+   target.writeInt(serializer.getVersion());

Review comment:
   Right now the serializer will be just for `Committables` in the Sink 
interface so having the version won't harm as there won't be much data volume. 
If it is used for regular additional version might be significant.
   
   Having said that I am fine with either, leaving or removing.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] SteNicholas commented on a change in pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


SteNicholas commented on a change in pull request #13825:
URL: https://github.com/apache/flink/pull/13825#discussion_r513419048



##
File path: 
flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * A {@link TypeSerializer} that delegates to an underlying {@link 
SimpleVersionedSerializer}.
+ */
+@Internal
+public class SimpleVersionedSerializerTypeSerializerProxy extends 
TypeSerializer implements Serializable {
+
+   private final SerializableSupplier> 
serializerSupplier;
+   private transient SimpleVersionedSerializer cachedSerializer;
+
+   public SimpleVersionedSerializerTypeSerializerProxy(
+   SerializableSupplier> 
serializerSupplier) {
+   this.serializerSupplier = serializerSupplier;
+   }
+
+   @Override
+   public boolean isImmutableType() {
+   return false;
+   }
+
+   @Override
+   public TypeSerializer duplicate() {
+   final byte[] serializedSerializer;
+   try {
+   serializedSerializer = 
InstantiationUtil.serializeObject(serializerSupplier);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not serialize 
SimpleVersionedSerializer.", e);
+   }
+   try {
+   return new 
SimpleVersionedSerializerTypeSerializerProxy<>(
+   InstantiationUtil.deserializeObject(
+   serializedSerializer,
+   
serializerSupplier.getClass().getClassLoader()));
+   } catch (ClassNotFoundException | IOException e) {
+   throw new RuntimeException("Could not duplicate 
SimpleVersionedSerializer.", e);
+   }
+   }
+
+   @Override
+   public T createInstance() {
+   return null;
+   }
+
+   @Override
+   public T copy(T from) {
+   SimpleVersionedSerializer serializer = getSerializer();
+   try {
+   byte[] serializedFrom = serializer.serialize(from);
+   return serializer.deserialize(
+   serializer.getVersion(),
+   serializedFrom);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not copy element.", 
e);
+   }
+   }
+
+   @Override
+   public T copy(T from, T reuse) {
+   // the reuse is optional, we can just ignore it
+   return copy(from);
+   }
+
+   @Override
+   public int getLength() {
+   return -1;
+   }
+
+   @Override
+   public void serialize(T record, DataOutputView target) throws 
IOException {
+   SimpleVersionedSerializer serializer = getSerializer();
+   target.writeInt(serializer.getVersion());

Review comment:
   @aljoscha I have concerned whether `TypeSerializerSnapshot` need 
`SimpleVersionedTypeSerializerSnapshotProxy` like this. What do you think about 
this?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13826: [BP-1.11][FLINK-19587][table-planner-blink] Fix error result when casting binary as varchar

2020-10-28 Thread GitBox


flinkbot commented on pull request #13826:
URL: https://github.com/apache/flink/pull/13826#issuecomment-717912982


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit a56eb8b921be50cddf4eaf489da3d24561d0e05d (Wed Oct 28 
12:54:01 UTC 2020)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] wangxlong opened a new pull request #13826: [BP-1.11][FLINK-19587][table-planner-blink] Fix error result when casting binary as varchar

2020-10-28 Thread GitBox


wangxlong opened a new pull request #13826:
URL: https://github.com/apache/flink/pull/13826


   
   ## What is the purpose of the change
   
   1. This is a backport of FLINK-19587, cherry picked from commit 
d9b0ac97ee4675aebdab1592af663b95fdc5051b.
   2. Cherry pick commit b2f88660b2b0897e57565e5b197ba83f20a03228 to support 
new type system for `ExpressionTestBase`. This  also allow to use new type 
system in `ExpressionTestBase` for 1.11 later.
   
   
   ## Brief change log
   
   cherry d9b0ac97ee4675aebdab1592af663b95fdc5051b
   cherry b2f88660b2b0897e57565e5b197ba83f20a03228



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] aljoscha commented on a change in pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


aljoscha commented on a change in pull request #13825:
URL: https://github.com/apache/flink/pull/13825#discussion_r513411521



##
File path: 
flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * A {@link TypeSerializer} that delegates to an underlying {@link 
SimpleVersionedSerializer}.
+ */
+@Internal
+public class SimpleVersionedSerializerTypeSerializerProxy extends 
TypeSerializer implements Serializable {
+
+   private final SerializableSupplier> 
serializerSupplier;
+   private transient SimpleVersionedSerializer cachedSerializer;
+
+   public SimpleVersionedSerializerTypeSerializerProxy(
+   SerializableSupplier> 
serializerSupplier) {
+   this.serializerSupplier = serializerSupplier;
+   }
+
+   @Override
+   public boolean isImmutableType() {
+   return false;
+   }
+
+   @Override
+   public TypeSerializer duplicate() {
+   final byte[] serializedSerializer;
+   try {
+   serializedSerializer = 
InstantiationUtil.serializeObject(serializerSupplier);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not serialize 
SimpleVersionedSerializer.", e);
+   }
+   try {
+   return new 
SimpleVersionedSerializerTypeSerializerProxy<>(
+   InstantiationUtil.deserializeObject(
+   serializedSerializer,
+   
serializerSupplier.getClass().getClassLoader()));
+   } catch (ClassNotFoundException | IOException e) {
+   throw new RuntimeException("Could not duplicate 
SimpleVersionedSerializer.", e);
+   }
+   }
+
+   @Override
+   public T createInstance() {
+   return null;
+   }
+
+   @Override
+   public T copy(T from) {
+   SimpleVersionedSerializer serializer = getSerializer();
+   try {
+   byte[] serializedFrom = serializer.serialize(from);
+   return serializer.deserialize(
+   serializer.getVersion(),
+   serializedFrom);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not copy element.", 
e);
+   }
+   }
+
+   @Override
+   public T copy(T from, T reuse) {
+   // the reuse is optional, we can just ignore it
+   return copy(from);
+   }
+
+   @Override
+   public int getLength() {
+   return -1;
+   }
+
+   @Override
+   public void serialize(T record, DataOutputView target) throws 
IOException {
+   SimpleVersionedSerializer serializer = getSerializer();
+   target.writeInt(serializer.getVersion());

Review comment:
   Yes, this whole things a pickle. I don't want to add a snapshot because 
that's complicated and would require me to serialize the supplier, which we 
shouldn't do.
   
   I'm fine with removing the version but I think leaving it in would make the 
code a tiny bit more future proof.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] zentol commented on pull request #13796: [FLINK-19810][CI] Automatically run a basic NOTICE file check on CI

2020-10-28 Thread GitBox


zentol commented on pull request #13796:
URL: https://github.com/apache/flink/pull/13796#issuecomment-717906443


   Having such a mapping was my idea as well; the explicit failure if a 
dependency was not found also serves as a reminder to check the license for 
compatibility.
   I don't think this would be tooo difficult tbh.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] zentol commented on pull request #13796: [FLINK-19810][CI] Automatically run a basic NOTICE file check on CI

2020-10-28 Thread GitBox


zentol commented on pull request #13796:
URL: https://github.com/apache/flink/pull/13796#issuecomment-717905033


   Let's add them all here.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] dawidwys commented on a change in pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


dawidwys commented on a change in pull request #13825:
URL: https://github.com/apache/flink/pull/13825#discussion_r513408053



##
File path: 
flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * A {@link TypeSerializer} that delegates to an underlying {@link 
SimpleVersionedSerializer}.
+ */
+@Internal
+public class SimpleVersionedSerializerTypeSerializerProxy extends 
TypeSerializer implements Serializable {
+
+   private final SerializableSupplier> 
serializerSupplier;
+   private transient SimpleVersionedSerializer cachedSerializer;
+
+   public SimpleVersionedSerializerTypeSerializerProxy(
+   SerializableSupplier> 
serializerSupplier) {
+   this.serializerSupplier = serializerSupplier;
+   }
+
+   @Override
+   public boolean isImmutableType() {
+   return false;
+   }
+
+   @Override
+   public TypeSerializer duplicate() {
+   final byte[] serializedSerializer;
+   try {
+   serializedSerializer = 
InstantiationUtil.serializeObject(serializerSupplier);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not serialize 
SimpleVersionedSerializer.", e);
+   }
+   try {
+   return new 
SimpleVersionedSerializerTypeSerializerProxy<>(
+   InstantiationUtil.deserializeObject(
+   serializedSerializer,
+   
serializerSupplier.getClass().getClassLoader()));
+   } catch (ClassNotFoundException | IOException e) {
+   throw new RuntimeException("Could not duplicate 
SimpleVersionedSerializer.", e);
+   }
+   }
+
+   @Override
+   public T createInstance() {
+   return null;
+   }
+
+   @Override
+   public T copy(T from) {
+   SimpleVersionedSerializer serializer = getSerializer();
+   try {
+   byte[] serializedFrom = serializer.serialize(from);
+   return serializer.deserialize(
+   serializer.getVersion(),
+   serializedFrom);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not copy element.", 
e);
+   }
+   }
+
+   @Override
+   public T copy(T from, T reuse) {
+   // the reuse is optional, we can just ignore it
+   return copy(from);
+   }
+
+   @Override
+   public int getLength() {
+   return -1;
+   }
+
+   @Override
+   public void serialize(T record, DataOutputView target) throws 
IOException {
+   SimpleVersionedSerializer serializer = getSerializer();
+   target.writeInt(serializer.getVersion());

Review comment:
   Hah, I actually removed a part about unaligned checkpoints from my 
initial comment ;) I dropped it when I actually realized that we don't support 
unaligned savepoints and thus version upgrades. (or am I wrong?)
   
   Shouldn't we then implement the `snapshotConfiguration`? This method is 
mandatory if we want to consider version upgrades, imo...





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this 

[GitHub] [flink] aljoscha commented on a change in pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


aljoscha commented on a change in pull request #13825:
URL: https://github.com/apache/flink/pull/13825#discussion_r513406057



##
File path: 
flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * A {@link TypeSerializer} that delegates to an underlying {@link 
SimpleVersionedSerializer}.
+ */
+@Internal
+public class SimpleVersionedSerializerTypeSerializerProxy extends 
TypeSerializer implements Serializable {
+
+   private final SerializableSupplier> 
serializerSupplier;
+   private transient SimpleVersionedSerializer cachedSerializer;
+
+   public SimpleVersionedSerializerTypeSerializerProxy(
+   SerializableSupplier> 
serializerSupplier) {
+   this.serializerSupplier = serializerSupplier;
+   }
+
+   @Override
+   public boolean isImmutableType() {
+   return false;
+   }
+
+   @Override
+   public TypeSerializer duplicate() {
+   final byte[] serializedSerializer;
+   try {
+   serializedSerializer = 
InstantiationUtil.serializeObject(serializerSupplier);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not serialize 
SimpleVersionedSerializer.", e);
+   }
+   try {
+   return new 
SimpleVersionedSerializerTypeSerializerProxy<>(
+   InstantiationUtil.deserializeObject(
+   serializedSerializer,
+   
serializerSupplier.getClass().getClassLoader()));
+   } catch (ClassNotFoundException | IOException e) {
+   throw new RuntimeException("Could not duplicate 
SimpleVersionedSerializer.", e);
+   }
+   }
+
+   @Override
+   public T createInstance() {
+   return null;
+   }
+
+   @Override
+   public T copy(T from) {
+   SimpleVersionedSerializer serializer = getSerializer();
+   try {
+   byte[] serializedFrom = serializer.serialize(from);
+   return serializer.deserialize(
+   serializer.getVersion(),
+   serializedFrom);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not copy element.", 
e);
+   }
+   }
+
+   @Override
+   public T copy(T from, T reuse) {
+   // the reuse is optional, we can just ignore it
+   return copy(from);
+   }
+
+   @Override
+   public int getLength() {
+   return -1;
+   }
+
+   @Override
+   public void serialize(T record, DataOutputView target) throws 
IOException {
+   SimpleVersionedSerializer serializer = getSerializer();
+   target.writeInt(serializer.getVersion());

Review comment:
   You are right. I tried to be as general as possible just in case someone 
came upon this code and also tried to use it. Also, with unaligned checkpoints 
in-flight records are checkpointed so it becomes relevant for that. Although we 
currently say that UC cannot be used for version upgrades (of both Flink or the 
user job).





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact 

[GitHub] [flink-statefun] tzulitai closed pull request #168: [FLINK-19692] Write all assigned key groups into the keyed raw stream with a Header.

2020-10-28 Thread GitBox


tzulitai closed pull request #168:
URL: https://github.com/apache/flink-statefun/pull/168


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] shuiqiangchen commented on pull request #13756: [FLINK-19770][python][test] Changed the PythonProgramOptionTest to be an ITCase.

2020-10-28 Thread GitBox


shuiqiangchen commented on pull request #13756:
URL: https://github.com/apache/flink/pull/13756#issuecomment-717898835


   @flinkbot run azure



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] dawidwys commented on a change in pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


dawidwys commented on a change in pull request #13825:
URL: https://github.com/apache/flink/pull/13825#discussion_r513399180



##
File path: 
flink-core/src/main/java/org/apache/flink/core/io/SimpleVersionedSerializerTypeSerializerProxy.java
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.io;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.InstantiationUtil;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.function.Supplier;
+
+/**
+ * A {@link TypeSerializer} that delegates to an underlying {@link 
SimpleVersionedSerializer}.
+ */
+@Internal
+public class SimpleVersionedSerializerTypeSerializerProxy extends 
TypeSerializer implements Serializable {
+
+   private final SerializableSupplier> 
serializerSupplier;
+   private transient SimpleVersionedSerializer cachedSerializer;
+
+   public SimpleVersionedSerializerTypeSerializerProxy(
+   SerializableSupplier> 
serializerSupplier) {
+   this.serializerSupplier = serializerSupplier;
+   }
+
+   @Override
+   public boolean isImmutableType() {
+   return false;
+   }
+
+   @Override
+   public TypeSerializer duplicate() {
+   final byte[] serializedSerializer;
+   try {
+   serializedSerializer = 
InstantiationUtil.serializeObject(serializerSupplier);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not serialize 
SimpleVersionedSerializer.", e);
+   }
+   try {
+   return new 
SimpleVersionedSerializerTypeSerializerProxy<>(
+   InstantiationUtil.deserializeObject(
+   serializedSerializer,
+   
serializerSupplier.getClass().getClassLoader()));
+   } catch (ClassNotFoundException | IOException e) {
+   throw new RuntimeException("Could not duplicate 
SimpleVersionedSerializer.", e);
+   }
+   }
+
+   @Override
+   public T createInstance() {
+   return null;
+   }
+
+   @Override
+   public T copy(T from) {
+   SimpleVersionedSerializer serializer = getSerializer();
+   try {
+   byte[] serializedFrom = serializer.serialize(from);
+   return serializer.deserialize(
+   serializer.getVersion(),
+   serializedFrom);
+   } catch (IOException e) {
+   throw new RuntimeException("Could not copy element.", 
e);
+   }
+   }
+
+   @Override
+   public T copy(T from, T reuse) {
+   // the reuse is optional, we can just ignore it
+   return copy(from);
+   }
+
+   @Override
+   public int getLength() {
+   return -1;
+   }
+
+   @Override
+   public void serialize(T record, DataOutputView target) throws 
IOException {
+   SimpleVersionedSerializer serializer = getSerializer();
+   target.writeInt(serializer.getVersion());

Review comment:
   Do we need to write the version of the serializer? If I am not mistaken 
it will be used only for the network transfer and thus always work with a 
single version of the serializer.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


flinkbot commented on pull request #13825:
URL: https://github.com/apache/flink/pull/13825#issuecomment-717897004


   
   ## CI report:
   
   * 267ea2ff011c4dc74f3209d574d66023835f501e UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13824: [FLINK-19736] Add the SinkTransformation

2020-10-28 Thread GitBox


flinkbot commented on pull request #13824:
URL: https://github.com/apache/flink/pull/13824#issuecomment-717895470


   
   ## CI report:
   
   * a10e3dcdaf7ff215ec455ce60ca7594651ad8d04 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13822: [FLINK-19856][network] Emit EndOfChannelRecoveryEvent

2020-10-28 Thread GitBox


flinkbot commented on pull request #13822:
URL: https://github.com/apache/flink/pull/13822#issuecomment-717894411


   
   ## CI report:
   
   * 612c241aa46314c2822a6f350faf3116c1b5cce0 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13821: [FLINK-19855][network] Specify channel AND gate in resumeConsumption()

2020-10-28 Thread GitBox


flinkbot commented on pull request #13821:
URL: https://github.com/apache/flink/pull/13821#issuecomment-717893540


   
   ## CI report:
   
   * b3158c4d03ddc32916d3d8e584709c50ae82715c UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13820: [FLINK-19671][codestyle] Revert .editorconfig change violating our coding style

2020-10-28 Thread GitBox


flinkbot commented on pull request #13820:
URL: https://github.com/apache/flink/pull/13820#issuecomment-717892896


   
   ## CI report:
   
   * c0e721f67b0b4b21b5080900954df10a05890d87 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] wangyang0918 commented on pull request #13644: [FLINK-19542][k8s]Implement LeaderElectionService and LeaderRetrievalService based on Kubernetes API

2020-10-28 Thread GitBox


wangyang0918 commented on pull request #13644:
URL: https://github.com/apache/flink/pull/13644#issuecomment-717892502


   @tillrohrmann I have addressed the comments you left and force pushed. They 
are all promising changes.
   
   You could continue the review :)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink-statefun] tzulitai commented on pull request #168: [FLINK-19692] Write all assigned key groups into the keyed raw stream with a Header.

2020-10-28 Thread GitBox


tzulitai commented on pull request #168:
URL: https://github.com/apache/flink-statefun/pull/168#issuecomment-717892546


   Thanks for addressing my comments @igalshilman!
   I'll proceed to merge this now to `master` and `release-2.2`.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13819: [hotfix][docs]fix some mistakes in StreamingFileSink

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13819:
URL: https://github.com/apache/flink/pull/13819#issuecomment-717834371


   
   ## CI report:
   
   * e4cf3eb0b429e4ffe12a5053e0d8110c2eb50693 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8493)
 
   * 1967ce552a0ee509cac0e0ef2a3aa29718dcd3ad UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13789: [FLINK-19727][table-runtime] Implement ParallelismProvider for sink i…

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13789:
URL: https://github.com/apache/flink/pull/13789#issuecomment-716299701


   
   ## CI report:
   
   * 934e4f0acf9044de5c6cb73ae25d094396bfc146 UNKNOWN
   * fd0aa012ed1a15803fe2ed75180d60e6c9fe8d6d Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8491)
 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8504)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13763: [FLINK-19779][avro] Remove the record_ field name prefix for Confluen…

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13763:
URL: https://github.com/apache/flink/pull/13763#issuecomment-715195599


   
   ## CI report:
   
   * f004220668e20dcd9860026b69566868d473db33 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8455)
 
   * 4c5a06b1ca2833fe7f63c25503660ed7acf9b77d UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink-statefun] tzulitai removed a comment on pull request #168: [FLINK-19692] Write all assigned key groups into the keyed raw stream with a Header.

2020-10-28 Thread GitBox


tzulitai removed a comment on pull request #168:
URL: https://github.com/apache/flink-statefun/pull/168#issuecomment-717889642


   Thanks for addressing my comments @igalshilman!
   I'll merge the PR now  



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13750: [FLINK-19394][docs-zh] Translate the 'Monitoring Checkpointing' page of 'Debugging & Monitoring' into Chinese

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13750:
URL: https://github.com/apache/flink/pull/13750#issuecomment-714567487


   
   ## CI report:
   
   * 1ec411ddcfd58a0ab5f22c7ebe5433439b217d27 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8119)
 
   * 275378338d8c0968a73489764de84a0cb2096e2b UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13749: [FLINK-19712][Coordination] Do not restart CREATED executions in RestartPipelinedRegionFailoverStrategy

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13749:
URL: https://github.com/apache/flink/pull/13749#issuecomment-714539634


   
   ## CI report:
   
   * d1a4068273d54499e90fb54f00044d4955dc5789 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8319)
 
   * a87afea927d59ffab8d757ae57929f50241141c9 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] AHeise commented on a change in pull request #13741: [FLINK-19680][checkpointing] Announce timeoutable CheckpointBarriers

2020-10-28 Thread GitBox


AHeise commented on a change in pull request #13741:
URL: https://github.com/apache/flink/pull/13741#discussion_r513391531



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/CheckpointBarrierAnnouncement.java
##
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.io.network.api;
+
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.runtime.event.RuntimeEvent;
+
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * {@link CheckpointBarrierAnnouncement} is announcing presence or receiving 
of a {@link CheckpointBarrier}.
+ * That {@link #announcedBarrier} is identified by it's sequence number.
+ */
+public class CheckpointBarrierAnnouncement extends RuntimeEvent {

Review comment:
   Okay let's go with specialized and generalize when a second use case 
pops up.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13605: [FLINK-19599][table] Introduce Filesystem format factories to integrate new FileSource to table

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13605:
URL: https://github.com/apache/flink/pull/13605#issuecomment-707521684


   
   ## CI report:
   
   * a9df8a1384eb6306656b4fd952edd4be5d7a857d UNKNOWN
   * fa0dc913f56aa2567c30073c044f688f9ed74fee Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8485)
 
   * 96468e31f7614ef314c655f97a63fcabe83505a8 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] shouweikun commented on pull request #13789: [FLINK-19727][table-runtime] Implement ParallelismProvider for sink i…

2020-10-28 Thread GitBox


shouweikun commented on pull request #13789:
URL: https://github.com/apache/flink/pull/13789#issuecomment-717890188


   @flinkbot run azure



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink-statefun] tzulitai commented on pull request #168: [FLINK-19692] Write all assigned key groups into the keyed raw stream with a Header.

2020-10-28 Thread GitBox


tzulitai commented on pull request #168:
URL: https://github.com/apache/flink-statefun/pull/168#issuecomment-717889642


   Thanks for addressing my comments @igalshilman!
   I'll merge the PR now  



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


flinkbot commented on pull request #13825:
URL: https://github.com/apache/flink/pull/13825#issuecomment-717888135


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 267ea2ff011c4dc74f3209d574d66023835f501e (Wed Oct 28 
12:03:21 UTC 2020)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] aljoscha opened a new pull request #13825: [FLINK-19836] Add SimpleVersionedSerializerTypeSerializerProxy

2020-10-28 Thread GitBox


aljoscha opened a new pull request #13825:
URL: https://github.com/apache/flink/pull/13825


   This allows using a SimpleVersionedSerializer, which the Source and Sink
   API provide, in places where we need a TypeSerializer, such as when
   setting the serializer that is used for records in a DataStream/that are
   send between operators.
   
   ## Brief change log
   
   * add the class along with tests
   
   ## Verifying this change
   
   * newly added tests
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: yes
 - The runtime per-record code paths (performance sensitive): yes
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? no
 - If yes, how is the feature documented? not applicable
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13824: [FLINK-19736] Add the SinkTransformation

2020-10-28 Thread GitBox


flinkbot commented on pull request #13824:
URL: https://github.com/apache/flink/pull/13824#issuecomment-717883808


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit a10e3dcdaf7ff215ec455ce60ca7594651ad8d04 (Wed Oct 28 
11:54:03 UTC 2020)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
* **This pull request references an unassigned [Jira 
ticket](https://issues.apache.org/jira/browse/FLINK-19736).** According to the 
[code contribution 
guide](https://flink.apache.org/contributing/contribute-code.html), tickets 
need to be assigned before starting with the implementation work.
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] guoweiM opened a new pull request #13824: [FLINK-19736] Add the SinkTransformation

2020-10-28 Thread GitBox


guoweiM opened a new pull request #13824:
URL: https://github.com/apache/flink/pull/13824


   
   
   ## What is the purpose of the change
   
   Translate the new sink to the physical operator topology.
   
   
   
   
   ## Brief change log
   
   1. Introduce 'SinkTransformation' to represent the new sink api.
   2. Introduce 'SinkTransformationTranslator' that translates the 
'SinkTransformation' to the corresponding runtime operators.
   3. Make DataStream Sdk support new sink api.
   
   ## Verifying this change
   
   
   
   This change added tests and can be verified as follows:
   
 - *Introduce `StreamSinkITCase` to test end to end result*
 - *Introduce `SinkTransformationTranslatorTest` to test build sink 
topology*
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): (no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes)
 - If yes, how is the feature documented? (JavaDocs )
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13803: [FLINK-19821][python] Add ProcessFunction and timer access for Python DataStream API.

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13803:
URL: https://github.com/apache/flink/pull/13803#issuecomment-717145782


   
   ## CI report:
   
   * c88ea42fe4ce5260b492d592ae8f5d79f54ca829 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8378)
 
   * 2ab68a5e26b158ab43b128e0d0dbbc62cb8dea17 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8498)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13819: [hotfix][docs]fix some mistakes in StreamingFileSink

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13819:
URL: https://github.com/apache/flink/pull/13819#issuecomment-717834371


   
   ## CI report:
   
   * e4cf3eb0b429e4ffe12a5053e0d8110c2eb50693 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8493)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13791: [FLINK-19749][docs] Improve documentation in 'Table API' page

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13791:
URL: https://github.com/apache/flink/pull/13791#issuecomment-716405856


   
   ## CI report:
   
   * bb4a09489cc7a59c5b860ade7780670b7d2bcca1 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8421)
 
   * 054cf380741ac078ac258663fa5f118fa413dbc2 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8497)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13756: [FLINK-19770][python][test] Changed the PythonProgramOptionTest to be an ITCase.

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13756:
URL: https://github.com/apache/flink/pull/13756#issuecomment-714895505


   
   ## CI report:
   
   * 37447c7adbd68701dbce2f515ff69af119be5a5b Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8477)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13784: [FLINK-19698][connectors/common] API improvements to the Sources.

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13784:
URL: https://github.com/apache/flink/pull/13784#issuecomment-716153889


   
   ## CI report:
   
   * ca695f256a825fdebadf79f576b0875d4faea7cd Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8450)
 
   * f0dec4fa5d8e69761377a45df57106a4fbfe8152 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8496)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13789: [FLINK-19727][table-runtime] Implement ParallelismProvider for sink i…

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13789:
URL: https://github.com/apache/flink/pull/13789#issuecomment-716299701


   
   ## CI report:
   
   * 934e4f0acf9044de5c6cb73ae25d094396bfc146 UNKNOWN
   * fd0aa012ed1a15803fe2ed75180d60e6c9fe8d6d Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8491)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13744: [FLINK-19766][table-runtime] Introduce File streaming compaction operators

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13744:
URL: https://github.com/apache/flink/pull/13744#issuecomment-714369975


   
   ## CI report:
   
   * 322bc2c3c11a0f5735db4a475864f894f38866da Azure: 
[CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8466)
 
   * a01ecf0bade9f8e4a56052fb5b5d25c5034fe511 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8486)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink-statefun] tzulitai commented on a change in pull request #168: [FLINK-19692] Write all assigned key groups into the keyed raw stream with a Header.

2020-10-28 Thread GitBox


tzulitai commented on a change in pull request #168:
URL: https://github.com/apache/flink-statefun/pull/168#discussion_r513379873



##
File path: 
statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/logger/UnboundedFeedbackLogger.java
##
@@ -99,11 +98,19 @@ private void flushToKeyedStateOutputStream() throws 
IOException {
 checkState(keyedStateOutputStream != null, "Trying to flush envelopes not 
in a logging state");
 
 final DataOutputView target = new 
DataOutputViewStreamWrapper(keyedStateOutputStream);
-for (Entry> entry : keyGroupStreams.entrySet()) 
{
-  checkpointedStreamOperations.startNewKeyGroup(keyedStateOutputStream, 
entry.getKey());
+final Iterable assignedKeyGroupIds =
+checkpointedStreamOperations.keyGroupList(keyedStateOutputStream);
+// the underlying checkpointed raw stream, requires that all key groups 
assigned
+// to this operator must be written to the underlying stream.

Review comment:
   I don’t have a strong opinion on whether or not the empty key groups 
should stay there in the long term, so fine by me to keep this as is without 
the TODO comment to revisit  
   





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] wuchong commented on pull request #13823: [hot-fix][table-planner-blink] Fix TableScanTest which used temporal …

2020-10-28 Thread GitBox


wuchong commented on pull request #13823:
URL: https://github.com/apache/flink/pull/13823#issuecomment-717879422


   Verified in the local machine. Merge it as it blocks builds.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13823: [hot-fix][table-planner-blink] Fix TableScanTest which used temporal …

2020-10-28 Thread GitBox


flinkbot commented on pull request #13823:
URL: https://github.com/apache/flink/pull/13823#issuecomment-717879581


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 521fc89541a33024db75670cfc93051857bba357 (Wed Oct 28 
11:44:41 UTC 2020)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
* **Invalid pull request title: No valid Jira ID provided**
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] wuchong merged pull request #13823: [hot-fix][table-planner-blink] Fix TableScanTest which used temporal …

2020-10-28 Thread GitBox


wuchong merged pull request #13823:
URL: https://github.com/apache/flink/pull/13823


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] leonardBang commented on pull request #13823: [hot-fix][table-planner-blink] Fix TableScanTest which used temporal …

2020-10-28 Thread GitBox


leonardBang commented on pull request #13823:
URL: https://github.com/apache/flink/pull/13823#issuecomment-717878191


   cc @wuchong Could you have a quick look ? thanks 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] leonardBang opened a new pull request #13823: [hot-fix][table-planner-blink] Fix TableScanTest which used temporal …

2020-10-28 Thread GitBox


leonardBang opened a new pull request #13823:
URL: https://github.com/apache/flink/pull/13823


   ## What is the purpose of the change
   
   * This pull hot-fix table plan test 
TableScanTest.testTemporalJoinOnUpsertSource() .
   
   
   ## Brief change log
   
 - Update TableScanTest.testTemporalJoinOnUpsertSource() using right 
temporal join order
   
   
   ## Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (no)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
 - The serializers: (no)
 - The runtime per-record code paths (performance sensitive): ( no)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
 - The S3 file system connector: (no)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (no)
 - If yes, how is the feature documented? (not applicable / docs / JavaDocs 
/ not documented)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] vthinkxie commented on a change in pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI

2020-10-28 Thread GitBox


vthinkxie commented on a change in pull request #13458:
URL: https://github.com/apache/flink/pull/13458#discussion_r513374506



##
File path: 
flink-runtime-web/web-dashboard/src/app/pages/job/checkpoints/detail/job-checkpoints-detail.component.ts
##
@@ -55,21 +59,38 @@ export class JobCheckpointsDetailComponent implements 
OnInit {
   }
 
   refresh() {
-this.isLoading = true;
-if (this.jobDetail && this.jobDetail.jid) {
-  this.jobService.loadCheckpointDetails(this.jobDetail.jid, 
this.checkPoint.id).subscribe(
-data => {
-  this.checkPointDetail = data;
-  this.isLoading = false;
-  this.cdr.markForCheck();
-},
-() => {
-  this.isLoading = false;
-  this.cdr.markForCheck();
-}
-  );
+  this.isLoading = true;
+  if (this.jobDetail && this.jobDetail.jid) {
+forkJoin([
+  this.jobService.loadCheckpointConfig(this.jobDetail.jid),
+  this.jobService.loadCheckpointDetails(this.jobDetail.jid, 
this.checkPoint.id)
+]).subscribe(
+  ([config, detail]) => {
+this.checkPointConfig = config;
+this.checkPointDetail = detail;
+if (this.checkPointDetail.checkpoint_type === 'CHECKPOINT') {
+  if (this.checkPointConfig.unaligned_checkpoints) {
+this.checkPointType = 'unaligned checkpoint';
+  } else {
+this.checkPointType = 'aligned checkpoint';
+  }
+} else if (this.checkPointDetail.checkpoint_type === 
'SYNC_SAVEPOINT') {
+  this.checkPointType = 'savepoint on cancel';
+} else if (this.checkPointDetail.checkpoint_type === 'SAVEPOINT') {
+  this.checkPointType = 'savepoint';
+} else {
+  this.checkPointType = '-';
+}
+this.isLoading = false;
+this.cdr.markForCheck();
+  },
+  () => {
+this.isLoading = false;
+this.cdr.markForCheck();
+  }
+);
+  }
 }

Review comment:
   There are two extra spaces in each line, the others look good to me





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13822: [FLINK-19856][network] Emit EndOfChannelRecoveryEvent

2020-10-28 Thread GitBox


flinkbot commented on pull request #13822:
URL: https://github.com/apache/flink/pull/13822#issuecomment-717875371


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 1e8d476b327183f27d5d05ef17f31144bdaed00c (Wed Oct 28 
11:35:12 UTC 2020)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] rkhachatryan opened a new pull request #13822: [FLINK-19856][network] Emit EndOfChannelRecoveryEvent

2020-10-28 Thread GitBox


rkhachatryan opened a new pull request #13822:
URL: https://github.com/apache/flink/pull/13822


   ## What is the purpose of the change
   
   Depends on #13821.
   
   The emitted event would allow to tear down "virtual channels" used to read 
channel state on recovery with unaligned checkpoints and rescaling.
   
   Upstream subpartitions are blocked upon sending this event and unblocked by 
the downstream when all channels receive it.
   
   ## Verifying this change
- Added `ChannelPersistenceITCase.testUpstreamBlocksAfterRecoveringState` 
to test emission and blocking on the upstream side
- Added `CheckpointedInputGateTest` to test alignment and unblocking on the 
downstream side
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? no
 - If yes, how is the feature documented? no
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] danny0405 commented on a change in pull request #13763: [FLINK-19779][avro] Remove the record_ field name prefix for Confluen…

2020-10-28 Thread GitBox


danny0405 commented on a change in pull request #13763:
URL: https://github.com/apache/flink/pull/13763#discussion_r513370899



##
File path: 
flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/typeutils/AvroSchemaConverter.java
##
@@ -297,17 +299,22 @@ private static DataType convertToDataType(Schema schema) {
 * @return Avro's {@link Schema} matching this logical type.
 */
public static Schema convertToSchema(LogicalType logicalType) {
-   return convertToSchema(logicalType, "record");
+   return convertToSchema(logicalType, "record", true);
}
 
/**
 * Converts Flink SQL {@link LogicalType} (can be nested) into an Avro 
schema.
 *
 * @param logicalType logical type
 * @param rowName the record name
+* @param top whether it is parsing the root record,
+*if it is, the logical type nullability would be 
ignored
 * @return Avro's {@link Schema} matching this logical type.
 */
-   public static Schema convertToSchema(LogicalType logicalType, String 
rowName) {
+   public static Schema convertToSchema(
+   LogicalType logicalType,
+   String rowName,
+   boolean top) {

Review comment:
   I had offline discussion with @wuchong and @dawidwys , and after some 
research, we found that a non-nullable row type is more reasonable.
   
   But because the change is huge(many codes that convert a type info to data 
type assumes nullable true before), me and @wuchong decide to change the method 
signature to `convertToSchema(RowType schema)` and add a notion to the method 
doc that the passed in `schema` must be the top level record type.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13821: [FLINK-19855][network] Specify channel AND gate in resumeConsumption()

2020-10-28 Thread GitBox


flinkbot commented on pull request #13821:
URL: https://github.com/apache/flink/pull/13821#issuecomment-717872665


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit ac72a653a769aee44f6e5ad6da9109ed0bfd5881 (Wed Oct 28 
11:29:27 UTC 2020)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink-statefun] igalshilman commented on a change in pull request #168: [FLINK-19692] Write all assigned key groups into the keyed raw stream with a Header.

2020-10-28 Thread GitBox


igalshilman commented on a change in pull request #168:
URL: https://github.com/apache/flink-statefun/pull/168#discussion_r513369798



##
File path: 
statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/logger/UnboundedFeedbackLogger.java
##
@@ -99,11 +98,19 @@ private void flushToKeyedStateOutputStream() throws 
IOException {
 checkState(keyedStateOutputStream != null, "Trying to flush envelopes not 
in a logging state");
 
 final DataOutputView target = new 
DataOutputViewStreamWrapper(keyedStateOutputStream);
-for (Entry> entry : keyGroupStreams.entrySet()) 
{
-  checkpointedStreamOperations.startNewKeyGroup(keyedStateOutputStream, 
entry.getKey());
+final Iterable assignedKeyGroupIds =
+checkpointedStreamOperations.keyGroupList(keyedStateOutputStream);
+// the underlying checkpointed raw stream, requires that all key groups 
assigned
+// to this operator must be written to the underlying stream.

Review comment:
   I think that it would be better to write down the empty groups anyways, 
because they are presented to us on restore.
   And having a header there, would help us to differentiate between different 
versions.
   But I can definitely add a TODO: revist this after 19748 is merged. 
   would that work for you?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] rkhachatryan opened a new pull request #13821: [FLINK-19855][network] Specify channel AND gate in resumeConsumption()

2020-10-28 Thread GitBox


rkhachatryan opened a new pull request #13821:
URL: https://github.com/apache/flink/pull/13821


   ## What is the purpose of the change
   
   Replace `channelIndex` with `ChannelInfo` in resumeConsumption methods of 
`InputGate` and `CheckpointableInput`.
   
   Given `channelIndex` only, `UnionInputGate` has to guess which wrapped 
`inputGate` this channel belongs to.
   For that, `UnionInputGate` expects `channelIndex` with an `inputGate` offset.
   However, some clients (e.g. `AlignedController`) pass channel index without 
offset.
   
   ## Verifying this change
   
   TBD
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): no
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: no
 - The serializers: no
 - The runtime per-record code paths (performance sensitive): no
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
 - The S3 file system connector: no
   
   ## Documentation
   
 - Does this pull request introduce a new feature? no
 - If yes, how is the feature documented? no
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13820: [FLINK-19671][codestyle] Revert .editorconfig change violating our coding style

2020-10-28 Thread GitBox


flinkbot commented on pull request #13820:
URL: https://github.com/apache/flink/pull/13820#issuecomment-717870794


   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit c0e721f67b0b4b21b5080900954df10a05890d87 (Wed Oct 28 
11:25:36 UTC 2020)
   
   **Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up 
to date!
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13073: [FLINK-18820] SourceOperator should send MAX_WATERMARK to downstream operator when closed

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13073:
URL: https://github.com/apache/flink/pull/13073#issuecomment-669650373


   
   ## CI report:
   
   * 145c86b135ee64462c449dacc81643ee8908e68b Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5219)
 
   * 406f3764484bb5b004988002f620bd20c0c34fd6 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8494)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] pnowojski opened a new pull request #13820: [FLINK-19671][codestyle] Revert .editorconfig change violating our coding style

2020-10-28 Thread GitBox


pnowojski opened a new pull request #13820:
URL: https://github.com/apache/flink/pull/13820


   
https://flink.apache.org/contributing/code-style-and-quality-formatting.html#breaking-the-lines-of-too-long-statements
   
   > Rules about breaking the long lines:
   >
   > Break the argument list or chain of calls if the line exceeds limit or 
earlier if you believe that the breaking would improve the code readability
   > If you break the line then each argument/call should have a separate line, 
including the first one
   > Each new line should have one extra indentation (or two for a function 
declaration) relative to the line of the parent function name or the called 
entity
   
   ## Does this pull request potentially affect one of the following parts:
   
 - Dependencies (does it add or upgrade a dependency): (yes / **no**)
 - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (yes / **no**)
 - The serializers: (yes / **no** / don't know)
 - The runtime per-record code paths (performance sensitive): (yes / **no** 
/ don't know)
 - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / 
don't know)
 - The S3 file system connector: (yes / **no** / don't know)
   
   ## Documentation
   
 - Does this pull request introduce a new feature? (yes / **no**)
 - If yes, how is the feature documented? (**not applicable** / docs / 
JavaDocs / not documented)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] StephanEwen commented on pull request #13595: [FLINK-19582][network] Introduce sort-merge based blocking shuffle to Flink

2020-10-28 Thread GitBox


StephanEwen commented on pull request #13595:
URL: https://github.com/apache/flink/pull/13595#issuecomment-717865867


   @wsry Your suggestion sounds good to me!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] rmetzger commented on pull request #13796: [FLINK-19810][CI] Automatically run a basic NOTICE file check on CI

2020-10-28 Thread GitBox


rmetzger commented on pull request #13796:
URL: https://github.com/apache/flink/pull/13796#issuecomment-717859165


   Once I'm done with the license check for this release, I can make an 
assessment of how well the tool works.
   If I'm confident that it can also handle edge-cases, I could look into 
automating it. What we need is basically a mapping from dependency --> license. 
We could make the build fail if this mapping doesn't exist for a dependency.
   
   I will proceed with addressing the licensing issues I've found so far. I was 
planning to open separate PRs for different areas of Flink (python, table, 
connectors), or shall I add them all here to this PR? (this would be easier for 
me, but only works if you are willing to review them all at once)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] tillrohrmann commented on a change in pull request #13644: [FLINK-19542][k8s]Implement LeaderElectionService and LeaderRetrievalService based on Kubernetes API

2020-10-28 Thread GitBox


tillrohrmann commented on a change in pull request #13644:
URL: https://github.com/apache/flink/pull/13644#discussion_r513355235



##
File path: 
flink-runtime/src/test/java/org/apache/flink/runtime/leaderelection/TestingLeaderBase.java
##
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.leaderelection;
+
+import java.util.concurrent.TimeoutException;
+import java.util.function.Supplier;
+
+/**
+ * Base class which provides some convenience functions for testing purposes 
of {@link LeaderContender} and
+ * {@link LeaderElectionEventHandler}.
+ */
+public class TestingLeaderBase {
+
+   protected boolean leader = false;
+   protected Throwable error = null;
+
+   protected final Object lock = new Object();
+   private final Object errorLock = new Object();
+
+   /**
+* Waits until the contender becomes the leader or until the timeout 
has been exceeded.
+*
+* @param timeout
+* @throws TimeoutException
+*/
+   public void waitForLeader(long timeout) throws TimeoutException {
+   waitFor(this::isLeader, timeout, "Contender was not elected as 
the leader within " + timeout + "ms");
+   }
+
+   /**
+* Waits until the contender revokes the leader or until the timeout 
has been exceeded.
+*
+* @param timeout
+* @throws TimeoutException
+*/
+   public void waitForRevokeLeader(long timeout) throws TimeoutException {
+   waitFor(() -> !isLeader(), timeout, "Contender was not revoked 
within " + timeout + "ms");
+   }
+
+   protected void waitFor(Supplier supplier, long timeout, String 
msg) throws TimeoutException {
+   long start = System.currentTimeMillis();
+   long curTimeout;
+
+   while (!supplier.get() && (curTimeout = timeout - 
System.currentTimeMillis() + start) > 0) {
+   synchronized (lock) {
+   try {
+   lock.wait(curTimeout);
+   } catch (InterruptedException e) {
+   // we got interrupted so check again 
for the condition
+   }
+   }
+   }
+
+   if (!supplier.get()) {
+   throw new TimeoutException(msg);
+   }

Review comment:
   This is not strictly needed right now. We could tackle this as part of a 
follow up issue.

##
File path: 
flink-runtime/src/test/java/org/apache/flink/runtime/leaderelection/TestingRetrievalBase.java
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.leaderelection;
+
+import org.apache.flink.runtime.leaderretrieval.LeaderRetrievalListener;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.UUID;
+import java.util.concurrent.TimeoutException;
+
+/**
+ * Base class which provides some convenience functions for testing purposes 
of {@link LeaderRetrievalListener} and
+ * {@link 
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalEventHandler}.
+ */
+public class TestingRetrievalBase {
+
+   protected final Logger logger = LoggerFactory.getLogger(getClass());
+   private final Object lock = new Object();
+
+   private String address;
+  

[GitHub] [flink] tillrohrmann commented on a change in pull request #13644: [FLINK-19542][k8s]Implement LeaderElectionService and LeaderRetrievalService based on Kubernetes API

2020-10-28 Thread GitBox


tillrohrmann commented on a change in pull request #13644:
URL: https://github.com/apache/flink/pull/13644#discussion_r513354997



##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/leaderretrieval/ZooKeeperLeaderRetrievalDriver.java
##
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.leaderretrieval;
+
+import org.apache.flink.runtime.leaderelection.LeaderInformation;
+import org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionDriver;
+import org.apache.flink.util.ExceptionUtils;
+import org.apache.flink.util.FlinkException;
+
+import 
org.apache.flink.shaded.curator4.org.apache.curator.framework.CuratorFramework;
+import 
org.apache.flink.shaded.curator4.org.apache.curator.framework.api.UnhandledErrorListener;
+import 
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.ChildData;
+import 
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.NodeCache;
+import 
org.apache.flink.shaded.curator4.org.apache.curator.framework.recipes.cache.NodeCacheListener;
+import 
org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionState;
+import 
org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateListener;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.util.UUID;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * The counterpart to the {@link ZooKeeperLeaderElectionDriver}.
+ * {@link LeaderRetrievalService} implementation for Zookeeper. It retrieves 
the current leader which has
+ * been elected by the {@link ZooKeeperLeaderElectionDriver}.
+ * The leader address as well as the current leader session ID is retrieved 
from ZooKeeper.
+ */
+public class ZooKeeperLeaderRetrievalDriver implements LeaderRetrievalDriver, 
NodeCacheListener, UnhandledErrorListener {
+   private static final Logger LOG = 
LoggerFactory.getLogger(ZooKeeperLeaderRetrievalDriver.class);
+
+   /** Connection to the used ZooKeeper quorum. */
+   private final CuratorFramework client;
+
+   /** Curator recipe to watch changes of a specific ZooKeeper node. */
+   private final NodeCache cache;
+
+   private final String retrievalPath;
+
+   private final ConnectionStateListener connectionStateListener = 
(client, newState) -> handleStateChange(newState);
+
+   private final LeaderRetrievalEventHandler leaderRetrievalEventHandler;
+
+   private volatile boolean running;
+
+   /**
+* Creates a leader retrieval service which uses ZooKeeper to retrieve 
the leader information.
+*
+* @param client Client which constitutes the connection to the 
ZooKeeper quorum
+* @param retrievalPath Path of the ZooKeeper node which contains the 
leader information
+* @param leaderRetrievalEventHandler handler to notify the leader 
changes.
+*/
+   public ZooKeeperLeaderRetrievalDriver(
+   CuratorFramework client,
+   String retrievalPath,
+   LeaderRetrievalEventHandler 
leaderRetrievalEventHandler) throws Exception {
+   this.client = checkNotNull(client, "CuratorFramework client");
+   this.cache = new NodeCache(client, retrievalPath);
+   this.retrievalPath = checkNotNull(retrievalPath);
+
+   this.leaderRetrievalEventHandler = 
checkNotNull(leaderRetrievalEventHandler);
+
+   client.getUnhandledErrorListenable().addListener(this);
+   cache.getListenable().addListener(this);
+   cache.start();
+
+   
client.getConnectionStateListenable().addListener(connectionStateListener);
+
+   running = true;
+   }
+
+   @Override
+   public void close() throws Exception {
+   if (!running) {
+   return;
+   }
+
+   running = false;

Review comment:
   I think we just need to make sure that we don't introduce a deadlock by 
requesting 

[GitHub] [flink] flinkbot edited a comment on pull request #13819: [hotfix][docs]fix some mistakes in StreamingFileSink

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13819:
URL: https://github.com/apache/flink/pull/13819#issuecomment-717834371


   
   ## CI report:
   
   * e4cf3eb0b429e4ffe12a5053e0d8110c2eb50693 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8493)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13817: [HotFix][docs] Fix broken link of docker.md

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13817:
URL: https://github.com/apache/flink/pull/13817#issuecomment-717771629


   
   ## CI report:
   
   * f7fcad2291f072819e6576b291327abd16947445 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8481)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13803: [FLINK-19821][python] Add ProcessFunction and timer access for Python DataStream API.

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13803:
URL: https://github.com/apache/flink/pull/13803#issuecomment-717145782


   
   ## CI report:
   
   * c88ea42fe4ce5260b492d592ae8f5d79f54ca829 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8378)
 
   * 2ab68a5e26b158ab43b128e0d0dbbc62cb8dea17 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13791: [FLINK-19749][docs] Improve documentation in 'Table API' page

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13791:
URL: https://github.com/apache/flink/pull/13791#issuecomment-716405856


   
   ## CI report:
   
   * bb4a09489cc7a59c5b860ade7780670b7d2bcca1 Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8421)
 
   * 054cf380741ac078ac258663fa5f118fa413dbc2 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13789: [FLINK-19727][table-runtime] Implement ParallelismProvider for sink i…

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13789:
URL: https://github.com/apache/flink/pull/13789#issuecomment-716299701


   
   ## CI report:
   
   * 6329930d7acc5d2d4d84e777561e79bd25ac9367 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8265)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8487)
 
   * 934e4f0acf9044de5c6cb73ae25d094396bfc146 UNKNOWN
   * fd0aa012ed1a15803fe2ed75180d60e6c9fe8d6d UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13784: [FLINK-19698][connectors/common] API improvements to the Sources.

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13784:
URL: https://github.com/apache/flink/pull/13784#issuecomment-716153889


   
   ## CI report:
   
   * ca695f256a825fdebadf79f576b0875d4faea7cd Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8450)
 
   * f0dec4fa5d8e69761377a45df57106a4fbfe8152 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13697: [FLINK-19357][FLINK-19357][fs-connector] Introduce createBucketWriter to BucketsBuilder & Introduce FileLifeCycleListener to Buckets

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13697:
URL: https://github.com/apache/flink/pull/13697#issuecomment-712672697


   
   ## CI report:
   
   * b45885955bfdee4d4288f9273171e1d52379773f Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7914)
 
   * bd25177243a782fad2e7f9d9ff508e6ba3758303 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8490)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13620: [hotfix][doc] the first FSDataOutputStream should be FSDataInputStream in FileSystem#L178

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13620:
URL: https://github.com/apache/flink/pull/13620#issuecomment-707904494


   
   ## CI report:
   
   * 16f95599c033e72ce486b06dd3d527abd656b1a9 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7546)
 
   * 4290129d21b0600e5d17b1c9db54c9416a861b2c Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8489)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] StephanEwen commented on a change in pull request #13784: [FLINK-19698][connectors/common] API improvements to the Sources.

2020-10-28 Thread GitBox


StephanEwen commented on a change in pull request #13784:
URL: https://github.com/apache/flink/pull/13784#discussion_r513286480



##
File path: flink-core/src/main/java/org/apache/flink/util/ExceptionUtils.java
##
@@ -565,6 +565,20 @@ else if (t instanceof Error) {
return Optional.empty();
}
 
+   /**
+* Find the root cause of the given throwable chain.
+*
+* @param throwable the throwable chain to check.
+* @return the root cause of the throwable chain.
+*/
+   public static Throwable findRootCause(Throwable throwable) {

Review comment:
   I think this needs an extra check against cyclic cause chains, which are 
possible and nasty.
   
   We previously used the Apache Commons Lang for that: 
https://github.com/apache/flink/blob/master/flink-runtime/src/test/java/org/apache/flink/runtime/blob/BlobCacheGetTest.java#L597
   
   I think at the very least, we need a set to check duplicates and stop 
traversing once we see an exception we saw before.

##
File path: 
flink-core/src/main/java/org/apache/flink/api/common/state/CheckpointListener.java
##
@@ -0,0 +1,51 @@
+/*
+ Licensed to the Apache Software Foundation (ASF) under one

Review comment:
   Nit: The header here uses a different style than the rest of the code 
(other files have * at every line start).

##
File path: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointListener.java
##
@@ -1,19 +1,19 @@
 /*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ Licensed to the Apache Software Foundation (ASF) under one

Review comment:
   Nit: The header here uses a different style than the rest of the code 
(other files have * at every line start).





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13073: [FLINK-18820] SourceOperator should send MAX_WATERMARK to downstream operator when closed

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13073:
URL: https://github.com/apache/flink/pull/13073#issuecomment-669650373


   
   ## CI report:
   
   * 145c86b135ee64462c449dacc81643ee8908e68b Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5219)
 
   * 406f3764484bb5b004988002f620bd20c0c34fd6 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] Sxnan commented on a change in pull request #13073: [FLINK-18820] SourceOperator should send MAX_WATERMARK to downstream operator when closed

2020-10-28 Thread GitBox


Sxnan commented on a change in pull request #13073:
URL: https://github.com/apache/flink/pull/13073#discussion_r513334483



##
File path: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java
##
@@ -163,6 +164,7 @@ public void sendSourceEventToCoordinator(SourceEvent event) 
{
@Override
public void close() throws Exception {
eventTimeLogic.stopPeriodicWatermarkEmits();
+   currentMainOutput.emitWatermark(Watermark.MAX_WATERMARK);

Review comment:
   Thanks for pointing out the problem and the detailed explaination. I 
have made the change at 406f376, please take a look. 
   I also think the the SourceoperatorStreamTask should override the 
`advanceToEndOfEventTime`. If @aljoscha can confirm that, I'd love to made that 
change in this pr as well.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] kl0u commented on a change in pull request #13819: [hotfix][docs]fix some mistakes in StreamingFileSink

2020-10-28 Thread GitBox


kl0u commented on a change in pull request #13819:
URL: https://github.com/apache/flink/pull/13819#discussion_r513331116



##
File path: 
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java
##
@@ -137,12 +137,12 @@ protected StreamingFileSink(
}
 
/**
-* Creates the builder for a {@link StreamingFileSink} with 
row-encoding format.
+* Creates the builder for a {@link StreamingFileSink} with 
bulk-factory format.

Review comment:
   "bulk-factory" -> "bulk-encoding"





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] becketqin commented on pull request #13574: [FLINK-18323] Add a Kafka source implementation based on FLIP-27.

2020-10-28 Thread GitBox


becketqin commented on pull request #13574:
URL: https://github.com/apache/flink/pull/13574#issuecomment-717837286


   @StephanEwen I am trying to address the blocking close() method issue. 
However, it is a little more complicated than I thought. I want to align some 
design principle with you and see what might be the solution there. 
   
   From what I understand the current design principle for the 
`SplitEnumerator` is that all the exception handling are synchronous, i.e. the 
implementation should throw exception when a method is invoked. If no exception 
was thrown, the method invocation is considered successful.
   
   In this case, if we want to allow the asynchronous handling and exception 
propagation in `SplitEnumerator`, such as asynchronous close(), we will need to 
have a `failJob()` method in the `SplitEnumeratorContext`. So users can call 
`failJob()` asynchronously instead of throw exceptions when closing in a 
non-blocking manner. The only caveat to this solution is that now users can 
either throw exception or call `failJob()` when the methods are invoked and 
people may wonder what is the difference.
   
   To avoid the above caveat, I was thinking that we can add the async close to 
the `SourceCoordinator` so we don't have to add `failJob()` method to the 
`SplitEnumerator`. However, in practice, sometimes the previous instance of 
`SplitEnumerator` must be successfully closed before the next instance can be 
created. Otherwise there might be conflicts. Therefore, naively having 
non-blocking closing in the `SourceCoordinator` won't work in all cases.
   
   Given all the above thoughts, I am falling back to the solution of adding a 
`failJob()` method to the `SplitEnumeratorContext` so the `SplitEnumerator` 
implementations can decide by themselves what to do in each method. And any 
exception thrown from the method invocation will just result in the job failure.
   
   Do you have any suggestion?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13819: [hotfix][docs]fix some mistakes in StreamingFileSink

2020-10-28 Thread GitBox


flinkbot commented on pull request #13819:
URL: https://github.com/apache/flink/pull/13819#issuecomment-717834371


   
   ## CI report:
   
   * e4cf3eb0b429e4ffe12a5053e0d8110c2eb50693 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] liuyongvs closed pull request #13818: [FLINK-19848][docs] fix docs bug of build flink.

2020-10-28 Thread GitBox


liuyongvs closed pull request #13818:
URL: https://github.com/apache/flink/pull/13818


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot commented on pull request #13818: [FLINK-19848][docs] fix docs bug of build flink.

2020-10-28 Thread GitBox


flinkbot commented on pull request #13818:
URL: https://github.com/apache/flink/pull/13818#issuecomment-717833523


   
   ## CI report:
   
   * 80c826228cfa96218b23bd087f4047673e91f734 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13812: [FLINK-19674][docs-zh] Translate "Docker" of "Clusters & Depolyment" page into Chinese

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13812:
URL: https://github.com/apache/flink/pull/13812#issuecomment-717687160


   
   ## CI report:
   
   * 74a934ea4dd99e6bc44ac84a9f314468faf8376b Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8470)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13789: [FLINK-19727][table-runtime] Implement ParallelismProvider for sink i…

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13789:
URL: https://github.com/apache/flink/pull/13789#issuecomment-716299701


   
   ## CI report:
   
   * 6329930d7acc5d2d4d84e777561e79bd25ac9367 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8265)
 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8487)
 
   * 934e4f0acf9044de5c6cb73ae25d094396bfc146 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13697: [FLINK-19357][FLINK-19357][fs-connector] Introduce createBucketWriter to BucketsBuilder & Introduce FileLifeCycleListener to Buckets

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13697:
URL: https://github.com/apache/flink/pull/13697#issuecomment-712672697


   
   ## CI report:
   
   * b45885955bfdee4d4288f9273171e1d52379773f Azure: 
[SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7914)
 
   * bd25177243a782fad2e7f9d9ff508e6ba3758303 UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13744: [FLINK-19766][table-runtime] Introduce File streaming compaction operators

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13744:
URL: https://github.com/apache/flink/pull/13744#issuecomment-714369975


   
   ## CI report:
   
   * 164c8b5b7750ed567813d70f160ef9f064d0a3b0 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8458)
 
   * 322bc2c3c11a0f5735db4a475864f894f38866da Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8466)
 
   * a01ecf0bade9f8e4a56052fb5b5d25c5034fe511 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8486)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13620: [hotfix][doc] the first FSDataOutputStream should be FSDataInputStream in FileSystem#L178

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13620:
URL: https://github.com/apache/flink/pull/13620#issuecomment-707904494


   
   ## CI report:
   
   * 16f95599c033e72ce486b06dd3d527abd656b1a9 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=7546)
 
   * 4290129d21b0600e5d17b1c9db54c9416a861b2c UNKNOWN
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13605: [FLINK-19599][table] Introduce Filesystem format factories to integrate new FileSource to table

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13605:
URL: https://github.com/apache/flink/pull/13605#issuecomment-707521684


   
   ## CI report:
   
   * a9df8a1384eb6306656b4fd952edd4be5d7a857d UNKNOWN
   * fa0dc913f56aa2567c30073c044f688f9ed74fee Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8485)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] RocMarshal commented on a change in pull request #13791: [FLINK-19749][docs] Improve documentation in 'Table API' page

2020-10-28 Thread GitBox


RocMarshal commented on a change in pull request #13791:
URL: https://github.com/apache/flink/pull/13791#discussion_r513201162



##
File path: docs/dev/table/tableApi.zh.md
##
@@ -3172,7 +3176,7 @@ Table table = input
   
   
 Similar to a GroupBy Aggregation. Groups the rows on the 
grouping keys with the following running table aggregation operator to 
aggregate rows group-wise. The difference from an AggregateFunction is that 
TableAggregateFunction may return 0 or more records for a group. You have to 
close the "flatAggregate" with a select statement. And the select statement 
does not support aggregate functions.
-Instead of using emitValue to output results, you can 
also use the emitUpdateWithRetract method. Different from 
emitValue, emitUpdateWithRetract is used to emit 
values that have been updated. This method outputs data incrementally in 
retract mode, i.e., once there is an update, we have to retract old records 
before sending new updated ones. The emitUpdateWithRetract method 
will be used in preference to the emitValue method if both methods 
are defined in the table aggregate function, because the method is treated to 
be more efficient than emitValue as it can output values 
incrementally. See Table Aggregation 
Functions for details.
+Instead of using emitValue to output results, you can 
also use the emitUpdateWithRetract method. Different from 
emitValue, emitUpdateWithRetract is used to emit 
values that have been updated. This method outputs data incrementally in 
retract mode, i.e., once there is an update, we have to retract old records 
before sending new updated ones. The emitUpdateWithRetract method 
will be used in preference to the emitValue method if both methods 
are defined in the table aggregate function, because the method is treated to 
be more efficient than emitValue as it can output values 
incrementally. See Table Aggregation Functions for details.

Review comment:
   ```suggestion
   Instead of using emitValue to output results, you 
can also use the emitUpdateWithRetract method. Different from 
emitValue, emitUpdateWithRetract is used to emit 
values that have been updated. This method outputs data incrementally in 
retract mode, i.e., once there is an update, we have to retract old records 
before sending new updated ones. The emitUpdateWithRetract method 
will be used in preference to the emitValue method if both methods 
are defined in the table aggregate function, because the method is treated to 
be more efficient than emitValue as it can output values 
incrementally. See Table Aggregation Functions for details.
   ```

##
File path: docs/dev/table/tableApi.zh.md
##
@@ -3571,9 +3575,9 @@ Please see the dedicated page about [data 
types](types.html).
 
 Generic types and (nested) composite types (e.g., POJOs, tuples, rows, Scala 
case classes) can be fields of a row as well.
 
-Fields of composite types with arbitrary nesting can be accessed with [value 
access functions]({{ site.baseurl 
}}/dev/table/functions/systemFunctions.html#value-access-functions).
+Fields of composite types with arbitrary nesting can be accessed with [value 
access functions]({% link dev/table/functions/systemFunctions.md 
%}#value-access-functions).
 
-Generic types are treated as a black box and can be passed on or processed by 
[user-defined functions]({{ site.baseurl }}/dev/table/functions/udfs.html).
+Generic types are treated as a black box and can be passed on or processed by 
[user-defined functions]({% link dev/table/functions/udfs.md %}).

Review comment:
   ```suggestion
   Generic types are treated as a black box and can be passed on or processed 
by [user-defined functions]({% link dev/table/functions/udfs.zh.md %}).
   ```

##
File path: docs/dev/table/tableApi.zh.md
##
@@ -3571,9 +3575,9 @@ Please see the dedicated page about [data 
types](types.html).
 
 Generic types and (nested) composite types (e.g., POJOs, tuples, rows, Scala 
case classes) can be fields of a row as well.
 
-Fields of composite types with arbitrary nesting can be accessed with [value 
access functions]({{ site.baseurl 
}}/dev/table/functions/systemFunctions.html#value-access-functions).
+Fields of composite types with arbitrary nesting can be accessed with [value 
access functions]({% link dev/table/functions/systemFunctions.md 
%}#value-access-functions).

Review comment:
   ```suggestion
   Fields of composite types with arbitrary nesting can be accessed with [value 
access functions]({% link dev/table/functions/systemFunctions.zh.md 
%}#value-access-functions).
   ```

##
File path: docs/dev/table/tableApi.zh.md
##
@@ -3398,7 +3402,7 @@ val table = input
   
   
 Similar to a GroupBy Aggregation. Groups the rows on the 
grouping keys with the following running table aggregation operator to 
aggregate rows group-wise. The difference from an AggregateFunction is 

[GitHub] [flink] flinkbot edited a comment on pull request #13458: FLINK-18851 [runtime] Add checkpoint type to checkpoint history entries in Web UI

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13458:
URL: https://github.com/apache/flink/pull/13458#issuecomment-697041219


   
   ## CI report:
   
   * 7311b0d12d19a645391ea0359a9aa6318806363b UNKNOWN
   *  Unknown: [CANCELED](TBD) 
   * c416edd7f8696bcf2ed6b77d0238ae7312282a51 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8484)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13307: [FLINK-19078][table-runtime] Import rowtime join temporal operator

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13307:
URL: https://github.com/apache/flink/pull/13307#issuecomment-685763264


   
   ## CI report:
   
   * a558cb0fd984dc5f256d7710f67239d0851204b3 Azure: 
[FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8483)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] zentol commented on pull request #13796: [FLINK-19810][CI] Automatically run a basic NOTICE file check on CI

2020-10-28 Thread GitBox


zentol commented on pull request #13796:
URL: https://github.com/apache/flink/pull/13796#issuecomment-717821843


   I'm still not a fan of automating the verification; people _will_ see the 
tool as the source of truth (which it isn't, as you pointed out), and any 
attempt at automating the generation can now be shut down with "well we have a 
tool to check the correctness, so let's not spend time on it".
   
   That said, it is obvious that it does find issues; particularly subtle ones 
that can _easily_ be missed during a manual review, `com` vs `org` is subtle AF 
and not really the focus of such a review (as it is more concerned with 
anything being listed at all and the version being correct).
   
   So I am, begrudgingly, fine with it.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] guoweiM commented on pull request #13808: [FLINK-19834] Make the TestSink reusable in all the sink related tests.

2020-10-28 Thread GitBox


guoweiM commented on pull request #13808:
URL: https://github.com/apache/flink/pull/13808#issuecomment-717821255


   > +1 for merging from me as soon as AZP given green, but I think @aljoscha 
's comments are not in the branch.
   
   
   
   > I merged this, but I renamed all the builder methods to `setFoo()` instead 
of `addFoo()` because they are setting a thing and not adding to a list of 
things.
   
   Sorry I miss this stuff 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] wuchong commented on a change in pull request #13307: [FLINK-19078][table-runtime] Import rowtime join temporal operator

2020-10-28 Thread GitBox


wuchong commented on a change in pull request #13307:
URL: https://github.com/apache/flink/pull/13307#discussion_r513259441



##
File path: 
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/TemporalJoinITCase.scala
##
@@ -260,6 +390,152 @@ class TemporalJoinITCase(state: StateBackendMode)
 tEnv.executeSql(sql).await()
   }
 
+  @Test
+  def testEventTimeTemporalJoin(): Unit = {
+val sql = "INSERT INTO rowtime_default_sink " +
+  " SELECT o.order_id, o.currency, o.amount, o.order_time, r.rate, 
r.currency_time " +
+  " FROM orders_rowtime AS o JOIN versioned_currency_with_single_key " +
+  " FOR SYSTEM_TIME AS OF o.order_time as r " +
+  " ON o.currency = r.currency"
+
+tEnv.executeSql(sql).await()
+val rawResult = getRawResults("rowtime_default_sink")
+val expected = List(
+  "+I(1,Euro,12,2020-08-15T00:01,114,2020-08-15T00:00)",
+  "+I(2,US Dollar,1,2020-08-15T00:02,102,2020-08-15T00:00)",
+  "+I(3,RMB,40,2020-08-15T00:03,702,2020-08-15T00:00)",
+  "+I(4,Euro,14,2020-08-16T00:02,118,2020-08-16T00:01)",
+  "+U(5,US Dollar,18,2020-08-16T00:03,106,2020-08-16T00:02)")
+assertEquals(expected.sorted, rawResult.sorted)
+  }
+
+  @Test
+  def testEventTimeTemporalJoinWithFilter(): Unit = {
+tEnv.executeSql("CREATE VIEW v1 AS" +
+  " SELECT * FROM versioned_currency_with_single_key WHERE rate > 114")

Review comment:
   I think this is not a good case, because there is no updates in the 
source that changing from matching condition to not matching. Maybe `rate < 
115` is better. 

##
File path: 
flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/TemporalJoinITCase.scala
##
@@ -59,10 +64,68 @@ class TemporalJoinITCase(state: StateBackendMode)
 changelogRow("+I","Euro", "no1", toJLong(118)),
 changelogRow("+I","US Dollar", "no2", toJLong(106)))
 
+  // test data for Event-Time temporal table join
+  val rowTimeOrderData = List(
+changelogRow("+I", toJLong(1), "Euro", "no1", toJLong(12),
+  toDateTime("2020-08-15T00:01:00")),
+changelogRow("+I", toJLong(2), "US Dollar", "no1", toJLong(1),
+  toDateTime("2020-08-15T00:02:00")),
+changelogRow("+I", toJLong(3), "RMB", "no1", toJLong(40),
+  toDateTime("2020-08-15T00:03:00")),
+changelogRow("+I", toJLong(4), "Euro", "no1", toJLong(14),
+  toDateTime("2020-08-16T00:02:00")),
+// test left stream could be changelog,
+// -U or -D message may retract fail in collection connector sink 
implementation

Review comment:
   I would suggest to add full events, it's weird to have only +U but no -U 
in a non-upsert source. 
   
   For event-time temporal join, I think it's safe to retract rows in sink. 

##
File path: 
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/join/temporal/TemporalRowTimeJoinOperator.java
##
@@ -236,21 +253,30 @@ private long emitResultAndCleanUpState(long 
timerTimestamp) throws Exception {
}
}
 
-   cleanupState(timerTimestamp, rightRowsSorted);
+   cleanupExpiredVersionInState(currentWatermark, rightRowsSorted);
return lastUnprocessedTime;
}
 
+   private void collectJoinedRow(RowData leftSideRow, RowData rightRow) {
+   outRow.setRowKind(leftSideRow.getRowKind());
+   outRow.replace(leftSideRow, rightRow);
+   collector.collect(outRow);
+   }
+
/**
-* Removes all right entries older then the watermark, except the 
latest one. For example with:
-* rightState = [1, 5, 9]
-* and
-* watermark = 6
-* we can not remove "5" from rightState, because left elements with 
rowtime of 7 or 8 could
-* be joined with it later
+* Removes all expired version in the versioned table's state according 
to current watermark.
+* For example with: rightState = [1(+I), 4(-U), 4(+U), 7(-U), 7(+U), 
9(-D), 12(I)],
+*
+* If watermark = 6 we can not remove "4(+U)" from rightState 
because accumulate message means
+* the start of version, the left elements with row time of 5 or 6 
could be joined with (+U,4) later.
+*
+* If watermark = 10 we can remove "9(-D)" from rightState because 
retract message means the
+* end of version, a trick is we do not remove it but do not correlate 
it if it is a retract message.

Review comment:
   I don't think this is a trick...

##
File path: 
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/join/temporal/TemporalRowTimeJoinOperator.java
##
@@ -294,6 +320,13 @@ private int indexOfFirstElementNewerThanTimer(long 
timerTimestamp, List
/**
 * Binary search {@code rightRowsSorted} to find the latest right 

[GitHub] [flink] aljoscha closed pull request #13808: [FLINK-19834] Make the TestSink reusable in all the sink related tests.

2020-10-28 Thread GitBox


aljoscha closed pull request #13808:
URL: https://github.com/apache/flink/pull/13808


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] aljoscha commented on pull request #13808: [FLINK-19834] Make the TestSink reusable in all the sink related tests.

2020-10-28 Thread GitBox


aljoscha commented on pull request #13808:
URL: https://github.com/apache/flink/pull/13808#issuecomment-717817391


   I merged this, but I renamed all the builder methods to `setFoo()` instead 
of `addFoo()` because they are setting a thing and not adding to a list of 
things.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




[GitHub] [flink] flinkbot edited a comment on pull request #13817: [HotFix][docs] Fix broken link of docker.md

2020-10-28 Thread GitBox


flinkbot edited a comment on pull request #13817:
URL: https://github.com/apache/flink/pull/13817#issuecomment-717771629


   
   ## CI report:
   
   * f7fcad2291f072819e6576b291327abd16947445 Azure: 
[PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8481)
 
   
   
   Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org




  1   2   3   4   5   6   7   8   9   10   >