[jira] [Updated] (FLINK-13865) Support custom config in Flink docker image

2019-08-26 Thread Dagang Wei (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dagang Wei updated FLINK-13865:
---
Component/s: (was: Deployment / Kubernetes)
 Deployment / Docker
Environment: I noticed FLINK-13380 which is helpful for Kubernetes, but it 
is still useful to support it at Docker image level.

> Support custom config in Flink docker image
> ---
>
> Key: FLINK-13865
> URL: https://issues.apache.org/jira/browse/FLINK-13865
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Docker
>Affects Versions: 1.6.3, 1.6.4, 1.7.2, 1.8.0, 1.8.1, 1.9.0
> Environment: I noticed FLINK-13380 which is helpful for Kubernetes, 
> but it is still useful to support it at Docker image level.
>Reporter: Dagang Wei
>Priority: Minor
>
> [Flink docker images](https://github.com/docker-flink/docker-flink) do not 
> support custom config, in order to do so, the user has to build their own 
> image based on the official image and modify config/flink-conf.yaml. It would 
> be much easier if the image accepts custom config through args and in 
> [docker-entrypoint.sh](https://github.com/docker-flink/docker-flink/blob/master/docker-entrypoint.sh)
>  it automatically merge the custom config into config/flink-conf.yaml.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-10995) Copy intermediate serialization results only once for broadcast mode

2019-08-26 Thread zhijiang (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916401#comment-16916401
 ] 

zhijiang commented on FLINK-10995:
--

I think it was not planned in release-1.10 before. We could discuss this design 
and plan for release-1.11 at some time.

> Copy intermediate serialization results only once for broadcast mode
> 
>
> Key: FLINK-10995
> URL: https://issues.apache.org/jira/browse/FLINK-10995
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Network
>Affects Versions: 1.8.0
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The emitted records from operator would be firstly serialized into 
> intermediate bytes array in {{RecordSerializer}}, then copy the intermediate 
> results into target buffers for different sub partitions.  For broadcast 
> mode, the same intermediate results would be copied as many times as the 
> number of sub partitions, and this would affect the performance seriously in 
> large scale jobs.
> We can copy to only one target buffer which would be shared by all the sub 
> partitions to reduce the overheads. For emitting latency marker in broadcast 
> mode, we should flush the previous shared target buffers first, and then 
> request a new buffer for the target sub partition to send latency marker.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13865) Support custom config in Flink docker image

2019-08-26 Thread Dagang Wei (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916402#comment-16916402
 ] 

Dagang Wei commented on FLINK-13865:


I noticed FLINK-13380 which is helpful for Kubernetes, but it is still useful 
to support it at Docker image level.

> Support custom config in Flink docker image
> ---
>
> Key: FLINK-13865
> URL: https://issues.apache.org/jira/browse/FLINK-13865
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Docker
>Affects Versions: 1.6.3, 1.6.4, 1.7.2, 1.8.0, 1.8.1, 1.9.0
>Reporter: Dagang Wei
>Priority: Minor
>
> [Flink docker images](https://github.com/docker-flink/docker-flink) do not 
> support custom config, in order to do so, the user has to build their own 
> image based on the official image and modify config/flink-conf.yaml. It would 
> be much easier if the image accepts custom config through args and in 
> [docker-entrypoint.sh](https://github.com/docker-flink/docker-flink/blob/master/docker-entrypoint.sh)
>  it automatically merge the custom config into config/flink-conf.yaml.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce 
TwoInputSelectionHandler for selecting input in 
StreamTwoInputSelectableProcessor
URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024
 
 
   
   ## CI report:
   
   * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123648363)
   * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123684855)
   * 7d75f94fed9684c020c7673a57431c79e2f0a543 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/124674615)
   * e21b3e5b91328bd7e9ce28fb8df9b848730b319a : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/124675311)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (FLINK-13865) Support custom config in Flink docker image

2019-08-26 Thread Dagang Wei (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dagang Wei updated FLINK-13865:
---
Environment: (was: I noticed FLINK-13380 which is helpful for 
Kubernetes, but it is still useful to support it at Docker image level.)

> Support custom config in Flink docker image
> ---
>
> Key: FLINK-13865
> URL: https://issues.apache.org/jira/browse/FLINK-13865
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Docker
>Affects Versions: 1.6.3, 1.6.4, 1.7.2, 1.8.0, 1.8.1, 1.9.0
>Reporter: Dagang Wei
>Priority: Minor
>
> [Flink docker images](https://github.com/docker-flink/docker-flink) do not 
> support custom config, in order to do so, the user has to build their own 
> image based on the official image and modify config/flink-conf.yaml. It would 
> be much easier if the image accepts custom config through args and in 
> [docker-entrypoint.sh](https://github.com/docker-flink/docker-flink/blob/master/docker-entrypoint.sh)
>  it automatically merge the custom config into config/flink-conf.yaml.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] zhijiangW commented on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread GitBox
zhijiangW commented on issue #9477: [FLINK-13765][task] Introduce 
TwoInputSelectionHandler for selecting input in 
StreamTwoInputSelectableProcessor
URL: https://github.com/apache/flink/pull/9477#issuecomment-525149250
 
 
   @flinkbot run travis


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13488:
---
Fix Version/s: (was: 1.9.1)
   (was: 1.10.0)
   1.9.0

> flink-python fails to build on Travis due to PackagesNotFoundError
> --
>
> Key: FLINK-13488
> URL: https://issues.apache.org/jira/browse/FLINK-13488
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Test Infrastructure
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: sunjincheng
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> https://api.travis-ci.org/v3/job/564925115/log.txt
> {code}
> install conda ... [SUCCESS]
> install miniconda... [SUCCESS]
> installing python environment...
> installing python2.7...
> install python2.7... [SUCCESS]
> installing python3.3...
> PackagesNotFoundError: The following packages are not available from current 
> channels:
>   - python=3.3
> Current channels:
>   - https://repo.anaconda.com/pkgs/main/linux-64
>   - https://repo.anaconda.com/pkgs/main/noarch
>   - https://repo.anaconda.com/pkgs/r/linux-64
>   - https://repo.anaconda.com/pkgs/r/noarch
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916399#comment-16916399
 ] 

Kurt Young commented on FLINK-13488:


Yeah, you're right. I will revert the change, thanks.

> flink-python fails to build on Travis due to PackagesNotFoundError
> --
>
> Key: FLINK-13488
> URL: https://issues.apache.org/jira/browse/FLINK-13488
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Test Infrastructure
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: sunjincheng
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> https://api.travis-ci.org/v3/job/564925115/log.txt
> {code}
> install conda ... [SUCCESS]
> install miniconda... [SUCCESS]
> installing python environment...
> installing python2.7...
> install python2.7... [SUCCESS]
> installing python3.3...
> PackagesNotFoundError: The following packages are not available from current 
> channels:
>   - python=3.3
> Current channels:
>   - https://repo.anaconda.com/pkgs/main/linux-64
>   - https://repo.anaconda.com/pkgs/main/noarch
>   - https://repo.anaconda.com/pkgs/r/linux-64
>   - https://repo.anaconda.com/pkgs/r/noarch
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] dianfu commented on issue #9496: [FLINK-13011][build] Add the Build logic for Python API release package

2019-08-26 Thread GitBox
dianfu commented on issue #9496: [FLINK-13011][build] Add the Build logic for 
Python API release package
URL: https://github.com/apache/flink/pull/9496#issuecomment-525145865
 
 
   @sunjincheng121 Thanks a lot for the PR. LGTM. +1 from my side.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add 
documentation for TopN and Deduplication in blink planner
URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126
 
 
   
   ## CI report:
   
   * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/124206586)
   * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124227220)
   * f1399e04785d2afecacfc1962d68bbf98416f68b : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/124676036)
   * 0e1c29aa3ddb1f602b4f63271382ebf7171cca50 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/124683423)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add 
documentation for TopN and Deduplication in blink planner
URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126
 
 
   
   ## CI report:
   
   * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/124206586)
   * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124227220)
   * f1399e04785d2afecacfc1962d68bbf98416f68b : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/124676036)
   * 0e1c29aa3ddb1f602b4f63271382ebf7171cca50 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add 
documentation for TopN and Deduplication in blink planner
URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126
 
 
   
   ## CI report:
   
   * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/124206586)
   * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124227220)
   * f1399e04785d2afecacfc1962d68bbf98416f68b : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/124676036)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend

2019-08-26 Thread GitBox
carp84 commented on a change in pull request #9501: [FLINK-12697] [State 
Backends] Support on-disk state storage for spill-able heap backend
URL: https://github.com/apache/flink/pull/9501#discussion_r317895384
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/ByteBufferUtils.java
 ##
 @@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.heap;
+
+import javax.annotation.Nonnull;
+
+import java.lang.reflect.Field;
+import java.nio.ByteBuffer;
+
+/**
+ * Utilities to get/put data to {@link ByteBuffer}. All methods don't change
+ * byte buffer's position.
+ */
+@SuppressWarnings({"WeakerAccess", "unused", "UnusedReturnValue"})
 
 Review comment:
   Will move this class to `org.apache.flink.core.memory` in flink-core, then 
we don't need the `WeakerAccess` warning suppress anymore. The current "unused" 
method might be used latter since this is only part of the whole work, will add 
test case to cover it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend

2019-08-26 Thread GitBox
carp84 commented on a change in pull request #9501: [FLINK-12697] [State 
Backends] Support on-disk state storage for spill-able heap backend
URL: https://github.com/apache/flink/pull/9501#discussion_r317895515
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/ByteBufferUtils.java
 ##
 @@ -0,0 +1,332 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.heap;
+
+import javax.annotation.Nonnull;
+
+import java.lang.reflect.Field;
+import java.nio.ByteBuffer;
+
+/**
+ * Utilities to get/put data to {@link ByteBuffer}. All methods don't change
+ * byte buffer's position.
+ */
+@SuppressWarnings({"WeakerAccess", "unused", "UnusedReturnValue"})
+public class ByteBufferUtils {
+
+   private static final boolean UNSAFE_AVAIL = UnsafeHelp.isAvailable();
+   private static final boolean UNSAFE_UNALIGNED = UnsafeHelp.unaligned();
+   private static final Field ACCESS_FIELD;
+
+   static {
+   try {
+   ACCESS_FIELD = 
java.nio.Buffer.class.getDeclaredField("address");
+   ACCESS_FIELD.setAccessible(true);
+   } catch (NoSuchFieldException e) {
+   throw new RuntimeException("Failed to get address 
method from java.nio.Buffer", e);
+   }
+   }
+
+   /**
+* Reads an int value at the given buffer's offset.
+*
+* @param buffer the given buffer
+* @param offset the given buffer's offset
+* @return int value at offset
+*/
+   public static int toInt(ByteBuffer buffer, int offset) {
+   if (UNSAFE_UNALIGNED) {
+   return UnsafeHelp.toInt(buffer, offset);
+   } else {
+   return buffer.getInt(offset);
+   }
+   }
+
+   /**
+* Reads a long value at the given buffer's offset.
+*
+* @param buffer the given buffer
+* @param offset the given buffer's offset
+* @return long value at offset
+*/
+   public static long toLong(ByteBuffer buffer, int offset) {
+   if (UNSAFE_UNALIGNED) {
+   return UnsafeHelp.toLong(buffer, offset);
+   } else {
+   return buffer.getLong(offset);
+   }
+   }
+
+   /**
+* Reads a short value at the given buffer's offset.
+*
+* @param buffer the given buffer
+* @param offset the given buffer's offset
+* @return short value at offset
+*/
+   public static short toShort(ByteBuffer buffer, int offset) {
+   if (UNSAFE_UNALIGNED) {
+   return UnsafeHelp.toShort(buffer, offset);
+   } else {
+   return buffer.getShort(offset);
+   }
+   }
+
+   public static byte toByte(ByteBuffer buffer, int offset) {
+   if (UnsafeHelp.isAvailable()) {
+   return UnsafeHelp.toByte(buffer, offset);
+   } else {
+   return buffer.get(offset);
+   }
+   }
+
+   public static void putInt(ByteBuffer buffer, int index, int val) {
+   if (UNSAFE_UNALIGNED) {
+   UnsafeHelp.putInt(buffer, index, val);
+   } else {
+   buffer.putInt(index, val);
+   }
+   }
+
+   public static void putLong(ByteBuffer buffer, int index, long val) {
+   if (UNSAFE_UNALIGNED) {
+   UnsafeHelp.putLong(buffer, index, val);
+   } else {
+   buffer.putLong(index, val);
+   }
+   }
+
+   /**
+* Copy from one buffer to another from given offset. This will be 
absolute positional copying and
+* won't affect the position of any of the buffers.
+*
+* @param inthe given buffer to read
+* @param out   the given buffer of destination
+* @param sourceOffset  the given buffer's offset of src
+* @param destinationOffset the 

[jira] [Comment Edited] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError

2019-08-26 Thread Dian Fu (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916370#comment-16916370
 ] 

Dian Fu edited comment on FLINK-13488 at 8/27/19 5:01 AM:
--

Hi [~ykt836], I guess this fix is actually contained in 
[1.9.0|https://github.com/apache/flink/commits/release-1.9.0?after=9c32ed989c0178a2bf3e059e897927c451188700+190].
 Pls correct me if I missed something:) Do you mean the PR 
[https://github.com/apache/flink/pull/9431] ? I guess it refers a wrong Jira 
number and linked to this Jira by mistake. [~gjy]


was (Author: dian.fu):
Hi [~ykt836], I guess this fix is actually contained in 
[1.9.0|https://github.com/apache/flink/commits/release-1.9.0?after=9c32ed989c0178a2bf3e059e897927c451188700+190].
 Pls correct me if I missed something:)

> flink-python fails to build on Travis due to PackagesNotFoundError
> --
>
> Key: FLINK-13488
> URL: https://issues.apache.org/jira/browse/FLINK-13488
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Test Infrastructure
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: sunjincheng
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> https://api.travis-ci.org/v3/job/564925115/log.txt
> {code}
> install conda ... [SUCCESS]
> install miniconda... [SUCCESS]
> installing python environment...
> installing python2.7...
> install python2.7... [SUCCESS]
> installing python3.3...
> PackagesNotFoundError: The following packages are not available from current 
> channels:
>   - python=3.3
> Current channels:
>   - https://repo.anaconda.com/pkgs/main/linux-64
>   - https://repo.anaconda.com/pkgs/main/noarch
>   - https://repo.anaconda.com/pkgs/r/linux-64
>   - https://repo.anaconda.com/pkgs/r/noarch
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError

2019-08-26 Thread Dian Fu (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916370#comment-16916370
 ] 

Dian Fu commented on FLINK-13488:
-

Hi [~ykt836], I guess this fix is actually contained in 
[1.9.0|https://github.com/apache/flink/commits/release-1.9.0?after=9c32ed989c0178a2bf3e059e897927c451188700+190].
 Pls correct me if I missed something:)

> flink-python fails to build on Travis due to PackagesNotFoundError
> --
>
> Key: FLINK-13488
> URL: https://issues.apache.org/jira/browse/FLINK-13488
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Test Infrastructure
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: sunjincheng
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> https://api.travis-ci.org/v3/job/564925115/log.txt
> {code}
> install conda ... [SUCCESS]
> install miniconda... [SUCCESS]
> installing python environment...
> installing python2.7...
> install python2.7... [SUCCESS]
> installing python3.3...
> PackagesNotFoundError: The following packages are not available from current 
> channels:
>   - python=3.3
> Current channels:
>   - https://repo.anaconda.com/pkgs/main/linux-64
>   - https://repo.anaconda.com/pkgs/main/noarch
>   - https://repo.anaconda.com/pkgs/r/linux-64
>   - https://repo.anaconda.com/pkgs/r/noarch
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce 
TwoInputSelectionHandler for selecting input in 
StreamTwoInputSelectableProcessor
URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024
 
 
   
   ## CI report:
   
   * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123648363)
   * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123684855)
   * 7d75f94fed9684c020c7673a57431c79e2f0a543 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/124674615)
   * e21b3e5b91328bd7e9ce28fb8df9b848730b319a : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124675311)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in 
ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#issuecomment-522053886
 
 
   
   ## CI report:
   
   * 009da497d7c551ba854dc7ed8fa658f2acd6d6ce : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123529451)
   * 14be3c57f3d625f3f9bbb96b7f74b6b0116fed3c : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124588576)
   * c43a462c92c6c5c145a6964aa5390a82b894fa4e : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/124673290)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (FLINK-13364) Add documentation for setting resources in blink planner

2019-08-26 Thread Jark Wu (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu updated FLINK-13364:

Fix Version/s: (was: 1.9.1)

> Add documentation for setting resources in blink planner
> 
>
> Key: FLINK-13364
> URL: https://issues.apache.org/jira/browse/FLINK-13364
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table SQL / Planner
>Reporter: Jark Wu
>Assignee: xupingyong
>Priority: Major
>
> Add documentation for setting resources.
> - “Performance Tuning / Resource Setting”: how to set resources in blink 
> planner, explain the configs and how they work together.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13361) Add documentation for JDBC connector for Table API & SQL

2019-08-26 Thread Jark Wu (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu updated FLINK-13361:

Fix Version/s: 1.10.0

> Add documentation for JDBC connector for Table API & SQL
> 
>
> Key: FLINK-13361
> URL: https://issues.apache.org/jira/browse/FLINK-13361
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / JDBC, Documentation
>Reporter: Jark Wu
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.10.0, 1.9.1
>
>
> Add documentation for JDBC connector for Table API & SQL
> - “Connect to External Systems”: Add DDL for JDBC in “Table Connector” 
> section. JDBC support batch-source & lookup & sink.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13365) Add documentation for reusing upstream operations when multi-sink is applied

2019-08-26 Thread Jark Wu (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu updated FLINK-13365:

Fix Version/s: (was: 1.9.1)
   (was: 1.10.0)

> Add documentation for reusing upstream operations when multi-sink is applied
> 
>
> Key: FLINK-13365
> URL: https://issues.apache.org/jira/browse/FLINK-13365
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
>
> Add documentation for reuse upstream operations when multi-sink
> - “Performance Tuning / Multi-Sink Optimization”: How to avoid executing 
> common operators repeatedly if there are multiple TableSinks in a job.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13360) Add documentation for HBase connector for Table API & SQL

2019-08-26 Thread Jark Wu (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu updated FLINK-13360:

Fix Version/s: 1.10.0

> Add documentation for HBase connector for Table API & SQL
> -
>
> Key: FLINK-13360
> URL: https://issues.apache.org/jira/browse/FLINK-13360
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / HBase, Documentation
>Reporter: Jark Wu
>Assignee: Caizhi Weng
>Priority: Major
> Fix For: 1.10.0, 1.9.1
>
>
> Add documentation for HBase connector for Table API & SQL
> - “Connect to External Systems”: Add DDL for HBase in “Table Connector” 
> section. HBase support batch-source & lookup & sink.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13363) Add documentation for streaming aggregate performance tunning.

2019-08-26 Thread Jark Wu (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu updated FLINK-13363:

Fix Version/s: 1.10.0

> Add documentation for streaming aggregate performance tunning.
> --
>
> Key: FLINK-13363
> URL: https://issues.apache.org/jira/browse/FLINK-13363
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Add documentation for streaming aggregate performance tuning
> - “Performance Tuning / Streaming Aggregation”: Explain how to solve 
> data-skew problem in streaming aggregation (non-windowed aggregate), the 
> internals, and the configurations.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13708) transformations should be cleared because a table environment could execute multiple job

2019-08-26 Thread Jark Wu (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu updated FLINK-13708:

Fix Version/s: 1.10.0

> transformations should be cleared because a table environment could execute 
> multiple job
> 
>
> Key: FLINK-13708
> URL: https://issues.apache.org/jira/browse/FLINK-13708
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> currently, if a table environment execute more than one sql jobs, the 
> following job contains transformations about the previous job. the reason is 
> the transformations is not cleared after execution



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13520) Add documentation for the different behavior of row count sliding window in blink planner

2019-08-26 Thread Jark Wu (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu updated FLINK-13520:

Fix Version/s: 1.10.0

> Add documentation for the different behavior of row count sliding window in 
> blink planner
> -
>
> Key: FLINK-13520
> URL: https://issues.apache.org/jira/browse/FLINK-13520
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table SQL / Planner
>Reporter: Jark Wu
>Priority: Minor
> Fix For: 1.10.0, 1.9.1
>
>
> As discussed in FLINK-13446, we want to keep the current behavior of row 
> count sliding window in blink planner. This is different with flink planner. 
> So it would be nice to document it. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13365) Add documentation for reusing upstream operations when multi-sink is applied

2019-08-26 Thread Jark Wu (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jark Wu updated FLINK-13365:

Fix Version/s: 1.10.0

> Add documentation for reusing upstream operations when multi-sink is applied
> 
>
> Key: FLINK-13365
> URL: https://issues.apache.org/jira/browse/FLINK-13365
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.10.0, 1.9.1
>
>
> Add documentation for reuse upstream operations when multi-sink
> - “Performance Tuning / Multi-Sink Optimization”: How to avoid executing 
> common operators repeatedly if there are multiple TableSinks in a job.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13359) Add documentation for DDL introduction

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13359:
---
Fix Version/s: 1.10.0

> Add documentation for DDL introduction
> --
>
> Key: FLINK-13359
> URL: https://issues.apache.org/jira/browse/FLINK-13359
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table SQL / API
>Reporter: Jark Wu
>Assignee: Danny Chan
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Add documentation for DDL introduction
> -  “Concepts & Common API”: Add a section to describe how to execute DDL on 
> TableEnvironment.
> - “SQL Client”: Add a section and example in SQL CLI page too?



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13362) Add documentation for Kafka & ES & FileSystem DDL

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13362:
---
Fix Version/s: 1.10.0

> Add documentation for Kafka & ES & FileSystem DDL
> -
>
> Key: FLINK-13362
> URL: https://issues.apache.org/jira/browse/FLINK-13362
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Jark Wu
>Assignee: Danny Chan
>Priority: Major
> Fix For: 1.10.0, 1.9.1
>
>
> Add documentation for Kafka & ES & FileSystem DDL
> - “Connect to External Systems”: Add DDL for Kafka & ES & FileSystem.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13105) Add documentation for blink planner's built-in functions

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13105:
---
Fix Version/s: 1.10.0

> Add documentation for blink planner's built-in functions
> 
>
> Key: FLINK-13105
> URL: https://issues.apache.org/jira/browse/FLINK-13105
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table SQL / Planner
>Reporter: Zhenghua Gao
>Assignee: Zhenghua Gao
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Blink planner intros some built-in functions which need to be documented.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13354) Add documentation for how to use blink planner

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13354:
---
Fix Version/s: 1.10.0

> Add documentation for how to use blink planner
> --
>
> Key: FLINK-13354
> URL: https://issues.apache.org/jira/browse/FLINK-13354
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Table SQL / Planner
>Reporter: Jark Wu
>Assignee: godfrey he
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Add documentation for how to use different planner
> - “Overview”: add pom dependency
> -  “Concepts & Common API”:  add description about how to use different 
> planner in code



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13564) blink planner should also throw exception if constant with YEAR TO MONTH resolution was used for group windows

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13564:
---
Fix Version/s: 1.10.0

> blink planner should also throw exception if constant with YEAR TO MONTH 
> resolution was used for group windows 
> ---
>
> Key: FLINK-13564
> URL: https://issues.apache.org/jira/browse/FLINK-13564
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.10.0
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> just as [FLINK-11017|https://issues.apache.org/jira/browse/FLINK-11017], 
> blink planner should also throw exception if constant with YEAR TO MONTH 
> resolution was used for group windows 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend

2019-08-26 Thread GitBox
carp84 commented on a change in pull request #9501: [FLINK-12697] [State 
Backends] Support on-disk state storage for spill-able heap backend
URL: https://github.com/apache/flink/pull/9501#discussion_r317887398
 
 

 ##
 File path: 
flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/ByteBufferInputStreamWithPos.java
 ##
 @@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.state.heap;
 
 Review comment:
   Agree that we'd better use another package name. How about we rename the 
package as the last step after all review comments addressed just to convenient 
the review?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (FLINK-13599) Kinesis end-to-end test failed on Travis

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916349#comment-16916349
 ] 

Kurt Young commented on FLINK-13599:


I've updated the fix version to 1.9.1 since this is not included in 1.9.0 
release.

> Kinesis end-to-end test failed on Travis
> 
>
> Key: FLINK-13599
> URL: https://issues.apache.org/jira/browse/FLINK-13599
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Kinesis, Tests
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Till Rohrmann
>Assignee: Andrey Zagrebin
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The {{Kinesis end-to-end test}} failed on Travis with 
> {code}
> 2019-08-06 08:48:20,177 ERROR org.apache.flink.client.cli.CliFrontend 
>   - Error while running the command.
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error: Unable to execute HTTP request: Connect to localhost:4567 
> [localhost/127.0.0.1] failed: Connection refused (Connection refused)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
>   at 
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> Caused by: org.apache.flink.kinesis.shaded.com.amazonaws.SdkClientException: 
> Unable to execute HTTP request: Connect to localhost:4567 
> [localhost/127.0.0.1] failed: Connection refused (Connection refused)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1116)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1066)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2388)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2364)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.executeDescribeStream(AmazonKinesisClient.java:754)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:729)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:766)
>   at 
> org.apache.flink.streaming.kinesis.test.KinesisPubsubClient.createTopic(KinesisPubsubClient.java:63)
>   at 
> org.apache.flink.streaming.kinesis.test.KinesisExampleTest.main(KinesisExampleTest.java:57)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
>   ... 9 more
> Caused by: 
> 

[jira] [Updated] (FLINK-13599) Kinesis end-to-end test failed on Travis

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13599:
---
Fix Version/s: (was: 1.9.0)
   1.9.1

> Kinesis end-to-end test failed on Travis
> 
>
> Key: FLINK-13599
> URL: https://issues.apache.org/jira/browse/FLINK-13599
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Kinesis, Tests
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Till Rohrmann
>Assignee: Andrey Zagrebin
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The {{Kinesis end-to-end test}} failed on Travis with 
> {code}
> 2019-08-06 08:48:20,177 ERROR org.apache.flink.client.cli.CliFrontend 
>   - Error while running the command.
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error: Unable to execute HTTP request: Connect to localhost:4567 
> [localhost/127.0.0.1] failed: Connection refused (Connection refused)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
>   at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
>   at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
>   at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
>   at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
>   at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
>   at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
>   at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
>   at 
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>   at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> Caused by: org.apache.flink.kinesis.shaded.com.amazonaws.SdkClientException: 
> Unable to execute HTTP request: Connect to localhost:4567 
> [localhost/127.0.0.1] failed: Connection refused (Connection refused)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1116)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1066)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2388)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2364)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.executeDescribeStream(AmazonKinesisClient.java:754)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:729)
>   at 
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:766)
>   at 
> org.apache.flink.streaming.kinesis.test.KinesisPubsubClient.createTopic(KinesisPubsubClient.java:63)
>   at 
> org.apache.flink.streaming.kinesis.test.KinesisExampleTest.main(KinesisExampleTest.java:57)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
>   ... 9 more
> Caused by: 
> 

[GitHub] [flink] flinkbot edited a comment on issue #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9540: [FLINK-13859][docs] 
JSONDeserializationSchema spell error
URL: https://github.com/apache/flink/pull/9540#issuecomment-525128086
 
 
   
   ## CI report:
   
   * 5cb9691c978214d5f868c0e6a43f782b05b29ff6 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/124678443)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (FLINK-13277) add documentation of Hive source/sink

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916347#comment-16916347
 ] 

Kurt Young commented on FLINK-13277:


Looks like this issue already merged, [~phoenixjiangnan] could you please close 
this jira?

> add documentation of Hive source/sink
> -
>
> Key: FLINK-13277
> URL: https://issues.apache.org/jira/browse/FLINK-13277
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Hive, Documentation
>Reporter: Bowen Li
>Assignee: Rui Li
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> add documentation of Hive source/sink in {{batch/connector.md}}
> its corresponding Chinese one is FLINK-13278
> cc [~xuefuz] [~lirui] [~Terry1897]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13738) NegativeArraySizeException in LongHybridHashTable

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13738:
---
Fix Version/s: 1.10.0

> NegativeArraySizeException in LongHybridHashTable
> -
>
> Key: FLINK-13738
> URL: https://issues.apache.org/jira/browse/FLINK-13738
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Runtime
>Affects Versions: 1.9.0
>Reporter: Robert Metzger
>Assignee: Jingsong Lee
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Executing this (meaningless) query:
> {code:java}
> INSERT INTO sinkTable ( SELECT CONCAT( CAST( id AS VARCHAR), CAST( COUNT(*) 
> AS VARCHAR)) as something, 'const' FROM CsvTable, table1  WHERE sometxt LIKE 
> 'a%' AND id = key GROUP BY id ) {code}
> leads to the following exception:
> {code:java}
> Caused by: java.lang.NegativeArraySizeException
>  at 
> org.apache.flink.table.runtime.hashtable.LongHybridHashTable.tryDenseMode(LongHybridHashTable.java:216)
>  at 
> org.apache.flink.table.runtime.hashtable.LongHybridHashTable.endBuild(LongHybridHashTable.java:105)
>  at LongHashJoinOperator$36.endInput1$(Unknown Source)
>  at LongHashJoinOperator$36.endInput(Unknown Source)
>  at 
> org.apache.flink.streaming.runtime.tasks.OperatorChain.endInput(OperatorChain.java:256)
>  at 
> org.apache.flink.streaming.runtime.io.StreamTwoInputSelectableProcessor.checkFinished(StreamTwoInputSelectableProcessor.java:359)
>  at 
> org.apache.flink.streaming.runtime.io.StreamTwoInputSelectableProcessor.processInput(StreamTwoInputSelectableProcessor.java:193)
>  at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.performDefaultAction(StreamTask.java:276)
>  at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298)
>  at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403)
>  at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:687)
>  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:517)
>  at java.lang.Thread.run(Thread.java:748){code}
> This is the plan:
>  
> {code:java}
> == Abstract Syntax Tree ==
> LogicalSink(name=[sinkTable], fields=[f0, f1])
> +- LogicalProject(something=[CONCAT(CAST($0):VARCHAR(2147483647) CHARACTER 
> SET "UTF-16LE", CAST($1):VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT 
> NULL)], EXPR$1=[_UTF-16LE'const'])
>+- LogicalAggregate(group=[{0}], agg#0=[COUNT()])
>   +- LogicalProject(id=[$1])
>  +- LogicalFilter(condition=[AND(LIKE($0, _UTF-16LE'a%'), =($1, 
> CAST($2):BIGINT))])
> +- LogicalJoin(condition=[true], joinType=[inner])
>:- LogicalTableScan(table=[[default_catalog, default_database, 
> CsvTable, source: [CsvTableSource(read fields: sometxt, id)]]])
>+- LogicalTableScan(table=[[default_catalog, default_database, 
> table1, source: [GeneratorTableSource(key, rowtime, payload)]]])
> == Optimized Logical Plan ==
> Sink(name=[sinkTable], fields=[f0, f1]): rowcount = 1498810.6659336376, 
> cumulative cost = {4.459964319978008E8 rows, 1.879799762133187E10 cpu, 4.8E9 
> io, 8.4E8 network, 1.799524266373455E8 memory}
> +- Calc(select=[CONCAT(CAST(id), CAST($f1)) AS something, _UTF-16LE'const' AS 
> EXPR$1]): rowcount = 1498810.6659336376, cumulative cost = 
> {4.444976213318672E8 rows, 1.8796498810665936E10 cpu, 4.8E9 io, 8.4E8 
> network, 1.799524266373455E8 memory}
>+- HashAggregate(isMerge=[false], groupBy=[id], select=[id, COUNT(*) AS 
> $f1]): rowcount = 1498810.6659336376, cumulative cost = {4.429988106659336E8 
> rows, 1.8795E10 cpu, 4.8E9 io, 8.4E8 network, 1.799524266373455E8 memory}
>   +- Calc(select=[id]): rowcount = 1.575E7, cumulative cost = {4.415E8 
> rows, 1.848E10 cpu, 4.8E9 io, 8.4E8 network, 1.2E8 memory}
>  +- HashJoin(joinType=[InnerJoin], where=[=(id, key0)], select=[id, 
> key0], build=[left]): rowcount = 1.575E7, cumulative cost = {4.2575E8 rows, 
> 1.848E10 cpu, 4.8E9 io, 8.4E8 network, 1.2E8 memory}
> :- Exchange(distribution=[hash[id]]): rowcount = 500.0, 
> cumulative cost = {1.1E8 rows, 8.4E8 cpu, 2.0E9 io, 4.0E7 network, 0.0 memory}
> :  +- Calc(select=[id], where=[LIKE(sometxt, _UTF-16LE'a%')]): 
> rowcount = 500.0, cumulative cost = {1.05E8 rows, 0.0 cpu, 2.0E9 io, 0.0 
> network, 0.0 memory}
> : +- TableSourceScan(table=[[default_catalog, 
> default_database, CsvTable, source: [CsvTableSource(read fields: sometxt, 
> id)]]], fields=[sometxt, id]): rowcount = 1.0E8, cumulative cost = {1.0E8 
> rows, 0.0 cpu, 2.0E9 io, 0.0 network, 0.0 memory}
> +- Exchange(distribution=[hash[key0]]): rowcount = 1.0E8, 
> cumulative cost = {3.0E8 rows, 1.68E10 cpu, 2.8E9 io, 

[jira] [Updated] (FLINK-13739) BinaryRowTest.testWriteString() fails in some environments

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13739:
---
Fix Version/s: 1.10.0

> BinaryRowTest.testWriteString() fails in some environments
> --
>
> Key: FLINK-13739
> URL: https://issues.apache.org/jira/browse/FLINK-13739
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Runtime
>Affects Versions: 1.9.0
> Environment:  
>  
>Reporter: Robert Metzger
>Assignee: Jingsong Lee
>Priority: Major
>  Labels: pull-request-available, test-stability
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
>  
>  
> {code:java}
> Test set: org.apache.flink.table.dataformat.BinaryRowTest
> ---
> Tests run: 26, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.328 s <<< 
> FAILURE! - in org.apache.flink.table.dataformat.BinaryRowTest
> testWriteString(org.apache.flink.table.dataformat.BinaryRowTest)  Time 
> elapsed: 0.05 s  <<< FAILURE!
> org.junit.ComparisonFailure: 
> expected:<[<95><95><95><95><95><88><91><98>
> <90><9A><84><89><88><8C>]> 
> but was:<[?]>
>         at 
> org.apache.flink.table.dataformat.BinaryRowTest.testWriteString(BinaryRowTest.java:189)
> {code}
>  
> This error happens on a Google Cloud n2-standard-16 (16 vCPUs, 64 GB memory) 
> machine.
> {code}$ lsb_release -a
> No LSB modules are available.
> Distributor ID:Debian
> Description:Debian GNU/Linux 9.9 (stretch)
> Release:9.9
> Codename:stretch{code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13742) Fix code generation when aggregation contains both distinct aggregate with and without filter

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13742:
---
Fix Version/s: 1.10.0

> Fix code generation when aggregation contains both distinct aggregate with 
> and without filter
> -
>
> Key: FLINK-13742
> URL: https://issues.apache.org/jira/browse/FLINK-13742
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Shuo Cheng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The following test will fail when the aggregation contains {{COUNT(DISTINCT 
> c)}} and {{COUNT(DISTINCT c) filter ...}}.
> {code:java}
> @Test
>   def testDistinctWithMultiFilter(): Unit = {
> val sqlQuery =
>   "SELECT b, " +
> "  SUM(DISTINCT (a * 3)), " +
> "  COUNT(DISTINCT SUBSTRING(c FROM 1 FOR 2))," +
> "  COUNT(DISTINCT c)," +
> "  COUNT(DISTINCT c) filter (where MOD(a, 3) = 0)," +
> "  COUNT(DISTINCT c) filter (where MOD(a, 3) = 1) " +
> "FROM MyTable " +
> "GROUP BY b"
> val t = 
> failingDataSource(StreamTestData.get3TupleData).toTable(tEnv).as('a, 'b, 'c)
> tEnv.registerTable("MyTable", t)
> val result = tEnv.sqlQuery(sqlQuery).toRetractStream[Row]
> val sink = new TestingRetractSink
> result.addSink(sink)
> env.execute()
> val expected = List(
>   "1,3,1,1,0,1",
>   "2,15,1,2,1,0",
>   "3,45,3,3,1,1",
>   "4,102,1,4,1,2",
>   "5,195,1,5,2,1",
>   "6,333,1,6,2,2")
> assertEquals(expected.sorted, sink.getRetractResults.sorted)
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13277) add documentation of Hive source/sink

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13277:
---
Fix Version/s: 1.10.0

> add documentation of Hive source/sink
> -
>
> Key: FLINK-13277
> URL: https://issues.apache.org/jira/browse/FLINK-13277
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Hive, Documentation
>Reporter: Bowen Li
>Assignee: Rui Li
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> add documentation of Hive source/sink in {{batch/connector.md}}
> its corresponding Chinese one is FLINK-13278
> cc [~xuefuz] [~lirui] [~Terry1897]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13737) flink-dist should add provided dependency on flink-examples-table

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916346#comment-16916346
 ] 

Kurt Young commented on FLINK-13737:


I've updated the fix version to 1.9.1 since this is not included in 1.9.0 
release.

> flink-dist should add provided dependency on flink-examples-table
> -
>
> Key: FLINK-13737
> URL: https://issues.apache.org/jira/browse/FLINK-13737
> Project: Flink
>  Issue Type: Bug
>  Components: Examples
>Affects Versions: 1.9.0
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In FLINK-13558 we changed the `flink-dist/bin.xml` to also include 
> flink-examples-table in the binary distribution. The flink-dist module though 
> does not depend on the flink-examples-table.
> If only the flink-dist module is built with its dependencies (this happens in 
> the release scripts). The table examples are not built and thus not included 
> in the distribution



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13737) flink-dist should add provided dependency on flink-examples-table

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13737:
---
Fix Version/s: (was: 1.9.0)
   1.9.1
   1.10.0

> flink-dist should add provided dependency on flink-examples-table
> -
>
> Key: FLINK-13737
> URL: https://issues.apache.org/jira/browse/FLINK-13737
> Project: Flink
>  Issue Type: Bug
>  Components: Examples
>Affects Versions: 1.9.0
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In FLINK-13558 we changed the `flink-dist/bin.xml` to also include 
> flink-examples-table in the binary distribution. The flink-dist module though 
> does not depend on the flink-examples-table.
> If only the flink-dist module is built with its dependencies (this happens in 
> the release scripts). The table examples are not built and thus not included 
> in the distribution



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13688) HiveCatalogUseBlinkITCase.testBlinkUdf constantly failed with 1.9.0-rc2

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916344#comment-16916344
 ] 

Kurt Young commented on FLINK-13688:


[~till.rohrmann] Thanks for the heads up, this was planned. I'm also start to 
scan all issues that are not included in 1.9.0 release and change the 
fixVersion now.

> HiveCatalogUseBlinkITCase.testBlinkUdf constantly failed with 1.9.0-rc2
> ---
>
> Key: FLINK-13688
> URL: https://issues.apache.org/jira/browse/FLINK-13688
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Hive, Tests
>Affects Versions: 1.9.0
> Environment: Linux server
> kernal version: 3.10.0
> java version: "1.8.0_102"
> processor count: 96
>Reporter: Kurt Young
>Assignee: Jingsong Lee
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> I tried to build flink 1.9.0-rc2 from source and ran all tests in a linux 
> server, HiveCatalogUseBlinkITCase.testBlinkUdf will be constantly fail. 
>  
> Fail trace:
> {code:java}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
> 313.228 s <<< FAILURE! - in 
> org.apache.flink.table.catalog.hive.HiveCatalogUseBlinkITCase
> [ERROR] 
> testBlinkUdf(org.apache.flink.table.catalog.hive.HiveCatalogUseBlinkITCase) 
> Time elapsed: 305.155 s <<< ERROR!
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at 
> org.apache.flink.table.catalog.hive.HiveCatalogUseBlinkITCase.testBlinkUdf(HiveCatalogUseBlinkITCase.java:180)
> Caused by: 
> org.apache.flink.runtime.resourcemanager.exceptions.UnfulfillableSlotRequestException:
>  Could not fulfill slot request 35cf6fdc1b525de9b6eed13894e2e31d. Requested 
> resource profile (ResourceProfile{cpuCores=0.0, heapMemoryInMB=0, 
> directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0, 
> managedMemoryInMB=128}) is unfulfillable.
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13706) add documentation of how to use Hive functions in Flink

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916339#comment-16916339
 ] 

Kurt Young commented on FLINK-13706:


I've updated the fix version to 1.9.1 since this is not included in 1.9.0 
release.

> add documentation of how to use Hive functions in Flink
> ---
>
> Key: FLINK-13706
> URL: https://issues.apache.org/jira/browse/FLINK-13706
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Hive, Documentation
>Affects Versions: 1.9.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13706) add documentation of how to use Hive functions in Flink

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13706:
---
Fix Version/s: (was: 1.9.0)
   1.9.1
   1.10.0

> add documentation of how to use Hive functions in Flink
> ---
>
> Key: FLINK-13706
> URL: https://issues.apache.org/jira/browse/FLINK-13706
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / Hive, Documentation
>Affects Versions: 1.9.0
>Reporter: Bowen Li
>Assignee: Bowen Li
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13501) Fixes a few issues in documentation for Hive integration

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13501:
---
Fix Version/s: (was: 1.9.0)
   1.9.1
   1.10.0

> Fixes a few issues in documentation for Hive integration
> 
>
> Key: FLINK-13501
> URL: https://issues.apache.org/jira/browse/FLINK-13501
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Hive, Table SQL / API
>Affects Versions: 1.9.0
>Reporter: Xuefu Zhang
>Assignee: Terry Wang
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
> Attachments: Screen Shot 2019-07-30 at 3.21.25 PM.png, Screen Shot 
> 2019-07-30 at 3.25.13 PM.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Going thru existing Hive doc I found the following issues that should be 
> addressed:
> 1. Section "Hive Integration" should come after "SQL client" (at the same 
> level).
> 2. In Catalog section, there are headers named "Hive Catalog". Also, some 
> information is duplicated with that in "Hive Integration"
> 3. "Data Type Mapping" is Hive specific and should probably move to "Hive 
> integration"



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13415) Document how to use hive connector in scala shell

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916338#comment-16916338
 ] 

Kurt Young commented on FLINK-13415:


I've updated the fix version to 1.9.1 since this is not included in 1.9.0 
release.

> Document how to use hive connector in scala shell
> -
>
> Key: FLINK-13415
> URL: https://issues.apache.org/jira/browse/FLINK-13415
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13415) Document how to use hive connector in scala shell

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13415:
---
Fix Version/s: (was: 1.9.0)
   1.9.1
   1.10.0

> Document how to use hive connector in scala shell
> -
>
> Key: FLINK-13415
> URL: https://issues.apache.org/jira/browse/FLINK-13415
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Jeff Zhang
>Assignee: Jeff Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13501) Fixes a few issues in documentation for Hive integration

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916337#comment-16916337
 ] 

Kurt Young commented on FLINK-13501:


I've updated the fix version to 1.9.1 since this is not included in 1.9.0 
release.

> Fixes a few issues in documentation for Hive integration
> 
>
> Key: FLINK-13501
> URL: https://issues.apache.org/jira/browse/FLINK-13501
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Hive, Table SQL / API
>Affects Versions: 1.9.0
>Reporter: Xuefu Zhang
>Assignee: Terry Wang
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.9.0
>
> Attachments: Screen Shot 2019-07-30 at 3.21.25 PM.png, Screen Shot 
> 2019-07-30 at 3.25.13 PM.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Going thru existing Hive doc I found the following issues that should be 
> addressed:
> 1. Section "Hive Integration" should come after "SQL client" (at the same 
> level).
> 2. In Catalog section, there are headers named "Hive Catalog". Also, some 
> information is duplicated with that in "Hive Integration"
> 3. "Data Type Mapping" is Hive specific and should probably move to "Hive 
> integration"



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13534) Unable to query Hive table with decimal column

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13534:
---
Fix Version/s: 1.10.0

> Unable to query Hive table with decimal column
> --
>
> Key: FLINK-13534
> URL: https://issues.apache.org/jira/browse/FLINK-13534
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Hive
>Affects Versions: 1.9.0
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Hit the following exception when access a Hive table with decimal column:
> {noformat}
> Caused by: org.apache.flink.table.api.TableException: TableSource of type 
> org.apache.flink.batch.connectors.hive.HiveTableSource returned a DataSet of 
> data type ROW<`x` LEGACY(BigDecimal)> that does not match with the data type 
> ROW<`x` DECIMAL(10, 0)> declared by the TableSource.getProducedDataType() 
> method. Please validate the implementation of the TableSource.
>  at 
> org.apache.flink.table.plan.nodes.dataset.BatchTableSourceScan.translateToPlan(BatchTableSourceScan.scala:118)
>  at 
> org.apache.flink.table.api.internal.BatchTableEnvImpl.translate(BatchTableEnvImpl.scala:303)
>  at 
> org.apache.flink.table.api.internal.BatchTableEnvImpl.translate(BatchTableEnvImpl.scala:281)
>  at 
> org.apache.flink.table.api.internal.BatchTableEnvImpl.writeToSink(BatchTableEnvImpl.scala:117)
>  at 
> org.apache.flink.table.api.internal.TableEnvImpl.insertInto(TableEnvImpl.scala:564)
>  at 
> org.apache.flink.table.api.internal.TableEnvImpl.insertInto(TableEnvImpl.scala:516)
>  at 
> org.apache.flink.table.api.internal.BatchTableEnvImpl.insertInto(BatchTableEnvImpl.scala:59)
>  at 
> org.apache.flink.table.api.internal.TableImpl.insertInto(TableImpl.java:428)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916336#comment-16916336
 ] 

Kurt Young commented on FLINK-13488:


I've updated the fix version to 1.9.1 since this is not included in 1.9.0 
release.

> flink-python fails to build on Travis due to PackagesNotFoundError
> --
>
> Key: FLINK-13488
> URL: https://issues.apache.org/jira/browse/FLINK-13488
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Test Infrastructure
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: sunjincheng
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> https://api.travis-ci.org/v3/job/564925115/log.txt
> {code}
> install conda ... [SUCCESS]
> install miniconda... [SUCCESS]
> installing python environment...
> installing python2.7...
> install python2.7... [SUCCESS]
> installing python3.3...
> PackagesNotFoundError: The following packages are not available from current 
> channels:
>   - python=3.3
> Current channels:
>   - https://repo.anaconda.com/pkgs/main/linux-64
>   - https://repo.anaconda.com/pkgs/main/noarch
>   - https://repo.anaconda.com/pkgs/r/linux-64
>   - https://repo.anaconda.com/pkgs/r/noarch
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13488:
---
Fix Version/s: (was: 1.9.0)
   1.9.1
   1.10.0

> flink-python fails to build on Travis due to PackagesNotFoundError
> --
>
> Key: FLINK-13488
> URL: https://issues.apache.org/jira/browse/FLINK-13488
> Project: Flink
>  Issue Type: Bug
>  Components: API / Python, Test Infrastructure
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: sunjincheng
>Priority: Blocker
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> https://api.travis-ci.org/v3/job/564925115/log.txt
> {code}
> install conda ... [SUCCESS]
> install miniconda... [SUCCESS]
> installing python environment...
> installing python2.7...
> install python2.7... [SUCCESS]
> installing python3.3...
> PackagesNotFoundError: The following packages are not available from current 
> channels:
>   - python=3.3
> Current channels:
>   - https://repo.anaconda.com/pkgs/main/linux-64
>   - https://repo.anaconda.com/pkgs/main/noarch
>   - https://repo.anaconda.com/pkgs/r/linux-64
>   - https://repo.anaconda.com/pkgs/r/noarch
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13517) Restructure Hive Catalog documentation

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13517:
---
Fix Version/s: 1.10.0

> Restructure Hive Catalog documentation
> --
>
> Key: FLINK-13517
> URL: https://issues.apache.org/jira/browse/FLINK-13517
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Hive, Documentation
>Reporter: Seth Wiesman
>Assignee: Seth Wiesman
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Hive documentation is currently spread across a number of pages and 
> fragmented. In particular: 
> 1) An example was added to getting-started/examples, however, this section is 
> being removed
> 2) There is a dedicated page on hive integration but also a lot of hive 
> specific information is on the catalog page
> We should
> 1) Inline the example into the hive integration page
> 2) Move the hive specific information on catalogs.md to hive_integration.md
> 3) Make catalogs.md be just about catalogs in general and link to the hive 
> integration. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13490) Fix return null in JDBCUtils::getFieldFromResultSet

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13490:
---
Fix Version/s: 1.10.0

> Fix return null in JDBCUtils::getFieldFromResultSet
> ---
>
> Key: FLINK-13490
> URL: https://issues.apache.org/jira/browse/FLINK-13490
> Project: Flink
>  Issue Type: Sub-task
>  Components: Connectors / JDBC
>Reporter: Caizhi Weng
>Assignee: Caizhi Weng
>Priority: Critical
> Fix For: 1.10.0, 1.9.1
>
>
> The null checking in `JDBCUtils::getFieldFromResultSet` is incorrect. Under 
> the current implementation, if one column is null in the result set, the 
> following calls to this method using the same result set will always return 
> null, no matter what the content of the column is.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13526) Switching to a non existing catalog or database crashes sql-client

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13526:
---
Fix Version/s: 1.10.0

> Switching to a non existing catalog or database crashes sql-client
> --
>
> Key: FLINK-13526
> URL: https://issues.apache.org/jira/browse/FLINK-13526
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Client
>Affects Versions: 1.9.0
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> sql-client crashes if user tries to switch to a non-existing DB:
> {noformat}
> Exception in thread "main" org.apache.flink.table.client.SqlClientException: 
> Unexpected exception. This is a bug. Please consider filing an issue.
>   at org.apache.flink.table.client.SqlClient.main(SqlClient.java:206)
> Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A 
> database with name [foo] does not exist in the catalog: [myhive].
>   at 
> org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:286)
>   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:398)
>   at 
> org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$useDatabase$5(LocalExecutor.java:258)
>   at 
> org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:216)
>   at 
> org.apache.flink.table.client.gateway.local.LocalExecutor.useDatabase(LocalExecutor.java:256)
>   at 
> org.apache.flink.table.client.cli.CliClient.callUseDatabase(CliClient.java:434)
>   at 
> org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:282)
>   at java.util.Optional.ifPresent(Optional.java:159)
>   at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200)
>   at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:123)
>   at org.apache.flink.table.client.SqlClient.start(SqlClient.java:105)
>   at org.apache.flink.table.client.SqlClient.main(SqlClient.java:194)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13563) TumblingGroupWindow should implement toString method

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13563:
---
Fix Version/s: 1.10.0

> TumblingGroupWindow should implement toString method
> 
>
> Key: FLINK-13563
> URL: https://issues.apache.org/jira/browse/FLINK-13563
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.10.0
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> {code:scala}
>   @Test
>   def testAllEventTimeTumblingGroupWindowOverTime(): Unit = {
> val util = streamTestUtil()
> val table = util.addDataStream[(Long, Int, String)](
>   "T1", 'long, 'int, 'string, 'rowtime.rowtime)
> val windowedTable = table
>   .window(Tumble over 5.millis on 'rowtime as 'w)
>   .groupBy('w)
>   .select('int.count)
> util.verifyPlan(windowedTable)
>   }
> {code}
> currently, it's physical plan is 
> {code:java}
> HashWindowAggregate(window=[TumblingGroupWindow], 
> select=[Final_COUNT(count$0) AS EXPR$0])
> +- Exchange(distribution=[single])
>+- LocalHashWindowAggregate(window=[TumblingGroupWindow], 
> select=[Partial_COUNT(int) AS count$0])
>   +- TableSourceScan(table=[[default_catalog, default_database, Table1, 
> source: [TestTableSource(long, int, string)]]], fields=[long, int, string])
> {code}
> we know nothing about the TumblingGroupWindow except its name. the expected 
> plan is
> {code:java}
> HashWindowAggregate(window=[TumblingGroupWindow('w, long, 5)], 
> select=[Final_COUNT(count$0) AS EXPR$0])
> +- Exchange(distribution=[single])
>+- LocalHashWindowAggregate(window=[TumblingGroupWindow('w, long, 5)], 
> select=[Partial_COUNT(int) AS count$0])
>   +- TableSourceScan(table=[[default_catalog, default_database, Table1, 
> source: [TestTableSource(long, int, string)]]], fields=[long, int, string])
> {code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13562) throws exception when FlinkRelMdColumnInterval meets two stage stream group aggregate

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13562:
---
Fix Version/s: 1.10.0

> throws exception when FlinkRelMdColumnInterval meets two stage stream group 
> aggregate
> -
>
> Key: FLINK-13562
> URL: https://issues.apache.org/jira/browse/FLINK-13562
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> test case:
> {code:scala}
>   @Test
>   def testTwoDistinctAggregateWithNonDistinctAgg(): Unit = {
> util.addTableSource[(Int, Long, String)]("MyTable", 'a, 'b, 'c)
> util.verifyPlan("SELECT c, SUM(DISTINCT a), SUM(a), COUNT(DISTINCT b) 
> FROM MyTable GROUP BY c")
>   }
> {code}
> org.apache.flink.table.api.TableException: Sum aggregate function does not 
> support type: ''VARCHAR''.
> Please re-check the data type.
>   at 
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createSumAggFunction(AggFunctionFactory.scala:191)
>   at 
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createAggFunction(AggFunctionFactory.scala:74)
>   at 
> org.apache.flink.table.planner.plan.utils.AggregateUtil$$anonfun$9.apply(AggregateUtil.scala:285)
>   at 
> org.apache.flink.table.planner.plan.utils.AggregateUtil$$anonfun$9.apply(AggregateUtil.scala:279)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> org.apache.flink.table.planner.plan.utils.AggregateUtil$.transformToAggregateInfoList(AggregateUtil.scala:279)
>   at 
> org.apache.flink.table.planner.plan.utils.AggregateUtil$.getOutputIndexToAggCallIndexMap(AggregateUtil.scala:154)
>   at 
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdColumnInterval.getAggCallIndexInLocalAgg$1(FlinkRelMdColumnInterval.scala:504)
>   at 
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdColumnInterval.estimateColumnIntervalOfAggregate(FlinkRelMdColumnInterval.scala:526)
>   at 
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdColumnInterval.getColumnInterval(FlinkRelMdColumnInterval.scala:417)
>   at GeneratedMetadataHandler_ColumnInterval.getColumnInterval_$(Unknown 
> Source)
>   at GeneratedMetadataHandler_ColumnInterval.getColumnInterval(Unknown 
> Source)
>   at 
> org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.getColumnInterval(FlinkRelMetadataQuery.java:122)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13637) Anchors not working in document(building.md, common.md, queryable_state.md)

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13637:
---
Fix Version/s: 1.10.0

> Anchors not working in document(building.md, common.md, queryable_state.md)
> ---
>
> Key: FLINK-13637
> URL: https://issues.apache.org/jira/browse/FLINK-13637
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.10.0, 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Anchors not working in document(building.md, common.md, queryable_state.md).
> The format should be:
> {code:java}
> [create an anchor](#anchors-in-markdown)
> {code}
> - Add - characters between each word in the heading and wrap the value in 
> parens
> - All letters should be lowercase.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] flinkbot commented on issue #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error

2019-08-26 Thread GitBox
flinkbot commented on issue #9540: [FLINK-13859][docs] 
JSONDeserializationSchema spell error
URL: https://github.com/apache/flink/pull/9540#issuecomment-525128086
 
 
   
   ## CI report:
   
   * 5cb9691c978214d5f868c0e6a43f782b05b29ff6 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (FLINK-13517) Restructure Hive Catalog documentation

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916323#comment-16916323
 ] 

Kurt Young commented on FLINK-13517:


I've updated the fix version to 1.9.1 since this is not included in 1.9.0 
release.

> Restructure Hive Catalog documentation
> --
>
> Key: FLINK-13517
> URL: https://issues.apache.org/jira/browse/FLINK-13517
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Hive, Documentation
>Reporter: Seth Wiesman
>Assignee: Seth Wiesman
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Hive documentation is currently spread across a number of pages and 
> fragmented. In particular: 
> 1) An example was added to getting-started/examples, however, this section is 
> being removed
> 2) There is a dedicated page on hive integration but also a lot of hive 
> specific information is on the catalog page
> We should
> 1) Inline the example into the hive integration page
> 2) Move the hive specific information on catalogs.md to hive_integration.md
> 3) Make catalogs.md be just about catalogs in general and link to the hive 
> integration. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13637) Anchors not working in document(building.md, common.md, queryable_state.md)

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916320#comment-16916320
 ] 

Kurt Young commented on FLINK-13637:


I've updated the fix version to 1.9.1 since this is not included in 1.9.0 
release.

> Anchors not working in document(building.md, common.md, queryable_state.md)
> ---
>
> Key: FLINK-13637
> URL: https://issues.apache.org/jira/browse/FLINK-13637
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Anchors not working in document(building.md, common.md, queryable_state.md).
> The format should be:
> {code:java}
> [create an anchor](#anchors-in-markdown)
> {code}
> - Add - characters between each word in the heading and wrap the value in 
> parens
> - All letters should be lowercase.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13517) Restructure Hive Catalog documentation

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13517:
---
Fix Version/s: (was: 1.9.0)
   1.9.1

> Restructure Hive Catalog documentation
> --
>
> Key: FLINK-13517
> URL: https://issues.apache.org/jira/browse/FLINK-13517
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Hive, Documentation
>Reporter: Seth Wiesman
>Assignee: Seth Wiesman
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Hive documentation is currently spread across a number of pages and 
> fragmented. In particular: 
> 1) An example was added to getting-started/examples, however, this section is 
> being removed
> 2) There is a dedicated page on hive integration but also a lot of hive 
> specific information is on the catalog page
> We should
> 1) Inline the example into the hive integration page
> 2) Move the hive specific information on catalogs.md to hive_integration.md
> 3) Make catalogs.md be just about catalogs in general and link to the hive 
> integration. 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13526) Switching to a non existing catalog or database crashes sql-client

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13526:
---
Fix Version/s: (was: 1.9.0)
   1.9.1

> Switching to a non existing catalog or database crashes sql-client
> --
>
> Key: FLINK-13526
> URL: https://issues.apache.org/jira/browse/FLINK-13526
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Client
>Affects Versions: 1.9.0
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> sql-client crashes if user tries to switch to a non-existing DB:
> {noformat}
> Exception in thread "main" org.apache.flink.table.client.SqlClientException: 
> Unexpected exception. This is a bug. Please consider filing an issue.
>   at org.apache.flink.table.client.SqlClient.main(SqlClient.java:206)
> Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A 
> database with name [foo] does not exist in the catalog: [myhive].
>   at 
> org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:286)
>   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:398)
>   at 
> org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$useDatabase$5(LocalExecutor.java:258)
>   at 
> org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:216)
>   at 
> org.apache.flink.table.client.gateway.local.LocalExecutor.useDatabase(LocalExecutor.java:256)
>   at 
> org.apache.flink.table.client.cli.CliClient.callUseDatabase(CliClient.java:434)
>   at 
> org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:282)
>   at java.util.Optional.ifPresent(Optional.java:159)
>   at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200)
>   at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:123)
>   at org.apache.flink.table.client.SqlClient.start(SqlClient.java:105)
>   at org.apache.flink.table.client.SqlClient.main(SqlClient.java:194)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-13526) Switching to a non existing catalog or database crashes sql-client

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916319#comment-16916319
 ] 

Kurt Young commented on FLINK-13526:


I've updated the fix version to 1.9.1 since this is not included in 1.9.0 
release.

> Switching to a non existing catalog or database crashes sql-client
> --
>
> Key: FLINK-13526
> URL: https://issues.apache.org/jira/browse/FLINK-13526
> Project: Flink
>  Issue Type: Bug
>  Components: Table SQL / Client
>Affects Versions: 1.9.0
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> sql-client crashes if user tries to switch to a non-existing DB:
> {noformat}
> Exception in thread "main" org.apache.flink.table.client.SqlClientException: 
> Unexpected exception. This is a bug. Please consider filing an issue.
>   at org.apache.flink.table.client.SqlClient.main(SqlClient.java:206)
> Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A 
> database with name [foo] does not exist in the catalog: [myhive].
>   at 
> org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:286)
>   at 
> org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:398)
>   at 
> org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$useDatabase$5(LocalExecutor.java:258)
>   at 
> org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:216)
>   at 
> org.apache.flink.table.client.gateway.local.LocalExecutor.useDatabase(LocalExecutor.java:256)
>   at 
> org.apache.flink.table.client.cli.CliClient.callUseDatabase(CliClient.java:434)
>   at 
> org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:282)
>   at java.util.Optional.ifPresent(Optional.java:159)
>   at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200)
>   at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:123)
>   at org.apache.flink.table.client.SqlClient.start(SqlClient.java:105)
>   at org.apache.flink.table.client.SqlClient.main(SqlClient.java:194)
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Updated] (FLINK-13637) Anchors not working in document(building.md, common.md, queryable_state.md)

2019-08-26 Thread Kurt Young (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kurt Young updated FLINK-13637:
---
Fix Version/s: (was: 1.9.0)
   1.9.1

> Anchors not working in document(building.md, common.md, queryable_state.md)
> ---
>
> Key: FLINK-13637
> URL: https://issues.apache.org/jira/browse/FLINK-13637
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.1
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Anchors not working in document(building.md, common.md, queryable_state.md).
> The format should be:
> {code:java}
> [create an anchor](#anchors-in-markdown)
> {code}
> - Add - characters between each word in the heading and wrap the value in 
> parens
> - All letters should be lowercase.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-10995) Copy intermediate serialization results only once for broadcast mode

2019-08-26 Thread Kurt Young (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916315#comment-16916315
 ] 

Kurt Young commented on FLINK-10995:


Would it possible to open the Jira for writing only one file in BLOCKING 
broadcast shuffle first and start to discuss the design and plan? 

> Copy intermediate serialization results only once for broadcast mode
> 
>
> Key: FLINK-10995
> URL: https://issues.apache.org/jira/browse/FLINK-10995
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Network
>Affects Versions: 1.8.0
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The emitted records from operator would be firstly serialized into 
> intermediate bytes array in {{RecordSerializer}}, then copy the intermediate 
> results into target buffers for different sub partitions.  For broadcast 
> mode, the same intermediate results would be copied as many times as the 
> number of sub partitions, and this would affect the performance seriously in 
> large scale jobs.
> We can copy to only one target buffer which would be shared by all the sub 
> partitions to reduce the overheads. For emitting latency marker in broadcast 
> mode, we should flush the previous shared target buffers first, and then 
> request a new buffer for the target sub partition to send latency marker.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] flinkbot commented on issue #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error

2019-08-26 Thread GitBox
flinkbot commented on issue #9540: [FLINK-13859][docs] 
JSONDeserializationSchema spell error
URL: https://github.com/apache/flink/pull/9540#issuecomment-525126046
 
 
   Thanks a lot for your contribution to the Apache Flink project. I'm the 
@flinkbot. I help the community
   to review your pull request. We will use this comment to track the progress 
of the review.
   
   
   ## Automated Checks
   Last check on commit 5cb9691c978214d5f868c0e6a43f782b05b29ff6 (Tue Aug 27 
03:55:13 UTC 2019)
   
   **Warnings:**
* **This pull request references an unassigned [Jira 
ticket](https://issues.apache.org/jira/browse/FLINK-13859).** According to the 
[code contribution 
guide](https://flink.apache.org/contributing/contribute-code.html), tickets 
need to be assigned before starting with the implementation work.
   
   
   Mention the bot in a comment to re-run the automated checks.
   ## Review Progress
   
   * ❓ 1. The [description] looks good.
   * ❓ 2. There is [consensus] that the contribution should go into to Flink.
   * ❓ 3. Needs [attention] from.
   * ❓ 4. The change fits into the overall [architecture].
   * ❓ 5. Overall code [quality] is good.
   
   Please see the [Pull Request Review 
Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full 
explanation of the review process.
The Bot is tracking the review progress through labels. Labels are applied 
according to the order of the review items. For consensus, approval by a Flink 
committer of PMC member is required Bot commands
 The @flinkbot bot supports the following commands:
   
- `@flinkbot approve description` to approve one or more aspects (aspects: 
`description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until 
`architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's 
attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (FLINK-13859) JSONDeserializationSchema spell error

2019-08-26 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-13859:
---
Labels: pull-request-available  (was: )

> JSONDeserializationSchema spell error
> -
>
> Key: FLINK-13859
> URL: https://issues.apache.org/jira/browse/FLINK-13859
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.9.0
>Reporter: limbo
>Priority: Minor
>  Labels: pull-request-available
> Attachments: image-2019-08-26-20-14-20-075.png
>
>
> In kafka page the JsonDeserializationSchema would be JSONDeserializationSchema
> !image-2019-08-26-20-14-20-075.png!
>  
> [https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html]



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] flinkbot edited a comment on issue #9539: [hotfix][docs] Fix typo

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9539: [hotfix][docs] Fix typo
URL: https://github.com/apache/flink/pull/9539#issuecomment-525098010
 
 
   
   ## CI report:
   
   * 9430aca9b85c3a226cf5d5546e174d57daa8174e : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/124669373)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] taizilongxu opened a new pull request #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error

2019-08-26 Thread GitBox
taizilongxu opened a new pull request #9540: [FLINK-13859][docs] 
JSONDeserializationSchema spell error
URL: https://github.com/apache/flink/pull/9540
 
 
   ##  What is the purpose of the change
   
   The doc spell error
   
   ## Brief change log
   
   JsonDeserializationSchema -> JSONDeserializationSchema
   
   ## Verifying this change
   
   none
   
   ## Does this pull request potentially affect one of the following parts:
   
   none
   
   ## Documentation
   
   none
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (FLINK-13547) Verify and correct string function's semantic for Blink planner

2019-08-26 Thread Kevin Zhang (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916313#comment-16916313
 ] 

Kevin Zhang commented on FLINK-13547:
-

Thanks [~docete] and [~jark], we will have a look at the related issues.

> Verify and correct string function's semantic for Blink planner
> ---
>
> Key: FLINK-13547
> URL: https://issues.apache.org/jira/browse/FLINK-13547
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Zhenghua Gao
>Assignee: Zhenghua Gao
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently many string builtin functions in blink planner follow hive/spark 
> semantics, which should keep compatible with old planner. And some 
> non-standard functions(Blink planner intros) should be removed.
>  * concat/concat_ws function (null treatment)
>  * substring function (follow calcite/flink)
>  * from_base64 should return string not binary
>  * intro truncate function to blink planner
>  * uuid should be no-argument (remove the one-argument version)
>  * length/jsonvalue/keyvalue/substr (non-standard function should be removed)
>  * md5/sha1/sha2/sha224/sha256/sha384/sha512(remove the two-arguments version)
>  * ascii (operand type should beSqlTypeFamily.CHARACTER)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] flinkbot edited a comment on issue #9510: [FLINK-13807][tests] Read file with UTF-8 charset in TestBaseUtils.getResultReader

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9510: [FLINK-13807][tests] Read file with 
UTF-8 charset in TestBaseUtils.getResultReader
URL: https://github.com/apache/flink/pull/9510#issuecomment-523878082
 
 
   
   ## CI report:
   
   * 7e7bc9b213b2fadb8759449b65e5be19b11aa829 : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124206563)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add 
documentation for TopN and Deduplication in blink planner
URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126
 
 
   
   ## CI report:
   
   * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/124206586)
   * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124227220)
   * f1399e04785d2afecacfc1962d68bbf98416f68b : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/124676036)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add 
documentation for TopN and Deduplication in blink planner
URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126
 
 
   
   ## CI report:
   
   * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/124206586)
   * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124227220)
   * f1399e04785d2afecacfc1962d68bbf98416f68b : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] JingsongLi commented on a change in pull request #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner

2019-08-26 Thread GitBox
JingsongLi commented on a change in pull request #9511: 
[FLINK-13356][table][docs] Add documentation for TopN and Deduplication in 
blink planner
URL: https://github.com/apache/flink/pull/9511#discussion_r317879792
 
 

 ##
 File path: docs/dev/table/sql.md
 ##
 @@ -813,6 +813,222 @@ LIMIT 3
 
 {% top %}
 
+### Top-N
+
+TopN is used to calculate the maximum/minimum N records in a stream. It can be 
flexibly completed based on OVER window aggregation. The grammar is shown as 
below:
+
+Top-N queries ask for the N smallest or largest values ordered by columns. 
Both smallest and largest values sets are considered Top-N queries. Top-N 
queries are useful in cases where the need is to display only the N bottom-most 
or the N top-
+most records from batch/stream table on a condition. This result set can be 
used for further analysis.
+
+Flink uses the combination of a OVER window clause and a filter condition to 
express a Top-N query. With the power of OVER window `PARTITION BY` clause, 
Flink also supports per group Top-N. For example, the top five products per 
category that have the maximum sales in realtime. Top-N queries are supported 
for SQL on batch and streaming tables.
+
+The following shows the syntax of the TOP-N statement:
+
+{% highlight sql %}
+SELECT [column_list]
+FROM (
+   SELECT [column_list],
+ ROW_NUMBER() OVER ([PARTITION BY col1[, col2...]]
+   ORDER BY col1 [asc|desc][, col2 [asc|desc]...]) AS rownum
+   FROM table_name)
+WHERE rownum <= N [AND conditions]
+{% endhighlight %}
+
+**Parameter Specification:**
+
+- `ROW_NUMBER()`: Assigns an unique, sequential number to each row, starting 
with one, according to the ordering of rows within the partition. Currently, we 
only support `ROW_NUMBER` as the over window function. In the future, we will 
support `RANK()` and `DENSE_RANK()`.
+- `PARTITION BY col1[, col2...]`: Specifies the partition columns. Each 
partition will have a Top-N result.
+- `ORDER BY col1 [asc|desc][, col2 [asc|desc]...]`: Specifies the ordering 
columns. The ordering directions can be different on different columns.
+- `WHERE rownum <= N`: The `rownum <= N` is required for Flink to recognize 
this query is a Top-N query. The N represents the N smallest or largest records 
will be retained.
+- `[AND conditions]`: It is free to add other conditions in the where clause, 
but the other conditions can only be combined with `rownum <= N` using `AND` 
conjunction.
+
+Attention Flink SQL will sort the 
input data stream according to the order key, so if the top N records have been 
changed, the changed ones will be sent as retraction/update records to 
downstream. In addition, if the top N records need to be stored in external 
storage, the result table must have the same primary key with the Top-N query. 
By default, the primary key of Top-N query is the combination of partition 
columns + rownum column.
+
+The following examples show how to specify SQL queries with Top-N on streaming 
tables. This is an example to get "the top five products per category that have 
the maximum sales in realtime" we mentioned above.
+
+
+
+{% highlight java %}
+StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
+StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env);
+
+// ingest a DataStream from an external source
+DataStream> ds = env.addSource(...);
+// register the DataStream as table "ShopSales"
+tableEnv.registerDataStream("ShopSales", ds, "product_id, category, 
product_name, sales");
+
+// select top-5 products per category which have the maximum sales.
+Table result1 = tableEnv.sqlQuery(
+  "SELECT * " +
+  "FROM (" +
+  "   SELECT *," +
+  "   ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) as 
row_num" +
+  "   FROM ShopSales)" +
+  "WHERE row_num <= 5");
+{% endhighlight %}
+
+
+
+{% highlight scala %}
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+val tableEnv = TableEnvironment.getTableEnvironment(env)
+
+// read a DataStream from an external source
+val ds: DataStream[(String, String, String, Long)] = env.addSource(...)
+// register the DataStream under the name "ShopSales"
+tableEnv.registerDataStream("ShopSales", ds, 'product_id, 'category, 
'product_name, 'sales)
+
+
+// select top-5 products per category which have the maximum sales.
+val result1 = tableEnv.sqlQuery(
+"""
+  |SELECT *
+  |FROM (
+  |   SELECT *,
+  |   ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) as 
row_num
+  |   FROM ShopSales)
+  |WHERE row_num <= 5
+""".stripMargin)
+{% endhighlight %}
+
+
+
+ No Ranking Output Optimization
+
+As described above, the `rownum` field will be written into the result table 
as one field of the primary key, which may lead to a lot of records being 
written to the result table. For example, when the record (say `product-1001`) 
of ranking 9 is updated and its rank is upgraded to 1, all the records from 

[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce 
TwoInputSelectionHandler for selecting input in 
StreamTwoInputSelectableProcessor
URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024
 
 
   
   ## CI report:
   
   * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123648363)
   * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123684855)
   * 7d75f94fed9684c020c7673a57431c79e2f0a543 : CANCELED 
[Build](https://travis-ci.com/flink-ci/flink/builds/124674615)
   * e21b3e5b91328bd7e9ce28fb8df9b848730b319a : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/124675311)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce 
TwoInputSelectionHandler for selecting input in 
StreamTwoInputSelectableProcessor
URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024
 
 
   
   ## CI report:
   
   * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123648363)
   * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123684855)
   * 7d75f94fed9684c020c7673a57431c79e2f0a543 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/124674615)
   * e21b3e5b91328bd7e9ce28fb8df9b848730b319a : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] xuefuz commented on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line…

2019-08-26 Thread GitBox
xuefuz commented on issue #8738: [FLINK-12845][sql-client] Execute multiple 
statements in command line…
URL: https://github.com/apache/flink/pull/8738#issuecomment-525118500
 
 
   As a reference, Hive has been doing, which seems quite simple. 
https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L384


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] flinkbot edited a comment on issue #9538: [hotfix][docs] Fix typo

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9538: [hotfix][docs] Fix typo
URL: https://github.com/apache/flink/pull/9538#issuecomment-525097982
 
 
   
   ## CI report:
   
   * 07c04045c445ccd5d844ffc733d01ac7adbf88ee : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/124669351)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] zhijiangW commented on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread GitBox
zhijiangW commented on issue #9477: [FLINK-13765][task] Introduce 
TwoInputSelectionHandler for selecting input in 
StreamTwoInputSelectableProcessor
URL: https://github.com/apache/flink/pull/9477#issuecomment-525116894
 
 
   Thanks for the confirmation and I would merge it after travis green. 
@pnowojski 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (FLINK-13765) Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread zhijiang (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhijiang updated FLINK-13765:
-
Description: 
In StreamTwoInputSelectableProcessor there are three fields \{InputSelectable, 
InputSelection, availableInputsMask} to be used together for the function of 
selecting next available input index.

>From design aspect, these fields can be abstracted into a separate component 
>called TwoInputSelectionHandler, which is  passed into the constructor of 
>StreamTwoInputSelectableProcessor as a final field. So the internal 
>implementation details of TwoInputSelectionHandler is hidden from processor 
>view which only needs to interact with exposed methods from selection handler.

Another tiny benefit is that we make the StreamTwoInputSelectableProcessor a 
bit because two methods are removed from it.

  was:
In StreamTwoInputSelectableProcessor there are three fields \{InputSelectable, 
InputSelection, availableInputsMask} to be used together for the function of 
selecting next available input index.

>From design aspect, these fields can be abstracted into a separate component 
>called TwoInputSelectionHandler, which is  passed into the constructor of 
>StreamTwoInputSelectableProcessor as a final field. So the internal 
>implementation details of TwoInputSelectionHandler is hidden from processor 
>view which only needs to interact with exposed methods from selection handler.

Another tiny benefits is that we make the StreamTwoInputSelectableProcessor a 
bit because two methods are removed from it.


> Introduce TwoInputSelectionHandler for selecting input in 
> StreamTwoInputSelectableProcessor
> ---
>
> Key: FLINK-13765
> URL: https://issues.apache.org/jira/browse/FLINK-13765
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Task
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In StreamTwoInputSelectableProcessor there are three fields 
> \{InputSelectable, InputSelection, availableInputsMask} to be used together 
> for the function of selecting next available input index.
> From design aspect, these fields can be abstracted into a separate component 
> called TwoInputSelectionHandler, which is  passed into the constructor of 
> StreamTwoInputSelectableProcessor as a final field. So the internal 
> implementation details of TwoInputSelectionHandler is hidden from processor 
> view which only needs to interact with exposed methods from selection handler.
> Another tiny benefit is that we make the StreamTwoInputSelectableProcessor a 
> bit because two methods are removed from it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce 
TwoInputSelectionHandler for selecting input in 
StreamTwoInputSelectableProcessor
URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024
 
 
   
   ## CI report:
   
   * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123648363)
   * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123684855)
   * 7d75f94fed9684c020c7673a57431c79e2f0a543 : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (FLINK-13765) Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread zhijiang (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhijiang updated FLINK-13765:
-
Description: 
In StreamTwoInputSelectableProcessor there are three fields \{InputSelectable, 
InputSelection, availableInputsMask} to be used together for the function of 
selecting next available input index.

>From design aspect, these fields can be abstracted into a separate component 
>called TwoInputSelectionHandler, which is  passed into the constructor of 
>StreamTwoInputSelectableProcessor as a final field. So the internal 
>implementation details of TwoInputSelectionHandler is hidden from processor 
>view which only needs to interact with exposed methods from selection handler.

Another tiny benefits is that we make the StreamTwoInputSelectableProcessor a 
bit because two methods are removed from it.

  was:
In StreamTwoInputSelectableProcessor there are three fields \{InputSelectable, 
InputSelection, availableInputsMask} to be used together for the function of 
selecting next available input index. It would bring two problems:
 * From design aspect, these fields should be abstracted into a separate 
component and passed into StreamTwoInputSelectableProcessor.
 * inputSelector.nextSelection() is called while processing elements in  
StreamTwoInputSelectableProcessor, so it is the blocker for integrating task 
input/output for both StreamOneInputProcessor/StreamTwoInputSelectableProcessor 
later.


> Introduce TwoInputSelectionHandler for selecting input in 
> StreamTwoInputSelectableProcessor
> ---
>
> Key: FLINK-13765
> URL: https://issues.apache.org/jira/browse/FLINK-13765
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Task
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In StreamTwoInputSelectableProcessor there are three fields 
> \{InputSelectable, InputSelection, availableInputsMask} to be used together 
> for the function of selecting next available input index.
> From design aspect, these fields can be abstracted into a separate component 
> called TwoInputSelectionHandler, which is  passed into the constructor of 
> StreamTwoInputSelectableProcessor as a final field. So the internal 
> implementation details of TwoInputSelectionHandler is hidden from processor 
> view which only needs to interact with exposed methods from selection handler.
> Another tiny benefits is that we make the StreamTwoInputSelectableProcessor a 
> bit because two methods are removed from it.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in 
ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#issuecomment-522053886
 
 
   
   ## CI report:
   
   * 009da497d7c551ba854dc7ed8fa658f2acd6d6ce : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123529451)
   * 14be3c57f3d625f3f9bbb96b7f74b6b0116fed3c : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124588576)
   * c43a462c92c6c5c145a6964aa5390a82b894fa4e : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/124673290)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (FLINK-13765) Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread zhijiang (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhijiang updated FLINK-13765:
-
Summary: Introduce TwoInputSelectionHandler for selecting input in 
StreamTwoInputSelectableProcessor  (was: Introduce the TwoInputSelectionHandler 
for selecting input in StreamTwoInputSelectableProcessor)

> Introduce TwoInputSelectionHandler for selecting input in 
> StreamTwoInputSelectableProcessor
> ---
>
> Key: FLINK-13765
> URL: https://issues.apache.org/jira/browse/FLINK-13765
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Task
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In StreamTwoInputSelectableProcessor there are three fields 
> \{InputSelectable, InputSelection, availableInputsMask} to be used together 
> for the function of selecting next available input index. It would bring two 
> problems:
>  * From design aspect, these fields should be abstracted into a separate 
> component and passed into StreamTwoInputSelectableProcessor.
>  * inputSelector.nextSelection() is called while processing elements in  
> StreamTwoInputSelectableProcessor, so it is the blocker for integrating task 
> input/output for both 
> StreamOneInputProcessor/StreamTwoInputSelectableProcessor later.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] Lemonjing commented on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
Lemonjing commented on issue #9467: [FLINK-9941][ScalaAPI] Flush in 
ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#issuecomment-525112986
 
 
   @flinkbot run travis


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Updated] (FLINK-13765) Introduce the TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor

2019-08-26 Thread zhijiang (Jira)


 [ 
https://issues.apache.org/jira/browse/FLINK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

zhijiang updated FLINK-13765:
-
Summary: Introduce the TwoInputSelectionHandler for selecting input in 
StreamTwoInputSelectableProcessor  (was: Introduce the InputSelectionHandler 
for selecting next input in StreamTwoInputSelectableProcessor)

> Introduce the TwoInputSelectionHandler for selecting input in 
> StreamTwoInputSelectableProcessor
> ---
>
> Key: FLINK-13765
> URL: https://issues.apache.org/jira/browse/FLINK-13765
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Task
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> In StreamTwoInputSelectableProcessor there are three fields 
> \{InputSelectable, InputSelection, availableInputsMask} to be used together 
> for the function of selecting next available input index. It would bring two 
> problems:
>  * From design aspect, these fields should be abstracted into a separate 
> component and passed into StreamTwoInputSelectableProcessor.
>  * inputSelector.nextSelection() is called while processing elements in  
> StreamTwoInputSelectableProcessor, so it is the blocker for integrating task 
> input/output for both 
> StreamOneInputProcessor/StreamTwoInputSelectableProcessor later.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in 
ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#issuecomment-522053886
 
 
   
   ## CI report:
   
   * 009da497d7c551ba854dc7ed8fa658f2acd6d6ce : SUCCESS 
[Build](https://travis-ci.com/flink-ci/flink/builds/123529451)
   * 14be3c57f3d625f3f9bbb96b7f74b6b0116fed3c : FAILURE 
[Build](https://travis-ci.com/flink-ci/flink/builds/124588576)
   * c43a462c92c6c5c145a6964aa5390a82b894fa4e : UNKNOWN
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] 
Flush in ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#discussion_r317871779
 
 

 ##
 File path: 
flink-scala/src/test/java/org/apache/flink/api/scala/operators/ScalaCsvOutputFormatTest.java
 ##
 @@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.operators;
+
+import org.apache.flink.api.common.io.FileOutputFormat;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.List;
+
+import scala.Tuple3;
+
+import static org.junit.Assert.fail;
+
+/**
+ * Tests for {@link ScalaCsvOutputFormat}.
+ */
+public class ScalaCsvOutputFormatTest {
+
+   private String path = null;
+
+   @Before
+   public void createFile() throws Exception {
+   path = File.createTempFile("scala_csv_output_test_file", 
".csv").getAbsolutePath();
+   }
+
+   @Test
+   public void testNullAllow() throws Exception {
+   final ScalaCsvOutputFormat> 
csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path));
+   try {
+   
csvOutputFormat.setWriteMode(FileSystem.WriteMode.OVERWRITE);
+   
csvOutputFormat.setOutputDirectoryMode(FileOutputFormat.OutputDirectoryMode.PARONLY);
+   csvOutputFormat.setAllowNullValues(true);
+   csvOutputFormat.open(0, 1);
+   csvOutputFormat.writeRecord(new Tuple3<>("One", null, 
8));
+   } finally {
+   csvOutputFormat.close();
+   }
+   java.nio.file.Path p = Paths.get(path);
+   Assert.assertTrue(Files.exists(p));
+   List lines = Files.readAllLines(Paths.get(path), 
StandardCharsets.UTF_8);
+   Assert.assertEquals(1, lines.size());
+   Assert.assertEquals("One,,8", lines.get(0));
+   }
+
+   @Test
+   public void testNullDisallowOnDefault() throws Exception {
+   final ScalaCsvOutputFormat> 
csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path));
 
 Review comment:
   Thanks for the review @StephanEwen . I updated the codes for addressing the 
comments.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] 
Flush in ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#discussion_r317868102
 
 

 ##
 File path: 
flink-scala/src/test/java/org/apache/flink/api/scala/operators/ScalaCsvOutputFormatTest.java
 ##
 @@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.operators;
+
+import org.apache.flink.api.common.io.FileOutputFormat;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.List;
+
+import scala.Tuple3;
+
+import static org.junit.Assert.fail;
+
+/**
+ * Tests for {@link ScalaCsvOutputFormat}.
+ */
+public class ScalaCsvOutputFormatTest {
+
+   private String path = null;
+
+   @Before
+   public void createFile() throws Exception {
+   path = File.createTempFile("scala_csv_output_test_file", 
".csv").getAbsolutePath();
+   }
+
+   @Test
+   public void testNullAllow() throws Exception {
+   final ScalaCsvOutputFormat> 
csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path));
+   try {
+   
csvOutputFormat.setWriteMode(FileSystem.WriteMode.OVERWRITE);
+   
csvOutputFormat.setOutputDirectoryMode(FileOutputFormat.OutputDirectoryMode.PARONLY);
+   csvOutputFormat.setAllowNullValues(true);
+   csvOutputFormat.open(0, 1);
+   csvOutputFormat.writeRecord(new Tuple3<>("One", null, 
8));
+   } finally {
+   csvOutputFormat.close();
+   }
+   java.nio.file.Path p = Paths.get(path);
+   Assert.assertTrue(Files.exists(p));
+   List lines = Files.readAllLines(Paths.get(path), 
StandardCharsets.UTF_8);
+   Assert.assertEquals(1, lines.size());
+   Assert.assertEquals("One,,8", lines.get(0));
+   }
+
+   @Test
+   public void testNullDisallowOnDefault() throws Exception {
+   final ScalaCsvOutputFormat> 
csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path));
 
 Review comment:
   > Minor: I would recommend to put these four lines (creation and 
configuration of format) into a setup helper method.
   
   yes, thanks for your review, update already.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] Lemonjing removed a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
Lemonjing removed a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in 
ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#issuecomment-522054960
 
 
   @flinkbot attention @yanghua @buptljy 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (FLINK-10995) Copy intermediate serialization results only once for broadcast mode

2019-08-26 Thread zhijiang (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916286#comment-16916286
 ] 

zhijiang commented on FLINK-10995:
--

[~ykt836], this Jira is for improving the serialization stack in RecordWriter, 
which could bring benefit for both PIPELINED and BLOCKING partitions at the 
moment. In detail the intermediate serialization data buffer would need copy 
only once for all the subpartitions.  But for the blocking subpartitions, the 
same referenced target buffer would still be persisted into separate file as 
you mentioned. In theory it can be persisted into only one file for all the 
subpartitions, but it is not in the scope of this jira. Further the same data 
could be transported only once in the network stack if many consumers are in 
the same TaskManager. We might further focus on these improvements future, but 
might not in release-1.10. But this Jira would be covered in release-1.10.

> Copy intermediate serialization results only once for broadcast mode
> 
>
> Key: FLINK-10995
> URL: https://issues.apache.org/jira/browse/FLINK-10995
> Project: Flink
>  Issue Type: Sub-task
>  Components: Runtime / Network
>Affects Versions: 1.8.0
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The emitted records from operator would be firstly serialized into 
> intermediate bytes array in {{RecordSerializer}}, then copy the intermediate 
> results into target buffers for different sub partitions.  For broadcast 
> mode, the same intermediate results would be copied as many times as the 
> number of sub partitions, and this would affect the performance seriously in 
> large scale jobs.
> We can copy to only one target buffer which would be shared by all the sub 
> partitions to reduce the overheads. For emitting latency marker in broadcast 
> mode, we should flush the previous shared target buffers first, and then 
> request a new buffer for the target sub partition to send latency marker.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[jira] [Commented] (FLINK-9477) Support SQL 2016 JSON functions in Flink SQL

2019-08-26 Thread Jark Wu (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916283#comment-16916283
 ] 

Jark Wu commented on FLINK-9477:


Hi [~x1q1j1], it would be great if you can summarize the features/functions 
that we actually want to support first in a little design document and how to 
integrate with Flink (especially Table API). Because you are the contributor of 
JSON functions in Calcite, maybe you know better about the functions.

Once we have the design and is accepted by the community, then we can create 
subtasks to cooperate together.

> Support SQL 2016 JSON functions in Flink SQL
> 
>
> Key: FLINK-9477
> URL: https://issues.apache.org/jira/browse/FLINK-9477
> Project: Flink
>  Issue Type: New Feature
>  Components: Table SQL / API
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] Lemonjing removed a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
Lemonjing removed a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in 
ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#issuecomment-525108063
 
 
   @flinkbot run travis


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] Lemonjing commented on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
Lemonjing commented on issue #9467: [FLINK-9941][ScalaAPI] Flush in 
ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#issuecomment-525108063
 
 
   @flinkbot run travis


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[jira] [Commented] (FLINK-13862) Remove or rewrite Execution Plan docs

2019-08-26 Thread TisonKun (Jira)


[ 
https://issues.apache.org/jira/browse/FLINK-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916282#comment-16916282
 ] 

TisonKun commented on FLINK-13862:
--

After a close look I would prefer removing it at least for now. Because it 
mainly talk about how to use the removed plan visualizer. For {{Web Interface}} 
part, it only talk about {{web.submit.enable}} which is also documented at 
[this 
page|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html].

We can anyway add back a document on how to preview a plan depend on our 
discussion about client api. It should cover this topic.

> Remove or rewrite Execution Plan docs
> -
>
> Key: FLINK-13862
> URL: https://issues.apache.org/jira/browse/FLINK-13862
> Project: Flink
>  Issue Type: Bug
>  Components: Documentation
>Affects Versions: 1.9.0
>Reporter: Stephan Ewen
>Priority: Blocker
> Fix For: 1.10.0, 1.9.1
>
>
> The *Execution Plans* section is totally outdated and refers to the old 
> {{tools/planVisalizer.html}} file that has been removed for two years.
> https://ci.apache.org/projects/flink/flink-docs-master/dev/execution_plans.html



--
This message was sent by Atlassian Jira
(v8.3.2#803003)


[GitHub] [flink] Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] 
Flush in ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#discussion_r317868102
 
 

 ##
 File path: 
flink-scala/src/test/java/org/apache/flink/api/scala/operators/ScalaCsvOutputFormatTest.java
 ##
 @@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.operators;
+
+import org.apache.flink.api.common.io.FileOutputFormat;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.List;
+
+import scala.Tuple3;
+
+import static org.junit.Assert.fail;
+
+/**
+ * Tests for {@link ScalaCsvOutputFormat}.
+ */
+public class ScalaCsvOutputFormatTest {
+
+   private String path = null;
+
+   @Before
+   public void createFile() throws Exception {
+   path = File.createTempFile("scala_csv_output_test_file", 
".csv").getAbsolutePath();
+   }
+
+   @Test
+   public void testNullAllow() throws Exception {
+   final ScalaCsvOutputFormat> 
csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path));
+   try {
+   
csvOutputFormat.setWriteMode(FileSystem.WriteMode.OVERWRITE);
+   
csvOutputFormat.setOutputDirectoryMode(FileOutputFormat.OutputDirectoryMode.PARONLY);
+   csvOutputFormat.setAllowNullValues(true);
+   csvOutputFormat.open(0, 1);
+   csvOutputFormat.writeRecord(new Tuple3<>("One", null, 
8));
+   } finally {
+   csvOutputFormat.close();
+   }
+   java.nio.file.Path p = Paths.get(path);
+   Assert.assertTrue(Files.exists(p));
+   List lines = Files.readAllLines(Paths.get(path), 
StandardCharsets.UTF_8);
+   Assert.assertEquals(1, lines.size());
+   Assert.assertEquals("One,,8", lines.get(0));
+   }
+
+   @Test
+   public void testNullDisallowOnDefault() throws Exception {
+   final ScalaCsvOutputFormat> 
csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path));
 
 Review comment:
   > Minor: I would recommend to put these four lines (creation and 
configuration of format) into a setup helper method.
   
   yes, thanks for your review, update already.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close

2019-08-26 Thread GitBox
Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] 
Flush in ScalaCsvOutputFormat before close
URL: https://github.com/apache/flink/pull/9467#discussion_r317623406
 
 

 ##
 File path: 
flink-scala/src/test/java/org/apache/flink/api/scala/operators/ScalaCsvOutputFormatTest.java
 ##
 @@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.operators;
+
+import org.apache.flink.api.common.io.FileOutputFormat;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.List;
+
+import scala.Tuple3;
+
+import static org.junit.Assert.fail;
+
+/**
+ * Tests for {@link ScalaCsvOutputFormat}.
+ */
+public class ScalaCsvOutputFormatTest {
+
+   private String path = null;
+
+   @Before
+   public void createFile() throws Exception {
+   path = File.createTempFile("scala_csv_output_test_file", 
".csv").getAbsolutePath();
+   }
+
+   @Test
+   public void testNullAllow() throws Exception {
+   final ScalaCsvOutputFormat> 
csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path));
+   try {
+   
csvOutputFormat.setWriteMode(FileSystem.WriteMode.OVERWRITE);
+   
csvOutputFormat.setOutputDirectoryMode(FileOutputFormat.OutputDirectoryMode.PARONLY);
+   csvOutputFormat.setAllowNullValues(true);
+   csvOutputFormat.open(0, 1);
+   csvOutputFormat.writeRecord(new Tuple3<>("One", null, 
8));
+   } finally {
+   csvOutputFormat.close();
+   }
+   java.nio.file.Path p = Paths.get(path);
+   Assert.assertTrue(Files.exists(p));
+   List lines = Files.readAllLines(Paths.get(path), 
StandardCharsets.UTF_8);
+   Assert.assertEquals(1, lines.size());
+   Assert.assertEquals("One,,8", lines.get(0));
+   }
+
+   @Test
+   public void testNullDisallowOnDefault() throws Exception {
+   final ScalaCsvOutputFormat> 
csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path));
 
 Review comment:
   thanks for your review, update already.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] wuchong commented on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line…

2019-08-26 Thread GitBox
wuchong commented on issue #8738: [FLINK-12845][sql-client] Execute multiple 
statements in command line…
URL: https://github.com/apache/flink/pull/8738#issuecomment-525106464
 
 
   > I agree the current PR is way too hacky. However, I don't see the need of 
extending SQL parser just to support the multi-command line either as there can 
be commands that's not SQL at all. As far as I can see, we need to split the 
command line by ';' (certainly take care of possible escaped ';'), pre-process 
each segment to determine the command type (which we are already doing), and 
invoke sql parser only if the command is a sql command.
   
   The problem is how to split command by semi-colon. Using regex is a hack way 
and we have met many issues because of it in the internal blink.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [flink] flinkbot edited a comment on issue #9510: [FLINK-13807][tests] Read file with UTF-8 charset in TestBaseUtils.getResultReader

2019-08-26 Thread GitBox
flinkbot edited a comment on issue #9510: [FLINK-13807][tests] Read file with 
UTF-8 charset in TestBaseUtils.getResultReader
URL: https://github.com/apache/flink/pull/9510#issuecomment-523878082
 
 
   
   ## CI report:
   
   * 7e7bc9b213b2fadb8759449b65e5be19b11aa829 : PENDING 
[Build](https://travis-ci.com/flink-ci/flink/builds/124206563)
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


  1   2   3   4   5   >