[jira] [Updated] (FLINK-13865) Support custom config in Flink docker image
[ https://issues.apache.org/jira/browse/FLINK-13865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated FLINK-13865: --- Component/s: (was: Deployment / Kubernetes) Deployment / Docker Environment: I noticed FLINK-13380 which is helpful for Kubernetes, but it is still useful to support it at Docker image level. > Support custom config in Flink docker image > --- > > Key: FLINK-13865 > URL: https://issues.apache.org/jira/browse/FLINK-13865 > Project: Flink > Issue Type: Improvement > Components: Deployment / Docker >Affects Versions: 1.6.3, 1.6.4, 1.7.2, 1.8.0, 1.8.1, 1.9.0 > Environment: I noticed FLINK-13380 which is helpful for Kubernetes, > but it is still useful to support it at Docker image level. >Reporter: Dagang Wei >Priority: Minor > > [Flink docker images](https://github.com/docker-flink/docker-flink) do not > support custom config, in order to do so, the user has to build their own > image based on the official image and modify config/flink-conf.yaml. It would > be much easier if the image accepts custom config through args and in > [docker-entrypoint.sh](https://github.com/docker-flink/docker-flink/blob/master/docker-entrypoint.sh) > it automatically merge the custom config into config/flink-conf.yaml. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-10995) Copy intermediate serialization results only once for broadcast mode
[ https://issues.apache.org/jira/browse/FLINK-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916401#comment-16916401 ] zhijiang commented on FLINK-10995: -- I think it was not planned in release-1.10 before. We could discuss this design and plan for release-1.11 at some time. > Copy intermediate serialization results only once for broadcast mode > > > Key: FLINK-10995 > URL: https://issues.apache.org/jira/browse/FLINK-10995 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.8.0 >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The emitted records from operator would be firstly serialized into > intermediate bytes array in {{RecordSerializer}}, then copy the intermediate > results into target buffers for different sub partitions. For broadcast > mode, the same intermediate results would be copied as many times as the > number of sub partitions, and this would affect the performance seriously in > large scale jobs. > We can copy to only one target buffer which would be shared by all the sub > partitions to reduce the overheads. For emitting latency marker in broadcast > mode, we should flush the previous shared target buffers first, and then > request a new buffer for the target sub partition to send latency marker. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13865) Support custom config in Flink docker image
[ https://issues.apache.org/jira/browse/FLINK-13865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916402#comment-16916402 ] Dagang Wei commented on FLINK-13865: I noticed FLINK-13380 which is helpful for Kubernetes, but it is still useful to support it at Docker image level. > Support custom config in Flink docker image > --- > > Key: FLINK-13865 > URL: https://issues.apache.org/jira/browse/FLINK-13865 > Project: Flink > Issue Type: Improvement > Components: Deployment / Docker >Affects Versions: 1.6.3, 1.6.4, 1.7.2, 1.8.0, 1.8.1, 1.9.0 >Reporter: Dagang Wei >Priority: Minor > > [Flink docker images](https://github.com/docker-flink/docker-flink) do not > support custom config, in order to do so, the user has to build their own > image based on the official image and modify config/flink-conf.yaml. It would > be much easier if the image accepts custom config through args and in > [docker-entrypoint.sh](https://github.com/docker-flink/docker-flink/blob/master/docker-entrypoint.sh) > it automatically merge the custom config into config/flink-conf.yaml. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024 ## CI report: * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123648363) * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123684855) * 7d75f94fed9684c020c7673a57431c79e2f0a543 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124674615) * e21b3e5b91328bd7e9ce28fb8df9b848730b319a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124675311) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13865) Support custom config in Flink docker image
[ https://issues.apache.org/jira/browse/FLINK-13865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dagang Wei updated FLINK-13865: --- Environment: (was: I noticed FLINK-13380 which is helpful for Kubernetes, but it is still useful to support it at Docker image level.) > Support custom config in Flink docker image > --- > > Key: FLINK-13865 > URL: https://issues.apache.org/jira/browse/FLINK-13865 > Project: Flink > Issue Type: Improvement > Components: Deployment / Docker >Affects Versions: 1.6.3, 1.6.4, 1.7.2, 1.8.0, 1.8.1, 1.9.0 >Reporter: Dagang Wei >Priority: Minor > > [Flink docker images](https://github.com/docker-flink/docker-flink) do not > support custom config, in order to do so, the user has to build their own > image based on the official image and modify config/flink-conf.yaml. It would > be much easier if the image accepts custom config through args and in > [docker-entrypoint.sh](https://github.com/docker-flink/docker-flink/blob/master/docker-entrypoint.sh) > it automatically merge the custom config into config/flink-conf.yaml. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] zhijiangW commented on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
zhijiangW commented on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor URL: https://github.com/apache/flink/pull/9477#issuecomment-525149250 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError
[ https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13488: --- Fix Version/s: (was: 1.9.1) (was: 1.10.0) 1.9.0 > flink-python fails to build on Travis due to PackagesNotFoundError > -- > > Key: FLINK-13488 > URL: https://issues.apache.org/jira/browse/FLINK-13488 > Project: Flink > Issue Type: Bug > Components: API / Python, Test Infrastructure >Reporter: Tzu-Li (Gordon) Tai >Assignee: sunjincheng >Priority: Blocker > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 1h > Remaining Estimate: 0h > > https://api.travis-ci.org/v3/job/564925115/log.txt > {code} > install conda ... [SUCCESS] > install miniconda... [SUCCESS] > installing python environment... > installing python2.7... > install python2.7... [SUCCESS] > installing python3.3... > PackagesNotFoundError: The following packages are not available from current > channels: > - python=3.3 > Current channels: > - https://repo.anaconda.com/pkgs/main/linux-64 > - https://repo.anaconda.com/pkgs/main/noarch > - https://repo.anaconda.com/pkgs/r/linux-64 > - https://repo.anaconda.com/pkgs/r/noarch > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError
[ https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916399#comment-16916399 ] Kurt Young commented on FLINK-13488: Yeah, you're right. I will revert the change, thanks. > flink-python fails to build on Travis due to PackagesNotFoundError > -- > > Key: FLINK-13488 > URL: https://issues.apache.org/jira/browse/FLINK-13488 > Project: Flink > Issue Type: Bug > Components: API / Python, Test Infrastructure >Reporter: Tzu-Li (Gordon) Tai >Assignee: sunjincheng >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 1h > Remaining Estimate: 0h > > https://api.travis-ci.org/v3/job/564925115/log.txt > {code} > install conda ... [SUCCESS] > install miniconda... [SUCCESS] > installing python environment... > installing python2.7... > install python2.7... [SUCCESS] > installing python3.3... > PackagesNotFoundError: The following packages are not available from current > channels: > - python=3.3 > Current channels: > - https://repo.anaconda.com/pkgs/main/linux-64 > - https://repo.anaconda.com/pkgs/main/noarch > - https://repo.anaconda.com/pkgs/r/linux-64 > - https://repo.anaconda.com/pkgs/r/noarch > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] dianfu commented on issue #9496: [FLINK-13011][build] Add the Build logic for Python API release package
dianfu commented on issue #9496: [FLINK-13011][build] Add the Build logic for Python API release package URL: https://github.com/apache/flink/pull/9496#issuecomment-525145865 @sunjincheng121 Thanks a lot for the PR. LGTM. +1 from my side. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126 ## CI report: * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124206586) * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124227220) * f1399e04785d2afecacfc1962d68bbf98416f68b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124676036) * 0e1c29aa3ddb1f602b4f63271382ebf7171cca50 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124683423) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126 ## CI report: * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124206586) * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124227220) * f1399e04785d2afecacfc1962d68bbf98416f68b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124676036) * 0e1c29aa3ddb1f602b4f63271382ebf7171cca50 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126 ## CI report: * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124206586) * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124227220) * f1399e04785d2afecacfc1962d68bbf98416f68b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124676036) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend
carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend URL: https://github.com/apache/flink/pull/9501#discussion_r317895384 ## File path: flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/ByteBufferUtils.java ## @@ -0,0 +1,332 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.state.heap; + +import javax.annotation.Nonnull; + +import java.lang.reflect.Field; +import java.nio.ByteBuffer; + +/** + * Utilities to get/put data to {@link ByteBuffer}. All methods don't change + * byte buffer's position. + */ +@SuppressWarnings({"WeakerAccess", "unused", "UnusedReturnValue"}) Review comment: Will move this class to `org.apache.flink.core.memory` in flink-core, then we don't need the `WeakerAccess` warning suppress anymore. The current "unused" method might be used latter since this is only part of the whole work, will add test case to cover it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend
carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend URL: https://github.com/apache/flink/pull/9501#discussion_r317895515 ## File path: flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/ByteBufferUtils.java ## @@ -0,0 +1,332 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.state.heap; + +import javax.annotation.Nonnull; + +import java.lang.reflect.Field; +import java.nio.ByteBuffer; + +/** + * Utilities to get/put data to {@link ByteBuffer}. All methods don't change + * byte buffer's position. + */ +@SuppressWarnings({"WeakerAccess", "unused", "UnusedReturnValue"}) +public class ByteBufferUtils { + + private static final boolean UNSAFE_AVAIL = UnsafeHelp.isAvailable(); + private static final boolean UNSAFE_UNALIGNED = UnsafeHelp.unaligned(); + private static final Field ACCESS_FIELD; + + static { + try { + ACCESS_FIELD = java.nio.Buffer.class.getDeclaredField("address"); + ACCESS_FIELD.setAccessible(true); + } catch (NoSuchFieldException e) { + throw new RuntimeException("Failed to get address method from java.nio.Buffer", e); + } + } + + /** +* Reads an int value at the given buffer's offset. +* +* @param buffer the given buffer +* @param offset the given buffer's offset +* @return int value at offset +*/ + public static int toInt(ByteBuffer buffer, int offset) { + if (UNSAFE_UNALIGNED) { + return UnsafeHelp.toInt(buffer, offset); + } else { + return buffer.getInt(offset); + } + } + + /** +* Reads a long value at the given buffer's offset. +* +* @param buffer the given buffer +* @param offset the given buffer's offset +* @return long value at offset +*/ + public static long toLong(ByteBuffer buffer, int offset) { + if (UNSAFE_UNALIGNED) { + return UnsafeHelp.toLong(buffer, offset); + } else { + return buffer.getLong(offset); + } + } + + /** +* Reads a short value at the given buffer's offset. +* +* @param buffer the given buffer +* @param offset the given buffer's offset +* @return short value at offset +*/ + public static short toShort(ByteBuffer buffer, int offset) { + if (UNSAFE_UNALIGNED) { + return UnsafeHelp.toShort(buffer, offset); + } else { + return buffer.getShort(offset); + } + } + + public static byte toByte(ByteBuffer buffer, int offset) { + if (UnsafeHelp.isAvailable()) { + return UnsafeHelp.toByte(buffer, offset); + } else { + return buffer.get(offset); + } + } + + public static void putInt(ByteBuffer buffer, int index, int val) { + if (UNSAFE_UNALIGNED) { + UnsafeHelp.putInt(buffer, index, val); + } else { + buffer.putInt(index, val); + } + } + + public static void putLong(ByteBuffer buffer, int index, long val) { + if (UNSAFE_UNALIGNED) { + UnsafeHelp.putLong(buffer, index, val); + } else { + buffer.putLong(index, val); + } + } + + /** +* Copy from one buffer to another from given offset. This will be absolute positional copying and +* won't affect the position of any of the buffers. +* +* @param inthe given buffer to read +* @param out the given buffer of destination +* @param sourceOffset the given buffer's offset of src +* @param destinationOffset the
[jira] [Comment Edited] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError
[ https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916370#comment-16916370 ] Dian Fu edited comment on FLINK-13488 at 8/27/19 5:01 AM: -- Hi [~ykt836], I guess this fix is actually contained in [1.9.0|https://github.com/apache/flink/commits/release-1.9.0?after=9c32ed989c0178a2bf3e059e897927c451188700+190]. Pls correct me if I missed something:) Do you mean the PR [https://github.com/apache/flink/pull/9431] ? I guess it refers a wrong Jira number and linked to this Jira by mistake. [~gjy] was (Author: dian.fu): Hi [~ykt836], I guess this fix is actually contained in [1.9.0|https://github.com/apache/flink/commits/release-1.9.0?after=9c32ed989c0178a2bf3e059e897927c451188700+190]. Pls correct me if I missed something:) > flink-python fails to build on Travis due to PackagesNotFoundError > -- > > Key: FLINK-13488 > URL: https://issues.apache.org/jira/browse/FLINK-13488 > Project: Flink > Issue Type: Bug > Components: API / Python, Test Infrastructure >Reporter: Tzu-Li (Gordon) Tai >Assignee: sunjincheng >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 1h > Remaining Estimate: 0h > > https://api.travis-ci.org/v3/job/564925115/log.txt > {code} > install conda ... [SUCCESS] > install miniconda... [SUCCESS] > installing python environment... > installing python2.7... > install python2.7... [SUCCESS] > installing python3.3... > PackagesNotFoundError: The following packages are not available from current > channels: > - python=3.3 > Current channels: > - https://repo.anaconda.com/pkgs/main/linux-64 > - https://repo.anaconda.com/pkgs/main/noarch > - https://repo.anaconda.com/pkgs/r/linux-64 > - https://repo.anaconda.com/pkgs/r/noarch > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError
[ https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916370#comment-16916370 ] Dian Fu commented on FLINK-13488: - Hi [~ykt836], I guess this fix is actually contained in [1.9.0|https://github.com/apache/flink/commits/release-1.9.0?after=9c32ed989c0178a2bf3e059e897927c451188700+190]. Pls correct me if I missed something:) > flink-python fails to build on Travis due to PackagesNotFoundError > -- > > Key: FLINK-13488 > URL: https://issues.apache.org/jira/browse/FLINK-13488 > Project: Flink > Issue Type: Bug > Components: API / Python, Test Infrastructure >Reporter: Tzu-Li (Gordon) Tai >Assignee: sunjincheng >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 1h > Remaining Estimate: 0h > > https://api.travis-ci.org/v3/job/564925115/log.txt > {code} > install conda ... [SUCCESS] > install miniconda... [SUCCESS] > installing python environment... > installing python2.7... > install python2.7... [SUCCESS] > installing python3.3... > PackagesNotFoundError: The following packages are not available from current > channels: > - python=3.3 > Current channels: > - https://repo.anaconda.com/pkgs/main/linux-64 > - https://repo.anaconda.com/pkgs/main/noarch > - https://repo.anaconda.com/pkgs/r/linux-64 > - https://repo.anaconda.com/pkgs/r/noarch > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024 ## CI report: * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123648363) * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123684855) * 7d75f94fed9684c020c7673a57431c79e2f0a543 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124674615) * e21b3e5b91328bd7e9ce28fb8df9b848730b319a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124675311) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#issuecomment-522053886 ## CI report: * 009da497d7c551ba854dc7ed8fa658f2acd6d6ce : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123529451) * 14be3c57f3d625f3f9bbb96b7f74b6b0116fed3c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124588576) * c43a462c92c6c5c145a6964aa5390a82b894fa4e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124673290) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13364) Add documentation for setting resources in blink planner
[ https://issues.apache.org/jira/browse/FLINK-13364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13364: Fix Version/s: (was: 1.9.1) > Add documentation for setting resources in blink planner > > > Key: FLINK-13364 > URL: https://issues.apache.org/jira/browse/FLINK-13364 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Planner >Reporter: Jark Wu >Assignee: xupingyong >Priority: Major > > Add documentation for setting resources. > - “Performance Tuning / Resource Setting”: how to set resources in blink > planner, explain the configs and how they work together. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13361) Add documentation for JDBC connector for Table API & SQL
[ https://issues.apache.org/jira/browse/FLINK-13361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13361: Fix Version/s: 1.10.0 > Add documentation for JDBC connector for Table API & SQL > > > Key: FLINK-13361 > URL: https://issues.apache.org/jira/browse/FLINK-13361 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC, Documentation >Reporter: Jark Wu >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.10.0, 1.9.1 > > > Add documentation for JDBC connector for Table API & SQL > - “Connect to External Systems”: Add DDL for JDBC in “Table Connector” > section. JDBC support batch-source & lookup & sink. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13365) Add documentation for reusing upstream operations when multi-sink is applied
[ https://issues.apache.org/jira/browse/FLINK-13365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13365: Fix Version/s: (was: 1.9.1) (was: 1.10.0) > Add documentation for reusing upstream operations when multi-sink is applied > > > Key: FLINK-13365 > URL: https://issues.apache.org/jira/browse/FLINK-13365 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Planner >Reporter: Jark Wu >Priority: Major > > Add documentation for reuse upstream operations when multi-sink > - “Performance Tuning / Multi-Sink Optimization”: How to avoid executing > common operators repeatedly if there are multiple TableSinks in a job. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13360) Add documentation for HBase connector for Table API & SQL
[ https://issues.apache.org/jira/browse/FLINK-13360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13360: Fix Version/s: 1.10.0 > Add documentation for HBase connector for Table API & SQL > - > > Key: FLINK-13360 > URL: https://issues.apache.org/jira/browse/FLINK-13360 > Project: Flink > Issue Type: Sub-task > Components: Connectors / HBase, Documentation >Reporter: Jark Wu >Assignee: Caizhi Weng >Priority: Major > Fix For: 1.10.0, 1.9.1 > > > Add documentation for HBase connector for Table API & SQL > - “Connect to External Systems”: Add DDL for HBase in “Table Connector” > section. HBase support batch-source & lookup & sink. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13363) Add documentation for streaming aggregate performance tunning.
[ https://issues.apache.org/jira/browse/FLINK-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13363: Fix Version/s: 1.10.0 > Add documentation for streaming aggregate performance tunning. > -- > > Key: FLINK-13363 > URL: https://issues.apache.org/jira/browse/FLINK-13363 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Planner >Reporter: Jark Wu >Assignee: Jark Wu >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 10m > Remaining Estimate: 0h > > Add documentation for streaming aggregate performance tuning > - “Performance Tuning / Streaming Aggregation”: Explain how to solve > data-skew problem in streaming aggregation (non-windowed aggregate), the > internals, and the configurations. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13708) transformations should be cleared because a table environment could execute multiple job
[ https://issues.apache.org/jira/browse/FLINK-13708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13708: Fix Version/s: 1.10.0 > transformations should be cleared because a table environment could execute > multiple job > > > Key: FLINK-13708 > URL: https://issues.apache.org/jira/browse/FLINK-13708 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 10m > Remaining Estimate: 0h > > currently, if a table environment execute more than one sql jobs, the > following job contains transformations about the previous job. the reason is > the transformations is not cleared after execution -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13520) Add documentation for the different behavior of row count sliding window in blink planner
[ https://issues.apache.org/jira/browse/FLINK-13520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13520: Fix Version/s: 1.10.0 > Add documentation for the different behavior of row count sliding window in > blink planner > - > > Key: FLINK-13520 > URL: https://issues.apache.org/jira/browse/FLINK-13520 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Planner >Reporter: Jark Wu >Priority: Minor > Fix For: 1.10.0, 1.9.1 > > > As discussed in FLINK-13446, we want to keep the current behavior of row > count sliding window in blink planner. This is different with flink planner. > So it would be nice to document it. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13365) Add documentation for reusing upstream operations when multi-sink is applied
[ https://issues.apache.org/jira/browse/FLINK-13365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-13365: Fix Version/s: 1.10.0 > Add documentation for reusing upstream operations when multi-sink is applied > > > Key: FLINK-13365 > URL: https://issues.apache.org/jira/browse/FLINK-13365 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Planner >Reporter: Jark Wu >Priority: Major > Fix For: 1.10.0, 1.9.1 > > > Add documentation for reuse upstream operations when multi-sink > - “Performance Tuning / Multi-Sink Optimization”: How to avoid executing > common operators repeatedly if there are multiple TableSinks in a job. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13359) Add documentation for DDL introduction
[ https://issues.apache.org/jira/browse/FLINK-13359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13359: --- Fix Version/s: 1.10.0 > Add documentation for DDL introduction > -- > > Key: FLINK-13359 > URL: https://issues.apache.org/jira/browse/FLINK-13359 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / API >Reporter: Jark Wu >Assignee: Danny Chan >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Add documentation for DDL introduction > - “Concepts & Common API”: Add a section to describe how to execute DDL on > TableEnvironment. > - “SQL Client”: Add a section and example in SQL CLI page too? -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13362) Add documentation for Kafka & ES & FileSystem DDL
[ https://issues.apache.org/jira/browse/FLINK-13362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13362: --- Fix Version/s: 1.10.0 > Add documentation for Kafka & ES & FileSystem DDL > - > > Key: FLINK-13362 > URL: https://issues.apache.org/jira/browse/FLINK-13362 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Reporter: Jark Wu >Assignee: Danny Chan >Priority: Major > Fix For: 1.10.0, 1.9.1 > > > Add documentation for Kafka & ES & FileSystem DDL > - “Connect to External Systems”: Add DDL for Kafka & ES & FileSystem. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13105) Add documentation for blink planner's built-in functions
[ https://issues.apache.org/jira/browse/FLINK-13105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13105: --- Fix Version/s: 1.10.0 > Add documentation for blink planner's built-in functions > > > Key: FLINK-13105 > URL: https://issues.apache.org/jira/browse/FLINK-13105 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Planner >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Blink planner intros some built-in functions which need to be documented. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13354) Add documentation for how to use blink planner
[ https://issues.apache.org/jira/browse/FLINK-13354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13354: --- Fix Version/s: 1.10.0 > Add documentation for how to use blink planner > -- > > Key: FLINK-13354 > URL: https://issues.apache.org/jira/browse/FLINK-13354 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / Planner >Reporter: Jark Wu >Assignee: godfrey he >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Add documentation for how to use different planner > - “Overview”: add pom dependency > - “Concepts & Common API”: add description about how to use different > planner in code -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13564) blink planner should also throw exception if constant with YEAR TO MONTH resolution was used for group windows
[ https://issues.apache.org/jira/browse/FLINK-13564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13564: --- Fix Version/s: 1.10.0 > blink planner should also throw exception if constant with YEAR TO MONTH > resolution was used for group windows > --- > > Key: FLINK-13564 > URL: https://issues.apache.org/jira/browse/FLINK-13564 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.10.0 >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > just as [FLINK-11017|https://issues.apache.org/jira/browse/FLINK-11017], > blink planner should also throw exception if constant with YEAR TO MONTH > resolution was used for group windows -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend
carp84 commented on a change in pull request #9501: [FLINK-12697] [State Backends] Support on-disk state storage for spill-able heap backend URL: https://github.com/apache/flink/pull/9501#discussion_r317887398 ## File path: flink-state-backends/flink-statebackend-heap-spillable/src/main/java/org/apache/flink/runtime/state/heap/ByteBufferInputStreamWithPos.java ## @@ -0,0 +1,117 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.state.heap; Review comment: Agree that we'd better use another package name. How about we rename the package as the last step after all review comments addressed just to convenient the review? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13599) Kinesis end-to-end test failed on Travis
[ https://issues.apache.org/jira/browse/FLINK-13599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916349#comment-16916349 ] Kurt Young commented on FLINK-13599: I've updated the fix version to 1.9.1 since this is not included in 1.9.0 release. > Kinesis end-to-end test failed on Travis > > > Key: FLINK-13599 > URL: https://issues.apache.org/jira/browse/FLINK-13599 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis, Tests >Affects Versions: 1.9.0, 1.10.0 >Reporter: Till Rohrmann >Assignee: Andrey Zagrebin >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > The {{Kinesis end-to-end test}} failed on Travis with > {code} > 2019-08-06 08:48:20,177 ERROR org.apache.flink.client.cli.CliFrontend > - Error while running the command. > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: Unable to execute HTTP request: Connect to localhost:4567 > [localhost/127.0.0.1] failed: Connection refused (Connection refused) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) > at > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) > at > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) > at > org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) > Caused by: org.apache.flink.kinesis.shaded.com.amazonaws.SdkClientException: > Unable to execute HTTP request: Connect to localhost:4567 > [localhost/127.0.0.1] failed: Connection refused (Connection refused) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1116) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1066) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2388) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2364) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.executeDescribeStream(AmazonKinesisClient.java:754) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:729) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:766) > at > org.apache.flink.streaming.kinesis.test.KinesisPubsubClient.createTopic(KinesisPubsubClient.java:63) > at > org.apache.flink.streaming.kinesis.test.KinesisExampleTest.main(KinesisExampleTest.java:57) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) > ... 9 more > Caused by: >
[jira] [Updated] (FLINK-13599) Kinesis end-to-end test failed on Travis
[ https://issues.apache.org/jira/browse/FLINK-13599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13599: --- Fix Version/s: (was: 1.9.0) 1.9.1 > Kinesis end-to-end test failed on Travis > > > Key: FLINK-13599 > URL: https://issues.apache.org/jira/browse/FLINK-13599 > Project: Flink > Issue Type: Bug > Components: Connectors / Kinesis, Tests >Affects Versions: 1.9.0, 1.10.0 >Reporter: Till Rohrmann >Assignee: Andrey Zagrebin >Priority: Critical > Labels: pull-request-available, test-stability > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > The {{Kinesis end-to-end test}} failed on Travis with > {code} > 2019-08-06 08:48:20,177 ERROR org.apache.flink.client.cli.CliFrontend > - Error while running the command. > org.apache.flink.client.program.ProgramInvocationException: The main method > caused an error: Unable to execute HTTP request: Connect to localhost:4567 > [localhost/127.0.0.1] failed: Connection refused (Connection refused) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) > at > org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) > at > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) > at > org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) > Caused by: org.apache.flink.kinesis.shaded.com.amazonaws.SdkClientException: > Unable to execute HTTP request: Connect to localhost:4567 > [localhost/127.0.0.1] failed: Connection refused (Connection refused) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1116) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1066) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) > at > org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2388) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2364) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.executeDescribeStream(AmazonKinesisClient.java:754) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:729) > at > org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.describeStream(AmazonKinesisClient.java:766) > at > org.apache.flink.streaming.kinesis.test.KinesisPubsubClient.createTopic(KinesisPubsubClient.java:63) > at > org.apache.flink.streaming.kinesis.test.KinesisExampleTest.main(KinesisExampleTest.java:57) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) > ... 9 more > Caused by: >
[GitHub] [flink] flinkbot edited a comment on issue #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error
flinkbot edited a comment on issue #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error URL: https://github.com/apache/flink/pull/9540#issuecomment-525128086 ## CI report: * 5cb9691c978214d5f868c0e6a43f782b05b29ff6 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124678443) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13277) add documentation of Hive source/sink
[ https://issues.apache.org/jira/browse/FLINK-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916347#comment-16916347 ] Kurt Young commented on FLINK-13277: Looks like this issue already merged, [~phoenixjiangnan] could you please close this jira? > add documentation of Hive source/sink > - > > Key: FLINK-13277 > URL: https://issues.apache.org/jira/browse/FLINK-13277 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive, Documentation >Reporter: Bowen Li >Assignee: Rui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > add documentation of Hive source/sink in {{batch/connector.md}} > its corresponding Chinese one is FLINK-13278 > cc [~xuefuz] [~lirui] [~Terry1897] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13738) NegativeArraySizeException in LongHybridHashTable
[ https://issues.apache.org/jira/browse/FLINK-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13738: --- Fix Version/s: 1.10.0 > NegativeArraySizeException in LongHybridHashTable > - > > Key: FLINK-13738 > URL: https://issues.apache.org/jira/browse/FLINK-13738 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.9.0 >Reporter: Robert Metzger >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Executing this (meaningless) query: > {code:java} > INSERT INTO sinkTable ( SELECT CONCAT( CAST( id AS VARCHAR), CAST( COUNT(*) > AS VARCHAR)) as something, 'const' FROM CsvTable, table1 WHERE sometxt LIKE > 'a%' AND id = key GROUP BY id ) {code} > leads to the following exception: > {code:java} > Caused by: java.lang.NegativeArraySizeException > at > org.apache.flink.table.runtime.hashtable.LongHybridHashTable.tryDenseMode(LongHybridHashTable.java:216) > at > org.apache.flink.table.runtime.hashtable.LongHybridHashTable.endBuild(LongHybridHashTable.java:105) > at LongHashJoinOperator$36.endInput1$(Unknown Source) > at LongHashJoinOperator$36.endInput(Unknown Source) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.endInput(OperatorChain.java:256) > at > org.apache.flink.streaming.runtime.io.StreamTwoInputSelectableProcessor.checkFinished(StreamTwoInputSelectableProcessor.java:359) > at > org.apache.flink.streaming.runtime.io.StreamTwoInputSelectableProcessor.processInput(StreamTwoInputSelectableProcessor.java:193) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.performDefaultAction(StreamTask.java:276) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:687) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:517) > at java.lang.Thread.run(Thread.java:748){code} > This is the plan: > > {code:java} > == Abstract Syntax Tree == > LogicalSink(name=[sinkTable], fields=[f0, f1]) > +- LogicalProject(something=[CONCAT(CAST($0):VARCHAR(2147483647) CHARACTER > SET "UTF-16LE", CAST($1):VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT > NULL)], EXPR$1=[_UTF-16LE'const']) >+- LogicalAggregate(group=[{0}], agg#0=[COUNT()]) > +- LogicalProject(id=[$1]) > +- LogicalFilter(condition=[AND(LIKE($0, _UTF-16LE'a%'), =($1, > CAST($2):BIGINT))]) > +- LogicalJoin(condition=[true], joinType=[inner]) >:- LogicalTableScan(table=[[default_catalog, default_database, > CsvTable, source: [CsvTableSource(read fields: sometxt, id)]]]) >+- LogicalTableScan(table=[[default_catalog, default_database, > table1, source: [GeneratorTableSource(key, rowtime, payload)]]]) > == Optimized Logical Plan == > Sink(name=[sinkTable], fields=[f0, f1]): rowcount = 1498810.6659336376, > cumulative cost = {4.459964319978008E8 rows, 1.879799762133187E10 cpu, 4.8E9 > io, 8.4E8 network, 1.799524266373455E8 memory} > +- Calc(select=[CONCAT(CAST(id), CAST($f1)) AS something, _UTF-16LE'const' AS > EXPR$1]): rowcount = 1498810.6659336376, cumulative cost = > {4.444976213318672E8 rows, 1.8796498810665936E10 cpu, 4.8E9 io, 8.4E8 > network, 1.799524266373455E8 memory} >+- HashAggregate(isMerge=[false], groupBy=[id], select=[id, COUNT(*) AS > $f1]): rowcount = 1498810.6659336376, cumulative cost = {4.429988106659336E8 > rows, 1.8795E10 cpu, 4.8E9 io, 8.4E8 network, 1.799524266373455E8 memory} > +- Calc(select=[id]): rowcount = 1.575E7, cumulative cost = {4.415E8 > rows, 1.848E10 cpu, 4.8E9 io, 8.4E8 network, 1.2E8 memory} > +- HashJoin(joinType=[InnerJoin], where=[=(id, key0)], select=[id, > key0], build=[left]): rowcount = 1.575E7, cumulative cost = {4.2575E8 rows, > 1.848E10 cpu, 4.8E9 io, 8.4E8 network, 1.2E8 memory} > :- Exchange(distribution=[hash[id]]): rowcount = 500.0, > cumulative cost = {1.1E8 rows, 8.4E8 cpu, 2.0E9 io, 4.0E7 network, 0.0 memory} > : +- Calc(select=[id], where=[LIKE(sometxt, _UTF-16LE'a%')]): > rowcount = 500.0, cumulative cost = {1.05E8 rows, 0.0 cpu, 2.0E9 io, 0.0 > network, 0.0 memory} > : +- TableSourceScan(table=[[default_catalog, > default_database, CsvTable, source: [CsvTableSource(read fields: sometxt, > id)]]], fields=[sometxt, id]): rowcount = 1.0E8, cumulative cost = {1.0E8 > rows, 0.0 cpu, 2.0E9 io, 0.0 network, 0.0 memory} > +- Exchange(distribution=[hash[key0]]): rowcount = 1.0E8, > cumulative cost = {3.0E8 rows, 1.68E10 cpu, 2.8E9 io,
[jira] [Updated] (FLINK-13739) BinaryRowTest.testWriteString() fails in some environments
[ https://issues.apache.org/jira/browse/FLINK-13739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13739: --- Fix Version/s: 1.10.0 > BinaryRowTest.testWriteString() fails in some environments > -- > > Key: FLINK-13739 > URL: https://issues.apache.org/jira/browse/FLINK-13739 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.9.0 > Environment: > >Reporter: Robert Metzger >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available, test-stability > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > > > {code:java} > Test set: org.apache.flink.table.dataformat.BinaryRowTest > --- > Tests run: 26, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.328 s <<< > FAILURE! - in org.apache.flink.table.dataformat.BinaryRowTest > testWriteString(org.apache.flink.table.dataformat.BinaryRowTest) Time > elapsed: 0.05 s <<< FAILURE! > org.junit.ComparisonFailure: > expected:<[<95><95><95><95><95><88><91><98> > <90><9A><84><89><88><8C>]> > but was:<[?]> > at > org.apache.flink.table.dataformat.BinaryRowTest.testWriteString(BinaryRowTest.java:189) > {code} > > This error happens on a Google Cloud n2-standard-16 (16 vCPUs, 64 GB memory) > machine. > {code}$ lsb_release -a > No LSB modules are available. > Distributor ID:Debian > Description:Debian GNU/Linux 9.9 (stretch) > Release:9.9 > Codename:stretch{code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13742) Fix code generation when aggregation contains both distinct aggregate with and without filter
[ https://issues.apache.org/jira/browse/FLINK-13742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13742: --- Fix Version/s: 1.10.0 > Fix code generation when aggregation contains both distinct aggregate with > and without filter > - > > Key: FLINK-13742 > URL: https://issues.apache.org/jira/browse/FLINK-13742 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: Jark Wu >Assignee: Shuo Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > The following test will fail when the aggregation contains {{COUNT(DISTINCT > c)}} and {{COUNT(DISTINCT c) filter ...}}. > {code:java} > @Test > def testDistinctWithMultiFilter(): Unit = { > val sqlQuery = > "SELECT b, " + > " SUM(DISTINCT (a * 3)), " + > " COUNT(DISTINCT SUBSTRING(c FROM 1 FOR 2))," + > " COUNT(DISTINCT c)," + > " COUNT(DISTINCT c) filter (where MOD(a, 3) = 0)," + > " COUNT(DISTINCT c) filter (where MOD(a, 3) = 1) " + > "FROM MyTable " + > "GROUP BY b" > val t = > failingDataSource(StreamTestData.get3TupleData).toTable(tEnv).as('a, 'b, 'c) > tEnv.registerTable("MyTable", t) > val result = tEnv.sqlQuery(sqlQuery).toRetractStream[Row] > val sink = new TestingRetractSink > result.addSink(sink) > env.execute() > val expected = List( > "1,3,1,1,0,1", > "2,15,1,2,1,0", > "3,45,3,3,1,1", > "4,102,1,4,1,2", > "5,195,1,5,2,1", > "6,333,1,6,2,2") > assertEquals(expected.sorted, sink.getRetractResults.sorted) > } > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13277) add documentation of Hive source/sink
[ https://issues.apache.org/jira/browse/FLINK-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13277: --- Fix Version/s: 1.10.0 > add documentation of Hive source/sink > - > > Key: FLINK-13277 > URL: https://issues.apache.org/jira/browse/FLINK-13277 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive, Documentation >Reporter: Bowen Li >Assignee: Rui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > add documentation of Hive source/sink in {{batch/connector.md}} > its corresponding Chinese one is FLINK-13278 > cc [~xuefuz] [~lirui] [~Terry1897] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13737) flink-dist should add provided dependency on flink-examples-table
[ https://issues.apache.org/jira/browse/FLINK-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916346#comment-16916346 ] Kurt Young commented on FLINK-13737: I've updated the fix version to 1.9.1 since this is not included in 1.9.0 release. > flink-dist should add provided dependency on flink-examples-table > - > > Key: FLINK-13737 > URL: https://issues.apache.org/jira/browse/FLINK-13737 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.9.0 >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 40m > Remaining Estimate: 0h > > In FLINK-13558 we changed the `flink-dist/bin.xml` to also include > flink-examples-table in the binary distribution. The flink-dist module though > does not depend on the flink-examples-table. > If only the flink-dist module is built with its dependencies (this happens in > the release scripts). The table examples are not built and thus not included > in the distribution -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13737) flink-dist should add provided dependency on flink-examples-table
[ https://issues.apache.org/jira/browse/FLINK-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13737: --- Fix Version/s: (was: 1.9.0) 1.9.1 1.10.0 > flink-dist should add provided dependency on flink-examples-table > - > > Key: FLINK-13737 > URL: https://issues.apache.org/jira/browse/FLINK-13737 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.9.0 >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 40m > Remaining Estimate: 0h > > In FLINK-13558 we changed the `flink-dist/bin.xml` to also include > flink-examples-table in the binary distribution. The flink-dist module though > does not depend on the flink-examples-table. > If only the flink-dist module is built with its dependencies (this happens in > the release scripts). The table examples are not built and thus not included > in the distribution -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13688) HiveCatalogUseBlinkITCase.testBlinkUdf constantly failed with 1.9.0-rc2
[ https://issues.apache.org/jira/browse/FLINK-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916344#comment-16916344 ] Kurt Young commented on FLINK-13688: [~till.rohrmann] Thanks for the heads up, this was planned. I'm also start to scan all issues that are not included in 1.9.0 release and change the fixVersion now. > HiveCatalogUseBlinkITCase.testBlinkUdf constantly failed with 1.9.0-rc2 > --- > > Key: FLINK-13688 > URL: https://issues.apache.org/jira/browse/FLINK-13688 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Tests >Affects Versions: 1.9.0 > Environment: Linux server > kernal version: 3.10.0 > java version: "1.8.0_102" > processor count: 96 >Reporter: Kurt Young >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > I tried to build flink 1.9.0-rc2 from source and ran all tests in a linux > server, HiveCatalogUseBlinkITCase.testBlinkUdf will be constantly fail. > > Fail trace: > {code:java} > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 313.228 s <<< FAILURE! - in > org.apache.flink.table.catalog.hive.HiveCatalogUseBlinkITCase > [ERROR] > testBlinkUdf(org.apache.flink.table.catalog.hive.HiveCatalogUseBlinkITCase) > Time elapsed: 305.155 s <<< ERROR! > org.apache.flink.runtime.client.JobExecutionException: Job execution failed. > at > org.apache.flink.table.catalog.hive.HiveCatalogUseBlinkITCase.testBlinkUdf(HiveCatalogUseBlinkITCase.java:180) > Caused by: > org.apache.flink.runtime.resourcemanager.exceptions.UnfulfillableSlotRequestException: > Could not fulfill slot request 35cf6fdc1b525de9b6eed13894e2e31d. Requested > resource profile (ResourceProfile{cpuCores=0.0, heapMemoryInMB=0, > directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0, > managedMemoryInMB=128}) is unfulfillable. > {code} > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13706) add documentation of how to use Hive functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916339#comment-16916339 ] Kurt Young commented on FLINK-13706: I've updated the fix version to 1.9.1 since this is not included in 1.9.0 release. > add documentation of how to use Hive functions in Flink > --- > > Key: FLINK-13706 > URL: https://issues.apache.org/jira/browse/FLINK-13706 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive, Documentation >Affects Versions: 1.9.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13706) add documentation of how to use Hive functions in Flink
[ https://issues.apache.org/jira/browse/FLINK-13706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13706: --- Fix Version/s: (was: 1.9.0) 1.9.1 1.10.0 > add documentation of how to use Hive functions in Flink > --- > > Key: FLINK-13706 > URL: https://issues.apache.org/jira/browse/FLINK-13706 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive, Documentation >Affects Versions: 1.9.0 >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13501) Fixes a few issues in documentation for Hive integration
[ https://issues.apache.org/jira/browse/FLINK-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13501: --- Fix Version/s: (was: 1.9.0) 1.9.1 1.10.0 > Fixes a few issues in documentation for Hive integration > > > Key: FLINK-13501 > URL: https://issues.apache.org/jira/browse/FLINK-13501 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / API >Affects Versions: 1.9.0 >Reporter: Xuefu Zhang >Assignee: Terry Wang >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Attachments: Screen Shot 2019-07-30 at 3.21.25 PM.png, Screen Shot > 2019-07-30 at 3.25.13 PM.png > > Time Spent: 20m > Remaining Estimate: 0h > > Going thru existing Hive doc I found the following issues that should be > addressed: > 1. Section "Hive Integration" should come after "SQL client" (at the same > level). > 2. In Catalog section, there are headers named "Hive Catalog". Also, some > information is duplicated with that in "Hive Integration" > 3. "Data Type Mapping" is Hive specific and should probably move to "Hive > integration" -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13415) Document how to use hive connector in scala shell
[ https://issues.apache.org/jira/browse/FLINK-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916338#comment-16916338 ] Kurt Young commented on FLINK-13415: I've updated the fix version to 1.9.1 since this is not included in 1.9.0 release. > Document how to use hive connector in scala shell > - > > Key: FLINK-13415 > URL: https://issues.apache.org/jira/browse/FLINK-13415 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13415) Document how to use hive connector in scala shell
[ https://issues.apache.org/jira/browse/FLINK-13415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13415: --- Fix Version/s: (was: 1.9.0) 1.9.1 1.10.0 > Document how to use hive connector in scala shell > - > > Key: FLINK-13415 > URL: https://issues.apache.org/jira/browse/FLINK-13415 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13501) Fixes a few issues in documentation for Hive integration
[ https://issues.apache.org/jira/browse/FLINK-13501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916337#comment-16916337 ] Kurt Young commented on FLINK-13501: I've updated the fix version to 1.9.1 since this is not included in 1.9.0 release. > Fixes a few issues in documentation for Hive integration > > > Key: FLINK-13501 > URL: https://issues.apache.org/jira/browse/FLINK-13501 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive, Table SQL / API >Affects Versions: 1.9.0 >Reporter: Xuefu Zhang >Assignee: Terry Wang >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.0 > > Attachments: Screen Shot 2019-07-30 at 3.21.25 PM.png, Screen Shot > 2019-07-30 at 3.25.13 PM.png > > Time Spent: 20m > Remaining Estimate: 0h > > Going thru existing Hive doc I found the following issues that should be > addressed: > 1. Section "Hive Integration" should come after "SQL client" (at the same > level). > 2. In Catalog section, there are headers named "Hive Catalog". Also, some > information is duplicated with that in "Hive Integration" > 3. "Data Type Mapping" is Hive specific and should probably move to "Hive > integration" -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13534) Unable to query Hive table with decimal column
[ https://issues.apache.org/jira/browse/FLINK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13534: --- Fix Version/s: 1.10.0 > Unable to query Hive table with decimal column > -- > > Key: FLINK-13534 > URL: https://issues.apache.org/jira/browse/FLINK-13534 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.9.0 >Reporter: Rui Li >Assignee: Rui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Hit the following exception when access a Hive table with decimal column: > {noformat} > Caused by: org.apache.flink.table.api.TableException: TableSource of type > org.apache.flink.batch.connectors.hive.HiveTableSource returned a DataSet of > data type ROW<`x` LEGACY(BigDecimal)> that does not match with the data type > ROW<`x` DECIMAL(10, 0)> declared by the TableSource.getProducedDataType() > method. Please validate the implementation of the TableSource. > at > org.apache.flink.table.plan.nodes.dataset.BatchTableSourceScan.translateToPlan(BatchTableSourceScan.scala:118) > at > org.apache.flink.table.api.internal.BatchTableEnvImpl.translate(BatchTableEnvImpl.scala:303) > at > org.apache.flink.table.api.internal.BatchTableEnvImpl.translate(BatchTableEnvImpl.scala:281) > at > org.apache.flink.table.api.internal.BatchTableEnvImpl.writeToSink(BatchTableEnvImpl.scala:117) > at > org.apache.flink.table.api.internal.TableEnvImpl.insertInto(TableEnvImpl.scala:564) > at > org.apache.flink.table.api.internal.TableEnvImpl.insertInto(TableEnvImpl.scala:516) > at > org.apache.flink.table.api.internal.BatchTableEnvImpl.insertInto(BatchTableEnvImpl.scala:59) > at > org.apache.flink.table.api.internal.TableImpl.insertInto(TableImpl.java:428) > {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError
[ https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916336#comment-16916336 ] Kurt Young commented on FLINK-13488: I've updated the fix version to 1.9.1 since this is not included in 1.9.0 release. > flink-python fails to build on Travis due to PackagesNotFoundError > -- > > Key: FLINK-13488 > URL: https://issues.apache.org/jira/browse/FLINK-13488 > Project: Flink > Issue Type: Bug > Components: API / Python, Test Infrastructure >Reporter: Tzu-Li (Gordon) Tai >Assignee: sunjincheng >Priority: Blocker > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 1h > Remaining Estimate: 0h > > https://api.travis-ci.org/v3/job/564925115/log.txt > {code} > install conda ... [SUCCESS] > install miniconda... [SUCCESS] > installing python environment... > installing python2.7... > install python2.7... [SUCCESS] > installing python3.3... > PackagesNotFoundError: The following packages are not available from current > channels: > - python=3.3 > Current channels: > - https://repo.anaconda.com/pkgs/main/linux-64 > - https://repo.anaconda.com/pkgs/main/noarch > - https://repo.anaconda.com/pkgs/r/linux-64 > - https://repo.anaconda.com/pkgs/r/noarch > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13488) flink-python fails to build on Travis due to PackagesNotFoundError
[ https://issues.apache.org/jira/browse/FLINK-13488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13488: --- Fix Version/s: (was: 1.9.0) 1.9.1 1.10.0 > flink-python fails to build on Travis due to PackagesNotFoundError > -- > > Key: FLINK-13488 > URL: https://issues.apache.org/jira/browse/FLINK-13488 > Project: Flink > Issue Type: Bug > Components: API / Python, Test Infrastructure >Reporter: Tzu-Li (Gordon) Tai >Assignee: sunjincheng >Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 1h > Remaining Estimate: 0h > > https://api.travis-ci.org/v3/job/564925115/log.txt > {code} > install conda ... [SUCCESS] > install miniconda... [SUCCESS] > installing python environment... > installing python2.7... > install python2.7... [SUCCESS] > installing python3.3... > PackagesNotFoundError: The following packages are not available from current > channels: > - python=3.3 > Current channels: > - https://repo.anaconda.com/pkgs/main/linux-64 > - https://repo.anaconda.com/pkgs/main/noarch > - https://repo.anaconda.com/pkgs/r/linux-64 > - https://repo.anaconda.com/pkgs/r/noarch > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13517) Restructure Hive Catalog documentation
[ https://issues.apache.org/jira/browse/FLINK-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13517: --- Fix Version/s: 1.10.0 > Restructure Hive Catalog documentation > -- > > Key: FLINK-13517 > URL: https://issues.apache.org/jira/browse/FLINK-13517 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Documentation >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Hive documentation is currently spread across a number of pages and > fragmented. In particular: > 1) An example was added to getting-started/examples, however, this section is > being removed > 2) There is a dedicated page on hive integration but also a lot of hive > specific information is on the catalog page > We should > 1) Inline the example into the hive integration page > 2) Move the hive specific information on catalogs.md to hive_integration.md > 3) Make catalogs.md be just about catalogs in general and link to the hive > integration. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13490) Fix return null in JDBCUtils::getFieldFromResultSet
[ https://issues.apache.org/jira/browse/FLINK-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13490: --- Fix Version/s: 1.10.0 > Fix return null in JDBCUtils::getFieldFromResultSet > --- > > Key: FLINK-13490 > URL: https://issues.apache.org/jira/browse/FLINK-13490 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: Caizhi Weng >Assignee: Caizhi Weng >Priority: Critical > Fix For: 1.10.0, 1.9.1 > > > The null checking in `JDBCUtils::getFieldFromResultSet` is incorrect. Under > the current implementation, if one column is null in the result set, the > following calls to this method using the same result set will always return > null, no matter what the content of the column is. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13526) Switching to a non existing catalog or database crashes sql-client
[ https://issues.apache.org/jira/browse/FLINK-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13526: --- Fix Version/s: 1.10.0 > Switching to a non existing catalog or database crashes sql-client > -- > > Key: FLINK-13526 > URL: https://issues.apache.org/jira/browse/FLINK-13526 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.9.0 >Reporter: Rui Li >Assignee: Rui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > sql-client crashes if user tries to switch to a non-existing DB: > {noformat} > Exception in thread "main" org.apache.flink.table.client.SqlClientException: > Unexpected exception. This is a bug. Please consider filing an issue. > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:206) > Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A > database with name [foo] does not exist in the catalog: [myhive]. > at > org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:286) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:398) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$useDatabase$5(LocalExecutor.java:258) > at > org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:216) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.useDatabase(LocalExecutor.java:256) > at > org.apache.flink.table.client.cli.CliClient.callUseDatabase(CliClient.java:434) > at > org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:282) > at java.util.Optional.ifPresent(Optional.java:159) > at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200) > at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:123) > at org.apache.flink.table.client.SqlClient.start(SqlClient.java:105) > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:194) > {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13563) TumblingGroupWindow should implement toString method
[ https://issues.apache.org/jira/browse/FLINK-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13563: --- Fix Version/s: 1.10.0 > TumblingGroupWindow should implement toString method > > > Key: FLINK-13563 > URL: https://issues.apache.org/jira/browse/FLINK-13563 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.10.0 >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > {code:scala} > @Test > def testAllEventTimeTumblingGroupWindowOverTime(): Unit = { > val util = streamTestUtil() > val table = util.addDataStream[(Long, Int, String)]( > "T1", 'long, 'int, 'string, 'rowtime.rowtime) > val windowedTable = table > .window(Tumble over 5.millis on 'rowtime as 'w) > .groupBy('w) > .select('int.count) > util.verifyPlan(windowedTable) > } > {code} > currently, it's physical plan is > {code:java} > HashWindowAggregate(window=[TumblingGroupWindow], > select=[Final_COUNT(count$0) AS EXPR$0]) > +- Exchange(distribution=[single]) >+- LocalHashWindowAggregate(window=[TumblingGroupWindow], > select=[Partial_COUNT(int) AS count$0]) > +- TableSourceScan(table=[[default_catalog, default_database, Table1, > source: [TestTableSource(long, int, string)]]], fields=[long, int, string]) > {code} > we know nothing about the TumblingGroupWindow except its name. the expected > plan is > {code:java} > HashWindowAggregate(window=[TumblingGroupWindow('w, long, 5)], > select=[Final_COUNT(count$0) AS EXPR$0]) > +- Exchange(distribution=[single]) >+- LocalHashWindowAggregate(window=[TumblingGroupWindow('w, long, 5)], > select=[Partial_COUNT(int) AS count$0]) > +- TableSourceScan(table=[[default_catalog, default_database, Table1, > source: [TestTableSource(long, int, string)]]], fields=[long, int, string]) > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13562) throws exception when FlinkRelMdColumnInterval meets two stage stream group aggregate
[ https://issues.apache.org/jira/browse/FLINK-13562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13562: --- Fix Version/s: 1.10.0 > throws exception when FlinkRelMdColumnInterval meets two stage stream group > aggregate > - > > Key: FLINK-13562 > URL: https://issues.apache.org/jira/browse/FLINK-13562 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Critical > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > test case: > {code:scala} > @Test > def testTwoDistinctAggregateWithNonDistinctAgg(): Unit = { > util.addTableSource[(Int, Long, String)]("MyTable", 'a, 'b, 'c) > util.verifyPlan("SELECT c, SUM(DISTINCT a), SUM(a), COUNT(DISTINCT b) > FROM MyTable GROUP BY c") > } > {code} > org.apache.flink.table.api.TableException: Sum aggregate function does not > support type: ''VARCHAR''. > Please re-check the data type. > at > org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createSumAggFunction(AggFunctionFactory.scala:191) > at > org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createAggFunction(AggFunctionFactory.scala:74) > at > org.apache.flink.table.planner.plan.utils.AggregateUtil$$anonfun$9.apply(AggregateUtil.scala:285) > at > org.apache.flink.table.planner.plan.utils.AggregateUtil$$anonfun$9.apply(AggregateUtil.scala:279) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) > at scala.collection.AbstractTraversable.map(Traversable.scala:104) > at > org.apache.flink.table.planner.plan.utils.AggregateUtil$.transformToAggregateInfoList(AggregateUtil.scala:279) > at > org.apache.flink.table.planner.plan.utils.AggregateUtil$.getOutputIndexToAggCallIndexMap(AggregateUtil.scala:154) > at > org.apache.flink.table.planner.plan.metadata.FlinkRelMdColumnInterval.getAggCallIndexInLocalAgg$1(FlinkRelMdColumnInterval.scala:504) > at > org.apache.flink.table.planner.plan.metadata.FlinkRelMdColumnInterval.estimateColumnIntervalOfAggregate(FlinkRelMdColumnInterval.scala:526) > at > org.apache.flink.table.planner.plan.metadata.FlinkRelMdColumnInterval.getColumnInterval(FlinkRelMdColumnInterval.scala:417) > at GeneratedMetadataHandler_ColumnInterval.getColumnInterval_$(Unknown > Source) > at GeneratedMetadataHandler_ColumnInterval.getColumnInterval(Unknown > Source) > at > org.apache.flink.table.planner.plan.metadata.FlinkRelMetadataQuery.getColumnInterval(FlinkRelMetadataQuery.java:122) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13637) Anchors not working in document(building.md, common.md, queryable_state.md)
[ https://issues.apache.org/jira/browse/FLINK-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13637: --- Fix Version/s: 1.10.0 > Anchors not working in document(building.md, common.md, queryable_state.md) > --- > > Key: FLINK-13637 > URL: https://issues.apache.org/jira/browse/FLINK-13637 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0, 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Anchors not working in document(building.md, common.md, queryable_state.md). > The format should be: > {code:java} > [create an anchor](#anchors-in-markdown) > {code} > - Add - characters between each word in the heading and wrap the value in > parens > - All letters should be lowercase. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot commented on issue #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error
flinkbot commented on issue #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error URL: https://github.com/apache/flink/pull/9540#issuecomment-525128086 ## CI report: * 5cb9691c978214d5f868c0e6a43f782b05b29ff6 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13517) Restructure Hive Catalog documentation
[ https://issues.apache.org/jira/browse/FLINK-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916323#comment-16916323 ] Kurt Young commented on FLINK-13517: I've updated the fix version to 1.9.1 since this is not included in 1.9.0 release. > Restructure Hive Catalog documentation > -- > > Key: FLINK-13517 > URL: https://issues.apache.org/jira/browse/FLINK-13517 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Documentation >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Hive documentation is currently spread across a number of pages and > fragmented. In particular: > 1) An example was added to getting-started/examples, however, this section is > being removed > 2) There is a dedicated page on hive integration but also a lot of hive > specific information is on the catalog page > We should > 1) Inline the example into the hive integration page > 2) Move the hive specific information on catalogs.md to hive_integration.md > 3) Make catalogs.md be just about catalogs in general and link to the hive > integration. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13637) Anchors not working in document(building.md, common.md, queryable_state.md)
[ https://issues.apache.org/jira/browse/FLINK-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916320#comment-16916320 ] Kurt Young commented on FLINK-13637: I've updated the fix version to 1.9.1 since this is not included in 1.9.0 release. > Anchors not working in document(building.md, common.md, queryable_state.md) > --- > > Key: FLINK-13637 > URL: https://issues.apache.org/jira/browse/FLINK-13637 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Anchors not working in document(building.md, common.md, queryable_state.md). > The format should be: > {code:java} > [create an anchor](#anchors-in-markdown) > {code} > - Add - characters between each word in the heading and wrap the value in > parens > - All letters should be lowercase. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13517) Restructure Hive Catalog documentation
[ https://issues.apache.org/jira/browse/FLINK-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13517: --- Fix Version/s: (was: 1.9.0) 1.9.1 > Restructure Hive Catalog documentation > -- > > Key: FLINK-13517 > URL: https://issues.apache.org/jira/browse/FLINK-13517 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Documentation >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Hive documentation is currently spread across a number of pages and > fragmented. In particular: > 1) An example was added to getting-started/examples, however, this section is > being removed > 2) There is a dedicated page on hive integration but also a lot of hive > specific information is on the catalog page > We should > 1) Inline the example into the hive integration page > 2) Move the hive specific information on catalogs.md to hive_integration.md > 3) Make catalogs.md be just about catalogs in general and link to the hive > integration. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13526) Switching to a non existing catalog or database crashes sql-client
[ https://issues.apache.org/jira/browse/FLINK-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13526: --- Fix Version/s: (was: 1.9.0) 1.9.1 > Switching to a non existing catalog or database crashes sql-client > -- > > Key: FLINK-13526 > URL: https://issues.apache.org/jira/browse/FLINK-13526 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.9.0 >Reporter: Rui Li >Assignee: Rui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > sql-client crashes if user tries to switch to a non-existing DB: > {noformat} > Exception in thread "main" org.apache.flink.table.client.SqlClientException: > Unexpected exception. This is a bug. Please consider filing an issue. > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:206) > Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A > database with name [foo] does not exist in the catalog: [myhive]. > at > org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:286) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:398) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$useDatabase$5(LocalExecutor.java:258) > at > org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:216) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.useDatabase(LocalExecutor.java:256) > at > org.apache.flink.table.client.cli.CliClient.callUseDatabase(CliClient.java:434) > at > org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:282) > at java.util.Optional.ifPresent(Optional.java:159) > at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200) > at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:123) > at org.apache.flink.table.client.SqlClient.start(SqlClient.java:105) > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:194) > {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13526) Switching to a non existing catalog or database crashes sql-client
[ https://issues.apache.org/jira/browse/FLINK-13526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916319#comment-16916319 ] Kurt Young commented on FLINK-13526: I've updated the fix version to 1.9.1 since this is not included in 1.9.0 release. > Switching to a non existing catalog or database crashes sql-client > -- > > Key: FLINK-13526 > URL: https://issues.apache.org/jira/browse/FLINK-13526 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.9.0 >Reporter: Rui Li >Assignee: Rui Li >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > sql-client crashes if user tries to switch to a non-existing DB: > {noformat} > Exception in thread "main" org.apache.flink.table.client.SqlClientException: > Unexpected exception. This is a bug. Please consider filing an issue. > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:206) > Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A > database with name [foo] does not exist in the catalog: [myhive]. > at > org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:286) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:398) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$useDatabase$5(LocalExecutor.java:258) > at > org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:216) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.useDatabase(LocalExecutor.java:256) > at > org.apache.flink.table.client.cli.CliClient.callUseDatabase(CliClient.java:434) > at > org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:282) > at java.util.Optional.ifPresent(Optional.java:159) > at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200) > at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:123) > at org.apache.flink.table.client.SqlClient.start(SqlClient.java:105) > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:194) > {noformat} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-13637) Anchors not working in document(building.md, common.md, queryable_state.md)
[ https://issues.apache.org/jira/browse/FLINK-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Young updated FLINK-13637: --- Fix Version/s: (was: 1.9.0) 1.9.1 > Anchors not working in document(building.md, common.md, queryable_state.md) > --- > > Key: FLINK-13637 > URL: https://issues.apache.org/jira/browse/FLINK-13637 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.9.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Anchors not working in document(building.md, common.md, queryable_state.md). > The format should be: > {code:java} > [create an anchor](#anchors-in-markdown) > {code} > - Add - characters between each word in the heading and wrap the value in > parens > - All letters should be lowercase. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-10995) Copy intermediate serialization results only once for broadcast mode
[ https://issues.apache.org/jira/browse/FLINK-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916315#comment-16916315 ] Kurt Young commented on FLINK-10995: Would it possible to open the Jira for writing only one file in BLOCKING broadcast shuffle first and start to discuss the design and plan? > Copy intermediate serialization results only once for broadcast mode > > > Key: FLINK-10995 > URL: https://issues.apache.org/jira/browse/FLINK-10995 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.8.0 >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The emitted records from operator would be firstly serialized into > intermediate bytes array in {{RecordSerializer}}, then copy the intermediate > results into target buffers for different sub partitions. For broadcast > mode, the same intermediate results would be copied as many times as the > number of sub partitions, and this would affect the performance seriously in > large scale jobs. > We can copy to only one target buffer which would be shared by all the sub > partitions to reduce the overheads. For emitting latency marker in broadcast > mode, we should flush the previous shared target buffers first, and then > request a new buffer for the target sub partition to send latency marker. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot commented on issue #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error
flinkbot commented on issue #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error URL: https://github.com/apache/flink/pull/9540#issuecomment-525126046 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5cb9691c978214d5f868c0e6a43f782b05b29ff6 (Tue Aug 27 03:55:13 UTC 2019) **Warnings:** * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13859).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13859) JSONDeserializationSchema spell error
[ https://issues.apache.org/jira/browse/FLINK-13859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13859: --- Labels: pull-request-available (was: ) > JSONDeserializationSchema spell error > - > > Key: FLINK-13859 > URL: https://issues.apache.org/jira/browse/FLINK-13859 > Project: Flink > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.9.0 >Reporter: limbo >Priority: Minor > Labels: pull-request-available > Attachments: image-2019-08-26-20-14-20-075.png > > > In kafka page the JsonDeserializationSchema would be JSONDeserializationSchema > !image-2019-08-26-20-14-20-075.png! > > [https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9539: [hotfix][docs] Fix typo
flinkbot edited a comment on issue #9539: [hotfix][docs] Fix typo URL: https://github.com/apache/flink/pull/9539#issuecomment-525098010 ## CI report: * 9430aca9b85c3a226cf5d5546e174d57daa8174e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124669373) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] taizilongxu opened a new pull request #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error
taizilongxu opened a new pull request #9540: [FLINK-13859][docs] JSONDeserializationSchema spell error URL: https://github.com/apache/flink/pull/9540 ## What is the purpose of the change The doc spell error ## Brief change log JsonDeserializationSchema -> JSONDeserializationSchema ## Verifying this change none ## Does this pull request potentially affect one of the following parts: none ## Documentation none This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13547) Verify and correct string function's semantic for Blink planner
[ https://issues.apache.org/jira/browse/FLINK-13547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916313#comment-16916313 ] Kevin Zhang commented on FLINK-13547: - Thanks [~docete] and [~jark], we will have a look at the related issues. > Verify and correct string function's semantic for Blink planner > --- > > Key: FLINK-13547 > URL: https://issues.apache.org/jira/browse/FLINK-13547 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhenghua Gao >Assignee: Zhenghua Gao >Priority: Critical > Labels: pull-request-available > Fix For: 1.9.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Currently many string builtin functions in blink planner follow hive/spark > semantics, which should keep compatible with old planner. And some > non-standard functions(Blink planner intros) should be removed. > * concat/concat_ws function (null treatment) > * substring function (follow calcite/flink) > * from_base64 should return string not binary > * intro truncate function to blink planner > * uuid should be no-argument (remove the one-argument version) > * length/jsonvalue/keyvalue/substr (non-standard function should be removed) > * md5/sha1/sha2/sha224/sha256/sha384/sha512(remove the two-arguments version) > * ascii (operand type should beSqlTypeFamily.CHARACTER) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9510: [FLINK-13807][tests] Read file with UTF-8 charset in TestBaseUtils.getResultReader
flinkbot edited a comment on issue #9510: [FLINK-13807][tests] Read file with UTF-8 charset in TestBaseUtils.getResultReader URL: https://github.com/apache/flink/pull/9510#issuecomment-523878082 ## CI report: * 7e7bc9b213b2fadb8759449b65e5be19b11aa829 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124206563) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126 ## CI report: * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124206586) * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124227220) * f1399e04785d2afecacfc1962d68bbf98416f68b : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124676036) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner
flinkbot edited a comment on issue #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner URL: https://github.com/apache/flink/pull/9511#issuecomment-523878126 ## CI report: * 11889a791a35b40ef0a4f5456da889cec0ebf990 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124206586) * 71db2f4cc05e2ceacf09d2c17562bbda37cd906a : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124227220) * f1399e04785d2afecacfc1962d68bbf98416f68b : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] JingsongLi commented on a change in pull request #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner
JingsongLi commented on a change in pull request #9511: [FLINK-13356][table][docs] Add documentation for TopN and Deduplication in blink planner URL: https://github.com/apache/flink/pull/9511#discussion_r317879792 ## File path: docs/dev/table/sql.md ## @@ -813,6 +813,222 @@ LIMIT 3 {% top %} +### Top-N + +TopN is used to calculate the maximum/minimum N records in a stream. It can be flexibly completed based on OVER window aggregation. The grammar is shown as below: + +Top-N queries ask for the N smallest or largest values ordered by columns. Both smallest and largest values sets are considered Top-N queries. Top-N queries are useful in cases where the need is to display only the N bottom-most or the N top- +most records from batch/stream table on a condition. This result set can be used for further analysis. + +Flink uses the combination of a OVER window clause and a filter condition to express a Top-N query. With the power of OVER window `PARTITION BY` clause, Flink also supports per group Top-N. For example, the top five products per category that have the maximum sales in realtime. Top-N queries are supported for SQL on batch and streaming tables. + +The following shows the syntax of the TOP-N statement: + +{% highlight sql %} +SELECT [column_list] +FROM ( + SELECT [column_list], + ROW_NUMBER() OVER ([PARTITION BY col1[, col2...]] + ORDER BY col1 [asc|desc][, col2 [asc|desc]...]) AS rownum + FROM table_name) +WHERE rownum <= N [AND conditions] +{% endhighlight %} + +**Parameter Specification:** + +- `ROW_NUMBER()`: Assigns an unique, sequential number to each row, starting with one, according to the ordering of rows within the partition. Currently, we only support `ROW_NUMBER` as the over window function. In the future, we will support `RANK()` and `DENSE_RANK()`. +- `PARTITION BY col1[, col2...]`: Specifies the partition columns. Each partition will have a Top-N result. +- `ORDER BY col1 [asc|desc][, col2 [asc|desc]...]`: Specifies the ordering columns. The ordering directions can be different on different columns. +- `WHERE rownum <= N`: The `rownum <= N` is required for Flink to recognize this query is a Top-N query. The N represents the N smallest or largest records will be retained. +- `[AND conditions]`: It is free to add other conditions in the where clause, but the other conditions can only be combined with `rownum <= N` using `AND` conjunction. + +Attention Flink SQL will sort the input data stream according to the order key, so if the top N records have been changed, the changed ones will be sent as retraction/update records to downstream. In addition, if the top N records need to be stored in external storage, the result table must have the same primary key with the Top-N query. By default, the primary key of Top-N query is the combination of partition columns + rownum column. + +The following examples show how to specify SQL queries with Top-N on streaming tables. This is an example to get "the top five products per category that have the maximum sales in realtime" we mentioned above. + + + +{% highlight java %} +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +StreamTableEnvironment tableEnv = TableEnvironment.getTableEnvironment(env); + +// ingest a DataStream from an external source +DataStream> ds = env.addSource(...); +// register the DataStream as table "ShopSales" +tableEnv.registerDataStream("ShopSales", ds, "product_id, category, product_name, sales"); + +// select top-5 products per category which have the maximum sales. +Table result1 = tableEnv.sqlQuery( + "SELECT * " + + "FROM (" + + " SELECT *," + + " ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) as row_num" + + " FROM ShopSales)" + + "WHERE row_num <= 5"); +{% endhighlight %} + + + +{% highlight scala %} +val env = StreamExecutionEnvironment.getExecutionEnvironment +val tableEnv = TableEnvironment.getTableEnvironment(env) + +// read a DataStream from an external source +val ds: DataStream[(String, String, String, Long)] = env.addSource(...) +// register the DataStream under the name "ShopSales" +tableEnv.registerDataStream("ShopSales", ds, 'product_id, 'category, 'product_name, 'sales) + + +// select top-5 products per category which have the maximum sales. +val result1 = tableEnv.sqlQuery( +""" + |SELECT * + |FROM ( + | SELECT *, + | ROW_NUMBER() OVER (PARTITION BY category ORDER BY sales DESC) as row_num + | FROM ShopSales) + |WHERE row_num <= 5 +""".stripMargin) +{% endhighlight %} + + + + No Ranking Output Optimization + +As described above, the `rownum` field will be written into the result table as one field of the primary key, which may lead to a lot of records being written to the result table. For example, when the record (say `product-1001`) of ranking 9 is updated and its rank is upgraded to 1, all the records from
[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024 ## CI report: * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123648363) * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123684855) * 7d75f94fed9684c020c7673a57431c79e2f0a543 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/124674615) * e21b3e5b91328bd7e9ce28fb8df9b848730b319a : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124675311) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024 ## CI report: * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123648363) * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123684855) * 7d75f94fed9684c020c7673a57431c79e2f0a543 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124674615) * e21b3e5b91328bd7e9ce28fb8df9b848730b319a : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] xuefuz commented on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line…
xuefuz commented on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line… URL: https://github.com/apache/flink/pull/8738#issuecomment-525118500 As a reference, Hive has been doing, which seems quite simple. https://github.com/apache/hive/blob/8190d2be7b7165effa62bd21b7d60ef81fb0e4af/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java#L384 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9538: [hotfix][docs] Fix typo
flinkbot edited a comment on issue #9538: [hotfix][docs] Fix typo URL: https://github.com/apache/flink/pull/9538#issuecomment-525097982 ## CI report: * 07c04045c445ccd5d844ffc733d01ac7adbf88ee : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124669351) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhijiangW commented on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
zhijiangW commented on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor URL: https://github.com/apache/flink/pull/9477#issuecomment-525116894 Thanks for the confirmation and I would merge it after travis green. @pnowojski This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13765) Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
[ https://issues.apache.org/jira/browse/FLINK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-13765: - Description: In StreamTwoInputSelectableProcessor there are three fields \{InputSelectable, InputSelection, availableInputsMask} to be used together for the function of selecting next available input index. >From design aspect, these fields can be abstracted into a separate component >called TwoInputSelectionHandler, which is passed into the constructor of >StreamTwoInputSelectableProcessor as a final field. So the internal >implementation details of TwoInputSelectionHandler is hidden from processor >view which only needs to interact with exposed methods from selection handler. Another tiny benefit is that we make the StreamTwoInputSelectableProcessor a bit because two methods are removed from it. was: In StreamTwoInputSelectableProcessor there are three fields \{InputSelectable, InputSelection, availableInputsMask} to be used together for the function of selecting next available input index. >From design aspect, these fields can be abstracted into a separate component >called TwoInputSelectionHandler, which is passed into the constructor of >StreamTwoInputSelectableProcessor as a final field. So the internal >implementation details of TwoInputSelectionHandler is hidden from processor >view which only needs to interact with exposed methods from selection handler. Another tiny benefits is that we make the StreamTwoInputSelectableProcessor a bit because two methods are removed from it. > Introduce TwoInputSelectionHandler for selecting input in > StreamTwoInputSelectableProcessor > --- > > Key: FLINK-13765 > URL: https://issues.apache.org/jira/browse/FLINK-13765 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In StreamTwoInputSelectableProcessor there are three fields > \{InputSelectable, InputSelection, availableInputsMask} to be used together > for the function of selecting next available input index. > From design aspect, these fields can be abstracted into a separate component > called TwoInputSelectionHandler, which is passed into the constructor of > StreamTwoInputSelectableProcessor as a final field. So the internal > implementation details of TwoInputSelectionHandler is hidden from processor > view which only needs to interact with exposed methods from selection handler. > Another tiny benefit is that we make the StreamTwoInputSelectableProcessor a > bit because two methods are removed from it. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
flinkbot edited a comment on issue #9477: [FLINK-13765][task] Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor URL: https://github.com/apache/flink/pull/9477#issuecomment-522360024 ## CI report: * 71e7fae9a212e19c83e1fe6d656de49a39334aa2 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123648363) * 6dd3cd3345d771fee15314914869d0e692350689 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123684855) * 7d75f94fed9684c020c7673a57431c79e2f0a543 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13765) Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
[ https://issues.apache.org/jira/browse/FLINK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-13765: - Description: In StreamTwoInputSelectableProcessor there are three fields \{InputSelectable, InputSelection, availableInputsMask} to be used together for the function of selecting next available input index. >From design aspect, these fields can be abstracted into a separate component >called TwoInputSelectionHandler, which is passed into the constructor of >StreamTwoInputSelectableProcessor as a final field. So the internal >implementation details of TwoInputSelectionHandler is hidden from processor >view which only needs to interact with exposed methods from selection handler. Another tiny benefits is that we make the StreamTwoInputSelectableProcessor a bit because two methods are removed from it. was: In StreamTwoInputSelectableProcessor there are three fields \{InputSelectable, InputSelection, availableInputsMask} to be used together for the function of selecting next available input index. It would bring two problems: * From design aspect, these fields should be abstracted into a separate component and passed into StreamTwoInputSelectableProcessor. * inputSelector.nextSelection() is called while processing elements in StreamTwoInputSelectableProcessor, so it is the blocker for integrating task input/output for both StreamOneInputProcessor/StreamTwoInputSelectableProcessor later. > Introduce TwoInputSelectionHandler for selecting input in > StreamTwoInputSelectableProcessor > --- > > Key: FLINK-13765 > URL: https://issues.apache.org/jira/browse/FLINK-13765 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In StreamTwoInputSelectableProcessor there are three fields > \{InputSelectable, InputSelection, availableInputsMask} to be used together > for the function of selecting next available input index. > From design aspect, these fields can be abstracted into a separate component > called TwoInputSelectionHandler, which is passed into the constructor of > StreamTwoInputSelectableProcessor as a final field. So the internal > implementation details of TwoInputSelectionHandler is hidden from processor > view which only needs to interact with exposed methods from selection handler. > Another tiny benefits is that we make the StreamTwoInputSelectableProcessor a > bit because two methods are removed from it. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#issuecomment-522053886 ## CI report: * 009da497d7c551ba854dc7ed8fa658f2acd6d6ce : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123529451) * 14be3c57f3d625f3f9bbb96b7f74b6b0116fed3c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124588576) * c43a462c92c6c5c145a6964aa5390a82b894fa4e : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124673290) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13765) Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
[ https://issues.apache.org/jira/browse/FLINK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-13765: - Summary: Introduce TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor (was: Introduce the TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor) > Introduce TwoInputSelectionHandler for selecting input in > StreamTwoInputSelectableProcessor > --- > > Key: FLINK-13765 > URL: https://issues.apache.org/jira/browse/FLINK-13765 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In StreamTwoInputSelectableProcessor there are three fields > \{InputSelectable, InputSelection, availableInputsMask} to be used together > for the function of selecting next available input index. It would bring two > problems: > * From design aspect, these fields should be abstracted into a separate > component and passed into StreamTwoInputSelectableProcessor. > * inputSelector.nextSelection() is called while processing elements in > StreamTwoInputSelectableProcessor, so it is the blocker for integrating task > input/output for both > StreamOneInputProcessor/StreamTwoInputSelectableProcessor later. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] Lemonjing commented on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
Lemonjing commented on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#issuecomment-525112986 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13765) Introduce the TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor
[ https://issues.apache.org/jira/browse/FLINK-13765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhijiang updated FLINK-13765: - Summary: Introduce the TwoInputSelectionHandler for selecting input in StreamTwoInputSelectableProcessor (was: Introduce the InputSelectionHandler for selecting next input in StreamTwoInputSelectableProcessor) > Introduce the TwoInputSelectionHandler for selecting input in > StreamTwoInputSelectableProcessor > --- > > Key: FLINK-13765 > URL: https://issues.apache.org/jira/browse/FLINK-13765 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In StreamTwoInputSelectableProcessor there are three fields > \{InputSelectable, InputSelection, availableInputsMask} to be used together > for the function of selecting next available input index. It would bring two > problems: > * From design aspect, these fields should be abstracted into a separate > component and passed into StreamTwoInputSelectableProcessor. > * inputSelector.nextSelection() is called while processing elements in > StreamTwoInputSelectableProcessor, so it is the blocker for integrating task > input/output for both > StreamOneInputProcessor/StreamTwoInputSelectableProcessor later. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
flinkbot edited a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#issuecomment-522053886 ## CI report: * 009da497d7c551ba854dc7ed8fa658f2acd6d6ce : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/123529451) * 14be3c57f3d625f3f9bbb96b7f74b6b0116fed3c : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124588576) * c43a462c92c6c5c145a6964aa5390a82b894fa4e : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#discussion_r317871779 ## File path: flink-scala/src/test/java/org/apache/flink/api/scala/operators/ScalaCsvOutputFormatTest.java ## @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.scala.operators; + +import org.apache.flink.api.common.io.FileOutputFormat; +import org.apache.flink.core.fs.FileSystem; +import org.apache.flink.core.fs.Path; + +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.List; + +import scala.Tuple3; + +import static org.junit.Assert.fail; + +/** + * Tests for {@link ScalaCsvOutputFormat}. + */ +public class ScalaCsvOutputFormatTest { + + private String path = null; + + @Before + public void createFile() throws Exception { + path = File.createTempFile("scala_csv_output_test_file", ".csv").getAbsolutePath(); + } + + @Test + public void testNullAllow() throws Exception { + final ScalaCsvOutputFormat> csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path)); + try { + csvOutputFormat.setWriteMode(FileSystem.WriteMode.OVERWRITE); + csvOutputFormat.setOutputDirectoryMode(FileOutputFormat.OutputDirectoryMode.PARONLY); + csvOutputFormat.setAllowNullValues(true); + csvOutputFormat.open(0, 1); + csvOutputFormat.writeRecord(new Tuple3<>("One", null, 8)); + } finally { + csvOutputFormat.close(); + } + java.nio.file.Path p = Paths.get(path); + Assert.assertTrue(Files.exists(p)); + List lines = Files.readAllLines(Paths.get(path), StandardCharsets.UTF_8); + Assert.assertEquals(1, lines.size()); + Assert.assertEquals("One,,8", lines.get(0)); + } + + @Test + public void testNullDisallowOnDefault() throws Exception { + final ScalaCsvOutputFormat> csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path)); Review comment: Thanks for the review @StephanEwen . I updated the codes for addressing the comments. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#discussion_r317868102 ## File path: flink-scala/src/test/java/org/apache/flink/api/scala/operators/ScalaCsvOutputFormatTest.java ## @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.scala.operators; + +import org.apache.flink.api.common.io.FileOutputFormat; +import org.apache.flink.core.fs.FileSystem; +import org.apache.flink.core.fs.Path; + +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.List; + +import scala.Tuple3; + +import static org.junit.Assert.fail; + +/** + * Tests for {@link ScalaCsvOutputFormat}. + */ +public class ScalaCsvOutputFormatTest { + + private String path = null; + + @Before + public void createFile() throws Exception { + path = File.createTempFile("scala_csv_output_test_file", ".csv").getAbsolutePath(); + } + + @Test + public void testNullAllow() throws Exception { + final ScalaCsvOutputFormat> csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path)); + try { + csvOutputFormat.setWriteMode(FileSystem.WriteMode.OVERWRITE); + csvOutputFormat.setOutputDirectoryMode(FileOutputFormat.OutputDirectoryMode.PARONLY); + csvOutputFormat.setAllowNullValues(true); + csvOutputFormat.open(0, 1); + csvOutputFormat.writeRecord(new Tuple3<>("One", null, 8)); + } finally { + csvOutputFormat.close(); + } + java.nio.file.Path p = Paths.get(path); + Assert.assertTrue(Files.exists(p)); + List lines = Files.readAllLines(Paths.get(path), StandardCharsets.UTF_8); + Assert.assertEquals(1, lines.size()); + Assert.assertEquals("One,,8", lines.get(0)); + } + + @Test + public void testNullDisallowOnDefault() throws Exception { + final ScalaCsvOutputFormat> csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path)); Review comment: > Minor: I would recommend to put these four lines (creation and configuration of format) into a setup helper method. yes, thanks for your review, update already. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Lemonjing removed a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
Lemonjing removed a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#issuecomment-522054960 @flinkbot attention @yanghua @buptljy This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10995) Copy intermediate serialization results only once for broadcast mode
[ https://issues.apache.org/jira/browse/FLINK-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916286#comment-16916286 ] zhijiang commented on FLINK-10995: -- [~ykt836], this Jira is for improving the serialization stack in RecordWriter, which could bring benefit for both PIPELINED and BLOCKING partitions at the moment. In detail the intermediate serialization data buffer would need copy only once for all the subpartitions. But for the blocking subpartitions, the same referenced target buffer would still be persisted into separate file as you mentioned. In theory it can be persisted into only one file for all the subpartitions, but it is not in the scope of this jira. Further the same data could be transported only once in the network stack if many consumers are in the same TaskManager. We might further focus on these improvements future, but might not in release-1.10. But this Jira would be covered in release-1.10. > Copy intermediate serialization results only once for broadcast mode > > > Key: FLINK-10995 > URL: https://issues.apache.org/jira/browse/FLINK-10995 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.8.0 >Reporter: zhijiang >Assignee: zhijiang >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The emitted records from operator would be firstly serialized into > intermediate bytes array in {{RecordSerializer}}, then copy the intermediate > results into target buffers for different sub partitions. For broadcast > mode, the same intermediate results would be copied as many times as the > number of sub partitions, and this would affect the performance seriously in > large scale jobs. > We can copy to only one target buffer which would be shared by all the sub > partitions to reduce the overheads. For emitting latency marker in broadcast > mode, we should flush the previous shared target buffers first, and then > request a new buffer for the target sub partition to send latency marker. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-9477) Support SQL 2016 JSON functions in Flink SQL
[ https://issues.apache.org/jira/browse/FLINK-9477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916283#comment-16916283 ] Jark Wu commented on FLINK-9477: Hi [~x1q1j1], it would be great if you can summarize the features/functions that we actually want to support first in a little design document and how to integrate with Flink (especially Table API). Because you are the contributor of JSON functions in Calcite, maybe you know better about the functions. Once we have the design and is accepted by the community, then we can create subtasks to cooperate together. > Support SQL 2016 JSON functions in Flink SQL > > > Key: FLINK-9477 > URL: https://issues.apache.org/jira/browse/FLINK-9477 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Shuyi Chen >Assignee: Shuyi Chen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] Lemonjing removed a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
Lemonjing removed a comment on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#issuecomment-525108063 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Lemonjing commented on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
Lemonjing commented on issue #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#issuecomment-525108063 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13862) Remove or rewrite Execution Plan docs
[ https://issues.apache.org/jira/browse/FLINK-13862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916282#comment-16916282 ] TisonKun commented on FLINK-13862: -- After a close look I would prefer removing it at least for now. Because it mainly talk about how to use the removed plan visualizer. For {{Web Interface}} part, it only talk about {{web.submit.enable}} which is also documented at [this page|https://ci.apache.org/projects/flink/flink-docs-master/ops/config.html]. We can anyway add back a document on how to preview a plan depend on our discussion about client api. It should cover this topic. > Remove or rewrite Execution Plan docs > - > > Key: FLINK-13862 > URL: https://issues.apache.org/jira/browse/FLINK-13862 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.9.0 >Reporter: Stephan Ewen >Priority: Blocker > Fix For: 1.10.0, 1.9.1 > > > The *Execution Plans* section is totally outdated and refers to the old > {{tools/planVisalizer.html}} file that has been removed for two years. > https://ci.apache.org/projects/flink/flink-docs-master/dev/execution_plans.html -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#discussion_r317868102 ## File path: flink-scala/src/test/java/org/apache/flink/api/scala/operators/ScalaCsvOutputFormatTest.java ## @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.scala.operators; + +import org.apache.flink.api.common.io.FileOutputFormat; +import org.apache.flink.core.fs.FileSystem; +import org.apache.flink.core.fs.Path; + +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.List; + +import scala.Tuple3; + +import static org.junit.Assert.fail; + +/** + * Tests for {@link ScalaCsvOutputFormat}. + */ +public class ScalaCsvOutputFormatTest { + + private String path = null; + + @Before + public void createFile() throws Exception { + path = File.createTempFile("scala_csv_output_test_file", ".csv").getAbsolutePath(); + } + + @Test + public void testNullAllow() throws Exception { + final ScalaCsvOutputFormat> csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path)); + try { + csvOutputFormat.setWriteMode(FileSystem.WriteMode.OVERWRITE); + csvOutputFormat.setOutputDirectoryMode(FileOutputFormat.OutputDirectoryMode.PARONLY); + csvOutputFormat.setAllowNullValues(true); + csvOutputFormat.open(0, 1); + csvOutputFormat.writeRecord(new Tuple3<>("One", null, 8)); + } finally { + csvOutputFormat.close(); + } + java.nio.file.Path p = Paths.get(path); + Assert.assertTrue(Files.exists(p)); + List lines = Files.readAllLines(Paths.get(path), StandardCharsets.UTF_8); + Assert.assertEquals(1, lines.size()); + Assert.assertEquals("One,,8", lines.get(0)); + } + + @Test + public void testNullDisallowOnDefault() throws Exception { + final ScalaCsvOutputFormat> csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path)); Review comment: > Minor: I would recommend to put these four lines (creation and configuration of format) into a setup helper method. yes, thanks for your review, update already. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close
Lemonjing commented on a change in pull request #9467: [FLINK-9941][ScalaAPI] Flush in ScalaCsvOutputFormat before close URL: https://github.com/apache/flink/pull/9467#discussion_r317623406 ## File path: flink-scala/src/test/java/org/apache/flink/api/scala/operators/ScalaCsvOutputFormatTest.java ## @@ -0,0 +1,94 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.api.scala.operators; + +import org.apache.flink.api.common.io.FileOutputFormat; +import org.apache.flink.core.fs.FileSystem; +import org.apache.flink.core.fs.Path; + +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.List; + +import scala.Tuple3; + +import static org.junit.Assert.fail; + +/** + * Tests for {@link ScalaCsvOutputFormat}. + */ +public class ScalaCsvOutputFormatTest { + + private String path = null; + + @Before + public void createFile() throws Exception { + path = File.createTempFile("scala_csv_output_test_file", ".csv").getAbsolutePath(); + } + + @Test + public void testNullAllow() throws Exception { + final ScalaCsvOutputFormat> csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path)); + try { + csvOutputFormat.setWriteMode(FileSystem.WriteMode.OVERWRITE); + csvOutputFormat.setOutputDirectoryMode(FileOutputFormat.OutputDirectoryMode.PARONLY); + csvOutputFormat.setAllowNullValues(true); + csvOutputFormat.open(0, 1); + csvOutputFormat.writeRecord(new Tuple3<>("One", null, 8)); + } finally { + csvOutputFormat.close(); + } + java.nio.file.Path p = Paths.get(path); + Assert.assertTrue(Files.exists(p)); + List lines = Files.readAllLines(Paths.get(path), StandardCharsets.UTF_8); + Assert.assertEquals(1, lines.size()); + Assert.assertEquals("One,,8", lines.get(0)); + } + + @Test + public void testNullDisallowOnDefault() throws Exception { + final ScalaCsvOutputFormat> csvOutputFormat = new ScalaCsvOutputFormat<>(new Path(path)); Review comment: thanks for your review, update already. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wuchong commented on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line…
wuchong commented on issue #8738: [FLINK-12845][sql-client] Execute multiple statements in command line… URL: https://github.com/apache/flink/pull/8738#issuecomment-525106464 > I agree the current PR is way too hacky. However, I don't see the need of extending SQL parser just to support the multi-command line either as there can be commands that's not SQL at all. As far as I can see, we need to split the command line by ';' (certainly take care of possible escaped ';'), pre-process each segment to determine the command type (which we are already doing), and invoke sql parser only if the command is a sql command. The problem is how to split command by semi-colon. Using regex is a hack way and we have met many issues because of it in the internal blink. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9510: [FLINK-13807][tests] Read file with UTF-8 charset in TestBaseUtils.getResultReader
flinkbot edited a comment on issue #9510: [FLINK-13807][tests] Read file with UTF-8 charset in TestBaseUtils.getResultReader URL: https://github.com/apache/flink/pull/9510#issuecomment-523878082 ## CI report: * 7e7bc9b213b2fadb8759449b65e5be19b11aa829 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/124206563) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services