[RESULT] [VOTE] Release 1.10.0, release candidate #3

I'm happy to announce that we have unanimously approved this release. There are 16 approving votes, 5 of which are binding: * Kurt Young (binding) * Jark Wu (binding) * Kostas Kloudas (binding) * Thomas Weise (binding) * Jincheng Sun (binding) * Aihua Li (non-binding) * Zhu Zhu (non-binding) *

[jira] [Created] (FLINK-15987) SELECT 1.0e0 / 0.0e0 throws NumberFormatException

Caizhi Weng created FLINK-15987: --- Summary: SELECT 1.0e0 / 0.0e0 throws NumberFormatException Key: FLINK-15987 URL: https://issues.apache.org/jira/browse/FLINK-15987 Project: Flink Issue Type:

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

5.x seems to have a lot of users, is the 6.x completely compatible with 5.x ~ Best, Danny Chan 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz ，写道： > Hi all, > > As described in this https://issues.apache.org/jira/browse/FLINK-11720 > ticket our elasticsearch 5.x connector does not work out of the box

[jira] [Created] (FLINK-15986) support setting or changing session properties in Flink SQL

Bowen Li created FLINK-15986: Summary: support setting or changing session properties in Flink SQL Key: FLINK-15986 URL: https://issues.apache.org/jira/browse/FLINK-15986 Project: Flink Issue

[jira] [Created] (FLINK-15985) offload runtime params from DDL to table hints in DML/queries

Bowen Li created FLINK-15985: Summary: offload runtime params from DDL to table hints in DML/queries Key: FLINK-15985 URL: https://issues.apache.org/jira/browse/FLINK-15985 Project: Flink Issue

[jira] [Created] (FLINK-15984) support hive stream table sink

Bowen Li created FLINK-15984: Summary: support hive stream table sink Key: FLINK-15984 URL: https://issues.apache.org/jira/browse/FLINK-15984 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-15983) add native reader for Hive parquet files

Bowen Li created FLINK-15983: Summary: add native reader for Hive parquet files Key: FLINK-15983 URL: https://issues.apache.org/jira/browse/FLINK-15983 Project: Flink Issue Type: New Feature

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

+1 (non-binding) - Verified the signature and checksum - Pip installed the package successfully: pip install apache-flink-1.9.2.tar.gz - Run word count example successfully. Regards, Dian > 在 2020年2月11日，上午11:44，jincheng sun 写道： > > > +1 (binding) > > - Install the PyFlink by `pip install`

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

+1 (binding) - Install the PyFlink by `pip install` [SUCCESS] - Run word_count in both command line and IDE [SUCCESS] Best, Jincheng Wei Zhong 于2020年2月11日周二 上午11:17写道： > Hi, > > Thanks for driving this, Jincheng. > > +1 (non-binding) > > - Verified signatures and checksums. > - Verified

[jira] [Created] (FLINK-15982) 'Quickstarts Java nightly end-to-end test' is failed on travis

Jark Wu created FLINK-15982: --- Summary: 'Quickstarts Java nightly end-to-end test' is failed on travis Key: FLINK-15982 URL: https://issues.apache.org/jira/browse/FLINK-15982 Project: Flink Issue

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

Hi, Thanks for driving this, Jincheng. +1 (non-binding) - Verified signatures and checksums. - Verified README.md and setup.py. - Run `pip install apache-flink-1.9.2.tar.gz` in Python 2.7.15 and Python 3.7.5 successfully. - Start local pyflink shell in Python 2.7.15 and Python 3.7.5 via

Re: [VOTE] FLIP-55: Introduction of a Table API Java Expression DSL

+1 for this. I have some minor comments: - I'm +1 to use $ in both Java and Scala API. - I'm +1 to use lit(), Spark also provides lit() function to create a literal value. - Is it possible to have `isGreater` instead of `isGreaterThan` and `isGreaterOrEqual` instead of `isGreaterThanOrEqualTo` in

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (non-binding) - Build from source - Run mesos e2e tests(including unmerged heap state backend and rocks state backend case) Best, Yangze Guo On Tue, Feb 11, 2020 at 10:08 AM Yu Li wrote: > > Thanks for the reminder Patrick! According to the release process [1] we > will publish the

[jira] [Created] (FLINK-15981) Control the direct memory in FileChannelBoundedData.FileBufferReader

Jingsong Lee created FLINK-15981: Summary: Control the direct memory in FileChannelBoundedData.FileBufferReader Key: FLINK-15981 URL: https://issues.apache.org/jira/browse/FLINK-15981 Project: Flink

[jira] [Created] (FLINK-15980) The notFollowedBy in the end of GroupPattern may be ignored

shuai.xu created FLINK-15980: Summary: The notFollowedBy in the end of GroupPattern may be ignored Key: FLINK-15980 URL: https://issues.apache.org/jira/browse/FLINK-15980 Project: Flink Issue

[jira] [Created] (FLINK-15979) Fix the merged count is not accurate in CountDistinctWithMerge

Jark Wu created FLINK-15979: --- Summary: Fix the merged count is not accurate in CountDistinctWithMerge Key: FLINK-15979 URL: https://issues.apache.org/jira/browse/FLINK-15979 Project: Flink Issue

Re: [DISCUSS] Support scalar vectorized Python UDF in PyFlink

Hi Jingsong, You're right. I have updated the FLIP which reflects this. Thanks, Dian > 在 2020年2月11日，上午10:03，Jingsong Li 写道： > > Hi Dian and Jincheng, > > Thanks for your explanation. Think again. Maybe most of users don't want to > modify this parameters. > We all realize that "batch.size"

Re: [VOTE] FLIP-55: Introduction of a Table API Java Expression DSL

Hi Dawid, Thanks for driving. - adding $ in scala api looks good to me. - Just a question, what should be expected to java.lang.Object? literal object or expression? So the Object is the grammatical sugar of literal? Best, Jingsong Lee On Mon, Feb 10, 2020 at 9:40 PM Timo Walther wrote: > +1

Re: [VOTE] Release 1.10.0, release candidate #3

Thanks for the reminder Patrick! According to the release process [1] we will publish the Dockerfiles *after* the RC voting passed, to finalize the release. I have created FLINK-15978 [2] and prepared a PR [3] for it, will follow up after we conclude our RC vote. Thanks. Best Regards, Yu [1]

Re: [DISCUSS] Support scalar vectorized Python UDF in PyFlink

Hi Dian and Jincheng, Thanks for your explanation. Think again. Maybe most of users don't want to modify this parameters. We all realize that "batch.size" should be a larger value, so "bundle.size" must also be increased. Now the default value of "bundle.size" is only 1000. I think you can update

[jira] [Created] (FLINK-15978) Publish Docerfiles for release 1.10.0

Yu Li created FLINK-15978: - Summary: Publish Docerfiles for release 1.10.0 Key: FLINK-15978 URL: https://issues.apache.org/jira/browse/FLINK-15978 Project: Flink Issue Type: Task

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

Hi Rong, That's great! Looking forward to your feedback. Thanks, Hequn On Tue, Feb 11, 2020 at 1:06 AM Rong Rong wrote: > Yes. I think the argument is fairly valid - we can always adjust the API > in the future, in fact most of the APIs are labeled publicEvolving at this > moment. > I was

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (binding) - Download the source code [SUCCESS] - Build the source release with Scala 2.12 and Scala 2.11 [SUCCESS] - Checked/verified signatures and hashes [SUCCESS] - Pip install PyFlink by `pip3 install apache-flink-1.10.0.tar.gz` [SUCCESS] - Run a word_count.py both in command line and

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (non-binding) - source doesn't contain binary files & build from source - run against stability test cases - manually verify YARN CLI & deployment work as expected Best, tison. Thomas Weise 于2020年2月11日周二 上午2:48写道： > +1 (binding) > > - verified signatures and hashes > - rebased internal

[jira] [Created] (FLINK-15977) Update pull request template to include Kubernetes as deployment candidates

Zili Chen created FLINK-15977: - Summary: Update pull request template to include Kubernetes as deployment candidates Key: FLINK-15977 URL: https://issues.apache.org/jira/browse/FLINK-15977 Project: Flink

[jira] [Created] (FLINK-15976) Build snapshots for Scala 2.12

Leonid Ilyevsky created FLINK-15976: --- Summary: Build snapshots for Scala 2.12 Key: FLINK-15976 URL: https://issues.apache.org/jira/browse/FLINK-15976 Project: Flink Issue Type: Improvement

Total recovery time estimation after checkpoint recovery

??Hi, I am working with Flink at the moment and am interested in knowing how one could estimate the Total Recovery Time for an application after checkpoint recovery. What I am specifically interested in is knowing the time needed for the recovery of the state + the catch-up phase (since the

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (binding) - verified signatures and hashes - rebased internal deploy to rc3 and verified that previously reported issues are resolved Thank you for addressing these issues promptly! Thomas On Fri, Feb 7, 2020 at 1:54 PM Gary Yao wrote: > Hi everyone, > Please review and vote on the

[jira] [Created] (FLINK-15975) Use LinkedHashMap for deterministic iterations

testfixer0 created FLINK-15975: -- Summary: Use LinkedHashMap for deterministic iterations Key: FLINK-15975 URL: https://issues.apache.org/jira/browse/FLINK-15975 Project: Flink Issue Type:

Re: [DISCUSS] Include flink-ml-api and flink-ml-lib in opt

Yes. I think the argument is fairly valid - we can always adjust the API in the future, in fact most of the APIs are labeled publicEvolving at this moment. I was only trying to provide the info, that the interfaces in flink-ml-api might change in the near future, for others when voting. In fact,

Re: [VOTE] Release 1.10.0, release candidate #3

Hi all, +1 (binding) - Built Flink locally - Tested quickstart by writing simple, WordCount-like streaming jobs - Submitted them to Yarn both "per-job" and "session" mode - used the yarn-session CLI to start/stop sessions Thanks a lot Gary and Yu for managing the release. Cheers, Kostas On

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

+1 from dropping old versions because of jar hells etc. However, in the wild there are still a lot of 2.x clusters and definitely 5.x clusters that are having a hard time upgrading. We know because we assist those on a daily basis. It is very easy to create an HTTP based connector that works with

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

+1 for dropping them, this stuff is quite old by now. On 10.02.20 15:04, Benchao Li wrote: +1 for dropping 2.x - 5.x. FYI currently only 6.x and 7.x ES Connectors are supported by table api. Flavio Pompermaier 于2020年2月10日周一 下午10:03写道： +1 for dropping all Elasticsearch connectors < 6.x On

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

+1 for dropping 2.x - 5.x. FYI currently only 6.x and 7.x ES Connectors are supported by table api. Flavio Pompermaier 于2020年2月10日周一 下午10:03写道： > +1 for dropping all Elasticsearch connectors < 6.x > > On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz > wrote: > > > Hi all, > > > > As described

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

+1 for dropping all Elasticsearch connectors < 6.x On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz wrote: > Hi all, > > As described in this https://issues.apache.org/jira/browse/FLINK-11720 > ticket our elasticsearch 5.x connector does not work out of the box on > some systems and requires a

[jira] [Created] (FLINK-15974) Support to use the Python UDF directly in the Python Table API

Dian Fu created FLINK-15974: --- Summary: Support to use the Python UDF directly in the Python Table API Key: FLINK-15974 URL: https://issues.apache.org/jira/browse/FLINK-15974 Project: Flink Issue

Re: [DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

Thanks for starting this discussion! +1 to drop both On Mon, Feb 10, 2020 at 2:45 PM Dawid Wysakowicz wrote: > Hi all, > > As described in this https://issues.apache.org/jira/browse/FLINK-11720 > ticket our elasticsearch 5.x connector does not work out of the box on > some systems and requires

[DISCUSS] Drop connectors for Elasticsearch 2.x and 5.x

Hi all, As described in this https://issues.apache.org/jira/browse/FLINK-11720 ticket our elasticsearch 5.x connector does not work out of the box on some systems and requires a version bump. This also happens for our e2e. We cannot bump the version in es 5.x connector, because 5.x connector

[jira] [Created] (FLINK-15973) Optimize the execution plan where it refers the Python UDF result field in the where clause

Dian Fu created FLINK-15973: --- Summary: Optimize the execution plan where it refers the Python UDF result field in the where clause Key: FLINK-15973 URL: https://issues.apache.org/jira/browse/FLINK-15973

Re: [VOTE] FLIP-55: Introduction of a Table API Java Expression DSL

+1 for this. It will also help in making a TableEnvironment.fromElements() possible and reduces technical debt. One entry point of TypeInformation less in the API. Regards, Timo On 10.02.20 08:31, Dawid Wysakowicz wrote: Hi all, I wanted to resurrect the thread about introducing a Java

[jira] [Created] (FLINK-15972) Add Python building blocks to make sure the basic functionality of Python TableFunction could work

Huang Xingbo created FLINK-15972: Summary: Add Python building blocks to make sure the basic functionality of Python TableFunction could work Key: FLINK-15972 URL:

[jira] [Created] (FLINK-15971) Adjust the default value of bundle size and bundle time

Dian Fu created FLINK-15971: --- Summary: Adjust the default value of bundle size and bundle time Key: FLINK-15971 URL: https://issues.apache.org/jira/browse/FLINK-15971 Project: Flink Issue Type:

Re: [VOTE] Release 1.10.0, release candidate #3

Now that [FLINK-15828] Integrate docker-flink/docker-flink into Flink release process is complete, the Dockerfiles for 1.10.0 can be published as part of the release process. @Gary/@Yu: please let me know if you have any questions regarding the

[jira] [Created] (FLINK-15970) Optimize the Python UDF execution to only serialize the value

Dian Fu created FLINK-15970: --- Summary: Optimize the Python UDF execution to only serialize the value Key: FLINK-15970 URL: https://issues.apache.org/jira/browse/FLINK-15970 Project: Flink Issue

[jira] [Created] (FLINK-15969) Do not multiplex both PersistedValue and PersistedTable with a single MapState state handle

Tzu-Li (Gordon) Tai created FLINK-15969: --- Summary: Do not multiplex both PersistedValue and PersistedTable with a single MapState state handle Key: FLINK-15969 URL:

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (non-binding) - build from source - start standalone cluster, and run some examples - played with sql-client with some simple sql - run tests in IDE - run some sqls running in 1.9 internal version with 1.10.0-rc3, seems 1.10 behaves well. Xintong Song 于2020年2月10日周一 下午8:13写道： > +1

[jira] [Created] (FLINK-15968) LegacyTypeInfoDataTypeConverter should support conversion between BINARY/VARBINARY and BYTE_PRIMITIVE_ARRAY_TYPE_INFO

Zhenghua Gao created FLINK-15968: Summary: LegacyTypeInfoDataTypeConverter should support conversion between BINARY/VARBINARY and BYTE_PRIMITIVE_ARRAY_TYPE_INFO Key: FLINK-15968 URL:

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (non-binding) - build from source (with tests) - run nightly e2e tests - run example jobs in local/standalone/yarn setups - play around with memory configurations on local/standalone/yarn setups Thank you~ Xintong Song On Mon, Feb 10, 2020 at 7:55 PM Jark Wu wrote: > +1 (binding) > > -

Re: [DISCUSS] FLINK-15831: Add Docker image publication to release documentation

Thanks, Chesnay. I've updated the release guide with the new step. -- Patrick On Mon, Feb 10, 2020 at 12:34 PM Chesnay Schepler wrote: > @Patrick You should have the required wiki permissions now. > > On 10/02/2020

[jira] [Created] (FLINK-15967) Examples use nightly kafka connector

Zili Chen created FLINK-15967: - Summary: Examples use nightly kafka connector Key: FLINK-15967 URL: https://issues.apache.org/jira/browse/FLINK-15967 Project: Flink Issue Type: Improvement

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (binding) - build the source release with Scala 2.12 and Scala 2.11 successfully - checked/verified signatures and hashes - started cluster for both Scala 2.11 and 2.12, ran examples, verified web ui and log output, nothing unexpected - started cluster and run some e2e sql queries, all of them

[jira] [Created] (FLINK-15966) Capture the call stack of RPC ask() calls.

Stephan Ewen created FLINK-15966: Summary: Capture the call stack of RPC ask() calls. Key: FLINK-15966 URL: https://issues.apache.org/jira/browse/FLINK-15966 Project: Flink Issue Type:

Re: [DISCUSS] FLINK-15831: Add Docker image publication to release documentation

@Patrick You should have the required wiki permissions now. On 10/02/2020 12:12, Patrick Lucas wrote: Thanks for the feedback. Could someone (Ufuk or Till?) grant me access to to the FLINK space Confluence so I can make these changes? My Confluence username is plucas. Thanks, Patrick On Mon,

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (binding) - verified signatures and checksums - start local cluster, run some examples, randomly play some sql with sql client, no suspicious error/warn log found in log files - repeat above operation with both scala 2.11 and 2.12 binary Best, Kurt On Mon, Feb 10, 2020 at 6:38 PM Yang Wang

Re: [VOTE] Release 1.10.0, release candidate #3

+1 non-binding I’ve re-done the manual EMR tests, including running some examples. Piotrek > On 10 Feb 2020, at 11:37, Yang Wang wrote: > > +1 non-binding > > > - Building from source with all tests skipped > - Build a custom image with 1.10-rc3 > - K8s tests >* Deploy a standalone

[VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the PyFlink version 1.9.2, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * the official

Re: [DISCUSS] FLINK-15831: Add Docker image publication to release documentation

Thanks for the feedback. Could someone (Ufuk or Till?) grant me access to to the FLINK space Confluence so I can make these changes? My Confluence username is plucas. Thanks, Patrick On Mon, Feb 10, 2020 at 9:54 AM Ufuk Celebi wrote: > +1 to have the README as source of truth and link to the

Re: [VOTE] Release 1.10.0, release candidate #3

+1 non-binding - Building from source with all tests skipped - Build a custom image with 1.10-rc3 - K8s tests * Deploy a standalone session cluster on K8s and submit multiple jobs * Deploy a standalone per-job cluster * Deploy a native session cluster on K8s with/without HA

Re: [DISCUSS] FLIP-75: Flink Web UI Improvement Proposal

Hi all I have drafted the docs of top-level FLIPs for the individual changes proposed in FLIP-75. will update it to the cwiki page and start the voting stage soon if there is no objection. - FLIP-98: Better Back Pressure Detection

Re: [DISCUSS] FLINK-15831: Add Docker image publication to release documentation

+1 to have the README as source of truth and link to the repository from the Wiki page. – Ufuk On Mon, Feb 10, 2020 at 3:48 AM Yang Wang wrote: > +1 to make flink-docker repository self-contained, including the document. > And others refer > to it. > > > Best, > Yang > > Till Rohrmann

Re: [DISCUSS] Support scalar vectorized Python UDF in PyFlink

Hi Jincheng, Hequn & Jingsong, Thanks a lot for your suggestions. I have created FLIP-97[1] for this feature. > One little suggestion: maybe it would be nice if we can add some performance explanation in the document? (I just very curious:)) Thanks for the suggestion. I have updated the design

Re: [VOTE] Release 1.10.0, release candidate #3

Hi, +1 (non-binding) Thanks for driving this, Gary & Yu. There is an unfriendly error here: "OutOfMemoryError: Direct buffer memory" in FileChannelBoundedData$FileBufferReader. It forces our batch users to configure "taskmanager.memory.task.off-heap.size" in production jobs. And users are

Re: [DISCUSS] Support scalar vectorized Python UDF in PyFlink

Hi Jingsong, Thanks for your feedback! I would like to share my thoughts regarding the follows question: >> - Can we only configure one parameter and calculate another automatically? For example, if we just want to "pipeline", "bundle.size" is twice as much as "batch.size", is this work? I

Re: [Question] Anyone know where I can find performance test result?

Hi Xu, Do you mean the flink-benchmark [1] project for micro benchmark? If so, you could find the daily run result from our code-speed center [2]. Please note that the result of the benchmark is hardware sensitive, so you may find your local result different from the online ones. Best Regards,

[jira] [Created] (FLINK-15965) DATE literal issue in static partition spec

Rui Li created FLINK-15965: -- Summary: DATE literal issue in static partition spec Key: FLINK-15965 URL: https://issues.apache.org/jira/browse/FLINK-15965 Project: Flink Issue Type: Bug

Re: [VOTE] Release 1.10.0, release candidate #3

+1 (non-binding) for rc3 - Built from source (release tag) - Downloaded binaries and verified Hive catalog/connector by manually running some queries against Hive tables On Mon, Feb 10, 2020 at 2:28 PM Congxian Qiu wrote: > +1 (non-binding) for rc3 > > - build source successfully