[RESULT] [VOTE] FLIP-88: PyFlink User-Defined Function Resource Management

2019-12-05 Thread Dian Fu
Hi everyone, Thanks for the discussion and votes. So far we have received 4 approving votes, 3 of which are binding and there is no -1 votes: * Jincheng (binding) * Hequn (binding) * Jark (binding) * Jingsong (non-binding) Therefore, I'm happy to announce that FLIP-88 has been accepted.

[jira] [Created] (FLINK-15091) JoinITCase.testFullJoinWithNonEquiJoinPred failed in travis

2019-12-05 Thread Kurt Young (Jira)
Kurt Young created FLINK-15091: -- Summary: JoinITCase.testFullJoinWithNonEquiJoinPred failed in travis Key: FLINK-15091 URL: https://issues.apache.org/jira/browse/FLINK-15091 Project: Flink

[jira] [Created] (FLINK-15090) Reverse the dependency from flink-streaming-java to flink-client

2019-12-05 Thread Zili Chen (Jira)
Zili Chen created FLINK-15090: - Summary: Reverse the dependency from flink-streaming-java to flink-client Key: FLINK-15090 URL: https://issues.apache.org/jira/browse/FLINK-15090 Project: Flink

Re: [VOTE] FLIP-88: PyFlink User-Defined Function Resource Management

2019-12-05 Thread Jark Wu
Thanks for driving this Dian! The FLIP looks good to me. +1 (binding) Best, Jark On Thu, 5 Dec 2019 at 16:44, Hequn Cheng wrote: > +1 (binding) > > Best, > Hequn > > On Thu, Dec 5, 2019 at 4:41 PM jincheng sun > wrote: > > > +1(binding) > > > > Best, > > Jincheng > > > > Jingsong Li

[jira] [Created] (FLINK-15089) Pulsar Catalog

2019-12-05 Thread Yijie Shen (Jira)
Yijie Shen created FLINK-15089: -- Summary: Pulsar Catalog Key: FLINK-15089 URL: https://issues.apache.org/jira/browse/FLINK-15089 Project: Flink Issue Type: Sub-task Components:

[jira] [Created] (FLINK-15088) "misc" build fail on travis-ci

2019-12-05 Thread Dezhi Cai (Jira)
Dezhi Cai created FLINK-15088: - Summary: "misc" build fail on travis-ci Key: FLINK-15088 URL: https://issues.apache.org/jira/browse/FLINK-15088 Project: Flink Issue Type: Bug

Re: [DISCUSS] Drop Heap Backend Synchronous snapshots

2019-12-05 Thread Yun Tang
+1 from my side for I did not see any real benefits if using synchronous snapshots. Moreover, I think we should also remove the support of synchronous snapshots in DefaultOpeatorStateBackend and deprecate the config state.backend.async Best Yun Tang On 12/5/19, 8:06 PM, "Stephan Ewen"

Re: [DISCUSS] PyFlink User-Defined Function Resource Management

2019-12-05 Thread Dian Fu
Hi Jingsong, Appreciated for your sharing. It's very helpful as the Python operator will take the similar way. Thanks, Dian > 在 2019年12月6日,上午11:12,Jingsong Li 写道: > > Hi Dian, > > After [1] and [2], in the batch sql world, we will: > - [2] In client/compile side: we use memory weight

Re: [DISCUSS] PyFlink User-Defined Function Resource Management

2019-12-05 Thread Jingsong Li
Hi Dian, After [1] and [2], in the batch sql world, we will: - [2] In client/compile side: we use memory weight request memory for Transformation. - [1] In runtime side: we use memory fraction to compute memory size and allocate in StreamOperator. For your information. [1]

[jira] [Created] (FLINK-15087) JobManager is forced to shutdown JVM due to temporary loss of zookeeper connection

2019-12-05 Thread Abdul Qadeer (Jira)
Abdul Qadeer created FLINK-15087: Summary: JobManager is forced to shutdown JVM due to temporary loss of zookeeper connection Key: FLINK-15087 URL: https://issues.apache.org/jira/browse/FLINK-15087

[jira] [Created] (FLINK-15086) JobCluster may not be shutdown if executeAsync called

2019-12-05 Thread Zili Chen (Jira)
Zili Chen created FLINK-15086: - Summary: JobCluster may not be shutdown if executeAsync called Key: FLINK-15086 URL: https://issues.apache.org/jira/browse/FLINK-15086 Project: Flink Issue Type:

[jira] [Created] (FLINK-15085) History server overview page fails loading because of web-submit feature not in the config

2019-12-05 Thread chaganti spurthi (Jira)
chaganti spurthi created FLINK-15085: Summary: History server overview page fails loading because of web-submit feature not in the config Key: FLINK-15085 URL:

Re: [DISCUSS] Migrate build infrastructure from Travis CI to Azure Pipelines

2019-12-05 Thread Robert Metzger
Thanks for your comments Yun. If there's strong support for idea 2, it would actually make my life easier: the migration would be easier to do. I also noticed that the uploads to transfer.sh were broken, but this should be fixed in the "rmetzger.flink" builds (coming from rmetzger/flink). The

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Trevor Grant
If anyone wants to do a drive by PR link to vendor docs, just let them. Just have the menu set up so it doesn't get too visually busy? E.g.: Deployment & Ops -> Cluster Deployment -> 3rd Party Vendors -> [ GCP, AWS, Azure, Ververica, Oracle Cloud, IBM, Lightbend, Crazy Trevo's House of Streaming,

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Seth Wiesman
@chesnay I'm not sure, however, I don't know what we could do to improve the situation that wouldn't effectively be copying those vendors docs into our own. One option would be to do exactly that, but then I feel like we are committing to tracking changes on those systems and I just don't know

Re: [DISCUSS] Migrate build infrastructure from Travis CI to Azure Pipelines

2019-12-05 Thread Yun Tang
Hi Robert Really exciting to see this new more powerful CI tool to get rid of the 50 minutes limit of traivs-CI free account. After reading the wiki, I support idea 2 of AZP-setup version-2. However, after I dig into some failing builds at https://dev.azure.com/rmetzger/Flink/_build , I

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Robert Metzger
The bounce rate of these pages is not particularly bad. On Thu, Dec 5, 2019 at 3:48 PM Trevor Grant wrote: > You can infer that by looking at the "bounce rate" eg someone gets to the > page, looks at it, realizes its trash and clicks "back". > > > > On Thu, Dec 5, 2019 at 8:46 AM Chesnay

[jira] [Created] (FLINK-15084) Let MemoryManager allocate and track shared memory resources

2019-12-05 Thread Stephan Ewen (Jira)
Stephan Ewen created FLINK-15084: Summary: Let MemoryManager allocate and track shared memory resources Key: FLINK-15084 URL: https://issues.apache.org/jira/browse/FLINK-15084 Project: Flink

Re: [DISCUSS] Migrate build infrastructure from Travis CI to Azure Pipelines

2019-12-05 Thread Robert Metzger
I've created a first draft of my plans in the wiki: https://cwiki.apache.org/confluence/display/FLINK/%5Bpreview%5D+Azure+Pipelines. I'm looking forward to your comments. On Thu, Dec 5, 2019 at 12:37 PM Robert Metzger wrote: > Thank you all for the positive feedback. I will start putting

[jira] [Created] (FLINK-15083) Fix connectors only pick physical fields from TableSchema

2019-12-05 Thread Jark Wu (Jira)
Jark Wu created FLINK-15083: --- Summary: Fix connectors only pick physical fields from TableSchema Key: FLINK-15083 URL: https://issues.apache.org/jira/browse/FLINK-15083 Project: Flink Issue Type:

[jira] [Created] (FLINK-15082) Mesos App Master does not respect taskmanager.memory.total-process.size

2019-12-05 Thread Gary Yao (Jira)
Gary Yao created FLINK-15082: Summary: Mesos App Master does not respect taskmanager.memory.total-process.size Key: FLINK-15082 URL: https://issues.apache.org/jira/browse/FLINK-15082 Project: Flink

[jira] [Created] (FLINK-15081) Translate "Concepts & Common API" page of Table API into Chinese

2019-12-05 Thread Steve OU (Jira)
Steve OU created FLINK-15081: Summary: Translate "Concepts & Common API" page of Table API into Chinese Key: FLINK-15081 URL: https://issues.apache.org/jira/browse/FLINK-15081 Project: Flink

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Trevor Grant
You can infer that by looking at the "bounce rate" eg someone gets to the page, looks at it, realizes its trash and clicks "back". On Thu, Dec 5, 2019 at 8:46 AM Chesnay Schepler wrote: > Question now is whether the numbers are so low because the docs aren't > required or because they are so

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Chesnay Schepler
Question now is whether the numbers are so low because the docs aren't required or because they are so bad. On 05/12/2019 14:26, Robert Metzger wrote: I just checked GA: All numbers are for the last month, independent of the Flink version: aws.html: 918 pageviews mapr_setup.html: 108

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Trevor Grant
Based on that the only "maybe" is AWS, and I just googled it and AWS docs pretty well own the first page (flink.apache.org shows up 3/4 the way down on first page behind AWS docs). I revise my "vote" to +1 to dump the whole thing. On Thu, Dec 5, 2019 at 7:26 AM Robert Metzger wrote: > I just

[VOTE] Release 1.8.3, release candidate #3

2019-12-05 Thread Hequn Cheng
Hi everyone, Please review and vote on the release candidate #3 for the version 1.8.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1],

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Robert Metzger
I just checked GA: All numbers are for the last month, independent of the Flink version: aws.html: 918 pageviews mapr_setup.html: 108 pageviews gce_setup.html: 256 pageviews Some other deployment-related pages for reference: yarn_setup: 4687 cluster: 4284 kubernetes: 3428 On Thu, Dec 5,

[jira] [Created] (FLINK-15080) Deploy OSS filesystem to maven central

2019-12-05 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-15080: Summary: Deploy OSS filesystem to maven central Key: FLINK-15080 URL: https://issues.apache.org/jira/browse/FLINK-15080 Project: Flink Issue Type:

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Trevor Grant
Same as Ufuk (non-binding) In general, docs pages are great "first commits" to leave out there as newb-issues. Also though, worth checking how often people use the page (e.g. GA) 3rd option: add a `.bu` to AWS/GCE pages and open a JIRA ticket to fix them (and put a readme explaining why they

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Ufuk Celebi
+1 to drop the MapR page. For the other two I'm +0. I fully agree that the linked AWS and GCE pages are in bad shape and don't relate to a component developed by the community. Do we have any numbers from Google Analytics on how popular those pages are? If they are somewhat popular, I would

[jira] [Created] (FLINK-15079) BashJavaUtilsTest fails when running in a clean directory

2019-12-05 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-15079: Summary: BashJavaUtilsTest fails when running in a clean directory Key: FLINK-15079 URL: https://issues.apache.org/jira/browse/FLINK-15079 Project: Flink

[jira] [Created] (FLINK-15078) Report Python version exceptions in time

2019-12-05 Thread sunjincheng (Jira)
sunjincheng created FLINK-15078: --- Summary: Report Python version exceptions in time Key: FLINK-15078 URL: https://issues.apache.org/jira/browse/FLINK-15078 Project: Flink Issue Type:

[DISCUSS] Drop Heap Backend Synchronous snapshots

2019-12-05 Thread Stephan Ewen
Hi all! I am wondering if there is any case for retaining the option to make synchronous snapshots on the heap statebackend. Is anyone using that? Or could we clean that code up and remove it? Best, Stephan

[jira] [Created] (FLINK-15077) Support Semi/Anti LookupJoin in Blink planner

2019-12-05 Thread Jing Zhang (Jira)
Jing Zhang created FLINK-15077: -- Summary: Support Semi/Anti LookupJoin in Blink planner Key: FLINK-15077 URL: https://issues.apache.org/jira/browse/FLINK-15077 Project: Flink Issue Type: Bug

Re: [DISCUSS] Migrate build infrastructure from Travis CI to Azure Pipelines

2019-12-05 Thread Robert Metzger
Thank you all for the positive feedback. I will start putting together a page in the wiki. @Jark: Azure Pipelines provides a free services, that is even better than what Travis provides for free: 10 parallel builds with 6 hours timeouts. @Chesnay: I will answer your questions in the

Re: [DISCUSS] Migrate build infrastructure from Travis CI to Azure Pipelines

2019-12-05 Thread Arvid Heise
+1 I had good experiences with Azure pipelines in the past. On Thu, Dec 5, 2019 at 11:35 AM Aljoscha Krettek wrote: > +1 > > Thanks for the effort! The tooling seems to be quite a bit nicer and I > like that we can grow by adding more machines. > > Best, > Aljoscha > > > On 5. Dec 2019, at

[jira] [Created] (FLINK-15076) Source thread should be interrupted during the Task cancellation

2019-12-05 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-15076: -- Summary: Source thread should be interrupted during the Task cancellation Key: FLINK-15076 URL: https://issues.apache.org/jira/browse/FLINK-15076 Project: Flink

[jira] [Created] (FLINK-15075) Can not run "create table" in IT Case with expressions in TableSchema because the copied ParameterScope can not been seen

2019-12-05 Thread Danny Chen (Jira)
Danny Chen created FLINK-15075: -- Summary: Can not run "create table" in IT Case with expressions in TableSchema because the copied ParameterScope can not been seen Key: FLINK-15075 URL:

Re: [DISCUSS] Migrate build infrastructure from Travis CI to Azure Pipelines

2019-12-05 Thread Aljoscha Krettek
+1 Thanks for the effort! The tooling seems to be quite a bit nicer and I like that we can grow by adding more machines. Best, Aljoscha > On 5. Dec 2019, at 03:18, Jark Wu wrote: > > +1 for Azure pipeline because it promises better performance. > > However, I have 2 concerns: > > 1) Travis

Re: [DISCUSS] Add N-Ary Stream Operator

2019-12-05 Thread Piotr Nowojski
Hi, Thanks for the clarifications Jingsong. Indeed, if chaining doesn’t work with multiple output right now (doesn’t it?), that’s also a good future story. Re Kurt: I think this pattern could be easily handled if those two joins are implemented as a single 3 input operator, that internally is

Re: [VOTE] Improve the Pyflink command line options (Adjustment to FLIP-78)

2019-12-05 Thread Aljoscha Krettek
+1 (binding) > On 5. Dec 2019, at 10:58, Hequn Cheng wrote: > > +1 (binding) > > Best, > Hequn > > On Thu, Dec 5, 2019 at 5:43 PM jincheng sun > wrote: > >> +1 (binding) >> >> Best, >> Jincheng >> >> Dian Fu 于2019年12月5日周四 上午11:14写道: >> >>> +1 (non-binding) >>> >>> Regards, >>> Dian >>>

Re: [VOTE] Improve the Pyflink command line options (Adjustment to FLIP-78)

2019-12-05 Thread Hequn Cheng
+1 (binding) Best, Hequn On Thu, Dec 5, 2019 at 5:43 PM jincheng sun wrote: > +1 (binding) > > Best, > Jincheng > > Dian Fu 于2019年12月5日周四 上午11:14写道: > > > +1 (non-binding) > > > > Regards, > > Dian > > > > > 在 2019年12月5日,上午11:11,Jark Wu 写道: > > > > > > +1 (binding) > > > > > > Best, > > >

Re: [DISCUSS] Drop vendor specific deployment documentation.

2019-12-05 Thread Till Rohrmann
If the community cannot manage to keep the vendor-specific documentation up to date, then I believe it is better to drop it. Hence +1 for the proposal. Cheers, Till On Tue, Dec 3, 2019 at 3:12 PM Aljoscha Krettek wrote: > +1 > > Best, > Aljoscha > > > On 2. Dec 2019, at 18:38, Konstantin Knauf

Re: [VOTE] Improve the Pyflink command line options (Adjustment to FLIP-78)

2019-12-05 Thread jincheng sun
+1 (binding) Best, Jincheng Dian Fu 于2019年12月5日周四 上午11:14写道: > +1 (non-binding) > > Regards, > Dian > > > 在 2019年12月5日,上午11:11,Jark Wu 写道: > > > > +1 (binding) > > > > Best, > > Jark > > > > On Thu, 5 Dec 2019 at 10:45, Wei Zhong wrote: > > > >> Hi all, > >> > >> According to our previous

Re: [DISCUSS] Voting from apache.org addresses

2019-12-05 Thread Till Rohrmann
+1 for the proposal. Cheers, Till On Wed, Dec 4, 2019 at 12:43 PM Dian Fu wrote: > Hi Dawid, > > Thanks for the reply. Counting all the votes from non apache addresses as > non-binding makes sense. Just as Jark mentioned, we can always remind the > committer/PMC to vote again to use the apache

[jira] [Created] (FLINK-15074) Connection timed out, Standalone

2019-12-05 Thread gameking (Jira)
gameking created FLINK-15074: Summary: Connection timed out, Standalone Key: FLINK-15074 URL: https://issues.apache.org/jira/browse/FLINK-15074 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-15073) sql client fails to run same query multiple times

2019-12-05 Thread Kurt Young (Jira)
Kurt Young created FLINK-15073: -- Summary: sql client fails to run same query multiple times Key: FLINK-15073 URL: https://issues.apache.org/jira/browse/FLINK-15073 Project: Flink Issue Type:

Re: [DISCUSS] Add N-Ary Stream Operator

2019-12-05 Thread Jingsong Li
Kurt mentioned a very interesting thing, If we want to better performance to read simultaneously, To this pattern: We need to control not only the read order of inputs, but also the outputs of endInput. In this case, HashAggregate can only call its real endInput after the input of build2 is

Re: [VOTE] FLIP-88: PyFlink User-Defined Function Resource Management

2019-12-05 Thread Hequn Cheng
+1 (binding) Best, Hequn On Thu, Dec 5, 2019 at 4:41 PM jincheng sun wrote: > +1(binding) > > Best, > Jincheng > > Jingsong Li 于2019年12月3日周二 下午7:30写道: > > > +1 (non-binding) > > > > Best, > > Jingsong Lee > > > > On Mon, Dec 2, 2019 at 5:30 PM Dian Fu wrote: > > > > > Hi Jingsong, > > > > >

[jira] [Created] (FLINK-15072) Hijack executeAsync instead of execute in context environment

2019-12-05 Thread Zili Chen (Jira)
Zili Chen created FLINK-15072: - Summary: Hijack executeAsync instead of execute in context environment Key: FLINK-15072 URL: https://issues.apache.org/jira/browse/FLINK-15072 Project: Flink

Re: [DISCUSS] Add N-Ary Stream Operator

2019-12-05 Thread Kurt Young
During implementing n-ary input operator in table, please keep this pattern in mind: Build1 ---+ | +---> HshJoin1 --—> HashAgg ---+ | | Probe1 ---+ +---> HashJoin2

Re: [VOTE] FLIP-88: PyFlink User-Defined Function Resource Management

2019-12-05 Thread jincheng sun
+1(binding) Best, Jincheng Jingsong Li 于2019年12月3日周二 下午7:30写道: > +1 (non-binding) > > Best, > Jingsong Lee > > On Mon, Dec 2, 2019 at 5:30 PM Dian Fu wrote: > > > Hi Jingsong, > > > > It's fine. :) Appreciated the comments! > > > > I have replied you in the discussion thread as I also think

Re: [DISCUSS] Add N-Ary Stream Operator

2019-12-05 Thread Jingsong Li
Hi Piotr, > a) two input operator X -> one input operator Y -> one input operator Z (ALLOWED) > b) n input operator X -> one input operator Y -> one input operator Z (ALLOWED) > c) two input operator X -> one input operator Y -> two input operator Z (NOT ALLOWED as a single chain) NOT ALLOWED to

[jira] [Created] (FLINK-15071) YARN vcore capacity check can not pass when use large slotPerTaskManager

2019-12-05 Thread huweihua (Jira)
huweihua created FLINK-15071: Summary: YARN vcore capacity check can not pass when use large slotPerTaskManager Key: FLINK-15071 URL: https://issues.apache.org/jira/browse/FLINK-15071 Project: Flink

[jira] [Created] (FLINK-15070) Supplement the case of bounded blocking partition for benchmark

2019-12-05 Thread zhijiang (Jira)
zhijiang created FLINK-15070: Summary: Supplement the case of bounded blocking partition for benchmark Key: FLINK-15070 URL: https://issues.apache.org/jira/browse/FLINK-15070 Project: Flink

[jira] [Created] (FLINK-15069) Supplement the compression case for benchmark

2019-12-05 Thread zhijiang (Jira)
zhijiang created FLINK-15069: Summary: Supplement the compression case for benchmark Key: FLINK-15069 URL: https://issues.apache.org/jira/browse/FLINK-15069 Project: Flink Issue Type: Task

[jira] [Created] (FLINK-15068) Disable RocksDB's local LOG by default

2019-12-05 Thread Nico Kruber (Jira)
Nico Kruber created FLINK-15068: --- Summary: Disable RocksDB's local LOG by default Key: FLINK-15068 URL: https://issues.apache.org/jira/browse/FLINK-15068 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-15067) Pass execution configuration from TableEnvironment to StreamExecutionEnvironment

2019-12-05 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-15067: Summary: Pass execution configuration from TableEnvironment to StreamExecutionEnvironment Key: FLINK-15067 URL: https://issues.apache.org/jira/browse/FLINK-15067

[jira] [Created] (FLINK-15066) Cannot run multiple `insert into csvTable values ()`

2019-12-05 Thread Kurt Young (Jira)
Kurt Young created FLINK-15066: -- Summary: Cannot run multiple `insert into csvTable values ()` Key: FLINK-15066 URL: https://issues.apache.org/jira/browse/FLINK-15066 Project: Flink Issue Type:

Re: [DISCUSS] Drop Kafka 0.8/0.9

2019-12-05 Thread Zhenghua Gao
+1 for dropping. *Best Regards,* *Zhenghua Gao* On Thu, Dec 5, 2019 at 11:08 AM Dian Fu wrote: > +1 for dropping them. > > Just FYI: there was a similar discussion few months ago [1]. > > [1] >