Re: [DISCUSS] Releasing Flink 1.8.2

Thanks for the work Jincheng! I have moved remaining major issues to 1.8.3 except FLINK-13586. Hi @Aljoscha Krettek , is that possible to merge FLINK-13586 today? Best, Jark On Wed, 4 Sep 2019 at 10:47, jincheng sun wrote: > Thanks for the udpate Jark! > > I have add the new version 1.8.3

Re: [DISCUSS] META-FLIP: Sticking (or not) to a strict FLIP voting process

Thanks for the summary, Yu. Hi all, Just want to loop back on this thread. The META-FLIP actually does not change much in the current FLIP process. It did add a few more details that are currently not explicitly written down in the FLIP wiki page. More specifically: 1. In general FLIPs should

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi, I agree with Xuefu that the main controversial points are mainly the two places. My thoughts on them: 1) Determinism of referencing Hive built-in functions. We can either remove Hive built-in functions from ambiguous function resolution and require users to use special syntax for their

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Thank you for your wonderful points. I like timo's proposal to enrich built-in functions to flexible function modules (For example, the financial model is useful to bank system). But I agree with bowen, I don't think hive functions deserves be a function module. I think all function modules

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

>From what I have seen, there are a couple of focal disagreements: 1. Resolution order: temp function --> flink built-in function --> catalog function vs flink built-in function --> temp function -> catalog function. 2. "External" built-in functions: how to treat built-in functions in external

[jira] [Created] (FLINK-13951) Unable to call limit without sort for batch mode

Jeff Zhang created FLINK-13951: -- Summary: Unable to call limit without sort for batch mode Key: FLINK-13951 URL: https://issues.apache.org/jira/browse/FLINK-13951 Project: Flink Issue Type:

Re: [DISCUSS] Releasing Flink 1.8.2

Thanks for the udpate Jark! I have add the new version 1.8.3 in JIRA, could you please remark the JIRAs(Such as FLINK-13689) which we do not want merge into the 1.8.2 release :) You are right, I think FLINK-13586 is better to be contained in 1.8.2 release! Thanks, Jincheng Jark Wu

[jira] [Created] (FLINK-13950) should report an error if hive table partition keys are not in the last schema fields

Hongtao Zhang created FLINK-13950: - Summary: should report an error if hive table partition keys are not in the last schema fields Key: FLINK-13950 URL: https://issues.apache.org/jira/browse/FLINK-13950

Re: [DISCUSS] Releasing Flink 1.8.2

Hi all, I am very happy to say that all the blockers and critical issues for release 1.8.2 have been resolved! Great thanks to everyone who contribute to the release. I hope to create the first RC on Sep 05, at 10:00 UTC+8. If you find some other blocker issues for 1.8.2, please let me know

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Thanks Timo & Bowen for the feedback. Bowen was right, my proposal is the same as Bowen's. But after thinking about it, I'm currently lean to Timo's suggestion. The reason is backward compatibility. If we follow Bowen's approach, let's say we first find function in Flink's built-in functions, and

[jira] [Created] (FLINK-13949) Delete deduplicating JobVertexDetailsInfo.VertexTaskDetail

lining created FLINK-13949: -- Summary: Delete deduplicating JobVertexDetailsInfo.VertexTaskDetail Key: FLINK-13949 URL: https://issues.apache.org/jira/browse/FLINK-13949 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

Hi Timo, Dawid and Aljoscha, Thanks for clarifying the goals. It is very helpful to understand the motivation here. It would be great to add them to the FLIP wiki. I agree that the current FLIP design achieves the two goals it wants to achieve. But I am trying to see is if the current approach

Re: [DISCUSS] FLIP-55: Introduction of a Table API Java Expression DSL

Thanks for putting together the proposal @Timo and sorry for joining the discussion thread late. I also share the same thought with Fabian on the ease-of-use front. However I was wondering if we need to start the expression design with them? One thing I can think of is: is it possible to support

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi Yun, Since I was the main driver behind FLINK-9641 and FLINK-9168, let me try to add more context on this. FLINK-9641 and FLINK-9168 was created for bringing Pulsar as source and sink for Flink. The integration was done with Flink 1.6.0. We sent out pull requests about a year ago and we ended

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi all, Thanks for the feedback. Just a kindly reminder that the [Proposal] section in the google doc was updated, please take a look first and let me know if you have more questions. On Tue, Sep 3, 2019 at 4:57 PM Bowen Li wrote: > Hi Timo, > > Re> 1) We should not have the restriction "hive

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi Timo, Re> 1) We should not have the restriction "hive built-in functions can only > be used when current catalog is hive catalog". Switching a catalog > should only have implications on the cat.db.object resolution but not > functions. It would be quite convinient for users to use Hive

[jira] [Created] (FLINK-13948) Fix loss of state for Identical Windows merging after initial merge

Scott Waller created FLINK-13948: Summary: Fix loss of state for Identical Windows merging after initial merge Key: FLINK-13948 URL: https://issues.apache.org/jira/browse/FLINK-13948 Project: Flink

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi Jingsong, Re> 1.Hive built-in functions is an intermediate solution. So we should > not introduce interfaces to influence the framework. To make > Flink itself more powerful, we should implement the functions > we need to add. Yes, please see the doc. Re> 2.Non-flink built-in functions are

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi Kurt, Re: > What I want to propose is we can merge #3 and #4, make them both under >"catalog" concept, by extending catalog function to make it have ability to >have built-in catalog functions. Some benefits I can see from this approach: >1. We don't have to introduce new concept like external

[jira] [Created] (FLINK-13947) Check Hive shim serialization in Hive UDF wrapper classes and test coverage

Xuefu Zhang created FLINK-13947: --- Summary: Check Hive shim serialization in Hive UDF wrapper classes and test coverage Key: FLINK-13947 URL: https://issues.apache.org/jira/browse/FLINK-13947 Project:

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi Yijie I can see that Pulsar becomes more and more popular recently and very glad to see more people willing to contribute to Flink ecosystem. Before any further discussion, would you please give some explanation of the relationship between this thread to current existing JIRAs of pulsar

Re: Potential block size issue with S3 binary files

Hi Arvid, Thanks for following up… > On Sep 2, 2019, at 3:09 AM, Arvid Heise wrote: > > Hi Ken, > > that's indeed a very odd issue that you found. I had a hard time to connect > block size with S3 in the beginning and had to dig into the code. I still > cannot fully understand why you got two

[jira] [Created] (FLINK-13946) Remove deactivated JobSession-related code.

Kostas Kloudas created FLINK-13946: -- Summary: Remove deactivated JobSession-related code. Key: FLINK-13946 URL: https://issues.apache.org/jira/browse/FLINK-13946 Project: Flink Issue Type:

Re: [DISCUSS] Reducing build times

+1 for Azure Pipelines, had very good experiences in the past with it and the open source and payment models are much better. The upcoming Github CI/CD seems also like a promising alternative, but from the first looks, it seems like the small brother of Azure Pipeline. So, any effort going into

Re: [DISCUSS] Reducing build times

Hi all, I wanted to give a short update on this: - Arvid, Aljoscha and I have started working on a Gradle PoC, currently working on making all modules compile and test with Gradle. We've also identified some problematic areas (shading being the most obvious one) which we will analyse as part of

Re: [DISCUSS] Repository split

Thanks a lot for your summary Chesnay. I agree with you that we have no consensus in the community for splitting up the repository immediately, and I agree with you that we should have a separate discussion about reducing the build time (which is already making good progress). Also, I will keep

Re: [DISCUSS] Releasing Flink 1.8.2

Yes, I will do that Jark! Kostas On Tue, Sep 3, 2019 at 4:19 PM Jark Wu wrote: > > Thanks Kostas for the quick fixing. > > However, I find that FLINK-13940 still target to 1.8.2 as a blocker. If I > understand correctly, FLINK-13940 is aiming for a nicer and better solution > in the future. >

[jira] [Created] (FLINK-13945) Vendor-repos Maven profile doesn't exist in flink-shaded

Elise Ramé created FLINK-13945: -- Summary: Vendor-repos Maven profile doesn't exist in flink-shaded Key: FLINK-13945 URL: https://issues.apache.org/jira/browse/FLINK-13945 Project: Flink Issue

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

Answers inline. On 9/3/19 4:01 PM, Till Rohrmann wrote: How so? Does your REPL add the generated classes to the system class loader? I assume the system class loader is used to load the Flink classes. No, it does not. It cannot on JDK >= 9 (or would have to hack into

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi Kurt, it should not affect the functions and operations we currently have in SQL. It just categorizes the available built-in functions. It is kind of an orthogonal concept to the catalog API but built-in functions deserve this special kind of treatment. CatalogFunction still fits perfectly

Re: [ANNOUNCE] Kinesis connector becomes part of Flink releases

Good news! Thanks for your efforts, Bowen! Best, Vino Yu Li 于2019年9月2日周一 上午6:04写道： > Great to know, thanks for the efforts Bowen! > > And I believe it worth a release note in the original JIRA, wdyt? Thanks. > > Best Regards, > Yu > > > On Sat, 31 Aug 2019 at 11:01, Bowen Li wrote: > >> Hi

Re: State of FLIPs

Hi Chesnay, I created FLIP-44 two months ago. Originally, the discussion of local aggregation feature happened in the ML thread.[1] After many people agree with this feature, I created an umbrella issue and split into some subtasks and opened one PR. However, @Aljoscha Krettek thinks it's a big

Re: [DISCUSS] Releasing Flink 1.8.2

Thanks Kostas for the quick fixing. However, I find that FLINK-13940 still target to 1.8.2 as a blocker. If I understand correctly, FLINK-13940 is aiming for a nicer and better solution in the future. So should we update the fixVersion of FLINK-13940? Best, Jark On Tue, 3 Sep 2019 at 21:33,

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

How so? Does your REPL add the generated classes to the system class loader? I assume the system class loader is used to load the Flink classes. Ideally, what you would like to have is the option to provide the parent class loader which is used load user code to the LocalEnvironment. This one

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

On the other hand, if you say, that the contract of LocalEnvironment is to execute as if it had all classes on its class loader, then it currently breaks this contract. :-) Jan On 9/3/19 3:45 PM, Jan Lukavský wrote: Hi Till, hmm, that sounds it might work. I would have to incorporate this

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

Hi Till, hmm, that sounds it might work. I would have to incorporate this (either as default, or on demand) into Apache Beam. Would you see any disadvantages of this approach? Would you suggest to make this default behavior for local beam FlinkRunner? I can introduce a configuration option

Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

I see the problem Jan. What about the following proposal: Instead of using the LocalEnvironment for local tests you always use the RemoteEnvironment but when testing it locally you spin up a MiniCluster in the same process and initialize the RemoteEnvironment with `MiniCluster#getRestAddress`.

Re: [DISCUSS] Releasing Flink 1.8.2

Thanks for waiting! A fix for FLINK-13940 has been merged on 1.8, 1.9 and the master under FLINK-13941. Cheers, Kostas On Tue, Sep 3, 2019 at 11:25 AM jincheng sun wrote: > > +1 FLINK-13940 is a > blocker, due to loss data is very important

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Does this only affect the functions and operations we currently have in SQL and have no effect on tables, right? Looks like this is an orthogonal concept with Catalog? If the answer are both yes, then the catalog function will be a weird concept? Best, Kurt On Tue, Sep 3, 2019 at 8:10 PM Danny

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

Thanks for creating this FLIP and starting the vote Xintong. +1 for the proposal from my side. I agree with Stephan that we might wanna revisit some of the configuration names. If I understood it correctly, then Task Off-heap memory represents the direct memory used by the user code, right? How

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Timo, Thanks for the quick reply ! :) I have added more example for #3 and #5 to the FLIP. That are great suggestions ! Regarding 2: There are two kind Serialization for CloudPickle(Which is different from Java): 1) For class and function which can be imported, CloudPickle only serialize

Re: Flink SQL - Support Computed Columns in DDL?

Hi Jark and Danny, Glad to hear your plan on this! One of our use cases is to define some column as rowtime (which is not of type timestamp). Computed column seems to be a natural fit for that. Thanks, Qi On Tue, Sep 3, 2019 at 7:46 PM Jark Wu wrote: > Hi Qi, > > The computed column is not

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

Hi, I think it’s important to keep in mind the original goals of this FLIP and not let the scope grow indefinitely. As I recall it, the goals are: - Extend the ConfigOption system enough to allow the Table API to configure options that are right now only available on CheckpointingOptions,

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

Thanks for starting the vote Xintong Also +1 for the proposed FLIP-49. @Stephan regarding namings: network vs shuffle. My understanding so far was that the network memory is what we basically give to Shuffle implementations and default netty implementation uses it in particular mostly for

Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

guaishushu1...@163.com From: guaishushu1...@163.com Date: 2019-09-03 20:25 To: dev Subject: Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader guaishushu1...@163.com From: guaishushu1...@163.com Date: 2019-09-03 20:23 To: dev Subject: Re: Re:

Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

guaishushu1...@163.com From: guaishushu1...@163.com Date: 2019-09-03 20:23 To: dev Subject: Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader guaishushu1...@163.com From: Jan Lukavský Date: 2019-09-03 20:17 To: dev Subject: Re: ClassLoader

Re: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

guaishushu1...@163.com From: Jan Lukavský Date: 2019-09-03 20:17 To: dev Subject: Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader Hi Till, the use-case is pretty much simple - I have a REPL shell in groovy, which generates classes at runtime. The

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

Hi Till, the use-case is pretty much simple - I have a REPL shell in groovy, which generates classes at runtime. The actual hierarchy is therefore  system class loader -> application classloader -> repl classloader (GroovyClassLoader actually) now, when a terminal (sink) operation in the

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

The way you proposed are basically the same as what Calcite does, I think we are in the same line. Best, Danny Chan 在 2019年9月3日 +0800 PM7:57，Timo Walther ，写道： > This sounds exactly as the module approach I mentioned, no? > > Regards, > Timo > > On 03.09.19 13:42, Danny Chan wrote: > > Thanks

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

Hi Danny, yes, this FLIP covers all the building blocks we need also for unification of the DDL properties. Regards, Timo On 03.09.19 13:45, Danny Chan wrote: with the new SQL DDL based on properties as well as more connectors and formats coming up, unified configuration becomes more

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

Hi Jan, I've talked with Aljoscha and Stephan offline and we concluded that we would like to avoid the usage of context class loaders if possible. The reason for this is that using the context class loader can easily mess up an otherwise clear class loader hierarchy which makes it hard to reason

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

This sounds exactly as the module approach I mentioned, no? Regards, Timo On 03.09.19 13:42, Danny Chan wrote: Thanks Bowen for bring up this topic, I think it’s a useful refactoring to make our function usage more user friendly. For the topic of how to organize the builtin operators and

Re: Flink SQL - Support Computed Columns in DDL?

Hi Qi, The computed column is not fully supported in 1.9. We will start a design discussion in the dev mailing list soon. Please stay tuned! Btw, could you share with us what's the case why do you want to use computed column? Best, Jark On Tue, 3 Sep 2019 at 19:25, Danny Chan wrote: > Yeah,

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

> with the new SQL DDL based on properties as well as more connectors and formats coming up, unified configuration becomes more important I Cann’t agree more, do you think we can unify the config options key format here for all the DDL properties ? Best, Danny Chan 在 2019年8月16日 +0800

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Thanks Bowen for bring up this topic, I think it’s a useful refactoring to make our function usage more user friendly. For the topic of how to organize the builtin operators and operators of Hive, here is a solution from Apache Calcite, the Calcite way is to make every dialect operators a

Re: Flink SQL - Support Computed Columns in DDL?

Yeah, we are planning to implement this feature in release-1.10, wait for our good news ! Best, Danny Chan 在 2019年9月3日 +0800 PM6:19，Qi Luo ，写道： > Hi folks, > > Computed columns in Flink SQL DDL is currently disabled in both old planner > and Blink planner (throws "Computed columns for DDL is not

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

Thanks Xintong for the explanation. For question #1, I think it's good as long as DataSet job behaviors remains the same. For question #2, agreed that the resource difference is small enough(at most 1 edge diff) in current supported point-wise execution edge connection patterns. Thanks, Zhu Zhu

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

+1 to the proposal in general A few things seems to be a bit put of sync with the latest discussions though. The section about JVM Parameters states that the -XX:MaxDirectMemorySize value is set to Task Off-heap Memory, Shuffle Memory and JVM Overhead. The way I understand the last discussion

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

Thanks for the comments, Zhu & Kurt. Andrey and I also had some discussions offline, and I would like to first post a summary of our discussion: 1. The motivation of the fraction based approach is to unify resource management for both operators with specified and unknown resource

Flink SQL - Support Computed Columns in DDL?

Hi folks, Computed columns in Flink SQL DDL is currently disabled in both old planner and Blink planner (throws "Computed columns for DDL is not supported yet!" exception in SqlToOperationConverter). I searched through the JIRA but found no relevant issues. Do we have any plans to support this

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Jincheng, thanks for your response. 2. Serializability of functions: Using some arbitrary serialization format for shipping a function to worker sounds fine to me. But once we store functions a the catalog we need to think about backwards compatibility and evolution of interfaces etc. I'm

Re: [SURVEY] Is the default restart delay of 0s causing problems?

The FLIP-62 discuss thread can be found here [1]. [1] https://lists.apache.org/thread.html/9602b342602a0181fcb618581f3b12e692ed2fad98c59fd6c1caeabd@%3Cdev.flink.apache.org%3E Cheers, Till On Tue, Sep 3, 2019 at 11:13 AM Till Rohrmann wrote: > Thanks everyone for the input again. I'll then

[DISCUSS] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

Hi everyone, I'd like to discuss changing the default restart delay for FixedDelay- and FailureRateRestartStrategy to "1 s" [1]. According to a user survey about the default value of the restart delay [2], it turned out that the current default value of "0 s" is not optimal. In practice Flink

[jira] [Created] (FLINK-13944) Table.toAppendStream: InvalidProgramException: Table program cannot be compiled.

Stefano created FLINK-13944: --- Summary: Table.toAppendStream: InvalidProgramException: Table program cannot be compiled. Key: FLINK-13944 URL: https://issues.apache.org/jira/browse/FLINK-13944 Project:

Re: [DISCUSS] Releasing Flink 1.8.2

+1 FLINK-13940 is a blocker, due to loss data is very important bug, And great thanks for helping fix it Kostas! Best, Jincheng Kostas Kloudas 于2019年9月2日周一 下午7:20写道： > Hi all, > > I think this should be also considered a blocker >

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi Bowen, thanks for your proposal. Here are some thoughts: 1) We should not have the restriction "hive built-in functions can only be used when current catalog is hive catalog". Switching a catalog should only have implications on the cat.db.object resolution but not functions. It would be

Re: [SURVEY] Is the default restart delay of 0s causing problems?

Thanks everyone for the input again. I'll then conclude this survey thread and start a discuss thread to set the default restart delay to 1s. @Arvid, I agree that a better documentation how to tune Flink with sane settings for certain scenarios is super helpful. However, as you've said it is

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

Hi Becket, it is definitely API but I would rather consider configuration as second-level API. Usually, a connector or plugin should have some nice builder pattern with helpful JavaDocs instead of just exposing the pure ConfigOptions. I think there is some general misunderstanding what a

Re: Please add me as contributor

Hi Dawid, thanks for the explanation. I got warning from PR [1] associated with JIRA [2]. Jan [1] https://github.com/apache/flink/pull/9579 [2] https://issues.apache.org/jira/browse/FLINK-13925 On 9/3/19 10:43 AM, Dawid Wysakowicz wrote: Hi Jan, Recently the community changed the

Re: Please add me as contributor

Hi Jan, Recently the community changed the contribution process a bit and there are no longer contributor privileges. The jira issues are supposed to be assigned by committers that are willing to help you with getting the contribution in. Please look at the contribution guidelines[1]. Do you have

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Thanks Bowen: +1 for this. And +1 to Kurt's suggestion. My other points are: 1.Hive built-in functions is an intermediate solution. So we should not introduce interfaces to influence the framework. To make Flink itself more powerful, we should implement the functions we need to add.

Please add me as contributor

Hi, I'd like to be able to assign JIRAs to myself, can I be added as contributor, please? My JIRA ID is 'janl'. Thanks,  Jan

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Thanks Bowen for driving this. +1 for the general idea. It makes the function resolved behavior more clear and deterministic. Besides, the user can use all hive built-in functions, which is a great feature. I only have one comment, but maybe it may touch your design so I think it would make

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

Thanks Xingtong for driving this effort, I haven't finished the whole document yet, but have couple of questions: 1. Regarding to network memory, the document said it will be derived by framework automatically. I'm wondering whether we should delete this dimension from user- facing API? 2.

Re: [SURVEY] Is the default restart delay of 0s causing problems?

Hi all, just wanted to share my experience with configurations with you. For non-expert users configurations of Flink can be very daunting. The list of common properties is already helping a lot [1], but it's not clear how they depend on each other and settings common for specific use cases are