Re: Re: [DISCUSS] Support JSON functions in Flink SQL

+1 to have JSON functions in Flink SQL JingsongLee 于2019年9月5日周四 下午4:46写道： > +1 > Nice document. I think it is easier to do after expression reworking[1]. > By the way, which planner do you want to start? > > [1] >

[jira] [Created] (FLINK-13974) isAssignable function return wrong result

forideal created FLINK-13974: Summary: isAssignable function return wrong result Key: FLINK-13974 URL: https://issues.apache.org/jira/browse/FLINK-13974 Project: Flink Issue Type: Bug

Re: [DISCUSS] Reducing build times

I do have a working Azure setup, yes. E2E tests are not included in the 3.5hrs. Yesterday, I became aware of a major blocker with Azure pipelines: Apache Infra does not allow it to be integrated with Apache GitHub repositories, because it requires write access (for a simple usability feature)

[jira] [Created] (FLINK-13978) Evaluate Azure Pipelines as a CI tool for Flink

Robert Metzger created FLINK-13978: -- Summary: Evaluate Azure Pipelines as a CI tool for Flink Key: FLINK-13978 URL: https://issues.apache.org/jira/browse/FLINK-13978 Project: Flink Issue

[jira] [Created] (FLINK-13979) Translate new streamfilesink docs to chinese

Gyula Fora created FLINK-13979: -- Summary: Translate new streamfilesink docs to chinese Key: FLINK-13979 URL: https://issues.apache.org/jira/browse/FLINK-13979 Project: Flink Issue Type: New

[jira] [Created] (FLINK-13976) Make moder kafka E2E test runnable on Java 11

Chesnay Schepler created FLINK-13976: Summary: Make moder kafka E2E test runnable on Java 11 Key: FLINK-13976 URL: https://issues.apache.org/jira/browse/FLINK-13976 Project: Flink Issue

[ANNOUNCE] Java 11 cron builds activated on master

Hello everyone, I just wanted to inform everyone that we now run Java 11 builds on Travis as part of the cron jobs, subsuming the existing Java 9 tests. All existing Java 9 build/test infrastructure has been removed. If you spot any test failures that appear to be specific to Java 11,

[jira] [Created] (FLINK-13975) Translate "Upcoming Events" on Chinese index.html

Fabian Hueske created FLINK-13975: - Summary: Translate "Upcoming Events" on Chinese index.html Key: FLINK-13975 URL: https://issues.apache.org/jira/browse/FLINK-13975 Project: Flink Issue

[jira] [Created] (FLINK-13977) Adapt HighAvailabilityServices#getWebMonitorLeaderElectionService to #getClusterRestEndpointLeaderRetriever

Till Rohrmann created FLINK-13977: - Summary: Adapt HighAvailabilityServices#getWebMonitorLeaderElectionService to #getClusterRestEndpointLeaderRetriever Key: FLINK-13977 URL:

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

+1 (non-binding) Minor: 1. Is it worth a separate "Limitations" section to contain all relative information like the Unsafe support issue in Java 12, just like many other FLIPs? 2. About the config names, if we change them later in PR, does it mean we will need to update the FLIP document? If so,

Re: instable checkpointing after migration to flink 1.8

Hi Bekir From what I could see, there should be two main factors influencing your time of sync execution checkpoint within that task. 1. Snapshot timers in heap to S3 [1] (network IO) 2. Creating local RocksDB checkpoint on disk [2] (disk IO) For the first part, unfortunately, there is

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 (non-binding) Best Regards, Yu On Thu, 5 Sep 2019 at 00:23, zhijiang wrote: > +1 > > Best, > Zhijiang > -- > From:Jark Wu > Send Time:2019年9月4日(星期三) 13:45 > To:dev > Subject:Re: [VOTE] FLIP-62: Set default restart delay for

[jira] [Created] (FLINK-13985) Use native memory for managed memory.

Xintong Song created FLINK-13985: Summary: Use native memory for managed memory. Key: FLINK-13985 URL: https://issues.apache.org/jira/browse/FLINK-13985 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13986) Clean-up of legacy mode

Xintong Song created FLINK-13986: Summary: Clean-up of legacy mode Key: FLINK-13986 URL: https://issues.apache.org/jira/browse/FLINK-13986 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13983) Launch task executor with new memory calculation logics

Xintong Song created FLINK-13983: Summary: Launch task executor with new memory calculation logics Key: FLINK-13983 URL: https://issues.apache.org/jira/browse/FLINK-13983 Project: Flink

[jira] [Created] (FLINK-13982) Implement memory calculation logics

Xintong Song created FLINK-13982: Summary: Implement memory calculation logics Key: FLINK-13982 URL: https://issues.apache.org/jira/browse/FLINK-13982 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-13984) Separate on-heap and off-heap managed memory pools

Xintong Song created FLINK-13984: Summary: Separate on-heap and off-heap managed memory pools Key: FLINK-13984 URL: https://issues.apache.org/jira/browse/FLINK-13984 Project: Flink Issue

Re: [DISCUSS] FLIP-66: Support time attribute in SQL DDL

Thanks Jark for this topic, This will be very useful. Best, ForwardXu Danny Chan 于2019年9月6日周五 上午11:26写道： > Thanks Jark for bring up this topic, this is definitely an import feature > for the SQL, especially the DDL users. > > I would spend some time to review this design doc, really thanks.

[VOTE] FLIP-53: Fine Grained Operator Resource Management

Hi all, I would like to start the voting process for FLIP-53 [1], which is discussed and reached consensus in this thread [2]. This voting will be open for at least 72 hours (excluding weekends). I'll try to close it Sep. 11, 04:00 UTC, unless there is an objection or not enough votes. Thank

Re: [DISCUSS] FLIP-66: Support time attribute in SQL DDL

Thanks Jark for bring up this topic, this is definitely an import feature for the SQL, especially the DDL users. I would spend some time to review this design doc, really thanks. Best, Danny Chan 在 2019年9月6日 +0800 AM11:19，Jark Wu ，写道： > Hi everyone, > > I would like to start discussion about

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

Thanks all for the votes. So far, we have - 4 binding +1 votes (Stephan, Andrey, Till, Zhijiang) - 2 un-binding +1 votes (Xintong, Yu) - No -1 votes There are more than 3 binding +1 votes and no -1 votes, and the voting time has past. According to the new bylaws, I'm happily to announce

[jira] [Created] (FLINK-13980) FLIP-49 Unified Memory Configuration for TaskExecutors

Xintong Song created FLINK-13980: Summary: FLIP-49 Unified Memory Configuration for TaskExecutors Key: FLINK-13980 URL: https://issues.apache.org/jira/browse/FLINK-13980 Project: Flink Issue

[jira] [Created] (FLINK-13981) Introduce a switch for enabling the new task executor memory configurations

Xintong Song created FLINK-13981: Summary: Introduce a switch for enabling the new task executor memory configurations Key: FLINK-13981 URL: https://issues.apache.org/jira/browse/FLINK-13981 Project:

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi, I think having a Pulsar connector in Flink can be a good mutual benefit to both communities. Another perspective is that Pulsar connector is the 1st streaming connector that integrates with Flink's metadata management system and Catalog APIs. It'll be cool to see how the integration turns

[DISCUSS] FLIP-66: Support time attribute in SQL DDL

Hi everyone, I would like to start discussion about how to support time attribute in SQL DDL. In Flink 1.9, we already introduced a basic SQL DDL to create a table. However, it doesn't support to define time attributes. This makes users can't apply window operations on the tables created by DDL

Re: [ANNOUNCE] Java 11 cron builds activated on master

Great news Chesnay! Thanks for your hard work and effort! Best, Vino Chesnay Schepler 于2019年9月5日周四 下午9:58写道： > Hello everyone, > > I just wanted to inform everyone that we now run Java 11 builds on > Travis as part of the cron jobs, subsuming the existing Java 9 tests. > All existing Java 9

Re: [VOTE] FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s

+1 (non-binding) Best, Vino Yu Li 于2019年9月6日周五 上午2:13写道： > +1 (non-binding) > > Best Regards, > Yu > > > On Thu, 5 Sep 2019 at 00:23, zhijiang > wrote: > > > +1 > > > > Best, > > Zhijiang > > -- > > From:Jark Wu > > Send

Re: [VOTE] FLIP-49: Unified Memory Configuration for TaskExecutors

+1 (non-binding) from my side. @Yu, thanks for the vote. - The current FLIP document already mentioned the issue that Unsafe is not supported in Java 12, in the section 'Unifying Explicit and Implicit Memory Allocation'. It makes sense to me to emphasize this by adding a separate limitation

[DISCUSS] Support notifyOnMaster for notifyCheckpointComplete

Hi devs: I try to implement streaming file sink for table[1] like StreamingFileSink. If the underlying is a HiveFormat, or a format that updates visibility through a metaStore, I have to update the metaStore in the notifyCheckpointComplete, but this operation occurs on the task side, which

[DISCUSS] FLIP-63: Rework table partition support

Hi everyone, thank you for your comments. Mail name was updated and streaming-related concepts were added. We would like to start a discussion thread on "FLIP-63: Rework table partition support"(Design doc: [1]), where we describe how to partition support in flink and how to integrate to hive

[DISCUSS] Support customize state in customized KeyedStateBackend

Hi every, I would like to start a discussion on supporting customize state in customized KeyedStateBackend. In Flink, users can customize KeyedStateBackend to support different type of data store. Although we can implement customized StateDescriptors for different kind of data structrues, we do

Re: [DISCUSS] Support notifyOnMaster for notifyCheckpointComplete

Hi Jingsong, Big fan of this idea. We faced the same problem and resolved by adding a distributed lock. It would be nice to have this feature in JobMaster, which can replace the lock. Best, Shimin JingsongLee 于2019年9月6日周五 下午12:20写道： > Hi devs: > > I try to implement streaming file sink for

Re: [DISCUSS] FLIP-66: Support time attribute in SQL DDL

Hi Jark, Thanks for bringing up this discussion and the detailed design doc. This is definitely a critical feature for streaming SQL jobs. I have left a few comments in the design doc. Thanks, Dian > 在 2019年9月6日，上午11:48，Forward Xu 写道： > > Thanks Jark for this topic, This will be very

Re: [DISCUSS] FLIP-64: Support for Temporary Objects in Table module

Thanks dawid, +1 for this approach. One concern is the removal of registerTableSink & registerTableSource in TableEnvironment. It has two alternatives: 1.the properties approach (DDL, descriptor). 2.from/toDataStream. #1 can only be properties, not java states, and some Connectors are

Re: [DISCUSS] Support JSON functions in Flink SQL

+1 to add JSON support to Flink. We also see lots of requirements for JSON related functions in our internal platform. Since these are already SQL standard, I think it's a good time to add them to Flink. Best, Kurt On Thu, Sep 5, 2019 at 10:37 AM Qi Luo wrote: > We also see strong demands

Re: [DISCUSS] Adding configure and close methods to the Kafka(De)SerializationSchema interfaces

Hi Arvid, The ConfluentRegistryAvroDeserializationSchema uses a checkAvroInitialized() call for every single record to initialize the schema for the first time. This is clearly an indication of a missing open/configure method. In addition some of the Kafka serializers rely on properties that are

Re: Re: [DISCUSS] Support JSON functions in Flink SQL

+1 Nice document. I think it is easier to do after expression reworking[1]. By the way, which planner do you want to start? [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-51%3A+Rework+of+the+Expression+Design Best, Jingsong Lee

Re: Storing offsets in Kafka

Hey, Yeah I am using the first case. Is there a specific requirement for checkpoints ? Like do they need to be externalized or so ? Best, Dom. czw., 5 wrz 2019 o 05:32 Becket Qin napisał(a): > Hi Dominik, > > There has not been any change to the offset committing logic in > KafkaConsumer for

Re: Re: [DISCUSS] Support JSON functions in Flink SQL

+1 I have done similar work before. Looking forward to discussing this feature. Best wenhui winifred.wenhui.t...@gmail.com From: Kurt Young Date: 2019-09-05 14:00 To: dev CC: Anyang Hu Subject: Re: [DISCUSS] Support JSON functions in Flink SQL +1 to add JSON support to Flink. We also see

[jira] [Created] (FLINK-13967) Generate full binary licensing via collect_license_files.sh

Chesnay Schepler created FLINK-13967: Summary: Generate full binary licensing via collect_license_files.sh Key: FLINK-13967 URL: https://issues.apache.org/jira/browse/FLINK-13967 Project: Flink

[jira] [Created] (FLINK-13968) Add travis check for the correctness of the binary licensing

Chesnay Schepler created FLINK-13968: Summary: Add travis check for the correctness of the binary licensing Key: FLINK-13968 URL: https://issues.apache.org/jira/browse/FLINK-13968 Project: Flink

[DISCUSS] Adding configure and close methods to the Kafka(De)SerializationSchema interfaces

Hi all! While implementing a new custom flink serialization schema that wraps an existing Kafka serializer, I realized we are missing 2 key methods that could be easily added: void configure(java.util.Map configs); void close(); We could rename configure to open but Kafka serializers have a

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi everyone, I'm wondering what the problem would be if we committed the Pulsar connector before the new source interface is ready. If I understood it correctly, then we need to support the old source interface anyway for the existing connectors. By checking it in early I could see the benefit

Re: [DISCUSS] Adding configure and close methods to the Kafka(De)SerializationSchema interfaces

Hi Gyula, when looking at the ConfluentRegistryAvroDeserializationSchema [1], it seems like the intended way is to pass all configuration parameters in the constructor. So you could call open there. Could you please line out in more details why this is not enough? What would you do in open and

Re: [DISCUSS] Reducing build times

I assume you already have a working (and verified) azure setup? Once we're running things on azure on the apache repo people will invariably use that as a source of truth because fancy check marks will yet again appear on commits. Hence I'm wary of running experiments here; I would prefer if

[jira] [Created] (FLINK-13970) Remove or move LifoSetQueue

TisonKun created FLINK-13970: Summary: Remove or move LifoSetQueue Key: FLINK-13970 URL: https://issues.apache.org/jira/browse/FLINK-13970 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi, Another thing to consider is the Scope of the FLIP. Currently, we try to support (stateful) AggregateFunctions. I have some concerns about whether or not DataView/MapView/ListView is a good interface because it requires quite some magic from the runners to make it work, such as messing

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi Till, You are right. It all depends on when the new source interface is going to be ready. Personally I think it would be there in about a month or so. But I could be too optimistic. It would also be good to hear what do Aljoscha and Stephan think as they are also involved in FLIP-27. In

[jira] [Created] (FLINK-13971) Add TaskManager In org.apache.flink.runtime.rest.messages.JobVertexTaskManagersInfo.TaskManagersInfo

lining created FLINK-13971: -- Summary: Add TaskManager In org.apache.flink.runtime.rest.messages.JobVertexTaskManagersInfo.TaskManagersInfo Key: FLINK-13971 URL: https://issues.apache.org/jira/browse/FLINK-13971

Re: Storing offsets in Kafka

No, I don't think so. As long as you have a successful checkpoint, The offset will be committed. Thanks, Jiangjie (Becket) Qin On Thu, Sep 5, 2019 at 4:56 PM Dominik Wosiński wrote: > Hey, > Yeah I am using the first case. Is there a specific requirement for > checkpoints ? Like do they need

[jira] [Created] (FLINK-13972) Move PreviewPlanEnvironment to flink-tests

Kostas Kloudas created FLINK-13972: -- Summary: Move PreviewPlanEnvironment to flink-tests Key: FLINK-13972 URL: https://issues.apache.org/jira/browse/FLINK-13972 Project: Flink Issue Type:

Re: Please add me as contributor

Hi Jan, I assigned FLINK-13925 to you. Thanks for your contribution! Best, Fabian Am Di., 3. Sept. 2019 um 10:52 Uhr schrieb Jan Lukavský : > Hi Dawid, > > thanks for the explanation. I got warning from PR [1] associated with > JIRA [2]. > > Jan > > [1]

Re: [DISCUSS] FLIP-54: Evolve ConfigOption and Configuration

Hi Dawid, Thanks a lot for the clarification. Got it now. A few more thoughts: 1. Naming. I agree that if the name of "Configurable" is a little misleading if we just want to use it to save POJOs. It would probably help to just name it something like "ConfigPojo". 2. Flat config map v.s.

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi Xuefu, Just wanted to summarize my opinion on the one topic (temporary functions). My preference would be to make temporary functions always 3-part qualified (as a result that would prohibit overriding built-in functions). Having said that if the community decides that it's better to allow

Re: [VOTE] FLIP-61 Simplify Flink's cluster level RestartStrategy configuration

+1 (non-binding) Zili Chen 于2019年9月5日周四 上午10:55写道： > +1 > > > zhijiang 于2019年9月5日周四 上午12:36写道： > > > +1 > > -- > > From:Till Rohrmann > > Send Time:2019年9月4日(星期三) 13:39 > > To:dev > > Cc:Zhu Zhu > > Subject:Re: [VOTE] FLIP-61

[ANNOUNCE] Flink Forward training registration closes on September 30th

Hi all, The registration for the Flink Forward Europe training sessions closes in four weeks. The training takes place in Berlin at October 7th and is followed by two days of talks by speakers from companies like Airbus, Goldman Sachs, Netflix, Pinterest, and Workday [1]. The following four

Re: [DISCUSS] Contribute Pulsar Flink connector back to Flink

Hi Sijie and Yijie, Thanks for sharing your thoughts. Just want to have some update on FLIP-27. Although the FLIP wiki and discussion thread has been quiet for some time, a few committer / contributors in Flink community were actually prototyping the entire thing. We have made some good progress

[jira] [Created] (FLINK-13966) Jar sorting in collect_license_files.sh is locale dependent

Chesnay Schepler created FLINK-13966: Summary: Jar sorting in collect_license_files.sh is locale dependent Key: FLINK-13966 URL: https://issues.apache.org/jira/browse/FLINK-13966 Project: Flink

Re: [DISCUSS] FLIP-53: Fine Grained Resource Management

Thanks all for joining the discussion. It seems to me that there is a consensus on the current FLIP document. So if there is no objection, I would like to start the voting process for this FLIP. Thank you~ Xintong Song On Wed, Sep 4, 2019 at 8:23 PM Andrey Zagrebin wrote: > Thanks for

[jira] [Created] (FLINK-13969) Resuming Externalized Checkpoint (rocks, incremental, scale down) end-to-end test fails on Travis

Till Rohrmann created FLINK-13969: - Summary: Resuming Externalized Checkpoint (rocks, incremental, scale down) end-to-end test fails on Travis Key: FLINK-13969 URL:

Re: [DISCUSS] FLIP-57 - Rework FunctionCatalog

Hi David, Thanks for sharing your thoughts and request for clarifications. I believe that I fully understood your proposal, which does has its merit. However, it's different from ours. Here are the answers to your questions: Re #1: yes, the temp functions in the proposal are global and have

Re: [DISCUSS] Flink Python User-Defined Function for Table API

Hi Aljoscha, Thanks for your comments! Regarding to the FLIP scope, it seems that we have agreed on the design of the stateless function support. What do you think about starting the development of the stateless function support firstly and continue the discussion of stateful function support?

Re: ClassLoader created by BlobLibraryCacheManager is not using context classloader

Hi Till and Aljoscha, I was investigating the other options, but unfortunately all of them look a little complicated (although possible, of course). But before going into a more complicated solutions, I'd like to know what issues do you actually see with using the context class loader. I can

Re: Fine grained batch recovery vs. native libraries

Thanks for reporting the problem David! Cheers, Fabian Am Mi., 4. Sept. 2019 um 14:09 Uhr schrieb David Morávek : > Hi Chesnay, I've created FLINK-13958 > to track the issue. > > Thanks, > D. > > On Wed, Sep 4, 2019 at 1:56 PM Chesnay Schepler