[jira] [Created] (FLINK-19422) Avro Confluent Schema Registry nightly end-to-end test failed with "Register operation timed out; error code: 50002"

2020-09-25 Thread Dian Fu (Jira)
Dian Fu created FLINK-19422: --- Summary: Avro Confluent Schema Registry nightly end-to-end test failed with "Register operation timed out; error code: 50002" Key: FLINK-19422 URL:

[jira] [Created] (FLINK-19421) Support Python UDAF in streaming mode

2020-09-25 Thread Dian Fu (Jira)
Dian Fu created FLINK-19421: --- Summary: Support Python UDAF in streaming mode Key: FLINK-19421 URL: https://issues.apache.org/jira/browse/FLINK-19421 Project: Flink Issue Type: Sub-task

Re: Avoiding "too many open files" during leftOuterJoin with Flink 1.11/batch

2020-09-25 Thread Ken Krugler
Hi Chesnay, Thanks, and you were right - it wasn’t a case of too many memory segments triggering too many open files. It was a configuration issue with Elasticsearch clients being used by a custom function. This just happened to start being executed at the same time as the leftOuterJoin &

Re: [DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-09-25 Thread liupengcheng
Hi, Jark, I'm very interested in this feature, and I'm also working on this recently. I just have a glance at the FLIP, it's good, but I found that there is no plan to add SESSION windows. Also, I think there can be more things we can do based on this new syntax. For

[DISCUSS] FLIP-145: Support SQL windowing table-valued function

2020-09-25 Thread Jark Wu
Hi everyone, I want to start a FLIP about supporting windowing table-valued functions (TVF). The main purpose of this FLIP is to improve the near real-time (NRT) experience of Flink. FLIP-145:

[jira] [Created] (FLINK-19420) Translate "Program Packaging" page of "Managing Execution" into Chinese

2020-09-25 Thread Xiao Huang (Jira)
Xiao Huang created FLINK-19420: -- Summary: Translate "Program Packaging" page of "Managing Execution" into Chinese Key: FLINK-19420 URL: https://issues.apache.org/jira/browse/FLINK-19420 Project: Flink

Re: [ANNOUNCE] Apache Flink 1.11.2 released

2020-09-25 Thread Etienne Chauchot
Congrats to everyone involved ! Etienne On 21/09/2020 08:34, Yu Li wrote: Thanks Zhu Zhu for being our release manager and everyone else who made the release possible! Best Regards, Yu On Thu, 17 Sep 2020 at 13:29, Zhu Zhu wrote: The Apache Flink community is very happy to announce the

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-25 Thread Steven Wu
I should clarify my last email a little more. For the example of commits for checkpoints 1-100 failed, the job is still up (processing records and uploading files). When commit for checkpoint 101 came, IcebergSink would prefer the framework to pass in all 101 GlobalCommT (100 old + 1 new) so that

Re: Timed out patterns handling using MATCH_RECOGNIZE

2020-09-25 Thread Kosma Grochowski
Thank you Julian for mentioning the anti-join. With its help, I managed to solve our particular case similarly as follows: ``` SELECT e.* FROM events e LEFT JOIN patterns p ON e.record_id = p.begin_record_id WHERE e.pattern_val = 'BEGIN' AND p.begin_record_id is null ``` However, I'm thinking

[jira] [Created] (FLINK-19419) "null-string-literal" does not work in HBaseSource decoder

2020-09-25 Thread CaoZhen (Jira)
CaoZhen created FLINK-19419: --- Summary: "null-string-literal" does not work in HBaseSource decoder Key: FLINK-19419 URL: https://issues.apache.org/jira/browse/FLINK-19419 Project: Flink Issue

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-25 Thread Seth Wiesman
Done Seth On Fri, Sep 25, 2020 at 2:47 AM Yu Li wrote: > *bq. I think it might help to highlight specific stumbling blocks users > have today and why I believe this change addresses those issues.* > Thanks for adding more details, I believe adding these blocks to the FLIP > doc could make the

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-25 Thread Steven Wu
> 1. The frame can not know which `GlobalCommT` to retry if we use the > List as parameter when the `commit` returns `RETRY`. > 2. Of course we can let the `commit` return more detailed info but it might > be too complicated. If commit(List) returns RETRY, it means the whole list needs to be

[jira] [Created] (FLINK-19418) Inline PRIMARY KEY constraint should be invalid

2020-09-25 Thread Konstantin Knauf (Jira)
Konstantin Knauf created FLINK-19418: Summary: Inline PRIMARY KEY constraint should be invalid Key: FLINK-19418 URL: https://issues.apache.org/jira/browse/FLINK-19418 Project: Flink

Re: [VOTE] FLIP-143: Unified Sink API

2020-09-25 Thread Aljoscha Krettek
+1 (binding) Aljoscha On 25.09.20 14:26, Guowei Ma wrote: From the discussion[1] we could find that FLIP focuses on providing an unified transactional sink API. So I updated the FLIP's title to "Unified Transactional Sink API". But I found that the old link could not be opened again. I would

Re: [DISCUSS] FLIP-146: Improve new TableSource and TableSink interfaces

2020-09-25 Thread admin
Hi everyone, Thanks for the proposal. In our company,we meet the same situation as @liu shouwei. We developed some features base on flink.Such as parallelism of sql source/sink connector, and kafka delay consumer which is adding a flatmap and a keyby transformation after the source Datastream.

Re: [VOTE] FLIP-143: Unified Sink API

2020-09-25 Thread Guowei Ma
>From the discussion[1] we could find that FLIP focuses on providing an unified transactional sink API. So I updated the FLIP's title to "Unified Transactional Sink API". But I found that the old link could not be opened again. I would update the link[2] here. Sorry for the inconvenience. [1]

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-25 Thread Guowei Ma
Hi, all >From the above discussion we could find that FLIP focuses on providing an unified transactional sink API. So I updated the FLIP's title to "Unified Transactional Sink API". But I found that the old link could not be opened again. I would update the link[1] here. Sorry for the

[jira] [Created] (FLINK-19417) Fix the bug of the method from_data_stream in table_environement

2020-09-25 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-19417: Summary: Fix the bug of the method from_data_stream in table_environement Key: FLINK-19417 URL: https://issues.apache.org/jira/browse/FLINK-19417 Project: Flink

Re: [DISCUSS] Support registering custom JobStatusListeners when scheduling a job

2020-09-25 Thread Aljoscha Krettek
Hi, I understand from your email that `StreamExecutionEnvironment.registerJobListener()` would not be enought for you because you want to be notified of changes on the cluster side, correct? That is when the job status changes on the master. Best, Aljoscha On 23.09.20 14:31, 季文昊 wrote: Hi

[jira] [Created] (FLINK-19416) Support Python datetime object in from_collection of Python DataStream

2020-09-25 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-19416: Summary: Support Python datetime object in from_collection of Python DataStream Key: FLINK-19416 URL: https://issues.apache.org/jira/browse/FLINK-19416 Project:

Re: [DISCUSS][Code-Style] The approach to implement singleton pattern

2020-09-25 Thread Xintong Song
Thanks for starting this discussion, Yangze. My personal preference for either singleton or non-initiatable classes is to use enum wherever it is possible, because it's briefer is safer. On the other hand, I'm also against private constructors. To my understanding, for most if not all

Re: [DISCUSS][Code-Style] The approach to implement singleton pattern

2020-09-25 Thread Yangze Guo
Thanks all for the valuable feedbacks! @Gael @Dawid Thanks for the explanation! I think you are right that this discussion is about a non-instantiable class that contains only static methods. @All My major proposal is actually to stick to one of two approaches in Flink. It seems that most devs

Re: [DISCUSS][Code-Style] The approach to implement singleton pattern

2020-09-25 Thread Jingsong Li
Hi, thanks for starting this discussion. I am +1 for using the private constructor for util class. We don't need to change it. I think few libraries use the enum, such as guava, common-utils, or even JDK, the private constructor is widely used. I don't quite understand why a util class is an

Re: [DISCUSS][Code-Style] The approach to implement singleton pattern

2020-09-25 Thread Dawid Wysakowicz
Hi all, First of all I very much agree with Gael. The discussion is not about a Singleton pattern. Secondly, similarly as @Timo and @Gael I find the pattern very confusing. Each time I see it I have a hard time figuring out why there are no enumerations in the enum. This is my preference though.

Re: [DISCUSS][Code-Style] The approach to implement singleton pattern

2020-09-25 Thread Gaël Renoux
Hi One small remark here: you should not call this a Singleton. For most people, a Singleton would refer to the implementation of the GoF Singleton pattern, where you have a single instance of the class (see for instance the corresponding Wikipedia page:

Re: [DISCUSS][Code-Style] The approach to implement singleton pattern

2020-09-25 Thread Timo Walther
Hi, honstely, I find using enums is more of a hack. `enum` stands for enumeration and is meant for listing flags or options. Using it for singleton patterns is just abusing a concept due to certain implementation details and less code. I feel this topic is like using Lombok for generating

Re: [VOTE] Apache Flink Stateful Functions 2.2.0, release candidate #2

2020-09-25 Thread Piotr Nowojski
+1 (binding) - build from the source and run all tests (successfully) - verified pom files between release-2.1 and release-2.2 branches for any relevant licensing changes Piotrek pt., 25 wrz 2020 o 10:21 Tzu-Li (Gordon) Tai napisał(a): > +1 (binding) > > - Verified signatures, and no binaries

[jira] [Created] (FLINK-19415) Move Hive document to "Table & SQL Connectors" from "Table API & SQL"

2020-09-25 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19415: Summary: Move Hive document to "Table & SQL Connectors" from "Table API & SQL" Key: FLINK-19415 URL: https://issues.apache.org/jira/browse/FLINK-19415 Project: Flink

Re: [DISCUSS] Move Hive document to "Table & SQL Connectors" from "Table API & SQL"

2020-09-25 Thread Jingsong Li
Thanks for your feedback. I created https://issues.apache.org/jira/browse/FLINK-19415 for track this. Best, Jingsong On Fri, Sep 25, 2020 at 11:11 AM Leonard Xu wrote: > +1 > > > 在 2020年9月24日,21:54,Seth Wiesman 写道: > > > > +1 > > > > On Thu, Sep 24, 2020 at 2:49 AM Rui Li wrote: > > > >> +1

Re: [DISCUSS][Code-Style] The approach to implement singleton pattern

2020-09-25 Thread Piotr Nowojski
Hi, I don't mind one way or the other. I guess enum way is somehow safer, however did we really have any issues with our current approach with `private` constructors? I mean, you are mentioning that using reflections could overcome private constructors, but is that a real concern in our code

[jira] [Created] (FLINK-19414) Introduce ParquetColumnarRowInputFormat

2020-09-25 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-19414: Summary: Introduce ParquetColumnarRowInputFormat Key: FLINK-19414 URL: https://issues.apache.org/jira/browse/FLINK-19414 Project: Flink Issue Type: Sub-task

[DISCUSS][Code-Style] The approach to implement singleton pattern

2020-09-25 Thread Yangze Guo
Hi, devs, Recently, in the PR of FLINK-19179[1], we have a discussion about how to implement singleton pattern in Flink. Currently, most of the utility classes implement singleton pattern through the private constructor. Seldom utility classes leverage the enum mechanism. From my perspective,

Re: [VOTE] Apache Flink Stateful Functions 2.2.0, release candidate #2

2020-09-25 Thread Tzu-Li (Gordon) Tai
+1 (binding) - Verified signatures, and no binaries in release artifacts - Built from source - Built Docker image from Dockerfiles in https://github.com/apache/flink-statefun-docker/pull/10. - Ran E2E tests with the built Dockerfiles (both Java 8 and 11 variants) - Ran the Python Greeter Example

Re: [DISCUSS] FLIP-146: Improve new TableSource and TableSink interfaces

2020-09-25 Thread Rui Li
Hi Jingsong, Thanks for driving this effort. I have two minor comments. 1. IMHO, parallelism is a concept that applies to all ScanTableSource. So instead of defining a new interface, is it more natural to incorporate parallel inference to existing interfaces, e.g. ScanTableSource or

[jira] [Created] (FLINK-19413) Translate "FileSystem" page of "Table & SQL Connectors" into Chinese

2020-09-25 Thread weizheng (Jira)
weizheng created FLINK-19413: Summary: Translate "FileSystem" page of "Table & SQL Connectors" into Chinese Key: FLINK-19413 URL: https://issues.apache.org/jira/browse/FLINK-19413 Project: Flink

[jira] [Created] (FLINK-19412) Re-layer Python Operation Make it Possible to Provide only Python implementation

2020-09-25 Thread Huang Xingbo (Jira)
Huang Xingbo created FLINK-19412: Summary: Re-layer Python Operation Make it Possible to Provide only Python implementation Key: FLINK-19412 URL: https://issues.apache.org/jira/browse/FLINK-19412

Re: [DISCUSS] FLIP-142: Disentangle StateBackends from Checkpointing

2020-09-25 Thread Yu Li
*bq. I think it might help to highlight specific stumbling blocks users have today and why I believe this change addresses those issues.* Thanks for adding more details, I believe adding these blocks to the FLIP doc could make the motivation more vivid and convincing. *bq. To be concrete I think

[jira] [Created] (FLINK-19411) MultipleInputStreamTask fails with RuntimeException when its input contains union

2020-09-25 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-19411: --- Summary: MultipleInputStreamTask fails with RuntimeException when its input contains union Key: FLINK-19411 URL: https://issues.apache.org/jira/browse/FLINK-19411

Re: [DISCUSS] FLIP-146: Improve new TableSource and TableSink interfaces

2020-09-25 Thread Jingsong Li
Hi Aljoscha, Thank you for your feedback, ## Connector parallelism Requirements: Set parallelism by user specified or inferred by connector. How to configure parallelism in DataStream: In the DataStream world, the only way to configure parallelism is

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-25 Thread Guowei Ma
Hi, Steven >>I also have a clarifying question regarding the WriterStateT. Since >>IcebergWriter won't need to checkpoint any state, should we set it to *Void* >>type? Since getWriterStateSerializer() returns Optional, that is clear and >>we can return Optional.empty(). Yes I think you could do

[jira] [Created] (FLINK-19410) RestAPIStabilityTest does not assert on enum changes

2020-09-25 Thread Matthias (Jira)
Matthias created FLINK-19410: Summary: RestAPIStabilityTest does not assert on enum changes Key: FLINK-19410 URL: https://issues.apache.org/jira/browse/FLINK-19410 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-143: Unified Sink API

2020-09-25 Thread Guowei Ma
Hi,Steven Thank you for reading the FLIP so carefully. 1. The frame can not know which `GlobalCommT` to retry if we use the List as parameter when the `commit` returns `RETRY`. 2. Of course we can let the `commit` return more detailed info but it might be too complicated. 3. On the other hand, I

Re: [DISCUSS] Release flink-shaded 12.0

2020-09-25 Thread Konstantin Knauf
+1 On Wed, Sep 23, 2020 at 9:13 AM Yu Li wrote: > +1 > > Best Regards, > Yu > > > On Tue, 22 Sep 2020 at 17:49, Robert Metzger wrote: > > > No concerns from my side. > > > > On Fri, Sep 18, 2020 at 8:25 AM Chesnay Schepler > > wrote: > > > > > Hello, > > > > > > I'd like to kickoff the next

[jira] [Created] (FLINK-19409) The comment for getValue has wrong code in class ListView

2020-09-25 Thread Liu (Jira)
Liu created FLINK-19409: --- Summary: The comment for getValue has wrong code in class ListView Key: FLINK-19409 URL: https://issues.apache.org/jira/browse/FLINK-19409 Project: Flink Issue Type:

[jira] [Created] (FLINK-19408) Update flink-statefun-docker release scripts for cross release Java 8 and 11

2020-09-25 Thread Tzu-Li (Gordon) Tai (Jira)
Tzu-Li (Gordon) Tai created FLINK-19408: --- Summary: Update flink-statefun-docker release scripts for cross release Java 8 and 11 Key: FLINK-19408 URL: https://issues.apache.org/jira/browse/FLINK-19408