Re: [DISCUSS] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

2022-05-16 Thread Jingsong Li
Hi Godfrey, Thanks for your reply. Sounds good to me. > I think we should also introduce a config option We can add this option to the FLIP. I prefer a option for FileSystemConnector, maybe a enum. Best, Jingsong On Tue, May 17, 2022 at 10:31 AM godfrey he wrote: > Hi Jingsong, > > Thanks

Re: [DISCUSS] FLIP-231: Introduce SupportStatisticReport to support reporting statistics from source connectors

2022-05-16 Thread godfrey he
Hi Jingsong, Thanks for the feedback. >One concern I have is that we read the footer for each file, and this may >be a bit costly in some cases. Is it possible for us to have some > hierarchical way yes, if there are thousands of orc/parquet files, it may take a long time. So we can introduce a

[jira] [Created] (FLINK-27656) Add parquet file format

2022-05-16 Thread Zheng Hu (Jira)
Zheng Hu created FLINK-27656: Summary: Add parquet file format Key: FLINK-27656 URL: https://issues.apache.org/jira/browse/FLINK-27656 Project: Flink Issue Type: Sub-task Reporter:

[jira] [Created] (FLINK-27655) Implement Avro File statistic collector

2022-05-16 Thread Zheng Hu (Jira)
Zheng Hu created FLINK-27655: Summary: Implement Avro File statistic collector Key: FLINK-27655 URL: https://issues.apache.org/jira/browse/FLINK-27655 Project: Flink Issue Type: Sub-task

[VOTE] FLIP-229: Introduces Join Hint for Flink SQL Batch Job

2022-05-16 Thread Xuyang
Hi, everyone. Thanks for your feedback for FLIP-229: Introduces Join Hint for Flink SQL Batch Job[1] on the discussion thread[2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an objection or not enough votes. -- Best! Xuyang [1]

Re: [DISCUSS] Next Flink Kubernetes Operator release timeline

2022-05-16 Thread Aitozi
Thanks Gyula. It looks good to me. I could do a favor during the release also. Please feel free to ping me to help the doc, release and test work :) Best, Aitozi Yang Wang 于2022年5月16日周一 21:57写道: > Thanks Gyula for sharing the progress. It is very likely we could have the > first release

[jira] [Created] (FLINK-27654) Older jackson-databind found in /flink-kubernetes-shaded-1.0-SNAPSHOT.jar

2022-05-16 Thread James Busche (Jira)
James Busche created FLINK-27654: Summary: Older jackson-databind found in /flink-kubernetes-shaded-1.0-SNAPSHOT.jar Key: FLINK-27654 URL: https://issues.apache.org/jira/browse/FLINK-27654 Project:

[jira] [Created] (FLINK-27653) Pulsar Connector bug: The startCursor has been setted default value of "MessageId.earliest", Every time to restart the job,the Consumer will do the seek operation.

2022-05-16 Thread wawa (Jira)
wawa created FLINK-27653: Summary: Pulsar Connector bug: The startCursor has been setted default value of "MessageId.earliest", Every time to restart the job,the Consumer will do the seek operation. Key: FLINK-27653

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

2022-05-16 Thread Lijie Wang
Hi Konstantin, Maybe change it to the following: 1. POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id} Merge is not allowed. If the {id} already exists, return error. Otherwise, create a new item. 2. POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge Merge is

[jira] [Created] (FLINK-27652) CompactManager.Rewriter cannot handle different partition keys invoked compaction

2022-05-16 Thread Jane Chan (Jira)
Jane Chan created FLINK-27652: - Summary: CompactManager.Rewriter cannot handle different partition keys invoked compaction Key: FLINK-27652 URL: https://issues.apache.org/jira/browse/FLINK-27652 Project:

[jira] [Created] (FLINK-27651) Support CREATE FUNCTION USING JAR syntax

2022-05-16 Thread dalongliu (Jira)
dalongliu created FLINK-27651: - Summary: Support CREATE FUNCTION USING JAR syntax Key: FLINK-27651 URL: https://issues.apache.org/jira/browse/FLINK-27651 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-27650) First environment variable of top level pod template is lost

2022-05-16 Thread Simon Paradis (Jira)
Simon Paradis created FLINK-27650: - Summary: First environment variable of top level pod template is lost Key: FLINK-27650 URL: https://issues.apache.org/jira/browse/FLINK-27650 Project: Flink

Re: [DISCUSS] Next Flink Kubernetes Operator release timeline

2022-05-16 Thread Yang Wang
Thanks Gyula for sharing the progress. It is very likely we could have the first release candidate next Monday. Best, Yang Gyula Fóra 于2022年5月16日周一 20:50写道: > Hi Devs! > > We are on track for our planned 1.0.0 release timeline. There are no > outstanding blocker issues on JIRA for the release.

Re: taskexecutor .out files

2022-05-16 Thread Weihua Hu
Sorry, the command is parsed as reference. The real command is : " > taskmanager.out " Best, Weihua > 2022年5月16日 下午9:52,Weihua Hu 写道: > > Hi, > > Flink redirects stdout to the taskmanager.out when starting TaskManager. > If taskmanager.out is deleted, Flink cannot automatically create >

Re: taskexecutor .out files

2022-05-16 Thread Weihua Hu
Hi, Flink redirects stdout to the taskmanager.out when starting TaskManager. If taskmanager.out is deleted, Flink cannot automatically create taskmanager.out, which means any subsequent output to stdout will be lost. If you want to clean up the content of taskmanager.out, you can try using:

[jira] [Created] (FLINK-27649) Reduce the number of outputted log lines by Elasticsearch6SinkE2ECase and Elasticsearch7SinkE2ECase

2022-05-16 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-27649: -- Summary: Reduce the number of outputted log lines by Elasticsearch6SinkE2ECase and Elasticsearch7SinkE2ECase Key: FLINK-27649 URL:

[jira] [Created] (FLINK-27648) Review example YAMLs in the documentation

2022-05-16 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-27648: -- Summary: Review example YAMLs in the documentation Key: FLINK-27648 URL: https://issues.apache.org/jira/browse/FLINK-27648 Project: Flink Issue Type:

[jira] [Created] (FLINK-27647) Improve Metrics documentation to include newly added metrics

2022-05-16 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-27647: -- Summary: Improve Metrics documentation to include newly added metrics Key: FLINK-27647 URL: https://issues.apache.org/jira/browse/FLINK-27647 Project: Flink

[jira] [Created] (FLINK-27646) Create Roadmap page for Flink Kubernetes operator

2022-05-16 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-27646: -- Summary: Create Roadmap page for Flink Kubernetes operator Key: FLINK-27646 URL: https://issues.apache.org/jira/browse/FLINK-27646 Project: Flink Issue Type:

[jira] [Created] (FLINK-27645) Update overview / supported features page for 1.0.0

2022-05-16 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-27645: -- Summary: Update overview / supported features page for 1.0.0 Key: FLINK-27645 URL: https://issues.apache.org/jira/browse/FLINK-27645 Project: Flink Issue Type:

[jira] [Created] (FLINK-27644) Update CRD documentation with new spec/status changes

2022-05-16 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-27644: -- Summary: Update CRD documentation with new spec/status changes Key: FLINK-27644 URL: https://issues.apache.org/jira/browse/FLINK-27644 Project: Flink Issue

[jira] [Created] (FLINK-27643) Document new deployment lifecycle features for the operator

2022-05-16 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-27643: -- Summary: Document new deployment lifecycle features for the operator Key: FLINK-27643 URL: https://issues.apache.org/jira/browse/FLINK-27643 Project: Flink

[jira] [Created] (FLINK-27642) Make sure that the Elasticsearch E2E tests only try a limited amount of retries in case of test failures

2022-05-16 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-27642: -- Summary: Make sure that the Elasticsearch E2E tests only try a limited amount of retries in case of test failures Key: FLINK-27642 URL:

Re: [DISCUSS] Next Flink Kubernetes Operator release timeline

2022-05-16 Thread Gyula Fóra
Hi Devs! We are on track for our planned 1.0.0 release timeline. There are no outstanding blocker issues on JIRA for the release. There are 3 outstanding new feature PRs. They are all in pretty good shape and should be merged within a day:

[jira] [Created] (FLINK-27641) Create view lost Time attribute in Hive Catalog

2022-05-16 Thread Robert Wu (Jira)
Robert Wu created FLINK-27641: - Summary: Create view lost Time attribute in Hive Catalog Key: FLINK-27641 URL: https://issues.apache.org/jira/browse/FLINK-27641 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-27640) Flink not compiling, flink-connector-hive_2.12 is missing pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde

2022-05-16 Thread Piotr Nowojski (Jira)
Piotr Nowojski created FLINK-27640: -- Summary: Flink not compiling, flink-connector-hive_2.12 is missing pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde Key: FLINK-27640 URL:

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

2022-05-16 Thread Konstantin Knauf
Hi Lijie, hm, maybe the following is more appropriate in that case POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge Best, Konstantin Am Mo., 16. Mai 2022 um 07:05 Uhr schrieb Lijie Wang < wangdachui9...@gmail.com>: > Hi Konstantin, > thanks for your feedback. > > From

[jira] [Created] (FLINK-27639) Flink JOIN uses the now() function when inserting data, resulting in data that cannot be deleted

2022-05-16 Thread lvycc (Jira)
lvycc created FLINK-27639: - Summary: Flink JOIN uses the now() function when inserting data, resulting in data that cannot be deleted Key: FLINK-27639 URL: https://issues.apache.org/jira/browse/FLINK-27639

Re: What is the alternative schema of DirectOutput

2022-05-16 Thread yuxia
Hi, Boot. For DirectedOutput, I guess you want to split stream. For splitting stream, you can use side_output[1] [1]: https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/side_output/ Best regards, Yuxia - 原始邮件 - 发件人: "Boot" <331233...@qq.com.INVALID> 收件人:

Fw: What is the alternative schema of DirectOutput

2022-05-16 Thread Boot
The correct class name is DirectedOutput ---Original--- From: "Boot"<331233...@qq.com Date: Mon, May 16, 2022 17:19 PM To: "user"

[jira] [Created] (FLINK-27638) failed to join with table function

2022-05-16 Thread Spongebob (Jira)
Spongebob created FLINK-27638: - Summary: failed to join with table function Key: FLINK-27638 URL: https://issues.apache.org/jira/browse/FLINK-27638 Project: Flink Issue Type: Bug

[RESULT][VOTE] FLIP-226: Introduce Schema Evolution on Table Store

2022-05-16 Thread Jingsong Li
Hi dev, FLIP-226 [1] Has been accepted. There [2] were 3 binding votes in favor. None against. Votes are in the order of arrival: - Binding: Jing Zhang - Binding: Jark Wu - Binding: Jingsong Lee [1]

Re: [VOTE] FLIP-226: Introduce Schema Evolution on Table Store

2022-05-16 Thread Jingsong Li
+1 (binding) Best, Jingsong On Mon, May 16, 2022 at 2:15 PM Jark Wu wrote: > +1 (binding) > > Best, > Jark > > On Mon, 16 May 2022 at 13:50, Jing Zhang wrote: > > > +1 > > Thanks @ Jingsong for driving this topic. > > > > Best, > > Jing Zhang > > > > Jingsong Li 于2022年5月12日周四 17:06写道: > > >

[jira] [Created] (FLINK-27637) Optimize the log information when the asynchronous part of checkpoint is canceled

2022-05-16 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-27637: -- Summary: Optimize the log information when the asynchronous part of checkpoint is canceled Key: FLINK-27637 URL: https://issues.apache.org/jira/browse/FLINK-27637

[jira] [Created] (FLINK-27636) Add data type coverage and sync / async tests for catalog in connector testing framework

2022-05-16 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-27636: - Summary: Add data type coverage and sync / async tests for catalog in connector testing framework Key: FLINK-27636 URL: https://issues.apache.org/jira/browse/FLINK-27636

[jira] [Created] (FLINK-27635) Add data type coverage and abilities test cases for table connectors in testing framework

2022-05-16 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-27635: - Summary: Add data type coverage and abilities test cases for table connectors in testing framework Key: FLINK-27635 URL: https://issues.apache.org/jira/browse/FLINK-27635

[jira] [Created] (FLINK-27634) Add component failure cases in connector testing framework

2022-05-16 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-27634: - Summary: Add component failure cases in connector testing framework Key: FLINK-27634 URL: https://issues.apache.org/jira/browse/FLINK-27634 Project: Flink

[jira] [Created] (FLINK-27633) Add FLIP-33 metric validation case in connector testing framework

2022-05-16 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-27633: - Summary: Add FLIP-33 metric validation case in connector testing framework Key: FLINK-27633 URL: https://issues.apache.org/jira/browse/FLINK-27633 Project: Flink

[jira] [Created] (FLINK-27632) Improve connector testing framework to support more cases

2022-05-16 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-27632: - Summary: Improve connector testing framework to support more cases Key: FLINK-27632 URL: https://issues.apache.org/jira/browse/FLINK-27632 Project: Flink

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-16 Thread Lincoln Lee
Hi Jinsong, Good question! The delayQueue is very similar to incompleteElements in UnorderedStreamElementQueue, it only records the references of in-flight retry elements, the core value is for the ease of a fast scan when force flush during endInput and less refactor for existing logic. Users

[jira] [Created] (FLINK-27631) Datastream job combined with table job

2022-05-16 Thread waywtdcc (Jira)
waywtdcc created FLINK-27631: Summary: Datastream job combined with table job Key: FLINK-27631 URL: https://issues.apache.org/jira/browse/FLINK-27631 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-27630) maven-source-plugin for table planner values connector for debug

2022-05-16 Thread jackylau (Jira)
jackylau created FLINK-27630: Summary: maven-source-plugin for table planner values connector for debug Key: FLINK-27630 URL: https://issues.apache.org/jira/browse/FLINK-27630 Project: Flink

Re: [ANNOUNCE] Problems with build due to tests using too much disk space

2022-05-16 Thread Martijn Visser
Hi all, Unfortunately, the issue has not been resolved yet. Will look into this again now (see https://issues.apache.org/jira/browse/FLINK-24433 for updates). Thanks, Martijn On Fri, 13 May 2022 at 13:52, Konstantin Knauf wrote: > Thanks a lot for taking care of this, Martijn! > > Am Fr.,

Re: [DISCUSS] FLIP-232: Add Retry Support For Async I/O In DataStream API

2022-05-16 Thread Jingsong Li
Thanks Lincoln for your reply. I'm a little confused about the relationship between Ordered/Unordered Queue and DelayQueue. Why do we need to have a DelayQueue? Can we remove the DelayQueue and put the state of the retry in the StreamRecordQueueEntry (seems like it's already in the FLIP) The

[jira] [Created] (FLINK-27629) Table Store throws NullPointerException when pushing down NotEqual predicate to a column consisting of nulls

2022-05-16 Thread Caizhi Weng (Jira)
Caizhi Weng created FLINK-27629: --- Summary: Table Store throws NullPointerException when pushing down NotEqual predicate to a column consisting of nulls Key: FLINK-27629 URL:

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

2022-05-16 Thread Timo Walther
Hi Shengkai, Hi Jark, thanks for the additional explanation and the update of the FLIP. This will help us in the future for documenting our decisions. The arguments why to include the Gateway into the main repo make a lot of sense to me. Esp. also because both CLI and gateway need some

Re: [VOTE] FLIP-226: Introduce Schema Evolution on Table Store

2022-05-16 Thread Jark Wu
+1 (binding) Best, Jark On Mon, 16 May 2022 at 13:50, Jing Zhang wrote: > +1 > Thanks @ Jingsong for driving this topic. > > Best, > Jing Zhang > > Jingsong Li 于2022年5月12日周四 17:06写道: > > > Hi, everyone > > > > Thanks all for your attention to FLIP-226: Introduce Schema Evolution on > > Table