[External] Access issue with ParquetColumnarRowInputFormat

hi Team, We recently started migrating flink from 1.14 to 1.15. We came across a challenge that till 1.14 ParquetColumnarRowInputFormat constructor was publicly accessible. But in version 1.15 it is accessible only within the package. As per documentation, constructors are allowed to be accessed

[jira] [Created] (FLINK-28385) Change the validator to return an error if the Jar URI is an empty string

Xin Hao created FLINK-28385: --- Summary: Change the validator to return an error if the Jar URI is an empty string Key: FLINK-28385 URL: https://issues.apache.org/jira/browse/FLINK-28385 Project: Flink

Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi everyone, Thank you to all those who participated in the discussion, the program has been gradually revised and improved, everyone has reached a consensus. I will relaunch vote soon. -- Best regards, Mang Zhang At 2022-07-05 11:54:07, "yuxia" wrote: Thanks for updating. Now, the

Re: [VOTE] FLIP-245: Source Supports Speculative Execution For Batch Job

+1 for the feature. Jing Zhang 于2022年7月5日周二 11:43写道： > Hi all, > > I'd like to start a vote for FLIP-245: Source Supports Speculative > Execution For Batch Job[1] on the discussion thread [2]. > > The vote will last for at least 72 hours unless there is an objection or > insufficient votes. > >

Re: Python Job Support for the Kubernetes Operator

Thank you for the info and help Dian :) Gyula On Tue, 5 Jul 2022 at 05:14, Yang Wang wrote: > Thanks Dian for the confirmation and nice help. > > Best, > Yang > > Dian Fu 于2022年7月5日周二 09:27写道： > > > @Yang, Yes, you are right. Python jobs could be seen as special JAR jobs > > whose main class

Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Thanks for updating. Now, the FLIP looks good to me. Best regards, Yuxia 发件人: "zhangmang1" 收件人: luoyu...@alumni.sjtu.edu.cn 抄送: "dev" , "Martijn Visser" , imj...@gmail.com 发送时间: 星期二, 2022年 7 月 05日 上午 11:35:35 主题: Re:Re: Re: Re: Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in

Re:Re: [VOTE] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi everyone, I'm sorry to bother you all, but since FLIP-218[1] has been updated, I'm going to relaunch VOTE. The main contents of the modification are: 1. remove rtas from option name 2. no longer introduce AtomicCatalog, add javadocs description for Catalog interface: If Catalog needs to

[VOTE] FLIP-245: Source Supports Speculative Execution For Batch Job

Hi all, I'd like to start a vote for FLIP-245: Source Supports Speculative Execution For Batch Job[1] on the discussion thread [2]. The vote will last for at least 72 hours unless there is an objection or insufficient votes. Best, Jing Zhang [1]

Re:Re: Re: Re: Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi yuxia, I updated the FLIP and adjusted your concern for RTAS and AtomicCatalog, not sure if it solved your concern, looking forward to your reply! -- Best regards, Mang Zhang At 2022-07-05 11:26:22, "Jark Wu" wrote: >Thanks for the update, the FLIP looks good to me now. > >Best,

RE: [DISCUSS] FLIP-236: Asynchronous Job Submission

Hello, sorry to bother but I want to know what’s the progress of this FLIP, we have facing the similar problem in Flink and FlinkSQL batch mode because of the resource shortage momentary and I think this may help us. Thanks! On 2022/05/30 16:11:24 "Afonso, Nuno" wrote: > Hello everyone, > >

RE: [DISCUSS] FLIP-236: Asynchronous Job Submission

Hello, sorry to bother but I want to know what’s the progress of this FLIP, we have facing the similar problem in Flink and FlinkSQL batch mode because of the resource shortage momentary and I think this may help us. Thanks! On 2022/05/30 16:11:24 "Afonso, Nuno" wrote: > Hello everyone, > >

Re: [DISCUSS] FLIP-245: Source Supports Speculative Execution For Batch Job

Hi everyone, Thanks a lot for all the feedback! I will open a vote for it since there is no more concern. Best, Jing Zhang Jing Zhang 于2022年7月5日周二 11:31写道： > Hi ZhuZhu, Jiangjie, > > Thanks a lot for your feedback. > > I agree that it's better to support most existing events. > I have updated

Re: [DISCUSS] FLIP-245: Source Supports Speculative Execution For Batch Job

Hi ZhuZhu, Jiangjie, Thanks a lot for your feedback. I agree that it's better to support most existing events. I have updated the FLIP to cover how to deal with the RequestSplitEvent/SourceEventWrapper/ReaderRegistrationEvent. The ReportedWatermarkEvent is only used in watermark alignment.

Re: Re: Re: Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Thanks for the update, the FLIP looks good to me now. Best, Jark On Tue, 5 Jul 2022 at 10:57, Mang Zhang wrote: > Hi Jark, > Regarding the two issues of concern to yuxia, we did some offline > discussions and adjusted the implementation plan. > > >1) RTAS > RTAS is not supported in FLIP, so we

Re: Python Job Support for the Kubernetes Operator

Thanks Dian for the confirmation and nice help. Best, Yang Dian Fu 于2022年7月5日周二 09:27写道： > @Yang, Yes, you are right. Python jobs could be seen as special JAR jobs > whose main class is always `org.apache.flink.client.python.PythonDriver`. > What we could do in Flink K8s operator is to make it

Re:Re: Re: Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi Jark, Regarding the two issues of concern to yuxia, we did some offline discussions and adjusted the implementation plan. >1) RTAS RTAS is not supported in FLIP, so we will remove rtas from option name and do option forward compatibility when it is supported in the future. >2)

[jira] [Created] (FLINK-28384) Add CatalogLock support for HiveCatalog in table store

Jingsong Lee created FLINK-28384: Summary: Add CatalogLock support for HiveCatalog in table store Key: FLINK-28384 URL: https://issues.apache.org/jira/browse/FLINK-28384 Project: Flink Issue

Re: Python Job Support for the Kubernetes Operator

@Yang, Yes, you are right. Python jobs could be seen as special JAR jobs whose main class is always `org.apache.flink.client.python.PythonDriver`. What we could do in Flink K8s operator is to make it more convenient and handle properly for the different kinds of dependencies[1]. @Gyula, I can

Re: [VOTE] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

+1 for the feature. Jark Wu 于2022年7月4日周一 17:33写道： > Hi Mang, > > I left a comment in the DISCUSS thread. > > Best, > Jark > > On Mon, 4 Jul 2022 at 15:24, Rui Fan <1996fan...@gmail.com> wrote: > > > Hi. > > > > Thanks Mang for this FLIP. I think it will be useful for users. > > > >

Re: [DISCUSS] FLIP-245: Source Supports Speculative Execution For Batch Job

Yes, that sounds reasonable to me. That said, supporting custom events might still be preferable if that does not complicate the design too much. It would be good to avoid having a tricky feature availability matrix when we add new features to the project. Thanks, Jiangjie (Becket) Qin On

Re: [DISCUSS] FLIP-238: Introduce FLIP-27-based Data Generator Source

Hi Becket, I updated and extended FLIP-238 accordingly. Here is also my POC branch [1]. DataGeneratorSourceV3 is the class that I currently converged on [2]. It is based on the expanded SourceReaderContext. A couple more relevant classes [3] [4] Would appreciate it if you could take a quick

Re: [DISCUSS] FLIP-243: Dedicated Opensearch connectors

Hi Martijn, Thanks a lot supporting the FLIP! Regarding your question, yes, Elasticsearch has a new client in v8 (deprecating HighLevelRestClient from now on) and, no, there won't be any impact on Opensearch (the connectors use Opensearch's own HighLevelRestClient, no deprecations yet). Thank

Re: Re: Re: Re: [DISCUSS] FLIP-239: Port JDBC Connector Source to FLIP-27

Hi Roc, Thanks for the FLIP and opening the discussion. I have a couple of initial questions/remarks: * The FLIP contains information for both Source and Sink, but nothing explicitly on the Lookup functionality. I'm assuming we also want to have that implementation covered while porting this to

[jira] [Created] (FLINK-28383) HDFS source: Trying to access closed classloader

Arseniy Tashoyan created FLINK-28383: Summary: HDFS source: Trying to access closed classloader Key: FLINK-28383 URL: https://issues.apache.org/jira/browse/FLINK-28383 Project: Flink

Re: [DISCUSS] FLIP-243: Dedicated Opensearch connectors

Hi Andriy, Thanks for creating the FLIP and opening the discussion. In general +1 for the FLIP, looking forward! One question: I recall that in order for Flink to support Elasticsearch 8, the connector needs to be overhauled to use Elastic's new client because only the new client has a

[jira] [Created] (FLINK-28382) Introduce new compression algorithms of higher compression ratio

Yingjie Cao created FLINK-28382: --- Summary: Introduce new compression algorithms of higher compression ratio Key: FLINK-28382 URL: https://issues.apache.org/jira/browse/FLINK-28382 Project: Flink

[jira] [Created] (FLINK-28381) Add Spark Reader documentation

Jingsong Lee created FLINK-28381: Summary: Add Spark Reader documentation Key: FLINK-28381 URL: https://issues.apache.org/jira/browse/FLINK-28381 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-28380) Produce one intermediate dataset for multiple consumers consuming the same data

Yingjie Cao created FLINK-28380: --- Summary: Produce one intermediate dataset for multiple consumers consuming the same data Key: FLINK-28380 URL: https://issues.apache.org/jira/browse/FLINK-28380

Re: [DISCUSS] Support partition pruning for streaming reading

Hi zoucao, Regarding the pruning function, maybe a simple filter function is enough, e.g. `FilterFunction partitionFilter`. Besides, it would be better to state clearly how the new `applyPartitionPuringFunction` method affects batch/bounded table sources. From my understanding, this method

Re: [DISCUSS] Support partition pruning for streaming reading

Hi zoucao, The FileSource does support streaming reading [1]. Best regards, Martijn [1] https://nightlies.apache.org/flink/flink-docs-release-1.15/api/java/org/apache/flink/connector/file/src/FileSource.html Op ma 4 jul. 2022 om 05:58 schreef godfrey he : > Hi zoucao, > > Look forward your

[jira] [Created] (FLINK-28379) Process produced no output for 900 seconds after/while running org.apache.flink.runtime.blob.BlobClientSslTest

Martijn Visser created FLINK-28379: -- Summary: Process produced no output for 900 seconds after/while running org.apache.flink.runtime.blob.BlobClientSslTest Key: FLINK-28379 URL:

Re: [DISCUSS] FLIP-238: Introduce FLIP-27-based Data Generator Source

Hi Becket, Exposing the RuntimeContext is potentially even more useful. Do you think it is worth having both currentParallelism() and getRuntimeContext() methods? One can always call getNumberOfParallelSubtasks() on the RuntimeContext directly if we expose it. Best, Alexander Fedulov On Mon,

Re: [NOTICE] Dev@ ML is not receiving jira issue created notifications

FYI, this has been fixed 4 days ago [1]. Cheers, Jark [1]: https://issues.apache.org/jira/browse/INFRA-23425?focusedCommentId=17560597=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17560597 On Wed, 29 Jun 2022 at 10:37, Xintong Song wrote: > Hi devs, > > I just

Re: [VOTE] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi Mang, I left a comment in the DISCUSS thread. Best, Jark On Mon, 4 Jul 2022 at 15:24, Rui Fan <1996fan...@gmail.com> wrote: > Hi. > > Thanks Mang for this FLIP. I think it will be useful for users. > > +1(non-binding) > > Best wishes > Rui Fan > > On Mon, Jul 4, 2022 at 3:01 PM Mang Zhang

Re: Re: Re:Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi Mang, I'm not sure whether your response has addressed Yuxia's concern or not. Would be better to receive a confirmation from participants before starting the vote. Actually, I have the same feeling with Yuxia's reply. 1) RTAS If it's hard to have a consistent behavior for RTAS between

[jira] [Created] (FLINK-28378) Use larger data reading buffer size for sort-shuffle

Yingjie Cao created FLINK-28378: --- Summary: Use larger data reading buffer size for sort-shuffle Key: FLINK-28378 URL: https://issues.apache.org/jira/browse/FLINK-28378 Project: Flink Issue

Re: [DISCUSS] FLIP-245: Source Supports Speculative Execution For Batch Job

Hi Jiangjie, Yes you are that the goals of watermark alignment and speculative execution do not conflict. For the example you gave, we can make it work by only aligning watermarks for executions that are pipelined connected (i.e. in the same execution attempt level pipelined region). Even not

[jira] [Created] (FLINK-28377) Enable to request less number of data reading buffers for sort-shuffle if there is no enough data

Yingjie Cao created FLINK-28377: --- Summary: Enable to request less number of data reading buffers for sort-shuffle if there is no enough data Key: FLINK-28377 URL: https://issues.apache.org/jira/browse/FLINK-28377

[jira] [Created] (FLINK-28376) Reduce the number of IO threads for sort-shuffle

Yingjie Cao created FLINK-28376: --- Summary: Reduce the number of IO threads for sort-shuffle Key: FLINK-28376 URL: https://issues.apache.org/jira/browse/FLINK-28376 Project: Flink Issue Type:

[jira] [Created] (FLINK-28375) Whether to consider adding other data type to support for last_value function

hehuiyuan created FLINK-28375: - Summary: Whether to consider adding other data type to support for last_value function Key: FLINK-28375 URL: https://issues.apache.org/jira/browse/FLINK-28375 Project:

[jira] [Created] (FLINK-28374) Some further improvements of sort-shuffle

Yingjie Cao created FLINK-28374: --- Summary: Some further improvements of sort-shuffle Key: FLINK-28374 URL: https://issues.apache.org/jira/browse/FLINK-28374 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-245: Source Supports Speculative Execution For Batch Job

Hi Zhu, I agree that if we are talking about a single execution region with blocking shuffle, watermark alignment may not be that helpful as the subtasks are running independently of each other. That said, I don't think watermark alignment and speculative execution necessarily conflict with each

[jira] [Created] (FLINK-28373) Read larger size of data sequentially for sort-shuffle

Yingjie Cao created FLINK-28373: --- Summary: Read larger size of data sequentially for sort-shuffle Key: FLINK-28373 URL: https://issues.apache.org/jira/browse/FLINK-28373 Project: Flink Issue

[jira] [Created] (FLINK-28372) Investigate Akka Artery

Chesnay Schepler created FLINK-28372: Summary: Investigate Akka Artery Key: FLINK-28372 URL: https://issues.apache.org/jira/browse/FLINK-28372 Project: Flink Issue Type: Technical Debt

Re: [DISCUSS] FLIP-245: Source Supports Speculative Execution For Batch Job

I had another thought and now I think watermark alignment is actually conceptually conflicted with speculative execution. This is because the idea of watermark alignment is to limit the progress of all sources to be around the progress of the slowest source in the watermark group. However,

Re: [VOTE] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi. Thanks Mang for this FLIP. I think it will be useful for users. +1(non-binding) Best wishes Rui Fan On Mon, Jul 4, 2022 at 3:01 PM Mang Zhang wrote: > Hi everyone, > > > > > Thanks for all the feedback so far. Based on the discussion [1], we seem > to have consensus. So, I would like to

[jira] [Created] (FLINK-28371) Upgrade netty version in flink-akka-rpc

Michael Groys created FLINK-28371: - Summary: Upgrade netty version in flink-akka-rpc Key: FLINK-28371 URL: https://issues.apache.org/jira/browse/FLINK-28371 Project: Flink Issue Type:

[jira] [Created] (FLINK-28370) Add close method for KafkaRecordSerializationSchema

Leonard Xu created FLINK-28370: -- Summary: Add close method for KafkaRecordSerializationSchema Key: FLINK-28370 URL: https://issues.apache.org/jira/browse/FLINK-28370 Project: Flink Issue Type:

[VOTE] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

Hi everyone, Thanks for all the feedback so far. Based on the discussion [1], we seem to have consensus. So, I would like to start a vote on FLIP-218 [2]. The vote will last for at least 72 hours unless there is an objection or insufficient votes. [1]

Re: [VOTE] Release 1.15.1, release candidate #1

Hi Dawid and Konstantin. +0 for this RC. Let's move on. Best, Jingsong On Mon, Jul 4, 2022 at 2:17 PM Konstantin Knauf wrote: > > Thanks Jingsong. I agree that this is a serious bug and we need to take > compatibility seriously. We shouldn't wait long until we release Flink > 1.15.2. Still I

Re: [DISCUSS] FLIP-245: Source Supports Speculative Execution For Batch Job

Thanks for updating the FLIP! I agree that at the moment users do not need watermark alignment(in which case ReportedWatermarkEvent would happen) in batch cases. However, I think the concept of watermark alignment is not conflicted with speculative execution. It can work with speculative

Re: [VOTE] Release 1.15.1, release candidate #1

Thanks Jingsong. I agree that this is a serious bug and we need to take compatibility seriously. We shouldn't wait long until we release Flink 1.15.2. Still I stand to my point, that we shouldn't abort the current release. --- +1 (binding) * checked release blog post * verified signatures and

Re: [VOTE] Apache Flink ML Release 2.1.0, release candidate #2

Thanks for raising this release candidate. +1 (non-binding) - Verified that the checksums and GPG files match the corresponding release files. - Verified that the source distributions do not contain any binaries. - Built the source distribution and ensured that all source files have Apache