[jira] [Created] (FLINK-25640) Enhance the document for blocking shuffle

2022-01-12 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-25640: --- Summary: Enhance the document for blocking shuffle Key: FLINK-25640 URL: https://issues.apache.org/jira/browse/FLINK-25640 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25639) Increase the default read buffer size of sort-shuffle to 64M

2022-01-12 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-25639: --- Summary: Increase the default read buffer size of sort-shuffle to 64M Key: FLINK-25639 URL: https://issues.apache.org/jira/browse/FLINK-25639 Project: Flink

[jira] [Created] (FLINK-25638) Increase the default write buffer size of sort-shuffle to 16M

2022-01-12 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-25638: --- Summary: Increase the default write buffer size of sort-shuffle to 16M Key: FLINK-25638 URL: https://issues.apache.org/jira/browse/FLINK-25638 Project: Flink

[jira] [Created] (FLINK-25637) Make sort-shuffle the default shuffle implementation for batch jobs

2022-01-12 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-25637: --- Summary: Make sort-shuffle the default shuffle implementation for batch jobs Key: FLINK-25637 URL: https://issues.apache.org/jira/browse/FLINK-25637 Project: Flink

[jira] [Created] (FLINK-25636) FLIP-199: Change some default config values of blocking shuffle for better usability

2022-01-12 Thread Yingjie Cao (Jira)
Yingjie Cao created FLINK-25636: --- Summary: FLIP-199: Change some default config values of blocking shuffle for better usability Key: FLINK-25636 URL: https://issues.apache.org/jira/browse/FLINK-25636

Re: Flink native k8s integration vs. operator

2022-01-12 Thread Thomas Weise
Hi everyone, Thanks for the feedback and discussion. A few additional thoughts: [Konstantin] > With respect to common lifecycle management operations: these features are > not available (within Apache Flink) for any of the other resource providers > (YARN, Standalone) either. From this

Re: [DISCUSS] Seek help for making JIRA links clickable in github

2022-01-12 Thread Jingsong Li
Thanks Yun Tang, flink-table-store and flink-ml can work now. Best, Jingsong On Wed, Jan 12, 2022 at 4:41 PM Jingsong Li wrote: > > Hi Yun Gao, > Yes, the issues of flink-statefun and flink-ml are also managed in the > issues.apache.org. But the FLINK-XX in github is not clickable. > > Thanks

[jira] [Created] (FLINK-25635) Using sql client create hive catalog throw exception

2022-01-12 Thread dalongliu (Jira)
dalongliu created FLINK-25635: - Summary: Using sql client create hive catalog throw exception Key: FLINK-25635 URL: https://issues.apache.org/jira/browse/FLINK-25635 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

2022-01-12 Thread Dong Lin
Hi Fabian, Thank you for the explanation. The current approach needs to add new constructors for SourceReaderBase and SingleThreadMultiplexSourceReaderBase. This proposed change has now been included in the Public Interfaces section in the FLIP. And yes, I also agree it would be preferred if

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

2022-01-12 Thread 张俊帆
Hi G, Thanks for starting the discussion. I think this is a important improvement for Flink. The proposal looks good to me. And I focus on one point. 1. Hope that keeping the consistent with current implementation, we rely on the config of  'security.kerberos.fetch.delegation-token’ to submit

[jira] [Created] (FLINK-25634) flink-read-onyarn-configuration

2022-01-12 Thread Jira
宇宙先生 created FLINK-25634: Summary: flink-read-onyarn-configuration Key: FLINK-25634 URL: https://issues.apache.org/jira/browse/FLINK-25634 Project: Flink Issue Type: Bug Affects Versions: 1.11.2

Re: [VOTE] Release 1.14.3, release candidate #1

2022-01-12 Thread Jingsong Li
+1 (non-binding) - Verified checksums and signatures - Build from source - Start a standalone cluster, web is OK - Start sql-client, everything looks good Best, Jingsong On Wed, Jan 12, 2022 at 11:00 PM Till Rohrmann wrote: > > +1 (binding) > > - Verified checksums and signatures > - Build

Re: [VOTE] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

2022-01-12 Thread Dian Fu
+1 (binding) Regards, Dian On Thu, Jan 13, 2022 at 6:50 AM Thomas Weise wrote: > +1 (binding) > > On Wed, Jan 12, 2022 at 4:58 AM Till Rohrmann > wrote: > > > > +1 (binding) > > > > Cheers, > > Till > > > > On Wed, Jan 12, 2022 at 11:07 AM Wei Zhong > wrote: > > > > > +1(binding) > > > > > >

Re: [VOTE] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

2022-01-12 Thread Thomas Weise
+1 (binding) On Wed, Jan 12, 2022 at 4:58 AM Till Rohrmann wrote: > > +1 (binding) > > Cheers, > Till > > On Wed, Jan 12, 2022 at 11:07 AM Wei Zhong wrote: > > > +1(binding) > > > > Best, > > Wei > > > > > 2022年1月12日 下午5:58,Xingbo Huang 写道: > > > > > > Hi all, > > > > > > I would like to start

[jira] [Created] (FLINK-25633) CPUResourceTest does not work with German Locale

2022-01-12 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-25633: - Summary: CPUResourceTest does not work with German Locale Key: FLINK-25633 URL: https://issues.apache.org/jira/browse/FLINK-25633 Project: Flink Issue

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

2022-01-12 Thread Fabian Paul
Hi Dong, I think I am beginning to understand your idea. Since SourceReaderBase is marked as PublicEvolving can you also update the FLIP with the changes you want to make to it? Ideally, connector developers do not have to change their SourceReaders to implement this new logic. My idea was to

[jira] [Created] (FLINK-25632) Makes Job name available in cleanup stage

2022-01-12 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-25632: - Summary: Makes Job name available in cleanup stage Key: FLINK-25632 URL: https://issues.apache.org/jira/browse/FLINK-25632 Project: Flink Issue Type:

Re: [VOTE] Release 1.14.3, release candidate #1

2022-01-12 Thread Till Rohrmann
+1 (binding) - Verified checksums and signatures - Build from source and ran StateMachineExample - Reviewed the flink-web PR - Verified that there were no other dependency changes than testcontainer, japicmp-plugin and log4j. Cheers, Till On Wed, Jan 12, 2022 at 9:58 AM Yun Tang wrote: > +1

Re: OutOfMemoryError: Java heap space while implmentating flink sql api

2022-01-12 Thread Martijn Visser
Hi Ronak, I would like to ask you to stop cross-posting to all the Flink mailing lists and then also post the same question to Stackoverflow. Both the mailing lists and Stackoverflow are designed for asynchronous communication and you should allow the community some days to address your question.

Re: OutOfMemoryError: Java heap space while implmentating flink sql api

2022-01-12 Thread Roman Khachatryan
Hi Ronak, You shared a screenshot of JM. Do you mean that exception also happens on JM? (I'd rather assume TM). Could you explain the join clause: left join ccmversionsumapTable cvsm ON (cdr.version = cvsm.ccmversion) "version" doesn't sound very selective, so maybe you end up with (almost)

Re: Could not find any factory for identifier 'jdbc'

2022-01-12 Thread Roman Khachatryan
Hi, I think Chesnay's suggestion to double-check the bundle makes sense. Additionally, I'd try flink-connector-jdbc_2.12 instead of flink-connector-jdbc_2.11. Regards, Roman On Wed, Jan 12, 2022 at 12:23 PM Chesnay Schepler wrote: > > I would try double-checking whether the jdbc connector was

Re: [VOTE] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

2022-01-12 Thread Till Rohrmann
+1 (binding) Cheers, Till On Wed, Jan 12, 2022 at 11:07 AM Wei Zhong wrote: > +1(binding) > > Best, > Wei > > > 2022年1月12日 下午5:58,Xingbo Huang 写道: > > > > Hi all, > > > > I would like to start the vote for FLIP-206[1], which was discussed and > > reached a consensus in the discussion

[jira] [Created] (FLINK-25631) Support enhanced `show tables` statement

2022-01-12 Thread Yubin Li (Jira)
Yubin Li created FLINK-25631: Summary: Support enhanced `show tables` statement Key: FLINK-25631 URL: https://issues.apache.org/jira/browse/FLINK-25631 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-25630) Introduce MergeTree writer and reader for table store

2022-01-12 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-25630: Summary: Introduce MergeTree writer and reader for table store Key: FLINK-25630 URL: https://issues.apache.org/jira/browse/FLINK-25630 Project: Flink Issue

[jira] [Created] (FLINK-25629) Introduce CompactStrategy and CompactManager for table store

2022-01-12 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-25629: Summary: Introduce CompactStrategy and CompactManager for table store Key: FLINK-25629 URL: https://issues.apache.org/jira/browse/FLINK-25629 Project: Flink

[jira] [Created] (FLINK-25628) Introduce RecordReader and related classes for table store

2022-01-12 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-25628: Summary: Introduce RecordReader and related classes for table store Key: FLINK-25628 URL: https://issues.apache.org/jira/browse/FLINK-25628 Project: Flink

Re: Could not find any factory for identifier 'jdbc'

2022-01-12 Thread Chesnay Schepler
I would try double-checking whether the jdbc connector was truly bundled in your jar, specifically whether org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory is. I can't think of a reason why this shouldn't work for the JDBC connector. On 12/01/2022 06:34, Ronak Beejawat (rbeejawa)

Re: Flink native k8s integration vs. operator

2022-01-12 Thread Konstantin Knauf
cc dev@ Hi Thomas, Hi everyone, Thank you for starting this discussion and sorry for chiming in late. I agree with Thomas' and David's assessment of Flink's "Native Kubernetes Integration", in particular, it does actually not integrate well with the Kubernetes ecosystem despite being called

[DISCUSS] Data quality by apache flink

2022-01-12 Thread tanjialiang
Hi everyone, I would like to start a discussion thread on "Flink SQL support data quality" For example, I have a SQL job, in this job i have a source table with a column named phone, and i want to set the data quality of the data in the column phone's pattern, such as it must match the pattern

Re: [VOTE] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

2022-01-12 Thread Wei Zhong
+1(binding) Best, Wei > 2022年1月12日 下午5:58,Xingbo Huang 写道: > > Hi all, > > I would like to start the vote for FLIP-206[1], which was discussed and > reached a consensus in the discussion thread[2]. > > The vote will be open for at least 72h, unless there is an objection or not > enough

OutOfMemoryError: Java heap space while implmentating flink sql api

2022-01-12 Thread Ronak Beejawat (rbeejawa)
Hi Team, I was trying to implement flink sql api join with 2 tables it is throwing error OutOfMemoryError: Java heap space . PFB screenshot for flink cluster memory details. [Flink Memory Model][1] [1]: https://i.stack.imgur.com/AOnQI.png **PFB below code snippet which I was trying:** ```

[VOTE] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

2022-01-12 Thread Xingbo Huang
Hi all, I would like to start the vote for FLIP-206[1], which was discussed and reached a consensus in the discussion thread[2]. The vote will be open for at least 72h, unless there is an objection or not enough votes. [1]

[jira] [Created] (FLINK-25627) Add basic structures of file store in table-store

2022-01-12 Thread Jingsong Lee (Jira)
Jingsong Lee created FLINK-25627: Summary: Add basic structures of file store in table-store Key: FLINK-25627 URL: https://issues.apache.org/jira/browse/FLINK-25627 Project: Flink Issue

Re: [DISCUSS] FLIP-206: Support PyFlink Runtime Execution in Thread Mode

2022-01-12 Thread Xingbo Huang
Hi Thomas, Thanks for the confirmation. I will now start a vote. Best, Xingbo Thomas Weise 于2022年1月12日周三 02:20写道: > Hi Xingbo, > > +1 from my side > > Thanks for the clarification. For your use case the parameter size and > therefore serialization overhead was the limiting factor. I have seen

Re: what is efficient way to write Left join in flink

2022-01-12 Thread Chesnay Schepler
Your best bet is to try out both approaches with some representative data. On 12/01/2022 08:11, Ronak Beejawat (rbeejawa) wrote: Hi Team, Can you please help me with the below query, I wanted to know which approach will be better and efficient for multiple left join within one min tumbling

Re: [VOTE] Release 1.14.3, release candidate #1

2022-01-12 Thread Yun Tang
+1 (non-binding) * Checked the signature of source code, some of binaries and some of python packages. * Launch a local cluster successfully on linux machine with correct flink-version and commit id and run the state machine example successfully as expected. * Reviewed the

Re: Use of JIRA fixVersion

2022-01-12 Thread Till Rohrmann
Hi Thomas, Thanks for creating this proposal. I agree with Konstantin and you that a bit more guidance for this topic is definitely helpful. I would be ok with adopting your proposal. Cheers, Till On Wed, Jan 12, 2022 at 9:23 AM Konstantin Knauf wrote: > Hi Thomas, > > I am also +1 to set

what is efficient way to write Left join in flink

2022-01-12 Thread Ronak Beejawat (rbeejawa)
Hi Team, Can you please help me with the below query, I wanted to know which approach will be better and efficient for multiple left join within one min tumbling window concept (Datastream Vs SQL API wrt. performance and memory management) Use case : 1. We have topic one (testtopic1) which

RE: Could not find any factory for identifier 'jdbc'

2022-01-12 Thread Ronak Beejawat (rbeejawa)
Hi Chesnay, How do you ensure that the connector is actually available at runtime? We are providing below mentioned dependency inside pom.xml with scope compile that will be available in class path and it was there in my fink job bundled jar. Same we are doing the same for other connector say

Re: [DISCUSS] Seek help for making JIRA links clickable in github

2022-01-12 Thread Jingsong Li
Hi Yun Gao, Yes, the issues of flink-statefun and flink-ml are also managed in the issues.apache.org. But the FLINK-XX in github is not clickable. Thanks Yun Tang! That's what I want. Best, Jingsong On Wed, Jan 12, 2022 at 4:08 PM Yun Tang wrote: > > Hi Jingsong, > > I have already created the

Re: [DISCUSS] Deprecate MapR FS

2022-01-12 Thread Martijn Visser
Thanks everyone. We've moved forward with removing the MapR file system in Flink 1.15. On Tue, 11 Jan 2022 at 10:37, Jingsong Li wrote: > +1 for dropping the MapR FS. Thanks for driving. > > Best, > Jingsong > > On Tue, Jan 11, 2022 at 5:22 PM Chang Li wrote: > > > > +1 for dropping the MapR

Re: Use of JIRA fixVersion

2022-01-12 Thread Konstantin Knauf
Hi Thomas, I am also +1 to set fixVersion more conservatively, in particular for patch releases. I am not sure we can or should solve this in a "strict" way. I am happy to give contributors and contributing teams a bit of freedom in how they use fixVersion for planning, but giving more concrete

Re: [VOTE] FLIP-199: Change some default config values of blocking shuffle for better usability

2022-01-12 Thread Guowei Ma
+1(binding) Best, Guowei On Wed, Jan 12, 2022 at 3:44 PM Jingsong Li wrote: > +1 Thanks Yingjie for driving. > > Best, > Jingsong Lee > > On Wed, Jan 12, 2022 at 3:16 PM 刘建刚 wrote: > > > > +1 for the proposal. In fact, we have used these params in our inner > flink > > version for good

Re: [DISCUSS] Seek help for making JIRA links clickable in github

2022-01-12 Thread Yun Tang
Hi Jingsong, I have already created the ticket [1] to ask for help from ASF infrastructure. Let's wait to see the progress to make it done in github. [1] https://issues.apache.org/jira/browse/INFRA-22729 Best Yun Tang From: Yun Gao Sent: Wednesday, January 12,

[jira] [Created] (FLINK-25626) Azure failed due to "Error occurred in starting fork"

2022-01-12 Thread Yun Gao (Jira)
Yun Gao created FLINK-25626: --- Summary: Azure failed due to "Error occurred in starting fork" Key: FLINK-25626 URL: https://issues.apache.org/jira/browse/FLINK-25626 Project: Flink Issue Type: Bug