[jira] [Created] (FLINK-31296) Add JoinConditionEqualityTransferRule to stream optimizer
Aitozi created FLINK-31296:
--
Summary: Add JoinConditionEqualityTransferRule to stream optimizer
Key: FLINK-31296
URL: https://issues.apache.org/jira/browse/FLINK-31296
Project: Flink
Issue Type: Improvement
Components: Table SQL / Planner
Reporter: Aitozi
I find that {{JoinConditionEqualityTransferRule}} is a common rule for batch
and stream mode. So it should be added to the stream optimizer which will bring
performance improvement in some case.
Maybe, other rules also need to be reviewed whether can be aligned in batch and
stream case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31295) When kerbero authentication is enabled, hadoop_ Proxy cannot take effect.
HunterXHunter created FLINK-31295: - Summary: When kerbero authentication is enabled, hadoop_ Proxy cannot take effect. Key: FLINK-31295 URL: https://issues.apache.org/jira/browse/FLINK-31295 Project: Flink Issue Type: Bug Reporter: HunterXHunter When I set ： security.kerberos.login.keytab: kerbero_user.keytab security.kerberos.login.principal: kerbero_user and set HADOOP_PROXY_USER = proxy_user Data is still written to hdfs as user kerbero_user. But : When I turn off kerbero authentication. data is written to hdfs as user proxy_user. Finally, I found the logic in HadoopModule#install would not be used as a hadoop proxy when kerbero authentication is enabled. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Flink minor version support policy for old releases
+1 (binding) Best Regards, Yu On Thu, 2 Mar 2023 at 09:53, Jark Wu wrote: > +1 (binding) > > Best, > Jark > > > 2023年3月2日 05:03，Gyula Fóra 写道： > > > > +1 (binding) > > > > Gyula > > > > On Wed, Mar 1, 2023 at 9:57 PM Thomas Weise wrote: > > > >> +1 (binding) > >> > >> Thanks, > >> Thomas > >> > >> On Tue, Feb 28, 2023 at 6:53 AM Sergey Nuyanzin > >> wrote: > >> > >>> +1 (non-binding) > >>> > >>> Thanks for driving this Danny. > >>> > >>> On Tue, Feb 28, 2023 at 9:41 AM Samrat Deb > >> wrote: > >>> > +1 (non binding) > > Thanks for driving it > > Bests, > Samrat > > On Tue, 28 Feb 2023 at 1:36 PM, Junrui Lee > >> wrote: > > > Thanks Danny for driving it. > > > > +1 (non-binding) > > > > Best regards, > > Junrui > > > > yuxia 于2023年2月28日周二 14:04写道： > > > >> Thanks Danny for driving it. > >> > >> +1 (non-binding) > >> > >> Best regards, > >> Yuxia > >> > >> - 原始邮件 - > >> 发件人: "Weihua Hu" > >> 收件人: "dev" > >> 发送时间: 星期二, 2023年 2 月 28日 下午 12:48:09 > >> 主题: Re: [VOTE] Flink minor version support policy for old releases > >> > >> Thanks, Danny. > >> > >> +1 (non-binding) > >> > >> Best, > >> Weihua > >> > >> > >> On Tue, Feb 28, 2023 at 12:38 PM weijie guo < > >>> guoweijieres...@gmail.com > > > >> wrote: > >> > >>> Thanks Danny for bring this. > >>> > >>> +1 (non-binding) > >>> > >>> Best regards, > >>> > >>> Weijie > >>> > >>> > >>> Jing Ge 于2023年2月27日周一 20:23写道： > >>> > +1 (non-binding) > > BTW, should we follow the content style [1] to describe the new > rule > >>> using > 1.2.x, 1.1.y, 1.1.z? > > [1] > > https://flink.apache.org/downloads/#update-policy-for-old-releases > > Best regards, > Jing > > On Mon, Feb 27, 2023 at 1:06 PM Matthias Pohl > wrote: > > > Thanks, Danny. Sounds good to me. > > > > +1 (non-binding) > > > > On Wed, Feb 22, 2023 at 10:11 AM Danny Cranmer < > >>> dannycran...@apache.org> > > wrote: > > > >> I am starting a vote to update the "Update Policy for old > > releases" > >>> [1] > > to > >> include additional bugfix support for end of life versions. > >> > >> As per the discussion thread [2], the change we are voting > >> on > is: > >> - Support policy: updated to include: "Upon release of a > >> new > > Flink > minor > >> version, the community will perform one final bugfix > >> release > for > resolved > >> critical/blocker issues in the Flink minor version losing > > support." > >> - Release process: add a step to start the discussion > >> thread > for > >> the > > final > >> patch version, if there are resolved critical/blocking > >> issues > to > >>> flush. > >> > >> Voting schema: since our bylaws [3] do not cover this > particular > > scenario, > >> and releases require PMC involvement, we will use a > >> consensus > > vote > >>> with > > PMC > >> binding votes. > >> > >> Thanks, > >> Danny > >> > >> [1] > > > >> > >>> https://flink.apache.org/downloads.html#update-policy-for-old-releases > >> [2] > >> https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv > >> [3] > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > >> > > > > >>> > >> > > > > >>> > >>> > >>> -- > >>> Best regards, > >>> Sergey > >>> > >> > >
[jira] [Created] (FLINK-31294) CommitterOperator forgot to close Committer when closing.
ming li created FLINK-31294:
---
Summary: CommitterOperator forgot to close Committer when closing.
Key: FLINK-31294
URL: https://issues.apache.org/jira/browse/FLINK-31294
Project: Flink
Issue Type: Bug
Components: Table Store
Reporter: ming li
{{CommitterOperator}} does not close the {{Committer}} when it closes, which
may lead to resource leaks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager
Thanks for the test plan, Weihua! Yes, it addresses my concerns. Thanks, John On Wed, Mar 1, 2023, at 22:38, Weihua Hu wrote: > Hi, everyone, > Thanks for your suggestions and ideas. > Thanks Xintong for sharing the detailed backgrounds of SlotManager. > > *@Matthias > > 1. Did you do a proper test coverage analysis? > > > Just as Xintong said, we already have a CI stage for fine grained resource > managers. > And I will make sure FineGrainedSlotManager as the default SlotManager can > pass all the tests of CI. > In addition, I will review all unit tests of DeclarativeSlotManager(DSM) to > ensure that there are no gaps in the > coverage provided by the FineGrainedSlotManager. > I also added the 'Test Plan' part to the FLIP. > @Matthias @John @Shammon Does this test plan address your concerns? > > 2. DeclarativeSlotManager and FineGrainedSlotManager feel quite big in > > terms of lines of code > > > IMO, the refactoring of SlotManager does not belong to this FLIP since it > may lead to some unstable risks. For > FineGrainedSlotManager(FGSM), we already split some reasonable components. > They are: > * TaskManagerTracker: Track task managers and their resources. > * ResourceTracker: track requirements of jobs > * ResourceAllocationStrategy: Try to fulfill the resource requirements with > available/pending resources. > * SlotStatusSyncer: communicate with TaskManager, for allocating/freeing > slot and reconciling the slot status > Maybe we can start a discussion about refactoring SlotManager in another > FLIP if there are some good suggestions. > WDYT > > 3. For me personally, having a more detailed summary comparing the >> subcomponents of both SlotManager implementations with where >> their functionality matches and where they differ might help understand the >> consequences of the changes proposed in FLIP-298 > > Good suggestion, I have updated the comparison in this FLIP. Looking > forward to any suggestions/thoughts > if they are not described clearly. > > *@John > > 4. In addition to changing the default, would it make sense to log a >> deprecation warning on initialization > > if the DeclarativeSlotManager is used? >> > SGTM, We should add Deprecated annotations to DSM for devs. And log a > deprecation warning for users. > > *@Shammon > > 1. For their functional differences, can you give some detailed tests to >> verify that the new FineGrainedSlotManager has these capabilities? This can >> effectively verify the new functions >> > As just maintained, there is already a CI stage of FGSM, and I will do more > review of unit tests for DSM. > > 2. I'm worried that many functions are not independent and it is difficult >> to migrate step-by-step. You can list the relationship between them in >> detail. > > As Xintong saied the DSM is a subset of FGSM by design. But as time goes > on, FGSM has some lacking > functions as I listed in this FLIP. And I have added the comparison between > DSM and FGSM in this FLIP. > > > Thanks again for all your thoughts. Any feedback is appreciated! > > Best, > Weihua > > > On Wed, Mar 1, 2023 at 2:17 PM Xintong Song wrote: > >> Thanks Weihua for preparing this FLIP. +1 for the proposal. >> >> >> As one of the contributors of the fine-grained slot manager, I'd like to >> share some backgrounds here. >> >> - There used to be a defaut slot manager implementation, which is >> non-declarative and has been removed now. The two features, declarative / >> reactive resource management and fine-grained resource management, were >> proposed at about the same time. We were aware that by design the >> declarative slot manager is a subset of fine-grained slot manager at that >> time, but still decided to implement two slot managers for the purpose of >> decoupling efforts and reducing cross-team synchronization overhead. >> Merging the two slot managers once they are proved stable is IMO a >> technical debt that was planned at the very beginning. >> >> - The FineGrainedSlotManager has been verified in Alibaba's internal >> production as well as Alibaba Cloud services as the default slot manager >> for about 2 years. >> >> >> Concerning test cases, we currently have a ci stage for fine grained >> resource management. To avoid adding too much burden, the stage only >> includes tests from flink-runtime and flink-test modules. I think switching >> the default slot manager and applying the whole set of tests on the >> fine-grained slot manager would help us to be more confident about it. >> >> >> Concerning cutting out functionalities out of slot manager, I think Yangze >> and I have tried our best to shape the FineGrainedSlotManager into >> reasonable components. I personally don't have other ideas to further >> disassemble the component, but I'm open to such suggestions. However, from >> the stability perspective, I'd be in favor of not introducing significant >> changes to the FineGrainedSlotManager while switching it to the default. >> Because the current implementation has
Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager
Hi, everyone, Thanks for your suggestions and ideas. Thanks Xintong for sharing the detailed backgrounds of SlotManager. *@Matthias 1. Did you do a proper test coverage analysis? Just as Xintong said, we already have a CI stage for fine grained resource managers. And I will make sure FineGrainedSlotManager as the default SlotManager can pass all the tests of CI. In addition, I will review all unit tests of DeclarativeSlotManager(DSM) to ensure that there are no gaps in the coverage provided by the FineGrainedSlotManager. I also added the 'Test Plan' part to the FLIP. @Matthias @John @Shammon Does this test plan address your concerns? 2. DeclarativeSlotManager and FineGrainedSlotManager feel quite big in terms of lines of code IMO, the refactoring of SlotManager does not belong to this FLIP since it may lead to some unstable risks. For FineGrainedSlotManager(FGSM), we already split some reasonable components. They are: * TaskManagerTracker: Track task managers and their resources. * ResourceTracker: track requirements of jobs * ResourceAllocationStrategy: Try to fulfill the resource requirements with available/pending resources. * SlotStatusSyncer: communicate with TaskManager, for allocating/freeing slot and reconciling the slot status Maybe we can start a discussion about refactoring SlotManager in another FLIP if there are some good suggestions. WDYT 3. For me personally, having a more detailed summary comparing the > subcomponents of both SlotManager implementations with where > their functionality matches and where they differ might help understand the > consequences of the changes proposed in FLIP-298 Good suggestion, I have updated the comparison in this FLIP. Looking forward to any suggestions/thoughts if they are not described clearly. *@John 4. In addition to changing the default, would it make sense to log a > deprecation warning on initialization if the DeclarativeSlotManager is used? > SGTM, We should add Deprecated annotations to DSM for devs. And log a deprecation warning for users. *@Shammon 1. For their functional differences, can you give some detailed tests to > verify that the new FineGrainedSlotManager has these capabilities? This can > effectively verify the new functions > As just maintained, there is already a CI stage of FGSM, and I will do more review of unit tests for DSM. 2. I'm worried that many functions are not independent and it is difficult > to migrate step-by-step. You can list the relationship between them in > detail. As Xintong saied the DSM is a subset of FGSM by design. But as time goes on, FGSM has some lacking functions as I listed in this FLIP. And I have added the comparison between DSM and FGSM in this FLIP. Thanks again for all your thoughts. Any feedback is appreciated! Best, Weihua On Wed, Mar 1, 2023 at 2:17 PM Xintong Song wrote: > Thanks Weihua for preparing this FLIP. +1 for the proposal. > > > As one of the contributors of the fine-grained slot manager, I'd like to > share some backgrounds here. > > - There used to be a defaut slot manager implementation, which is > non-declarative and has been removed now. The two features, declarative / > reactive resource management and fine-grained resource management, were > proposed at about the same time. We were aware that by design the > declarative slot manager is a subset of fine-grained slot manager at that > time, but still decided to implement two slot managers for the purpose of > decoupling efforts and reducing cross-team synchronization overhead. > Merging the two slot managers once they are proved stable is IMO a > technical debt that was planned at the very beginning. > > - The FineGrainedSlotManager has been verified in Alibaba's internal > production as well as Alibaba Cloud services as the default slot manager > for about 2 years. > > > Concerning test cases, we currently have a ci stage for fine grained > resource management. To avoid adding too much burden, the stage only > includes tests from flink-runtime and flink-test modules. I think switching > the default slot manager and applying the whole set of tests on the > fine-grained slot manager would help us to be more confident about it. > > > Concerning cutting out functionalities out of slot manager, I think Yangze > and I have tried our best to shape the FineGrainedSlotManager into > reasonable components. I personally don't have other ideas to further > disassemble the component, but I'm open to such suggestions. However, from > the stability perspective, I'd be in favor of not introducing significant > changes to the FineGrainedSlotManager while switching it to the default. > Because the current implementation has already been verified (or at least > partially verified because Alibaba does not cover all the Flink use cases), > and introducing more changes also means more chances of breaking things. > > > Best, > > Xintong > > > > On Wed, Mar 1, 2023 at 11:12 AM Shammon FY wrote: > > > Hi > > > > Thanks for
[jira] [Created] (FLINK-31293) Request memory segment from LocalBufferPool may hanging forever.
Weijie Guo created FLINK-31293: -- Summary: Request memory segment from LocalBufferPool may hanging forever. Key: FLINK-31293 URL: https://issues.apache.org/jira/browse/FLINK-31293 Project: Flink Issue Type: Bug Affects Versions: 1.17.0 Reporter: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31292) User HadoopUtils to get Configuration in CatalogContext
Jingsong Lee created FLINK-31292: Summary: User HadoopUtils to get Configuration in CatalogContext Key: FLINK-31292 URL: https://issues.apache.org/jira/browse/FLINK-31292 Project: Flink Issue Type: Improvement Components: Table Store Reporter: Jingsong Lee Fix For: table-store-0.4.0 At present, if HadoopConf is not passed in the CatalogContext, a new HadoopConf will be directly generated, which may not have the required parameters. We can refer to HadoopUtils to obtain hadoopConf from the configuration and environment variables. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31291) Document table.exec.sink.upsert-materialize to none
Jingsong Lee created FLINK-31291: Summary: Document table.exec.sink.upsert-materialize to none Key: FLINK-31291 URL: https://issues.apache.org/jira/browse/FLINK-31291 Project: Flink Issue Type: Improvement Components: Table Store Reporter: Jingsong Lee Fix For: table-store-0.4.0 The table store has the ability to correct disorder, such as: [https://nightlies.apache.org/flink/flink-table-store-docs-master/docs/concepts/primary-key-table/#sequence-field] But Flink SQL default sink materialize will result strange behavior, In particular, write to the agg table of the fts. We should document this, set table.exec.sink.upsert-materialize to none always, set 'sequence.field' to table in case of disorder. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31290) Remove features in documentation
Jingsong Lee created FLINK-31290: Summary: Remove features in documentation Key: FLINK-31290 URL: https://issues.apache.org/jira/browse/FLINK-31290 Project: Flink Issue Type: Improvement Components: Table Store Reporter: Jingsong Lee Fix For: table-store-0.4.0 Features is confused in documentation. Now, there are two pages in features, log system and lookup join. We can move log system to concepts. And move lookup join to how-to. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31289) Default aggregate-function for field can be last_non_null_value
Jingsong Lee created FLINK-31289: Summary: Default aggregate-function for field can be last_non_null_value Key: FLINK-31289 URL: https://issues.apache.org/jira/browse/FLINK-31289 Project: Flink Issue Type: Improvement Components: Table Store Reporter: Jingsong Lee Fix For: table-store-0.4.0 At present, when aggfunc is not configured, NPE will be generated. When the table is oriented to many fields, the configuration will be more troublesome. We can give the field the default aggfunc, such as last_ non_ null_ Value, which is consistent with the partial-update table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31288) Disable overdraft buffer for batch shuffle
Weijie Guo created FLINK-31288: -- Summary: Disable overdraft buffer for batch shuffle Key: FLINK-31288 URL: https://issues.apache.org/jira/browse/FLINK-31288 Project: Flink Issue Type: Bug Reporter: Weijie Guo Assignee: Weijie Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31287) Default value of 'changelog-producer.compaction-interval' can be zero
Jingsong Lee created FLINK-31287: Summary: Default value of 'changelog-producer.compaction-interval' can be zero Key: FLINK-31287 URL: https://issues.apache.org/jira/browse/FLINK-31287 Project: Flink Issue Type: Improvement Components: Table Store Reporter: Jingsong Lee Fix For: table-store-0.4.0 At present, the 30-minute interval is too conservative. We can set it to 0 by default, so that each checkpoint will have a full-compaction and generate a changelog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31286) Python processes are still alive when shutting down a session cluster directly without stopping the jobs
Dian Fu created FLINK-31286: --- Summary: Python processes are still alive when shutting down a session cluster directly without stopping the jobs Key: FLINK-31286 URL: https://issues.apache.org/jira/browse/FLINK-31286 Project: Flink Issue Type: Bug Components: API / Python Reporter: Dian Fu Assignee: Dian Fu Attachments: image-2023-03-02-10-55-34-863.png Reproduce steps: 1) start a standalone cluster: ./bin/start_cluster.sh 2) submit a PyFlink job which contains Python UDFs 3) stop the cluster: ./bin/stop_cluster.sh 4) Check if Python process still exists: ps aux | grep -i beam_boot !image-2023-03-02-10-55-34-863.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Flink minor version support policy for old releases
+1 (binding) Best, Jark > 2023年3月2日 05:03，Gyula Fóra 写道： > > +1 (binding) > > Gyula > > On Wed, Mar 1, 2023 at 9:57 PM Thomas Weise wrote: > >> +1 (binding) >> >> Thanks, >> Thomas >> >> On Tue, Feb 28, 2023 at 6:53 AM Sergey Nuyanzin >> wrote: >> >>> +1 (non-binding) >>> >>> Thanks for driving this Danny. >>> >>> On Tue, Feb 28, 2023 at 9:41 AM Samrat Deb >> wrote: >>> +1 (non binding) Thanks for driving it Bests, Samrat On Tue, 28 Feb 2023 at 1:36 PM, Junrui Lee >> wrote: > Thanks Danny for driving it. > > +1 (non-binding) > > Best regards, > Junrui > > yuxia 于2023年2月28日周二 14:04写道： > >> Thanks Danny for driving it. >> >> +1 (non-binding) >> >> Best regards, >> Yuxia >> >> - 原始邮件 - >> 发件人: "Weihua Hu" >> 收件人: "dev" >> 发送时间: 星期二, 2023年 2 月 28日 下午 12:48:09 >> 主题: Re: [VOTE] Flink minor version support policy for old releases >> >> Thanks, Danny. >> >> +1 (non-binding) >> >> Best, >> Weihua >> >> >> On Tue, Feb 28, 2023 at 12:38 PM weijie guo < >>> guoweijieres...@gmail.com > >> wrote: >> >>> Thanks Danny for bring this. >>> >>> +1 (non-binding) >>> >>> Best regards, >>> >>> Weijie >>> >>> >>> Jing Ge 于2023年2月27日周一 20:23写道： >>> +1 (non-binding) BTW, should we follow the content style [1] to describe the new rule >>> using 1.2.x, 1.1.y, 1.1.z? [1] > https://flink.apache.org/downloads/#update-policy-for-old-releases Best regards, Jing On Mon, Feb 27, 2023 at 1:06 PM Matthias Pohl wrote: > Thanks, Danny. Sounds good to me. > > +1 (non-binding) > > On Wed, Feb 22, 2023 at 10:11 AM Danny Cranmer < >>> dannycran...@apache.org> > wrote: > >> I am starting a vote to update the "Update Policy for old > releases" >>> [1] > to >> include additional bugfix support for end of life versions. >> >> As per the discussion thread [2], the change we are voting >> on is: >> - Support policy: updated to include: "Upon release of a >> new > Flink minor >> version, the community will perform one final bugfix >> release for resolved >> critical/blocker issues in the Flink minor version losing > support." >> - Release process: add a step to start the discussion >> thread for >> the > final >> patch version, if there are resolved critical/blocking >> issues to >>> flush. >> >> Voting schema: since our bylaws [3] do not cover this particular > scenario, >> and releases require PMC involvement, we will use a >> consensus > vote >>> with > PMC >> binding votes. >> >> Thanks, >> Danny >> >> [1] > >> >>> https://flink.apache.org/downloads.html#update-policy-for-old-releases >> [2] >> https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv >> [3] > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws >> > >>> >> > >>> >>> >>> -- >>> Best regards, >>> Sergey >>> >>
[jira] [Created] (FLINK-31285) FileSource should support reading files in order
Yaroslav Tkachenko created FLINK-31285: -- Summary: FileSource should support reading files in order Key: FLINK-31285 URL: https://issues.apache.org/jira/browse/FLINK-31285 Project: Flink Issue Type: New Feature Components: Connectors / FileSystem Affects Versions: 1.18.0 Reporter: Yaroslav Tkachenko Currently, Flink's *FileSource* uses *LocalityAwareSplitAssigner* as a default *FileSplitAssigner* and it doesn't guarantee any order. In many scenarios involving processing historical data, reading files in order can be a requirement, especially when using event-time processing. I believe a new FileSplitAssigner should be implemented that supports ordering. FileSourceBuilder should be extended to allow choosing a different FileSplitAssigner. It's also clear that the files may not be read in _perfect_ order with parallelism > 1. However, in some cases, using parallelism of 1 might be fine. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[DISCUSS] FLIP-299 Pub/Sub Lite Connector
Hello all, I'd like to start an official discuss thread for adding a Pub/Sub Lite Connector to Flink. We've had requests from our users to add flink support, and are willing to maintain and support this connector long term from the product team. The proposal is https://cwiki.apache.org/confluence/x/P51bDg, what would be people's thoughts on adding this connector? -Daniel
Re: Contributing a Google Cloud Pub/Sub Lite source and sink?
Thanks Danny! I've written a FLIP doc and will send out a new email thread. On Wed, Mar 1, 2023 at 2:43 PM Danny Cranmer wrote: > Hey Daniel, I have granted you access. Please try again. > > Thanks, > Danny > > On Wed, Mar 1, 2023 at 6:48 PM Daniel Collins > > wrote: > > > Hello, > > > > It appears that I still do not have the ability to create Confluence > pages? > > Could someone possibly grant me access? > > > > -Daniel > > > > On Tue, Feb 28, 2023 at 10:20 AM Daniel Collins > > wrote: > > > > > Thanks! My Jira username is `dpcollins-google`. > > > > > > -Daniel > > > > > > On Tue, Feb 28, 2023 at 10:19 AM Martijn Visser < > > martijnvis...@apache.org> > > > wrote: > > > > > >> Hi Daniel, > > >> > > >> If you can provide your Jira username, those permissions can be > granted. > > >> > > >> Best regards, > > >> > > >> Martijn > > >> > > >> On Tue, Feb 28, 2023 at 3:13 PM Daniel Collins > > >> > > >> wrote: > > >> > > >> > Hello, > > >> > > > >> > Absent any comments, I was attempting to create a connector FLIP as > > >> > described above, however it appears that I do not have permissions > to > > >> edit > > >> > confluence to do so. Can someone please add access for me? > > >> > > > >> > -Daniel > > >> > > > >> > On Mon, Feb 27, 2023 at 10:28 AM Daniel Collins < > dpcoll...@google.com > > > > > >> > wrote: > > >> > > > >> > > Hello Martijn, > > >> > > > > >> > > Thanks for the redirect! > > >> > > > > >> > > > The process for contributing a connector is to create a > Connector > > >> FLIP > > >> > > [1], which needs to be discussed and voted on in the Dev mailing > > list > > >> [2] > > >> > > > > >> > > I don't mind doing this, but would like to get a read of the room > > >> before > > >> > > doing so. If people object to it, then it probably doesn't make > > sense. > > >> > > > > >> > > > One thing in particular is who can help with the maintenance of > > the > > >> > > connector: will there be more volunteers who can help with bug > > fixes, > > >> new > > >> > > features etc. > > >> > > > > >> > > In this case, the product team is willing to handle feature > requests > > >> and > > >> > > bug reports for the connector, as we currently do for our beam and > > >> spark > > >> > > connectors. I don't know if there is any mechanism for sending > > emails > > >> > when > > >> > > jira tickets with a specific tag are opened? But if there is I can > > >> ensure > > >> > > that any tickets get routed to a member of my team to look at. > > >> > > > > >> > > What would people's thoughts be about this? > > >> > > > > >> > > -Daniel > > >> > > > > >> > > On Mon, Feb 27, 2023 at 3:53 AM Martijn Visser < > > >> martijnvis...@apache.org > > >> > > > > >> > > wrote: > > >> > > > > >> > >> Hi Daniel, > > >> > >> > > >> > >> Thanks for reaching out. Keep in mind that you weren't subscribed > > to > > >> the > > >> > >> Flink Dev mailing list, I've just pushed this through the mailing > > >> list > > >> > >> moderation. > > >> > >> > > >> > >> The process for contributing a connector is to create a Connector > > >> FLIP > > >> > >> [1], > > >> > >> which needs to be discussed and voted on in the Dev mailing list > > [2]. > > >> > One > > >> > >> thing in particular is who can help with the maintenance of the > > >> > connector: > > >> > >> will there be more volunteers who can help with bug fixes, new > > >> features > > >> > >> etc. As we've seen with the current PubSub connector, that's > > already > > >> > quite > > >> > >> hard: it's currently lacking volunteers overall. I do recall a > > >> proposal > > >> > to > > >> > >> contribute a PubSub Lite connector a while back, but that > > ultimately > > >> was > > >> > >> not followed through. > > >> > >> > > >> > >> Best regards, > > >> > >> > > >> > >> Martijn > > >> > >> > > >> > >> [1] > > >> > >> > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template > > >> > >> [2] > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > > >> > >> > > >> > >> On Mon, Feb 27, 2023 at 9:44 AM Daniel Collins > > >> > >> > > >> > >> wrote: > > >> > >> > > >> > >> > Hello flink devs, > > >> > >> > > > >> > >> > My name is Daniel, I'm the tech lead for Google's Pub/Sub Lite > > >> > streaming > > >> > >> > system, which is a lower cost streaming data system with > > semantics > > >> > more > > >> > >> > similar to OSS streaming solutions. I've authored a source and > > sink > > >> > >> > connector for flink and load tested it at GiB/s scale- what > would > > >> be > > >> > the > > >> > >> > process for contributing this to flink? > > >> > >> > > > >> > >> > I've opened a JIRA to do this > > >> > >> > > https://issues.apache.org/jira/projects/FLINK/issues/FLINK-31229 > > , > > >> if > > >> > >> this > > >> > >> > seems like a reasonable thing to do could someone assign it to > > me? > > >> > >> > > > >> > >> > The code for the connector currently lives here > > >> > >> > https://github.com/googleapis/java-pubsublite-flink, I
Re: [VOTE] Flink minor version support policy for old releases
+1 (binding) Gyula On Wed, Mar 1, 2023 at 9:57 PM Thomas Weise wrote: > +1 (binding) > > Thanks, > Thomas > > On Tue, Feb 28, 2023 at 6:53 AM Sergey Nuyanzin > wrote: > > > +1 (non-binding) > > > > Thanks for driving this Danny. > > > > On Tue, Feb 28, 2023 at 9:41 AM Samrat Deb > wrote: > > > > > +1 (non binding) > > > > > > Thanks for driving it > > > > > > Bests, > > > Samrat > > > > > > On Tue, 28 Feb 2023 at 1:36 PM, Junrui Lee > wrote: > > > > > > > Thanks Danny for driving it. > > > > > > > > +1 (non-binding) > > > > > > > > Best regards, > > > > Junrui > > > > > > > > yuxia 于2023年2月28日周二 14:04写道： > > > > > > > > > Thanks Danny for driving it. > > > > > > > > > > +1 (non-binding) > > > > > > > > > > Best regards, > > > > > Yuxia > > > > > > > > > > - 原始邮件 - > > > > > 发件人: "Weihua Hu" > > > > > 收件人: "dev" > > > > > 发送时间: 星期二, 2023年 2 月 28日 下午 12:48:09 > > > > > 主题: Re: [VOTE] Flink minor version support policy for old releases > > > > > > > > > > Thanks, Danny. > > > > > > > > > > +1 (non-binding) > > > > > > > > > > Best, > > > > > Weihua > > > > > > > > > > > > > > > On Tue, Feb 28, 2023 at 12:38 PM weijie guo < > > guoweijieres...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Thanks Danny for bring this. > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Weijie > > > > > > > > > > > > > > > > > > Jing Ge 于2023年2月27日周一 20:23写道： > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > BTW, should we follow the content style [1] to describe the new > > > rule > > > > > > using > > > > > > > 1.2.x, 1.1.y, 1.1.z? > > > > > > > > > > > > > > [1] > > > > https://flink.apache.org/downloads/#update-policy-for-old-releases > > > > > > > > > > > > > > Best regards, > > > > > > > Jing > > > > > > > > > > > > > > On Mon, Feb 27, 2023 at 1:06 PM Matthias Pohl > > > > > > > wrote: > > > > > > > > > > > > > > > Thanks, Danny. Sounds good to me. > > > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > On Wed, Feb 22, 2023 at 10:11 AM Danny Cranmer < > > > > > > dannycran...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > I am starting a vote to update the "Update Policy for old > > > > releases" > > > > > > [1] > > > > > > > > to > > > > > > > > > include additional bugfix support for end of life versions. > > > > > > > > > > > > > > > > > > As per the discussion thread [2], the change we are voting > on > > > is: > > > > > > > > > - Support policy: updated to include: "Upon release of a > new > > > > Flink > > > > > > > minor > > > > > > > > > version, the community will perform one final bugfix > release > > > for > > > > > > > resolved > > > > > > > > > critical/blocker issues in the Flink minor version losing > > > > support." > > > > > > > > > - Release process: add a step to start the discussion > thread > > > for > > > > > the > > > > > > > > final > > > > > > > > > patch version, if there are resolved critical/blocking > issues > > > to > > > > > > flush. > > > > > > > > > > > > > > > > > > Voting schema: since our bylaws [3] do not cover this > > > particular > > > > > > > > scenario, > > > > > > > > > and releases require PMC involvement, we will use a > consensus > > > > vote > > > > > > with > > > > > > > > PMC > > > > > > > > > binding votes. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Danny > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://flink.apache.org/downloads.html#update-policy-for-old-releases > > > > > > > > > [2] > > > > > https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv > > > > > > > > > [3] > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Best regards, > > Sergey > > >
Re: [VOTE] Flink minor version support policy for old releases
+1 (binding) Thanks, Thomas On Tue, Feb 28, 2023 at 6:53 AM Sergey Nuyanzin wrote: > +1 (non-binding) > > Thanks for driving this Danny. > > On Tue, Feb 28, 2023 at 9:41 AM Samrat Deb wrote: > > > +1 (non binding) > > > > Thanks for driving it > > > > Bests, > > Samrat > > > > On Tue, 28 Feb 2023 at 1:36 PM, Junrui Lee wrote: > > > > > Thanks Danny for driving it. > > > > > > +1 (non-binding) > > > > > > Best regards, > > > Junrui > > > > > > yuxia 于2023年2月28日周二 14:04写道： > > > > > > > Thanks Danny for driving it. > > > > > > > > +1 (non-binding) > > > > > > > > Best regards, > > > > Yuxia > > > > > > > > - 原始邮件 - > > > > 发件人: "Weihua Hu" > > > > 收件人: "dev" > > > > 发送时间: 星期二, 2023年 2 月 28日 下午 12:48:09 > > > > 主题: Re: [VOTE] Flink minor version support policy for old releases > > > > > > > > Thanks, Danny. > > > > > > > > +1 (non-binding) > > > > > > > > Best, > > > > Weihua > > > > > > > > > > > > On Tue, Feb 28, 2023 at 12:38 PM weijie guo < > guoweijieres...@gmail.com > > > > > > > wrote: > > > > > > > > > Thanks Danny for bring this. > > > > > > > > > > +1 (non-binding) > > > > > > > > > > Best regards, > > > > > > > > > > Weijie > > > > > > > > > > > > > > > Jing Ge 于2023年2月27日周一 20:23写道： > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > BTW, should we follow the content style [1] to describe the new > > rule > > > > > using > > > > > > 1.2.x, 1.1.y, 1.1.z? > > > > > > > > > > > > [1] > > > https://flink.apache.org/downloads/#update-policy-for-old-releases > > > > > > > > > > > > Best regards, > > > > > > Jing > > > > > > > > > > > > On Mon, Feb 27, 2023 at 1:06 PM Matthias Pohl > > > > > > wrote: > > > > > > > > > > > > > Thanks, Danny. Sounds good to me. > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > On Wed, Feb 22, 2023 at 10:11 AM Danny Cranmer < > > > > > dannycran...@apache.org> > > > > > > > wrote: > > > > > > > > > > > > > > > I am starting a vote to update the "Update Policy for old > > > releases" > > > > > [1] > > > > > > > to > > > > > > > > include additional bugfix support for end of life versions. > > > > > > > > > > > > > > > > As per the discussion thread [2], the change we are voting on > > is: > > > > > > > > - Support policy: updated to include: "Upon release of a new > > > Flink > > > > > > minor > > > > > > > > version, the community will perform one final bugfix release > > for > > > > > > resolved > > > > > > > > critical/blocker issues in the Flink minor version losing > > > support." > > > > > > > > - Release process: add a step to start the discussion thread > > for > > > > the > > > > > > > final > > > > > > > > patch version, if there are resolved critical/blocking issues > > to > > > > > flush. > > > > > > > > > > > > > > > > Voting schema: since our bylaws [3] do not cover this > > particular > > > > > > > scenario, > > > > > > > > and releases require PMC involvement, we will use a consensus > > > vote > > > > > with > > > > > > > PMC > > > > > > > > binding votes. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Danny > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > https://flink.apache.org/downloads.html#update-policy-for-old-releases > > > > > > > > [2] > > > > https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv > > > > > > > > [3] > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Best regards, > Sergey >
Re: Contributing a Google Cloud Pub/Sub Lite source and sink?
Hey Daniel, I have granted you access. Please try again. Thanks, Danny On Wed, Mar 1, 2023 at 6:48 PM Daniel Collins wrote: > Hello, > > It appears that I still do not have the ability to create Confluence pages? > Could someone possibly grant me access? > > -Daniel > > On Tue, Feb 28, 2023 at 10:20 AM Daniel Collins > wrote: > > > Thanks! My Jira username is `dpcollins-google`. > > > > -Daniel > > > > On Tue, Feb 28, 2023 at 10:19 AM Martijn Visser < > martijnvis...@apache.org> > > wrote: > > > >> Hi Daniel, > >> > >> If you can provide your Jira username, those permissions can be granted. > >> > >> Best regards, > >> > >> Martijn > >> > >> On Tue, Feb 28, 2023 at 3:13 PM Daniel Collins > >> > >> wrote: > >> > >> > Hello, > >> > > >> > Absent any comments, I was attempting to create a connector FLIP as > >> > described above, however it appears that I do not have permissions to > >> edit > >> > confluence to do so. Can someone please add access for me? > >> > > >> > -Daniel > >> > > >> > On Mon, Feb 27, 2023 at 10:28 AM Daniel Collins > > >> > wrote: > >> > > >> > > Hello Martijn, > >> > > > >> > > Thanks for the redirect! > >> > > > >> > > > The process for contributing a connector is to create a Connector > >> FLIP > >> > > [1], which needs to be discussed and voted on in the Dev mailing > list > >> [2] > >> > > > >> > > I don't mind doing this, but would like to get a read of the room > >> before > >> > > doing so. If people object to it, then it probably doesn't make > sense. > >> > > > >> > > > One thing in particular is who can help with the maintenance of > the > >> > > connector: will there be more volunteers who can help with bug > fixes, > >> new > >> > > features etc. > >> > > > >> > > In this case, the product team is willing to handle feature requests > >> and > >> > > bug reports for the connector, as we currently do for our beam and > >> spark > >> > > connectors. I don't know if there is any mechanism for sending > emails > >> > when > >> > > jira tickets with a specific tag are opened? But if there is I can > >> ensure > >> > > that any tickets get routed to a member of my team to look at. > >> > > > >> > > What would people's thoughts be about this? > >> > > > >> > > -Daniel > >> > > > >> > > On Mon, Feb 27, 2023 at 3:53 AM Martijn Visser < > >> martijnvis...@apache.org > >> > > > >> > > wrote: > >> > > > >> > >> Hi Daniel, > >> > >> > >> > >> Thanks for reaching out. Keep in mind that you weren't subscribed > to > >> the > >> > >> Flink Dev mailing list, I've just pushed this through the mailing > >> list > >> > >> moderation. > >> > >> > >> > >> The process for contributing a connector is to create a Connector > >> FLIP > >> > >> [1], > >> > >> which needs to be discussed and voted on in the Dev mailing list > [2]. > >> > One > >> > >> thing in particular is who can help with the maintenance of the > >> > connector: > >> > >> will there be more volunteers who can help with bug fixes, new > >> features > >> > >> etc. As we've seen with the current PubSub connector, that's > already > >> > quite > >> > >> hard: it's currently lacking volunteers overall. I do recall a > >> proposal > >> > to > >> > >> contribute a PubSub Lite connector a while back, but that > ultimately > >> was > >> > >> not followed through. > >> > >> > >> > >> Best regards, > >> > >> > >> > >> Martijn > >> > >> > >> > >> [1] > >> > >> > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template > >> > >> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > >> > >> > >> > >> On Mon, Feb 27, 2023 at 9:44 AM Daniel Collins > >> > >> > >> > >> wrote: > >> > >> > >> > >> > Hello flink devs, > >> > >> > > >> > >> > My name is Daniel, I'm the tech lead for Google's Pub/Sub Lite > >> > streaming > >> > >> > system, which is a lower cost streaming data system with > semantics > >> > more > >> > >> > similar to OSS streaming solutions. I've authored a source and > sink > >> > >> > connector for flink and load tested it at GiB/s scale- what would > >> be > >> > the > >> > >> > process for contributing this to flink? > >> > >> > > >> > >> > I've opened a JIRA to do this > >> > >> > https://issues.apache.org/jira/projects/FLINK/issues/FLINK-31229 > , > >> if > >> > >> this > >> > >> > seems like a reasonable thing to do could someone assign it to > me? > >> > >> > > >> > >> > The code for the connector currently lives here > >> > >> > https://github.com/googleapis/java-pubsublite-flink, I believe > it > >> is > >> > >> > following the FLIP-27 guidelines, but please let me know if I'm > >> > >> > implementing the wrong interfaces. Which repo and in what folder > >> > should > >> > >> I > >> > >> > move this code into? > >> > >> > > >> > >> > -Daniel > >> > >> > > >> > >> > >> > > > >> > > >> > > >
Re: Contributing a Google Cloud Pub/Sub Lite source and sink?
Hello, It appears that I still do not have the ability to create Confluence pages? Could someone possibly grant me access? -Daniel On Tue, Feb 28, 2023 at 10:20 AM Daniel Collins wrote: > Thanks! My Jira username is `dpcollins-google`. > > -Daniel > > On Tue, Feb 28, 2023 at 10:19 AM Martijn Visser > wrote: > >> Hi Daniel, >> >> If you can provide your Jira username, those permissions can be granted. >> >> Best regards, >> >> Martijn >> >> On Tue, Feb 28, 2023 at 3:13 PM Daniel Collins >> >> wrote: >> >> > Hello, >> > >> > Absent any comments, I was attempting to create a connector FLIP as >> > described above, however it appears that I do not have permissions to >> edit >> > confluence to do so. Can someone please add access for me? >> > >> > -Daniel >> > >> > On Mon, Feb 27, 2023 at 10:28 AM Daniel Collins >> > wrote: >> > >> > > Hello Martijn, >> > > >> > > Thanks for the redirect! >> > > >> > > > The process for contributing a connector is to create a Connector >> FLIP >> > > [1], which needs to be discussed and voted on in the Dev mailing list >> [2] >> > > >> > > I don't mind doing this, but would like to get a read of the room >> before >> > > doing so. If people object to it, then it probably doesn't make sense. >> > > >> > > > One thing in particular is who can help with the maintenance of the >> > > connector: will there be more volunteers who can help with bug fixes, >> new >> > > features etc. >> > > >> > > In this case, the product team is willing to handle feature requests >> and >> > > bug reports for the connector, as we currently do for our beam and >> spark >> > > connectors. I don't know if there is any mechanism for sending emails >> > when >> > > jira tickets with a specific tag are opened? But if there is I can >> ensure >> > > that any tickets get routed to a member of my team to look at. >> > > >> > > What would people's thoughts be about this? >> > > >> > > -Daniel >> > > >> > > On Mon, Feb 27, 2023 at 3:53 AM Martijn Visser < >> martijnvis...@apache.org >> > > >> > > wrote: >> > > >> > >> Hi Daniel, >> > >> >> > >> Thanks for reaching out. Keep in mind that you weren't subscribed to >> the >> > >> Flink Dev mailing list, I've just pushed this through the mailing >> list >> > >> moderation. >> > >> >> > >> The process for contributing a connector is to create a Connector >> FLIP >> > >> [1], >> > >> which needs to be discussed and voted on in the Dev mailing list [2]. >> > One >> > >> thing in particular is who can help with the maintenance of the >> > connector: >> > >> will there be more volunteers who can help with bug fixes, new >> features >> > >> etc. As we've seen with the current PubSub connector, that's already >> > quite >> > >> hard: it's currently lacking volunteers overall. I do recall a >> proposal >> > to >> > >> contribute a PubSub Lite connector a while back, but that ultimately >> was >> > >> not followed through. >> > >> >> > >> Best regards, >> > >> >> > >> Martijn >> > >> >> > >> [1] >> > >> >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template >> > >> [2] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws >> > >> >> > >> On Mon, Feb 27, 2023 at 9:44 AM Daniel Collins >> > >> >> > >> wrote: >> > >> >> > >> > Hello flink devs, >> > >> > >> > >> > My name is Daniel, I'm the tech lead for Google's Pub/Sub Lite >> > streaming >> > >> > system, which is a lower cost streaming data system with semantics >> > more >> > >> > similar to OSS streaming solutions. I've authored a source and sink >> > >> > connector for flink and load tested it at GiB/s scale- what would >> be >> > the >> > >> > process for contributing this to flink? >> > >> > >> > >> > I've opened a JIRA to do this >> > >> > https://issues.apache.org/jira/projects/FLINK/issues/FLINK-31229, >> if >> > >> this >> > >> > seems like a reasonable thing to do could someone assign it to me? >> > >> > >> > >> > The code for the connector currently lives here >> > >> > https://github.com/googleapis/java-pubsublite-flink, I believe it >> is >> > >> > following the FLIP-27 guidelines, but please let me know if I'm >> > >> > implementing the wrong interfaces. Which repo and in what folder >> > should >> > >> I >> > >> > move this code into? >> > >> > >> > >> > -Daniel >> > >> > >> > >> >> > > >> > >> >
[jira] [Created] (FLINK-31284) Increase KerberosLoginProvider test coverage
Gabor Somogyi created FLINK-31284: - Summary: Increase KerberosLoginProvider test coverage Key: FLINK-31284 URL: https://issues.apache.org/jira/browse/FLINK-31284 Project: Flink Issue Type: Technical Debt Components: Tests Affects Versions: 1.17.0 Reporter: Gabor Somogyi -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL
Reg. 2: > event gap emit strategy [...] no matter in DataStream or SQL Jark raised a very good point. I thought we only expose what is contained in DataStream API already. If this strategy is not part of DataStream API, would like to exclude it from the FLIP. We need to be careful which strategies we offer by default. Reg 1: This already has a JIRA ticket with additional thoughts on this topic: https://issues.apache.org/jira/browse/FLINK-25221 Regards, Timo On 01.03.23 12:31, Jark Wu wrote: Sorry, I forgot to remind you that Timo's concern about the changes to the CompiledPlan looks like is still not covered in the FLIP. Best, Jark On Wed, 1 Mar 2023 at 19:28, Jark Wu wrote: Hi Kui, Thank you for the great proposal, I think this is already in a good shape. Just a kind reminder, according to the community guidelines[1], if there are unresponsive reviewers, a typical reasonable time to wait for responses is one week, but be pragmatic about it. Regarding the FLIP, I have some comments below: 1. IIRC, this is the first time we introduce the framework-level connector options that the option is not recognized and handled by connectors. The FLIP should cover how framework filters the watermark related options to avoid discover connector factory failed, and what happens if the connector already supported the conflict options. 2. I'm not sure about the usage scenarios of event gap emit strategy. Do you have any specific use case of this strategy? I'm confused why no one requested this strategy before no matter in DataStream or SQL, but maybe I missed something. I'm not against to add this option, but just want to be careful when adding new API because it's hard to remove in the future. 3. Adding a "Public Interface"[2] section to summarize the proposed APIs and options would be better for developers to know the impact. Currently, the APIs are scattered in the long design sections. Best, Jark [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals [2]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template On Wed, 1 Mar 2023 at 16:56, Kui Yuan wrote: Hi all, Thanks for all discussions! Anyone else have questions or suggestions? if not, I will start a vote thread later. Best Kui Yuan kui yuan 于2023年2月27日周一 20:21写道： Hi Timo, Thanks for your advice. I totally agree with your suggestion of naming convention, I will rename these options and update the flip later, thanks very much. In our internal implementation we had put these options inside the `FactoryUtil`, just as you expect. We have also taken into account the changes to the CompiledPlan and we have packaged these options appropriately to minimize intrusiveness and ensure the compatibility to the `WatermarkPushDownSpec`. A hint to the implementation: I would suggest that we add those options to `FactoryUtil`. All cross-connector options should end up there. Please also consider the changes to the CompiledPlan in your FLIP. This change has implications on the JSON format as watermark strategy of ExecNode becomes more complex, see WatermarkPushDownSpec Best Kui Yuan Timo Walther 于2023年2月27日周一 18:05写道： Hi Kui Yuan, thanks for working on this FLIP. Let me also give some comments about the proposed changes. I support the direction of this FLIP about handling these watermark-specific properties through options and /*+OPTIONS(...) */ hints. Regarding naming, I would like to keep the options in sync with existing options: > 'watermark.emit.strategy'='ON_EVENT' Let's use lower case (e.g. `on-event`) that matches with properties like sink.partitioner [1] or sink.delivery-guarantee [2]. > 'source.idle-timeout'='1min' According to FLIP-122 [3], we want to prefix all scan-source related properties with `scan.*`. This clearly includes idle-timeout and actually also watermark strategies which don't apply for lookup sources. Summarizing the comments above, we should use the following options: 'scan.watermark.emit.strategy'='on-event', 'scan.watermark.emit.on-event.gap'='1', 'scan.idle-timeout'='1min', 'scan.watermark.alignment.group'='alignment-group-1', 'scan.watermark.alignment.max-drift'='1min', 'scan.watermark.alignment.update-interval'='1s' I know that this makes the keys even longer, but given that those options are for power users this should be acceptable. It also clearly indicates which options are for sinks, scans, and lookups. This potentially also helps in allow lists. A hint to the implementation: I would suggest that we add those options to `FactoryUtil`. All cross-connector options should end up there. Please also consider the changes to the CompiledPlan in your FLIP. This change has implications on the JSON format as watermark strategy of ExecNode becomes more complex, see WatermarkPushDownSpec [4]. Regards, Timo [1] https://nightlies.apache.org/flink/flink-docs-release-1.11/dev/table/connectors/kafka.html#sink-partitioner [2]
[jira] [Created] (FLINK-31283) Correct the description of building from source with scala version
Yun Tang created FLINK-31283: Summary: Correct the description of building from source with scala version Key: FLINK-31283 URL: https://issues.apache.org/jira/browse/FLINK-31283 Project: Flink Issue Type: Bug Components: Documentation Reporter: Yun Tang Assignee: Yun Tang Fix For: 1.17.0 After FLINK-20845, Flink dropped the support of Scala 2.11. However, the doc of "building from source" still has the description on scala-2.11. -- This message was sent by Atlassian Jira (v8.20.10#820010)
RE: [DISCUSS] Why does the Doc say "Flink requires at least Java 11 to build"?
Joachim Sauer[1] gave a good answer to my question. [1] https://stackoverflow.com/a/75604355/3741571 On 2023/03/01 13:21:33 Zhongpu Chen wrote: > I reported the same question at StackOverflow [1]. As I tested in my > computer, Java 8 is enough to build Flink from source. > > [1] https://stackoverflow.com/questions/75601233/ > > -- > Zhongpu Chen >
[DISCUSS] Why does the Doc say "Flink requires at least Java 11 to build"?
I reported the same question at StackOverflow [1]. As I tested in my computer, Java 8 is enough to build Flink from source. [1] https://stackoverflow.com/questions/75601233/ -- Zhongpu Chen
[jira] [Created] (FLINK-31282) Create and Close the zookeeper connection frequently
shenyue created FLINK-31282:
---
Summary: Create and Close the zookeeper connection frequently
Key: FLINK-31282
URL: https://issues.apache.org/jira/browse/FLINK-31282
Project: Flink
Issue Type: Bug
Components: Client / Job Submission
Affects Versions: 1.16.0
Reporter: shenyue
When I use sql " *select * from table* " , I found there are a lot logs about
zookeeper connection Information. There is *critical* for Zookeeper ，because
*the Connection* is *create and close frequently*
[
2023-03-01 19:38:38.423 INFO (Thread-262) [ ]
o.a.f.r.l.DefaultLeaderRetrievalService Starting DefaultLeaderRetrievalServic
e with
ZookeeperLeaderRetrievalDriver\{connectionInformationPath='/leader/rest_server/connection_info'}.
2023-03-01 19:38:38.424 INFO (Thread-262-SendThread(node53.hde.com:2181)) [
] o.a.f.s.z.o.a.z.ClientCnxn Socket connection
established, initiating session, client: /10.121.65.53:58428, server:
node53.hde.com/10.121.65.53:2181
2023-03-01 19:38:38.427 INFO (Thread-262-SendThread(node53.hde.com:2181)) [
] o.a.f.s.z.o.a.z.ClientCnxn Session establishm
ent complete on server node53.hde.com/10.121.65.53:2181, session id =
0x1038726412a, negotiated timeout = 6
2023-03-01 19:38:38.427 INFO (Thread-262-EventThread) [ ]
o.a.f.s.c.o.a.c.f.s.ConnectionStateManager State change: CONNECTE
D
2023-03-01 19:38:38.430 INFO (Thread-262-EventThread) [ ]
o.a.f.s.c.o.a.c.f.i.EnsembleTracker New config event received: {s
erver.1=node53.hde.com:2888:3888:participant, version=0,
server.3=node55.hde.com:2888:3888:participant, server.2=node54.hde.co
m:2888:3888:participant}
2023-03-01 19:38:38.431 INFO (Thread-262-EventThread) [ ]
o.a.f.s.c.o.a.c.f.i.EnsembleTracker New config event received: {s
erver.1=node53.hde.com:2888:3888:participant, version=0,
server.3=node55.hde.com:2888:3888:participant, server.2=node54.hde.co
m:2888:3888:participant}
2023-03-01 19:38:38.700 INFO (ForkJoinPool.commonPool-worker-11) [ ]
o.a.f.r.l.DefaultLeaderRetrievalService Stopping Defau
ltLeaderRetrievalService.
2023-03-01 19:38:38.700 INFO (ForkJoinPool.commonPool-worker-11) [ ]
o.a.f.r.l.ZooKeeperLeaderRetrievalDriver Closing Zooke
eperLeaderRetrievalDriver\{connectionInformationPath='/leader/rest_server/connection_info'}.
2023-03-01 19:38:38.702 INFO (Curator-Framework-0) [ ]
o.a.f.s.c.o.a.c.f.i.CuratorFrameworkImpl backgroundOperationsLoop ex
iting
2023-03-01 19:38:38.708 WARN (Thread-262-SendThread(node53.hde.com:2181)) [
] o.a.f.s.z.o.a.z.ClientCnxn An exception was t
hrown while closing send thread for session 0x1038726412a.
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$EndOfStreamException:
Unable to read additional data from s
erver sessionid 0x1038726412a, likely server has closed socket
at
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
~[fli
nk-shaded-zookeeper-3.6.3.jar:3.6.3-15.0]
at
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:35
0) ~[flink-shaded-zookeeper-3.6.3.jar:3.6.3-15.0]
at
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[flink-shad
ed-zookeeper-3.6.3.jar:3.6.3-15.0]
2023-03-01 19:38:38.810 INFO (Thread-262-EventThread) [ ]
o.a.f.s.z.o.a.z.ClientCnxn EventThread shut down for session: 0x1
038726412a
2023-03-01 19:38:38.810 INFO (ForkJoinPool.commonPool-worker-11) [ ]
o.a.f.s.z.o.a.z.ZooKeeper Session: 0x1038726412a c
losed
2023-03-01 19:38:38.813 INFO (Thread-262) [ ] o.a.f.r.u.ZooKeeperUtils
Enforcing default ACL for ZK connections
2023-03-01 19:38:38.813 INFO (Thread-262) [ ] o.a.f.r.u.ZooKeeperUtils Using
'/flink/application_1677479737242_0025' as Zoo
keeper namespace.
2023-03-01 19:38:38.814 INFO (Thread-262) [ ]
o.a.f.s.c.o.a.c.f.i.CuratorFrameworkImpl Starting
2023-03-01 19:38:38.814 INFO (Thread-262) [ ] o.a.f.s.z.o.a.z.ZooKeeper
Initiating client connection, connectString=node54.
hde.com:2181,node55.hde.com:2181,node53.hde.com:2181 sessionTimeout=6
watcher=org.apache.flink.shaded.curator5.org.apache.
curator.ConnectionState@74e5bc14
2023-03-01 19:38:38.814 INFO (Thread-262) [ ]
o.a.f.s.z.o.a.z.ClientCnxnSocket jute.maxbuffer value is 1048575 Bytes
2023-03-01 19:38:38.814 INFO (Thread-262) [ ] o.a.f.s.z.o.a.z.ClientCnxn
zookeeper.request.timeout value is 0. feature enab
led=false
2023-03-01 19:38:38.815 INFO (Thread-262) [ ]
o.a.f.s.c.o.a.c.f.i.CuratorFrameworkImpl Default schema
2023-03-01 19:38:38.815 INFO (Thread-262) [ ]
o.a.f.r.l.DefaultLeaderRetrievalService Starting DefaultLeaderRetrievalServic
e with
ZookeeperLeaderRetrievalDriver\{connectionInformationPath='/leader/rest_server/connection_info'}.
2023-03-01 19:38:38.817 INFO
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
+1 for the FLIP, thanks Shammon for driving this. JDBC is quite useful in OLAP scenarios, supporting JDBC would enable Flink to be used with existing tools, such as Tableau. Regarding the JDBC interfaces listed in the FLIP, I think they looks good already. Besides `java.sql.Driver`, have you considered also adding support for `javax.sql.DataSource` interface? Jingsong Li 于2023年3月1日周三 17:53写道： > Thanks Shammon for driving. > > Big +1 for this. > > I heard that many users want to use FlinkGateway + JDBC to do some > queries, but at present, only Hive JDBC can be used. It is Hive > dialect by default, and the experience is also different from > FlinkSQL. We need to have our own JDBC. > > I took a look at your `Public Interface` part, only > `FlinkResultSet.getRowKind` is a true new interface, others are just > implementations. > > If the user does not cast into a FlinkResultSet, will there be serious > consequences here (RowKind is ignored)? > > Best, > Jingsong > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY wrote: > > > > Hi devs, > > > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc > Driver > > For Sql Gateway[1]. > > > > FLIP-275[2] supports remote sql client based on gateway, users can > interact > > with gateway by flink console. However, for users who create session > > clusters with Flink, they'd like to use Jdbc Driver to interact with the > > gateway in their applications, such as olap queries.. > > > > I have discussed this proposal with @shengkaifang and @jinsonglee. In > this > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use Jdbc > > Driver to submit their queries and get results like a database in their > > applications. > > > > Looking forward to your feedback, thanks. > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway > > [2] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway > > > > > > Best, > > Shammon > -- Best, Benchao Li
[jira] [Created] (FLINK-31281) PythonFunctionRunner doesn't extend AutoCloseable but implements close
Ran Tao created FLINK-31281:
---
Summary: PythonFunctionRunner doesn't extend AutoCloseable but
implements close
Key: FLINK-31281
URL: https://issues.apache.org/jira/browse/FLINK-31281
Project: Flink
Issue Type: Improvement
Reporter: Ran Tao
The PythonFunctionRunner provides a {{close}} method (see
[PythonFunctionRunner|https://github.com/apache/flink/blob/0612a997ddcc791ee54f500fbf1299ce04987679/flink-python/src/main/java/org/apache/flink/python/PythonFunctionRunner.java])
but doesn't implement {{{}AutoCloseable{}}}. However {{AutoCloseable}} would
enable us to use Java's try-with-resources feature and more generic utility
classes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL
Sorry, I forgot to remind you that Timo's concern about the changes to the CompiledPlan looks like is still not covered in the FLIP. Best, Jark On Wed, 1 Mar 2023 at 19:28, Jark Wu wrote: > Hi Kui, > > Thank you for the great proposal, I think this is already in a good shape. > > Just a kind reminder, according to the community guidelines[1], > if there are unresponsive reviewers, a typical reasonable time > to wait for responses is one week, but be pragmatic about it. > > Regarding the FLIP, I have some comments below: > > 1. IIRC, this is the first time we introduce the framework-level connector > options that the option is not recognized and handled by connectors. > The FLIP should cover how framework filters the watermark related options > to avoid discover connector factory failed, and what happens if the > connector > already supported the conflict options. > > 2. I'm not sure about the usage scenarios of event gap emit strategy. Do > you have any specific use case of this strategy? I'm confused why no one > requested this strategy before no matter in DataStream or SQL, but maybe > I missed something. I'm not against to add this option, but just want to > be > careful when adding new API because it's hard to remove in the future. > > > 3. Adding a "Public Interface"[2] section to summarize the > proposed APIs and options would be better for developers to > know the impact. Currently, the APIs are scattered in the long > design sections. > > Best, > Jark > > > [1]: > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > [2]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template > > On Wed, 1 Mar 2023 at 16:56, Kui Yuan wrote: > >> Hi all, >> >> Thanks for all discussions! >> >> Anyone else have questions or suggestions? if not, I will start a vote >> thread later. >> >> Best >> Kui Yuan >> >> kui yuan 于2023年2月27日周一 20:21写道： >> >> > Hi Timo, >> > >> > Thanks for your advice. I totally agree with your suggestion of naming >> > convention, I will rename these options and update the flip later, >> thanks >> > very much. >> > >> > In our internal implementation we had put these options inside the >> > `FactoryUtil`, just as you expect. We have also taken into account the >> > changes to the CompiledPlan and we have packaged these options >> > appropriately to minimize intrusiveness and ensure the compatibility to >> the >> > `WatermarkPushDownSpec`. >> > >> > > A hint to the implementation: I would suggest that we add those >> options >> > > to `FactoryUtil`. All cross-connector options should end up there. >> > >> > >> > > Please also consider the changes to the CompiledPlan in your FLIP. >> This >> > > change has implications on the JSON format as watermark strategy of >> > > ExecNode becomes more complex, see WatermarkPushDownSpec >> > >> > Best >> > Kui Yuan >> > >> > Timo Walther 于2023年2月27日周一 18:05写道： >> > >> >> Hi Kui Yuan, >> >> >> >> thanks for working on this FLIP. Let me also give some comments about >> >> the proposed changes. >> >> >> >> I support the direction of this FLIP about handling these >> >> watermark-specific properties through options and /*+OPTIONS(...) */ >> >> hints. >> >> >> >> Regarding naming, I would like to keep the options in sync with >> existing >> >> options: >> >> >> >> > 'watermark.emit.strategy'='ON_EVENT' >> >> >> >> Let's use lower case (e.g. `on-event`) that matches with properties >> like >> >> sink.partitioner [1] or sink.delivery-guarantee [2]. >> >> >> >> > 'source.idle-timeout'='1min' >> >> >> >> According to FLIP-122 [3], we want to prefix all scan-source related >> >> properties with `scan.*`. This clearly includes idle-timeout and >> >> actually also watermark strategies which don't apply for lookup >> sources. >> >> >> >> Summarizing the comments above, we should use the following options: >> >> >> >> 'scan.watermark.emit.strategy'='on-event', >> >> 'scan.watermark.emit.on-event.gap'='1', >> >> 'scan.idle-timeout'='1min', >> >> 'scan.watermark.alignment.group'='alignment-group-1', >> >> 'scan.watermark.alignment.max-drift'='1min', >> >> 'scan.watermark.alignment.update-interval'='1s' >> >> >> >> I know that this makes the keys even longer, but given that those >> >> options are for power users this should be acceptable. It also clearly >> >> indicates which options are for sinks, scans, and lookups. This >> >> potentially also helps in allow lists. >> >> >> >> A hint to the implementation: I would suggest that we add those options >> >> to `FactoryUtil`. All cross-connector options should end up there. >> >> >> >> Please also consider the changes to the CompiledPlan in your FLIP. This >> >> change has implications on the JSON format as watermark strategy of >> >> ExecNode becomes more complex, see WatermarkPushDownSpec [4]. >> >> >> >> Regards, >> >> Timo >> >> >> >> >> >> [1] >> >> >> >> >> https://nightlies.apache.org/flink/flink-docs-release-1.11/dev/table/connectors/kafka.html#sink-partitioner >> >>
Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL
Hi Kui, Thank you for the great proposal, I think this is already in a good shape. Just a kind reminder, according to the community guidelines[1], if there are unresponsive reviewers, a typical reasonable time to wait for responses is one week, but be pragmatic about it. Regarding the FLIP, I have some comments below: 1. IIRC, this is the first time we introduce the framework-level connector options that the option is not recognized and handled by connectors. The FLIP should cover how framework filters the watermark related options to avoid discover connector factory failed, and what happens if the connector already supported the conflict options. 2. I'm not sure about the usage scenarios of event gap emit strategy. Do you have any specific use case of this strategy? I'm confused why no one requested this strategy before no matter in DataStream or SQL, but maybe I missed something. I'm not against to add this option, but just want to be careful when adding new API because it's hard to remove in the future. 3. Adding a "Public Interface"[2] section to summarize the proposed APIs and options would be better for developers to know the impact. Currently, the APIs are scattered in the long design sections. Best, Jark [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals [2]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template On Wed, 1 Mar 2023 at 16:56, Kui Yuan wrote: > Hi all, > > Thanks for all discussions! > > Anyone else have questions or suggestions? if not, I will start a vote > thread later. > > Best > Kui Yuan > > kui yuan 于2023年2月27日周一 20:21写道： > > > Hi Timo, > > > > Thanks for your advice. I totally agree with your suggestion of naming > > convention, I will rename these options and update the flip later, thanks > > very much. > > > > In our internal implementation we had put these options inside the > > `FactoryUtil`, just as you expect. We have also taken into account the > > changes to the CompiledPlan and we have packaged these options > > appropriately to minimize intrusiveness and ensure the compatibility to > the > > `WatermarkPushDownSpec`. > > > > > A hint to the implementation: I would suggest that we add those options > > > to `FactoryUtil`. All cross-connector options should end up there. > > > > > > > Please also consider the changes to the CompiledPlan in your FLIP. This > > > change has implications on the JSON format as watermark strategy of > > > ExecNode becomes more complex, see WatermarkPushDownSpec > > > > Best > > Kui Yuan > > > > Timo Walther 于2023年2月27日周一 18:05写道： > > > >> Hi Kui Yuan, > >> > >> thanks for working on this FLIP. Let me also give some comments about > >> the proposed changes. > >> > >> I support the direction of this FLIP about handling these > >> watermark-specific properties through options and /*+OPTIONS(...) */ > >> hints. > >> > >> Regarding naming, I would like to keep the options in sync with existing > >> options: > >> > >> > 'watermark.emit.strategy'='ON_EVENT' > >> > >> Let's use lower case (e.g. `on-event`) that matches with properties like > >> sink.partitioner [1] or sink.delivery-guarantee [2]. > >> > >> > 'source.idle-timeout'='1min' > >> > >> According to FLIP-122 [3], we want to prefix all scan-source related > >> properties with `scan.*`. This clearly includes idle-timeout and > >> actually also watermark strategies which don't apply for lookup sources. > >> > >> Summarizing the comments above, we should use the following options: > >> > >> 'scan.watermark.emit.strategy'='on-event', > >> 'scan.watermark.emit.on-event.gap'='1', > >> 'scan.idle-timeout'='1min', > >> 'scan.watermark.alignment.group'='alignment-group-1', > >> 'scan.watermark.alignment.max-drift'='1min', > >> 'scan.watermark.alignment.update-interval'='1s' > >> > >> I know that this makes the keys even longer, but given that those > >> options are for power users this should be acceptable. It also clearly > >> indicates which options are for sinks, scans, and lookups. This > >> potentially also helps in allow lists. > >> > >> A hint to the implementation: I would suggest that we add those options > >> to `FactoryUtil`. All cross-connector options should end up there. > >> > >> Please also consider the changes to the CompiledPlan in your FLIP. This > >> change has implications on the JSON format as watermark strategy of > >> ExecNode becomes more complex, see WatermarkPushDownSpec [4]. > >> > >> Regards, > >> Timo > >> > >> > >> [1] > >> > >> > https://nightlies.apache.org/flink/flink-docs-release-1.11/dev/table/connectors/kafka.html#sink-partitioner > >> [2] > >> > >> > https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/#sink-delivery-guarantee > >> [3] > >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory > >> [4] > >> > >> >
[DISCUSS] Whether the snapshot of all OperatorCoordinator should be executed at the beginning of the checkpoint
Hi, devs, At present, at the beginning of checkpoint, all OperatorCoordinator will first perform snapshot in JM, and operator will then perform snapshot according to the order of receiving checkpoint barrier. In this case, does the data stored in OperatorCoordinator really correspond to the checkpoint of Operator? For example, in the scenario of source -> map -> sink, we add an OperatorCoordinator to the operator of the map to calculate the max of all data. If we record the max value at the beginning of the checkpoint, however, when the operator of the map actually starts to make checkpoint, the max value may have been updated, so is the max value snapshoted by OperatorCoordinator really what this checkpoint should correspond to? Perhaps this example is not appropriate, but it can be really confusing, especially when troubleshooting by querying the state of checkpoints. Thanks, Ming Li
Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Thanks Shammon for driving. Big +1 for this. I heard that many users want to use FlinkGateway + JDBC to do some queries, but at present, only Hive JDBC can be used. It is Hive dialect by default, and the experience is also different from FlinkSQL. We need to have our own JDBC. I took a look at your `Public Interface` part, only `FlinkResultSet.getRowKind` is a true new interface, others are just implementations. If the user does not cast into a FlinkResultSet, will there be serious consequences here (RowKind is ignored)? Best, Jingsong On Wed, Mar 1, 2023 at 4:59 PM Shammon FY wrote: > > Hi devs, > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc Driver > For Sql Gateway[1]. > > FLIP-275[2] supports remote sql client based on gateway, users can interact > with gateway by flink console. However, for users who create session > clusters with Flink, they'd like to use Jdbc Driver to interact with the > gateway in their applications, such as olap queries.. > > I have discussed this proposal with @shengkaifang and @jinsonglee. In this > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use Jdbc > Driver to submit their queries and get results like a database in their > applications. > > Looking forward to your feedback, thanks. > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway > > > Best, > Shammon
Re: [DISCUSS] FLIP-297: Improve Auxiliary Sql Statements
Hi, Timo. thanks. I have copied the content of docs and updated the FLIP. Thanks for your advice about consistency(ILIKE). Because calcite already support parse ILIKE and exist SqlLibraryOperators.ILIKE(which means it's not std). It's will not difficult to support it, and we just need to support it in table & sql api and not need to change sql keywords or Parser.jj. However, in flink docs, we may let users know that this is not standard, just for better use. Best Regards, Ran Tao Timo Walther 于2023年2月28日周二 22:23写道： > Hi Ran Tao, > > Thanks for working on this FLIP. The FLIP is in a pretty good shape > already and I don't have much to add. > > Will we also support ILIKE in queries? Or is this a pure DDL > expressions? For consistency, we should support it in SELECT and Table > API as well. I hope this is not too much effort. I hope everybody is > aware that ILIKE is not standard compliant but seems to be used by a > variety of vendors. > > > Because it may be modified under discuss, we put it on the google > docs. please see FLIP-297: Improve Auxiliary Sql Statements Docs > > This comment confused me. It would be nice to have the Wiki page as the > single source of truth and abandon the Google doc. In the past we used > Google Docs more but nowadays I support using only the Wiki to avoid any > confusion. > > Regards, > Timo > > > On 28.02.23 12:51, Ran Tao wrote: > > thanks Sergey, sounds good. > > You can add in FLIP ticket[1]. > > > > [1] https://issues.apache.org/jira/browse/FLINK-31256 > > > > Best Regards, > > Ran Tao > > https://github.com/chucheng92 > > > > > > Sergey Nuyanzin 于2023年2月28日周二 19:44写道： > > > Currently I think we can load from the jar and check the services > file to > get the connector type. but is it necessary we may continue to > discuss. > Hi, Sergey, WDYT？ > >> > >> Another idea is FactoryUtil#discoverFactories and > >> check if it implements DynamicTableSourceFactory or > DynamicTableSinkFactory > >> with versions it could be trickier... > >> Moreover it seems the version could be a part of the name sometimes[1]. > >> I think name and type could be enough or please correct me if I'm wrong > >> > or can we open a single ticket under this FLIP? > >> I have a relatively old jira issue[2] for showing connectors with a poc > pr. > >> Could I propose to move this jira issue as a subtask under the FLIP one > and > >> revive it? > >> > >> [1] > >> > >> > https://github.com/apache/flink/blob/161014149e803bfd1d3653badb230b2ed36ce3cb/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/Factory.java#L65-L69 > >> [2] https://issues.apache.org/jira/browse/FLINK-25788 > >> > >> On Tue, Feb 28, 2023 at 11:56 AM Ran Tao wrote: > >> > >>> Hi, Jark. thanks. > About ILIKE > >>> I have updated the FLIP for ILIKE support (Including existing > showTables > >> & > >>> showColumns how to change). > >>> > About show connectors @Sergey, > >>> Currently I think we can load from the jar and check the services file > to > >>> get the connector type. but is it necessary we may continue to discuss. > >>> Hi, Sergey, WDYT？or can we open a single ticket under this FLIP? > >>> > >>> > >>> Best Regards, > >>> Ran Tao > >>> > >>> > >>> Jark Wu 于2023年2月28日周二 17:45写道： > >>> > Besides, if we introduce the ILIKE, we should also add this feature > for > the previous SHOW with LIKE statements. They should be included in > this > FLIP. > > Best, > Jark > > > 2023年2月28日 17:40，Jark Wu 写道： > > > > Hi Ran, > > > > Could you add descriptions about what’s the behavior and differences > between the LIKE and ILIKE? > > > > Besides, I don’t see the SHOW CONNECTOR syntax and description and > >> how > it works in the FLIP. Is it intended to be included in this FLIP? > > > > Best, > > Jark > > > > > >> 2023年2月28日 10:58，Ran Tao 写道： > >> > >> Hi, guys. thanks for advices. > >> > >> allow me to make a small summary: > >> > >> 1.Support ILIKE > >> 2.Using catalog api to support show operations > >> 3.Need a dedicated FLIP try to support INFORMATION_SCHEMA > >> 4.Support SHOW CONNECTORS > >> > >> If there are no other questions, i will try to start a VOTE for this > FLIP. > >> WDYT? > >> > >> Best Regards, > >> Ran Tao > >> > >> > >> Sergey Nuyanzin 于2023年2月27日周一 21:12写道： > >> > >>> Hi Jark, > >>> > >>> thanks for your comment. > >>> > Considering they > are orthogonal and information schema requires more complex design > >>> and > discussion, it deserves a separate FLIP > >>> I'm ok with a separate FLIP for INFORMATION_SCHEMA. > >>> > Sergey, are you willing to contribute this FLIP? > >>> Seems I need to have more research done for that. > >>> I would try to help/contribute here > >>> > >>> > >>> On Mon, Feb 27, 2023
[jira] [Created] (FLINK-31280) Make core surefire plugin configuration stricter
Matthias Pohl created FLINK-31280:
-
Summary: Make core surefire plugin configuration stricter
Key: FLINK-31280
URL: https://issues.apache.org/jira/browse/FLINK-31280
Project: Flink
Issue Type: Sub-task
Reporter: Matthias Pohl
We should update the {{flink-core}} surefire plugin configuration in order to
ease the debuging of FLINK-31278.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31279) fix times using number and interval type bug
jackylau created FLINK-31279:
Summary: fix times using number and interval type bug
Key: FLINK-31279
URL: https://issues.apache.org/jira/browse/FLINK-31279
Project: Flink
Issue Type: Bug
Affects Versions: 1.18.0
Reporter: jackylau
Fix For: 1.18.0
{code:java}
// code placeholder
Flink SQL> select interval 3 day * 2;
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.planner.codegen.CodeGenException: Interval expression
type expected. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] FLIP-291: Externalized Declarative Resource Management
+1 (non-binding) Best, Shammon On Wed, Mar 1, 2023 at 4:51 PM Roman Khachatryan wrote: > +1 (binding) > > Thanks David, and everyone involved :) > > Regards, > Roman > > > On Wed, Mar 1, 2023 at 8:01 AM Gyula Fóra wrote: > > > +1 (binding) > > > > Looking forward to this :) > > > > Gyula > > > > On Wed, 1 Mar 2023 at 04:02, feng xiangyu wrote: > > > > > +1 (non-binding) > > > > > > ConradJam 于2023年3月1日周三 10:37写道： > > > > > > > +1 (non-binding) > > > > > > > > Zhanghao Chen 于2023年3月1日周三 10:18写道： > > > > > > > > > Thanks for driving this. +1 (non-binding) > > > > > > > > > > Best, > > > > > Zhanghao Chen > > > > > > > > > > From: David Mor?vek > > > > > Sent: Tuesday, February 28, 2023 21:46 > > > > > To: dev > > > > > Subject: [VOTE] FLIP-291: Externalized Declarative Resource > > Management > > > > > > > > > > Hi Everyone, > > > > > > > > > > I want to start the vote on FLIP-291: Externalized Declarative > > Resource > > > > > Management [1]. The FLIP was discussed in this thread [2]. > > > > > > > > > > The goal of the FLIP is to enable external declaration of the > > resource > > > > > requirements of a running job. > > > > > > > > > > The vote will last for at least 72 hours (Friday, 3rd of March, > 15:00 > > > > CET) > > > > > unless > > > > > there is an objection or insufficient votes. > > > > > > > > > > [1] > https://lists.apache.org/thread/b8fnj127jsl5ljg6p4w3c4wvq30cnybh > > > > > [2] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management > > > > > > > > > > Best, > > > > > D. > > > > > > > > > > > > > > > > > -- > > > > Best > > > > > > > > ConradJam > > > > > > > > > >
[DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway
Hi devs, I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway[1]. FLIP-275[2] supports remote sql client based on gateway, users can interact with gateway by flink console. However, for users who create session clusters with Flink, they'd like to use Jdbc Driver to interact with the gateway in their applications, such as olap queries.. I have discussed this proposal with @shengkaifang and @jinsonglee. In this FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use Jdbc Driver to submit their queries and get results like a database in their applications. Looking forward to your feedback, thanks. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway Best, Shammon
Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL
Hi all,
Thanks for all discussions!
Anyone else have questions or suggestions? if not, I will start a vote
thread later.
Best
Kui Yuan
kui yuan 于2023年2月27日周一 20:21写道：
> Hi Timo,
>
> Thanks for your advice. I totally agree with your suggestion of naming
> convention, I will rename these options and update the flip later, thanks
> very much.
>
> In our internal implementation we had put these options inside the
> `FactoryUtil`, just as you expect. We have also taken into account the
> changes to the CompiledPlan and we have packaged these options
> appropriately to minimize intrusiveness and ensure the compatibility to the
> `WatermarkPushDownSpec`.
>
> > A hint to the implementation: I would suggest that we add those options
> > to `FactoryUtil`. All cross-connector options should end up there.
>
>
> > Please also consider the changes to the CompiledPlan in your FLIP. This
> > change has implications on the JSON format as watermark strategy of
> > ExecNode becomes more complex, see WatermarkPushDownSpec
>
> Best
> Kui Yuan
>
> Timo Walther 于2023年2月27日周一 18:05写道：
>
>> Hi Kui Yuan,
>>
>> thanks for working on this FLIP. Let me also give some comments about
>> the proposed changes.
>>
>> I support the direction of this FLIP about handling these
>> watermark-specific properties through options and /*+OPTIONS(...) */
>> hints.
>>
>> Regarding naming, I would like to keep the options in sync with existing
>> options:
>>
>> > 'watermark.emit.strategy'='ON_EVENT'
>>
>> Let's use lower case (e.g. `on-event`) that matches with properties like
>> sink.partitioner [1] or sink.delivery-guarantee [2].
>>
>> > 'source.idle-timeout'='1min'
>>
>> According to FLIP-122 [3], we want to prefix all scan-source related
>> properties with `scan.*`. This clearly includes idle-timeout and
>> actually also watermark strategies which don't apply for lookup sources.
>>
>> Summarizing the comments above, we should use the following options:
>>
>> 'scan.watermark.emit.strategy'='on-event',
>> 'scan.watermark.emit.on-event.gap'='1',
>> 'scan.idle-timeout'='1min',
>> 'scan.watermark.alignment.group'='alignment-group-1',
>> 'scan.watermark.alignment.max-drift'='1min',
>> 'scan.watermark.alignment.update-interval'='1s'
>>
>> I know that this makes the keys even longer, but given that those
>> options are for power users this should be acceptable. It also clearly
>> indicates which options are for sinks, scans, and lookups. This
>> potentially also helps in allow lists.
>>
>> A hint to the implementation: I would suggest that we add those options
>> to `FactoryUtil`. All cross-connector options should end up there.
>>
>> Please also consider the changes to the CompiledPlan in your FLIP. This
>> change has implications on the JSON format as watermark strategy of
>> ExecNode becomes more complex, see WatermarkPushDownSpec [4].
>>
>> Regards,
>> Timo
>>
>>
>> [1]
>>
>> https://nightlies.apache.org/flink/flink-docs-release-1.11/dev/table/connectors/kafka.html#sink-partitioner
>> [2]
>>
>> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/#sink-delivery-guarantee
>> [3]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory
>> [4]
>>
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/abilities/source/WatermarkPushDownSpec.java
>>
>>
>> On 24.02.23 04:55, kui yuan wrote:
>> > Hi all,
>> >
>> > I have updated the flip according to the discussion, and we will extend
>> the
>> > watermark-related features with both table options and 'OPTIONS' hint,
>> like
>> > this:
>> >
>> > ```
>> > -- configure in table options
>> > CREATE TABLE user_actions (
>> >...
>> >user_action_time TIMESTAMP(3),
>> >WATERMARK FOR user_action_time AS user_action_time - INTERVAL '5'
>> SECOND
>> > ) WITH (
>> >'watermark.emit.strategy'='ON_PERIODIC',
>> >...
>> > );
>> >
>> > -- use 'OPTIONS' hint
>> > select ... from source_table /*+ OPTIONS('watermark.emit.strategy'=
>> > 'ON_PERIODIC') */
>> > ```
>> >
>> > Does everybody have any other questions?
>> >
>> > Best
>> > Kui Yuan
>> >
>> > kui yuan 于2023年2月23日周四 20:05写道：
>> >
>> >> Hi all,
>> >>
>> >> Thanks for all suggestions.
>> >>
>> >> We will extend the watermark-related features in SQL layer with dynamic
>> >> table options and 'OPTIONS' hint, just as everyone expects. I will
>> modify
>> >> Flip-296 as discussed.
>> >>
>> >> @Martijn As far as I know, there is no hint interface in the table API,
>> >> so we can't use hint in table API directly. if we need to extend the
>> hint
>> >> interface in the table API, maybe we need another flip. However, if we
>> >> extend the watermark-related features in the dynamic table options,
>> maybe
>> >> we are able to use them indirectly in the table API like this[1]:
>> >>
>> >> ```
>> >> // register a table named "Orders"
>> >> tableEnv.executeSql("CREATE TABLE Orders (`user`
Re: [VOTE] FLIP-291: Externalized Declarative Resource Management
+1 (binding) Thanks David, and everyone involved :) Regards, Roman On Wed, Mar 1, 2023 at 8:01 AM Gyula Fóra wrote: > +1 (binding) > > Looking forward to this :) > > Gyula > > On Wed, 1 Mar 2023 at 04:02, feng xiangyu wrote: > > > +1 (non-binding) > > > > ConradJam 于2023年3月1日周三 10:37写道： > > > > > +1 (non-binding) > > > > > > Zhanghao Chen 于2023年3月1日周三 10:18写道： > > > > > > > Thanks for driving this. +1 (non-binding) > > > > > > > > Best, > > > > Zhanghao Chen > > > > > > > > From: David Mor?vek > > > > Sent: Tuesday, February 28, 2023 21:46 > > > > To: dev > > > > Subject: [VOTE] FLIP-291: Externalized Declarative Resource > Management > > > > > > > > Hi Everyone, > > > > > > > > I want to start the vote on FLIP-291: Externalized Declarative > Resource > > > > Management [1]. The FLIP was discussed in this thread [2]. > > > > > > > > The goal of the FLIP is to enable external declaration of the > resource > > > > requirements of a running job. > > > > > > > > The vote will last for at least 72 hours (Friday, 3rd of March, 15:00 > > > CET) > > > > unless > > > > there is an objection or insufficient votes. > > > > > > > > [1] https://lists.apache.org/thread/b8fnj127jsl5ljg6p4w3c4wvq30cnybh > > > > [2] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management > > > > > > > > Best, > > > > D. > > > > > > > > > > > > > -- > > > Best > > > > > > ConradJam > > > > > >
[jira] [Created] (FLINK-31278) exit code 137 (i.e. OutOfMemoryError) in core module
Matthias Pohl created FLINK-31278: - Summary: exit code 137 (i.e. OutOfMemoryError) in core module Key: FLINK-31278 URL: https://issues.apache.org/jira/browse/FLINK-31278 Project: Flink Issue Type: Bug Reporter: Matthias Pohl The following build failed due to a 137 exit code indicating an OutOfMemoryError: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46643=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=7847 This build ran on an Azure pipeline machine (Azure Pipelines 9) and, therefore, cannot be caused by FLINK-18356. That said, there was a concurrent 137 exit code build failure happening on agent "Azure Pipelines 21" (see [20230301.3|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46643=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=7847]) ~10mins later -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31277) JM Deployment recovery logic inconsistent with health related restart
Gyula Fora created FLINK-31277: -- Summary: JM Deployment recovery logic inconsistent with health related restart Key: FLINK-31277 URL: https://issues.apache.org/jira/browse/FLINK-31277 Project: Flink Issue Type: Bug Components: Kubernetes Operator Affects Versions: kubernetes-operator-1.4.0 Reporter: Gyula Fora Assignee: Gyula Fora Fix For: kubernetes-operator-1.5.0 The current JM Deployment logic that restarts missing deployments strictly requires HA metadata event for stateless deployments. This is inconsistent with how the cluster health check related restarts work which can cause the operator to delete an unhealthy deployment and potentially leave it missing if the first deploy attempt fails. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31276) VectorIndexerTest#testFitAndPredictWithHandleInvalid fails
Zhipeng Zhang created FLINK-31276: - Summary: VectorIndexerTest#testFitAndPredictWithHandleInvalid fails Key: FLINK-31276 URL: https://issues.apache.org/jira/browse/FLINK-31276 Project: Flink Issue Type: Bug Components: Library / Machine Learning Affects Versions: ml-2.2.0 Reporter: Zhipeng Zhang The unit test fails [1] and the error stack is: Error: Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.176 s <<< FAILURE! - in org.apache.flink.ml.feature.VectorIndexerTest [592|https://github.com/apache/flink-ml/actions/runs/4300867923/jobs/7497476426#step:4:593]Error: testFitAndPredictWithHandleInvalid Time elapsed: 4.178 s <<< FAILURE! [593|https://github.com/apache/flink-ml/actions/runs/4300867923/jobs/7497476426#step:4:594]java.lang.AssertionError: expected: but was: [594|https://github.com/apache/flink-ml/actions/runs/4300867923/jobs/7497476426#step:4:595] at org.junit.Assert.fail(Assert.java:89) [595|https://github.com/apache/flink-ml/actions/runs/4300867923/jobs/7497476426#step:4:596] at org.junit.Assert.failNotEquals(Assert.java:835) [596|https://github.com/apache/flink-ml/actions/runs/4300867923/jobs/7497476426#step:4:597] at org.junit.Assert.assertEquals(Assert.java:120) [597|https://github.com/apache/flink-ml/actions/runs/4300867923/jobs/7497476426#step:4:598] at org.junit.Assert.assertEquals(Assert.java:146) [598|https://github.com/apache/flink-ml/actions/runs/4300867923/jobs/7497476426#step:4:599] at org.apache.flink.ml.feature.VectorIndexerTest.testFitAndPredictWithHandleInvalid(VectorIndexerTest.java:186) [1] [https://github.com/apache/flink-ml/actions/runs/4300867923/jobs/7497476426] -- This message was sent by Atlassian Jira (v8.20.10#820010)
