Re: [DISCUSS] Connector Externalization Retrospective
Hi Danny, Thanks for bringing this up, I might haven't driven a connector release myself but I echo the pain and delay in releases for adding Flink version support. I am not really with the mono-repo approach for the following reasons 1- We will lose the flexibility we currently have for connectors (I mean we even had a major release 4.X for AWS connectors IIRC). 2- We will be undoing the gains for CI decoupling, the frequency of contributions in connectors like kafka and AWS are unmatched for others like GCP and RabbitMQ, with new connector added and major feature work I see unnecessary cost and delays for CI due to monorepo. I believe the only benefit of this approach (over the other proposed )would be forcing adopting common connectors changes to all connectors at once instead of relying on divided efforts to port the change, however for the reasons mentioned above I still wouldn't vote for this approach. I am in favor of dropping the version though, I understand it might be confusing for users but as Sergey mentioned; many times the changes of a new Flink version don't introduce compatibility issues to connectors so I believe I believe it might be an easier task than It sound initially. A question would be what do you think the best approach to when we do introduce backward compatible changes to connectors API like in this PR[1], in this case existing connectors would still work with the newly released Flink version but would rather accumulate technical debt and removing it would be an Adhoc task for maintainers which I believe is an accepted tradeoff but would love to hear the feedback. 1- https://github.com/apache/flink/pull/24180/files#diff-2ffade463560e5941912b91b12a07c888313a4cc7e39ca8700398ed6975b8e90 Best Regards Ahmed Hamdy On Tue, 11 Jun 2024 at 08:50, Sergey Nuyanzin wrote: > Thanks for starting this discussion Danny > > I will put my 5 cents here > > From one side yes, support of new Flink release takes time as it was > mentioned above > However from another side most of the connectors (main/master branches) > supported Flink 1.19 > even before it was released, same for 1.20 since they were testing against > master and supported version branches. > There are already nightly/weekly jobs (depending on connector) > running against the latest Flink SNAPSHOTs. And it has already helped to > catch some blocker issues like[1], [2]. > In fact there are more, I need to spend time retrieving all of them. > > I would also not vote for connector mono-repo release since we recently > just splitted it. > > The thing I would suggest: > since we already have nightly/weekly jobs for connectors testing against > Flink main repo master branch > we could add a requirement before the release of Flink itself having these > job results also green. > > [1] https://issues.apache.org/jira/browse/FLINK-34941 > [2] https://issues.apache.org/jira/browse/FLINK-32978#comment-17804459 > > On Tue, Jun 11, 2024 at 8:24 AM Xintong Song > wrote: > > > Thanks for bringing this up, Danny. This is indeed an important issue > that > > the community needs to improve on. > > > > Personally, I think a mono-repo might not be a bad idea, if we apply > > different rules for the connector releases. To be specific: > > - flink-connectors 1.19.x contains all connectors that are compatible > with > > Flink 1.19.x. > > - allow not only bug-fixes, but also new features for a third-digit > release > > (e.g., flink-connectors 1.19.1) > > > > This would allow us to immediately release flink-connectors 1.19.0 right > > after flink 1.19.0 is out, excluding connectors that are no longer > > compatible with flink 1.19. Then we can have a couple of flink-connectors > > 1.19.x releases, gradually adding the missing connectors back. In the > worst > > case, this would result in as many releases as having separated connector > > repose. The benefit comes from 1) there are chances to combine releasing > of > > multiple connectors into one release of the mono repo (if they are ready > > around the same time), and 2) no need to maintain a compatibility matrix > > and worrying about it being out-of-sync with the code base. > > > > However, one thing I don't like about this approach is that it requires > > combining all the repos we just separated from the main-repo to another > > mono-repo. That back-and-forth is annoying. So I'm just speaking out my > > ideas, but would not strongly insist on this. > > > > And big +1 for compatibility tools and ci checks. > > > > Best, > > > > Xintong > > > > > > > > On Tue, Jun 11, 2024 at 2:38 AM David Radley > > wrote: > > > > > Hi Danny, > > > I think your proposal is
Re: [Discuss] Non-retriable (partial) errors in Elasticsearch 8 connector
Hi Mingliang,
Yes sounds like a good solution, I am not very familiar with ElasticSearch
internals and APIs but will try to assist with the PR when ready.
Best Regards
Ahmed Hamdy
On Tue, 11 Jun 2024 at 07:07, Mingliang Liu wrote:
> Thank you Ahmed for the explanation.
>
> The current Elasticsearch 8 connector already uses the
> FatalExeptionClassifier for fatal / non-retriable requests [1]. It's very
> similar to what you linked in the AWS connectors. Currently this is only
> used for fully failed requests. The main problem I was concerned about is
> the partial failures, when the Future of the client bulk request was not
> completed exceptionally, but instead some items failed according to the
> response. For those failed entries in the partially failed request, we
> retry infinitely though retrying will not always help.
>
> To avoid problems of "too many failed but non-retryable request entries in
> the buffer", I was thinking we can fail fast instead of infinitely
> retrying. Alternatively, we can limit the maximum number of retrying per
> rerecord. Like FLINK-35541 you shared for AWS connectors, I think a similar
> approach in Elasticsearch 8 connector would be useful. Given a
> non-retriable request entry, it will retry the request entries anyway but
> will eventually fail after exhausting the retries. Having both sound like a
> more comprehensive solution, as following sample:
>
> void handlePartiallyUnprocessedRequest(
> Response response, Consumer requestResult) {
> List requestsToRetry = new ArrayList<>();
>
> for (Request r : response.failedItems()) {
> if (!isRetryable(r.errorCode())// we don't have this
> check for ES 8, which could be 400 / 404
> || r.retryCount++ > maxRetry) { // FLINK-35541 could
> help limit retries for all failed requests
> throw new FlinkRuntimeException();
> }
> requestsToRetry.add(r);
> }
>
> requestResult.accept(requestsToRetry);
> }
>
> [1]
>
> https://github.com/apache/flink-connector-elasticsearch/blob/da2ef1fa6d5edd3cf1328b11632929fd2c99f567/flink-connector-elasticsearch8/src/main/java/org/apache/flink/connector/elasticsearch/sink/Elasticsearch8AsyncWriter.java#L73-L82
>
>
> On Fri, Jun 7, 2024 at 3:42 AM Ahmed Hamdy wrote:
>
> > Hi Mingliang,
> >
> > We already have a mechanism for detecting and propagating
> > Fatal/Non-retryable exceptions[1]. We can use that in ElasticSearch
> similar
> > to what we do for AWS connectors[2]. Also, you can check AWS connectors
> for
> > how to add a fail-fast mechanism to disable retrying all along.
> >
> > > FLIP-451 proposes timeout for retrying which helps with un-acknowledged
> > > requests, but not addressing the case when request gets processed and
> > > failed items keep failing no matter how many times we retry. Correct me
> > if
> > > I'm wrong
> > >
> > yes you are correct, this is mainly to mitigate the issues arising from
> > incorrect handling of requests in sink implementers.
> > The Failure handling itself has always been assumed to be the Sink
> > implementation responsibility, this is done in 3 levels
> > - Classifying Fatal exceptions as mentioned above
> > - Adding configuration to disable retries as mentioned above as well.
> > - Adding mechanism to limit retries as in the proposed ticket for AWS
> > connectors[3]
> >
> > In my opinion at least 1 and 3 are useful in this case for Elasticsearch,
> > Adding classifiers and retry mechanisms for elasticsearch.
> >
> > Or we can allow users to configure
> > > "drop/fail" behavior for non-retriable errors
> > >
> >
> > I am not sure I follow this proposal, but in general while "Dropping"
> > records seems to boost reliability, it breaks the at-least-once semantics
> > and if you don't have proper tracing and debugging mechanisms we will be
> > shooting ourselves in the foot.
> >
> >
> > 1-
> >
> >
> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/throwable/FatalExceptionClassifier.java
> > 2-
> >
> >
> https://github.com/apache/flink-connector-aws/blob/c688a8545ac1001c8450e8c9c5fe8bbafa13aeba/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/sink/DynamoDbSinkWriter.java#L227
> >
> > 3-https://issues.apache.org/jira/browse/FLINK-35541
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Thu, 6 Jun 2024 at 06:53, Mingliang
Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
Congratulations Rui! Best Regards Ahmed Hamdy On Mon, 10 Jun 2024 at 09:10, David Radley wrote: > Congratulations, Rui! > > From: Sergey Nuyanzin > Date: Sunday, 9 June 2024 at 20:33 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui > Congratulations, Rui! > > On Fri, Jun 7, 2024 at 5:36 AM Xia Sun wrote: > > > Congratulations, Rui! > > > > Best, > > Xia > > > > Paul Lam 于2024年6月6日周四 11:59写道： > > > > > Congrats, Rui! > > > > > > Best, > > > Paul Lam > > > > > > > 2024年6月6日 11:02，Junrui Lee 写道： > > > > > > > > Congratulations, Rui. > > > > > > > > Best, > > > > Junrui > > > > > > > > Hang Ruan 于2024年6月6日周四 10:35写道： > > > > > > > >> Congratulations, Rui! > > > >> > > > >> Best, > > > >> Hang > > > >> > > > >> Samrat Deb 于2024年6月6日周四 10:28写道： > > > >> > > > >>> Congratulations Rui > > > >>> > > > >>> Bests, > > > >>> Samrat > > > >>> > > > >>> On Thu, 6 Jun 2024 at 7:45 AM, Yuxin Tan > > > wrote: > > > >>> > > > >>>> Congratulations, Rui! > > > >>>> > > > >>>> Best, > > > >>>> Yuxin > > > >>>> > > > >>>> > > > >>>> Xuannan Su 于2024年6月6日周四 09:58写道： > > > >>>> > > > >>>>> Congratulations! > > > >>>>> > > > >>>>> Best regards, > > > >>>>> Xuannan > > > >>>>> > > > >>>>> On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu > > > > >>> wrote: > > > >>>>>> > > > >>>>>> Congratulations, Rui ! > > > >>>>>> > > > >>>>>> On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee < > > lincoln.8...@gmail.com > > > >>> > > > >>>>> wrote: > > > >>>>>> > > > >>>>>>> Congratulations, Rui! > > > >>>>>>> > > > >>>>>>> Best, > > > >>>>>>> Lincoln Lee > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> Lijie Wang 于2024年6月6日周四 09:11写道： > > > >>>>>>> > > > >>>>>>>> Congratulations, Rui! > > > >>>>>>>> > > > >>>>>>>> Best, > > > >>>>>>>> Lijie > > > >>>>>>>> > > > >>>>>>>> Rodrigo Meneses 于2024年6月5日周三 21:35写道： > > > >>>>>>>> > > > >>>>>>>>> All the best > > > >>>>>>>>> > > > >>>>>>>>> On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng < > > > >>>> xiangyu...@gmail.com> > > > >>>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> Congratulations, Rui! > > > >>>>>>>>>> > > > >>>>>>>>>> Regards, > > > >>>>>>>>>> Xiangyu Feng > > > >>>>>>>>>> > > > >>>>>>>>>> Feng Jin 于2024年6月5日周三 20:42写道： > > > >>>>>>>>>> > > > >>>>>>>>>>> Congratulations, Rui! > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> Best, > > > >>>>>>>>>>> Feng Jin > > > >>>>>>>>>>> > > > >>>>>>>>>>> On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei < > > > >>>> fredia...@gmail.com > > > >>>>>> > > > >>>>>>>> wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>>> Congratu
Re: [VOTE] Release 1.19.1, release candidate #1
Hi Hong, Thanks for driving +1 (non-binding) - Verified signatures and hashes - Checked github release tag - Verified licenses - Checked that the source code does not contain binaries - Reviewed Web PR, nit: Could we address the comment of adding FLINK-34633 in the release Best Regards Ahmed Hamdy On Sat, 8 Jun 2024 at 22:22, Jeyhun Karimov wrote: > Hi Hong, > > Thanks for driving the release. > +1 (non-binding) > > - Verified gpg signature > - Reviewed the PR > - Verified sha512 > - Checked github release tag > - Checked that the source code does not contain binaries > > Regards, > Jeyhun > > On Sat, Jun 8, 2024 at 1:52 PM weijie guo > wrote: > > > Thanks Hong! > > > > +1(binding) > > > > - Verified gpg signature > > - Verified sha512 hash > > - Checked gh release tag > > - Checked all artifacts deployed to maven repo > > - Ran a simple wordcount job on local standalone cluster > > - Compiled from source code with JDK 1.8.0_291. > > > > Best regards, > > > > Weijie > > > > > > Xiqian YU 于2024年6月7日周五 18:23写道： > > > > > +1 (non-binding) > > > > > > > > > * Checked download links & release tags > > > * Verified that package checksums matched > > > * Compiled Flink from source code with JDK 8 / 11 > > > * Ran E2e data integration test jobs on local cluster > > > > > > Regards, > > > yux > > > > > > De : Rui Fan <1996fan...@gmail.com> > > > Date : vendredi, 7 juin 2024 à 17:14 > > > À : dev@flink.apache.org > > > Objet : Re: [VOTE] Release 1.19.1, release candidate #1 > > > +1(binding) > > > > > > - Reviewed the flink-web PR (Left some comments) > > > - Checked Github release tag > > > - Verified signatures > > > - Verified sha512 (hashsums) > > > - The source archives do not contain any binaries > > > - Build the source with Maven 3 and java8 (Checked the license as well) > > > - Start the cluster locally with jdk8, and run the StateMachineExample > > job, > > > it works fine. > > > > > > Best, > > > Rui > > > > > > On Thu, Jun 6, 2024 at 11:39 PM Hong Liang wrote: > > > > > > > Hi everyone, > > > > Please review and vote on the release candidate #1 for the flink > > v1.19.1, > > > > as follows: > > > > [ ] +1, Approve the release > > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > > > > > The complete staging area is available for your review, which > includes: > > > > * JIRA release notes [1], > > > > * the official Apache source release and binary convenience releases > to > > > be > > > > deployed to dist.apache.org [2], which are signed with the key with > > > > fingerprint B78A5EA1 [3], > > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > > * source code tag "release-1.19.1-rc1" [5], > > > > * website pull request listing the new release and adding > announcement > > > blog > > > > post [6]. > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > majority > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > Thanks, > > > > Hong > > > > > > > > [1] > > > > > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354399 > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.19.1-rc1/ > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > > [4] > > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1736/ > > > > [5] https://github.com/apache/flink/releases/tag/release-1.19.1-rc1 > > > > [6] https://github.com/apache/flink-web/pull/745 > > > > > > > > > >
Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn
+1 (non-binding) Best Regards Ahmed Hamdy On Sat, 8 Jun 2024 at 22:26, Jeyhun Karimov wrote: > Hi Yuxin, > > Thanks for driving this. > +1 (non-binding) > > Regards, > Jeyhun > > On Fri, Jun 7, 2024 at 6:05 PM Jim Hughes > wrote: > > > HI all, > > > > +1 (non-binding) > > > > Cheers, > > > > Jim > > > > On Fri, Jun 7, 2024 at 4:03 AM Yuxin Tan wrote: > > > > > Hi everyone, > > > > > > Thanks for all the feedback about the FLIP-459 Support Flink > > > hybrid shuffle integration with Apache Celeborn[1]. > > > The discussion thread is here [2]. > > > > > > I'd like to start a vote for it. The vote will be open for at least > > > 72 hours unless there is an objection or insufficient votes. > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > > [2] https://lists.apache.org/thread/gy7sm7qrf7yrv1rl5f4vtk5fo463ts33 > > > > > > Best, > > > Yuxin > > > > > >
Re: [DISCUSS] FLIP-299 Pub/Sub Lite Connector
Hi all, I wonder if we can revive the discussion here. It has been discussed in another thread[1] that the lack of maintenance for connector repos as in gcp-pubsub should be a motive to promote more committers who actually commit to maintaining the repos instead of migrating/adding connectors out of support of the community, and I don't see another blocker for such FLIP to progress given the effort done by Daniel on the external repo. I do have comments on the FLIP itself, and am happy to drive changes myself If Daniel is no longer available but this doesn't seem the core point of the discussion at the moment. I would love to hear your thoughts. 1-https://lists.apache.org/thread/63wdvo5hvr6koc3wzbqy2kw4krhmkfbx Best Regards Ahmed Hamdy On Fri, 17 Mar 2023 at 15:58, Daniel Collins wrote: > > would the repository ... be removed ... ? > > Yes, I would remove it once it is merged into a version of flink that is > supported by GCP dataproc. It exists now (and I am creating releases and > maven artifacts for it) to unblock users in the interim period. > > -Daniel > > On Thu, Mar 16, 2023 at 3:32 PM Martijn Visser > wrote: > > > Hi Daniel, > > > > > I don't know how to get to this point, it sounds like more of an > > organizational constraint than a technical one though- who is responsible > > for the same role for the standard Pub/Sub connector? I'm working with > the > > Pub/Sub team right now on prioritization of supporting the flink > connector > > and converting it to support the recommended delivery mechanism for that > > service. > > > > It's not so much an organizational constraint, it's more if there are one > > or more committers in the Flink community who have the bandwidth to help > > with reviewing and merging a new connector. The PubSub connector has > pretty > > much been unmaintained for the past couple of years; I have done a couple > > of outreaches to Google, but those were unfruitful. > > > > I'm hoping that someone from the Flink committers has the bandwidth for > > helping you out. @All, if you have bandwidth, please come forward. > > > > > I imagine our involvement would be similar to support for our > > self-managed client libraries > > > > I think that sounds fine. > > > > One question I have is if you envision that the code of > > https://github.com/googleapis/java-pubsublite-flink moves to > > https://github.com/apache/flink-connector-gcp-pubsub, would the > repository > > https://github.com/googleapis/java-pubsublite-flink be removed or do you > > propose to have both of them exist? I would be in favour of having one, > but > > wanted to check with you. > > > > Best regards, > > > > Martijn > > > > On Tue, Mar 14, 2023 at 4:09 AM Daniel Collins > > > > wrote: > > > > > Hi all, > > > > > > Thank you for the feedback. Responses inline. > > > > > > > we need feedback from a Committer who would review and help maintain > it > > > going forward. Ideally, this Committer would guide one or more > > contributors > > > from Google to Committership so that Google could step up and maintain > > > Flink > > > 's PubSub and PubSub Lite Connector in the future. > > > > > > I don't know how to get to this point, it sounds like more of an > > > organizational constraint than a technical one though- who is > responsible > > > for the same role for the standard Pub/Sub connector? I'm working with > > the > > > Pub/Sub team right now on prioritization of supporting the flink > > connector > > > and converting it to support the recommended delivery mechanism for > that > > > service. > > > > > > > For this, it would be good to understand how you envision the > > involvement > > > of the PubSub Lite team at Google. > > > > > > I imagine our involvement would be similar to support for our > > self-managed > > > client libraries, we would field feature requests and dependency update > > > requests as they come in, as well as debugging any bug reports, and > > > improving the library as new service features arrive. We would want to > > work > > > with the flink community to support new features that our users find > > > useful. Our team did not contribute the Pub/Sub connector, but as more > of > > > our customers come to use flink, we would like to bring it up to par > both > > > in performance and supportability of the rest of our supported clients. > > > > > > > I think that giving the bi
Re: [Discuss] Non-retriable (partial) errors in Elasticsearch 8 connector
Hi Mingliang, We already have a mechanism for detecting and propagating Fatal/Non-retryable exceptions[1]. We can use that in ElasticSearch similar to what we do for AWS connectors[2]. Also, you can check AWS connectors for how to add a fail-fast mechanism to disable retrying all along. > FLIP-451 proposes timeout for retrying which helps with un-acknowledged > requests, but not addressing the case when request gets processed and > failed items keep failing no matter how many times we retry. Correct me if > I'm wrong > yes you are correct, this is mainly to mitigate the issues arising from incorrect handling of requests in sink implementers. The Failure handling itself has always been assumed to be the Sink implementation responsibility, this is done in 3 levels - Classifying Fatal exceptions as mentioned above - Adding configuration to disable retries as mentioned above as well. - Adding mechanism to limit retries as in the proposed ticket for AWS connectors[3] In my opinion at least 1 and 3 are useful in this case for Elasticsearch, Adding classifiers and retry mechanisms for elasticsearch. Or we can allow users to configure > "drop/fail" behavior for non-retriable errors > I am not sure I follow this proposal, but in general while "Dropping" records seems to boost reliability, it breaks the at-least-once semantics and if you don't have proper tracing and debugging mechanisms we will be shooting ourselves in the foot. 1- https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/throwable/FatalExceptionClassifier.java 2- https://github.com/apache/flink-connector-aws/blob/c688a8545ac1001c8450e8c9c5fe8bbafa13aeba/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/sink/DynamoDbSinkWriter.java#L227 3-https://issues.apache.org/jira/browse/FLINK-35541 Best Regards Ahmed Hamdy On Thu, 6 Jun 2024 at 06:53, Mingliang Liu wrote: > Hi all, > > Currently the Elasticsearch 8 connector retries all items if the request > fails as a whole, and retries failed items if the request has partial > failures [1]. I think this infinitely retries might be problematic in some > cases when retrying can never eventually succeed. For example, if the > request is 400 (bad request) or 404 (not found), retries do not help. If > there are too many failed items non-retriable, new requests will get > processed less effectively. In extreme cases, it may stall the pipeline if > in-flight requests are occupied by those failed items. > > FLIP-451 proposes timeout for retrying which helps with un-acknowledged > requests, but not addressing the case when request gets processed and > failed items keep failing no matter how many times we retry. Correct me if > I'm wrong. > > One opinionated option is to fail fast for non-retriable errors like 400 / > 404 and to drop items for 409. Or we can allow users to configure > "drop/fail" behavior for non-retriable errors. I prefer the latter. I > checked how LogStash ingests data to Elasticsearch and it takes a similar > approach for non-retriable errors [2]. In my day job, we have a > dead-letter-queue in AsynSinkWriter for failed entries that exhaust > retries. I guess that is too specific to our setup and seems an overkill > here for Elasticsearch connector. > > Any thoughts on this? > > [1] > > https://github.com/apache/flink-connector-elasticsearch/blob/5d1f8d03e3cff197ed7fe30b79951e44808b48fe/flink-connector-elasticsearch8/src/main/java/org/apache/flink/connector/elasticsearch/sink/Elasticsearch8AsyncWriter.java#L152-L170 > [2] > > https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/main/lib/logstash/plugin_mixins/elasticsearch/common.rb#L283-L304 >
[jira] [Created] (FLINK-35548) Add E2E tests for PubSubSinkV2
Ahmed Hamdy created FLINK-35548: --- Summary: Add E2E tests for PubSubSinkV2 Key: FLINK-35548 URL: https://issues.apache.org/jira/browse/FLINK-35548 Project: Flink Issue Type: Sub-task Reporter: Ahmed Hamdy Assignee: Ahmed Hamdy Fix For: gcp-pubsub-3.2.0 Refactor Google PubSub source to use Unified Sink API [FLIP-143|https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo
Congratulations Weijie Best Regards Ahmed Hamdy On Tue, 4 Jun 2024 at 10:51, Matthias Pohl wrote: > Congratulations, Weijie! > > Matthias > > On Tue, Jun 4, 2024 at 11:12 AM Guowei Ma wrote: > > > Congratulations! > > > > Best, > > Guowei > > > > > > On Tue, Jun 4, 2024 at 4:55 PM gongzhongqiang > > > wrote: > > > > > Congratulations Weijie! Best, > > > Zhongqiang Gong > > > > > > Xintong Song 于2024年6月4日周二 14:46写道： > > > > > > > Hi everyone, > > > > > > > > On behalf of the PMC, I'm very happy to announce that Weijie Guo has > > > joined > > > > the Flink PMC! > > > > > > > > Weijie has been an active member of the Apache Flink community for > many > > > > years. He has made significant contributions in many components, > > > including > > > > runtime, shuffle, sdk, connectors, etc. He has driven / participated > in > > > > many FLIPs, authored and reviewed hundreds of PRs, been consistently > > > active > > > > on mailing lists, and also helped with release management of 1.20 and > > > > several other bugfix releases. > > > > > > > > Congratulations and welcome Weijie! > > > > > > > > Best, > > > > > > > > Xintong (on behalf of the Flink PMC) > > > > > > > > > >
[jira] [Created] (FLINK-35500) DynamoDb SinkWriter fails to delete elements due to key not found
Ahmed Hamdy created FLINK-35500:
---
Summary: DynamoDb SinkWriter fails to delete elements due to key
not found
Key: FLINK-35500
URL: https://issues.apache.org/jira/browse/FLINK-35500
Project: Flink
Issue Type: Bug
Components: Connectors / DynamoDB
Affects Versions: aws-connector-4.2.0, aws-connector-4.1.0,
aws-connector-4.0.0
Reporter: Ahmed Hamdy
Fix For: aws-connector-4.4.0
h2. Description
When DynamoDbSink is used with CDC sources, it fails to process {{DELETE}}
records and throws
{quote}org.apache.flink.connector.dynamodb.shaded.software.amazon.awssdk.services.dynamodb.model.DynamoDbException:
The provided key element does not match the schema{quote}
This is due to {{DynamoDbSinkWriter}} passing the whole DynamoDb Item as key
instead of the constructed primary Key[1].
Note: The issue is reported in user mailing list[2]
h2. Steps to Reproduce
(1) Create a new DynamoDB table in AWS. Command line:
aws dynamodb create-table \
--table-name orders \
--attribute-definitions AttributeName=userId,AttributeType=S \
--key-schema AttributeName=userId,KeyType=HASH \
--billing-mode PAY_PER_REQUEST
(2) Create an input file in Debezium-JSON format with the following rows to
start:
{"op": "c", "after": {"orderId": 1, "userId": "a", "price": 5}}
{"op": "c", "after": {"orderId": 2, "userId": "b", "price": 7}}
{"op": "c", "after": {"orderId": 3, "userId": "c", "price": 9}}
{"op": "c", "after": {"orderId": 4, "userId": "a", "price": 11}}
(3) Start the Flink SQL Client, and run the following, substituting in the
proper local paths for the Dynamo Connector JAR file and for this local sample
input file:
ADD JAR '/Users/robg/Downloads/flink-sql-connector-dynamodb-4.2.0-1.18.jar';
SET 'execution.runtime-mode' = 'streaming';
SET 'sql-client.execution.result-mode' = 'changelog';
CREATE TABLE Orders_CDC(
orderId BIGINT,
price float,
userId STRING
) WITH (
'connector' = 'filesystem',
'path' = '/path/to/input_file.jsonl',
'format' = 'debezium-json'
);
CREATE TABLE Orders_Dynamo (
orderId BIGINT,
price float,
userId STRING,
PRIMARY KEY (userId) NOT ENFORCED
) PARTITIONED BY ( userId )
WITH (
'connector' = 'dynamodb',
'table-name' = 'orders',
'aws.region' = 'us-east-1'
);
INSERT INTO Orders_Dynamo SELECT * FROM Orders_CDC ;
(4) At this point, we will see that things currently all work properly, and
these 4 rows are inserted properly to Dynamo, because they are "Insert"
operations. So far, so good!
(5) Now, add the following row to the input file. This represents a deletion
in Debezium format, which should then cause a Deletion on the corresponding
DynamoDB table:
{"op": "d", "before": {"orderId": 3, "userId": "c", "price": 9}}
(6) Re-Run the SQL statement:
INSERT INTO Orders_Dynamo SELECT * FROM Orders_CDC ;
h3. References
1-https://github.com/apache/flink-connector-aws/blob/main/flink-connector-aws/flink-connector-dynamodb/src/main/java/org/apache/flink/connector/dynamodb/sink/DynamoDbSinkWriter.java#L267
2- https://lists.apache.org/thread/ysvctpvn6n9kn0qlf5b24gxchfg64ylf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] Add Flink CDC Channel to Apache Flink Slack Workspace
Thanks Zhongqiang, +1 for sure. Best Regards Ahmed Hamdy On Wed, 29 May 2024 at 13:48, ConradJam wrote: > +1 best > > Hang Ruan 于2024年5月29日周三 11:28写道： > > > Hi, zhongqiang. > > > > Thanks for the proposal. +1 for it. > > > > Best, > > Hang > > > > Leonard Xu 于2024年5月28日周二 11:58写道： > > > > > > > > Thanks Zhongqiang for the proposal, we need the Channel and I should > have > > > been created it but not yet, +1 from my side. > > > > > > Best, > > > Leonard > > > > > > > 2024年5月28日 上午11:54，gongzhongqiang 写道： > > > > > > > > Hi devs, > > > > > > > > I would like to propose adding a dedicated Flink CDC channel to the > > > Apache > > > > Flink Slack workspace. > > > > > > > > Creating a channel focused on Flink CDC will help community members > > > easily > > > > find previous discussions > > > > and target new discussions and questions to the correct place. Flink > > CDC > > > is > > > > a sufficiently distinct component > > > > within the Apache Flink ecosystem, and having a dedicated channel > will > > > make > > > > it viable and useful for > > > > those specifically working with or interested in this technology. > > > > > > > > Looking forward to your feedback and support on this proposal. > > > > > > > > > > > > Best, > > > > Zhongqiang Gong > > > > > > > > > > > -- > Best > > ConradJam >
[jira] [Created] (FLINK-35454) Connector ArchTests fails due to dependency on fink.util.Preconditions
Ahmed Hamdy created FLINK-35454:
---
Summary: Connector ArchTests fails due to dependency on
fink.util.Preconditions
Key: FLINK-35454
URL: https://issues.apache.org/jira/browse/FLINK-35454
Project: Flink
Issue Type: Bug
Components: Test Infrastructure
Affects Versions: 1.20.0
Reporter: Ahmed Hamdy
Fix For: 1.20.0
h2. Description
- Arch Unit Rules for connectors limits dependencies of any classes in
connectors on @Public or @PublicEvolving with exceptions of connector package
classes, this is not true since we should be able to depend on internal util
classes like {{Preconditions}} and {{ExceptionsUtils}}
{code:java}
freeze(
javaClassesThat(resideInAnyPackage(CONNECTOR_PACKAGES))
.and()
.areNotAnnotatedWith(Deprecated.class)
.should()
.onlyDependOnClassesThat(
areFlinkClassesThatResideOutsideOfConnectorPackagesAndArePublic()
.or(
JavaClass.Predicates.resideOutsideOfPackages(
"org.apache.flink.."))
.or(
JavaClass.Predicates.resideInAnyPackage(
CONNECTOR_PACKAGES)) )
.as(
"Connector production code must depend only
on public API when outside of connector packages"));
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35445) Update Async Sink documentation for Timeout configuration
Ahmed Hamdy created FLINK-35445: --- Summary: Update Async Sink documentation for Timeout configuration Key: FLINK-35445 URL: https://issues.apache.org/jira/browse/FLINK-35445 Project: Flink Issue Type: Improvement Components: Connectors / Common, Documentation Reporter: Ahmed Hamdy Fix For: 1.20.0 Update Documentation for AsyncSink Changes introduced by [FLIP-451|https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35435) [FLIP-451] Introduce timeout configuration to AsyncSink
Ahmed Hamdy created FLINK-35435: --- Summary: [FLIP-451] Introduce timeout configuration to AsyncSink Key: FLINK-35435 URL: https://issues.apache.org/jira/browse/FLINK-35435 Project: Flink Issue Type: Improvement Components: Connectors / Common Reporter: Ahmed Hamdy Fix For: 1.20.0 Implementation Ticket for: https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API -- This message was sent by Atlassian Jira (v8.20.10#820010)
[RESULT][VOTE] FLIP-451: Introduce timeout configuration to AsyncSink
Hi all I'm happy to announce that we have unanimously approved FLIP-451[1] There were 9 votes, of which 4 were binding on the vote thread[2]. - Muhammet Orazov (non-binding) - Jeyhun Karimov (non-binding) - Lorenzo Affetti (non-binding) - Aleksandr Pilipenko (non-binding) - David Radley (non-binding) - Danny Cranmer (binding) - Hong Liang (binding) - Leonard Xu (binding) - Weijie Guo (binding) Therefore, FLIP-451 was accepted. Thanks for the feedback! 1- https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API 2-https://lists.apache.org/thread/9mkzdv0gzssxh8p0y1cl3dvj6lvwyrc0 Best Regards Ahmed Hamdy
Re: [VOTE] FLIP-451: Introduce timeout configuration to AsyncSink
Hi all, Thanks for all your votes, I hereby close the vote and I'll announce the results in a separate email. Best Regards Ahmed Hamdy On Wed, 22 May 2024 at 11:30, weijie guo wrote: > Thanks Ahmed for introducing this FLIP, nice improvement. > > +1(binding) > > Best regards, > > Weijie > > > Leonard Xu 于2024年5月22日周三 17:25写道： > > > > > After discuss with Ahmed, the updated FLIP looks good to me. > > > > +1(binding) > > > > > > Best, > > Leonard > > > > > 2024年5月21日 下午6:12，Hong Liang 写道： > > > > > > +1 (binding) > > > > > > Thanks Ahmed > > > > > > On Tue, May 14, 2024 at 11:51 AM David Radley > > > > wrote: > > > > > >> Thanks for the clarification Ahmed > > >> > > >> +1 (non-binding) > > >> > > >> From: Ahmed Hamdy > > >> Date: Monday, 13 May 2024 at 19:58 > > >> To: dev@flink.apache.org > > >> Subject: [EXTERNAL] Re: [VOTE] FLIP-451: Introduce timeout > configuration > > >> to AsyncSink > > >> Thanks David, > > >> I have replied to your question in the discussion thread. > > >> Best Regards > > >> Ahmed Hamdy > > >> > > >> > > >> On Mon, 13 May 2024 at 16:21, David Radley > > >> wrote: > > >> > > >>> Hi, > > >>> I raised a question on the discussion thread, around retriable > errors, > > as > > >>> a possible alternative, > > >>> Kind regards, David. > > >>> > > >>> > > >>> From: Aleksandr Pilipenko > > >>> Date: Monday, 13 May 2024 at 16:07 > > >>> To: dev@flink.apache.org > > >>> Subject: [EXTERNAL] Re: [VOTE] FLIP-451: Introduce timeout > > configuration > > >>> to AsyncSink > > >>> Thanks for driving this! > > >>> > > >>> +1 (non-binding) > > >>> > > >>> Thanks, > > >>> Aleksandr > > >>> > > >>> On Mon, 13 May 2024 at 14:08, > > > >>> wrote: > > >>> > > >>>> Thanks Ahmed! > > >>>> > > >>>> +1 non binding > > >>>> On May 13, 2024 at 12:40 +0200, Jeyhun Karimov < > je.kari...@gmail.com > > >, > > >>>> wrote: > > >>>>> Thanks for driving this Ahmed. > > >>>>> > > >>>>> +1 (non-binding) > > >>>>> > > >>>>> Regards, > > >>>>> Jeyhun > > >>>>> > > >>>>> On Mon, May 13, 2024 at 12:37 PM Muhammet Orazov > > >>>>> wrote: > > >>>>> > > >>>>>> Thanks Ahmed, +1 (non-binding) > > >>>>>> > > >>>>>> Best, > > >>>>>> Muhammet > > >>>>>> > > >>>>>> On 2024-05-13 09:50, Ahmed Hamdy wrote: > > >>>>>>>> Hi all, > > >>>>>>>> > > >>>>>>>> Thanks for the feedback on the discussion thread[1], I would > > >> like > > >>>> to > > >>>>>>>> start > > >>>>>>>> a vote on FLIP-451[2]: Introduce timeout configuration to > > >>> AsyncSink > > >>>>>>>> > > >>>>>>>> The vote will be open for at least 72 hours unless there is an > > >>>>>>>> objection or > > >>>>>>>> insufficient votes. > > >>>>>>>> > > >>>>>>>> 1- > > >>> https://lists.apache.org/thread/ft7wcw7kyftvww25n5fm4l925tlgdfg0 > > >>>>>>>> 2- > > >>>>>>>> > > >>>>>> > > >>>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API > > >>>>>>>> Best Regards > > >>>>>>>> Ahmed Hamdy > > >>>>>> > > >>>> > > >>> > > >>> Unless otherwise stated above: > > >>> > > >>> IBM United Kingdom Limited > > >>> Registered in England and Wales with number 741598 > > >>> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 > 3AU > > >>> > > >> > > >> Unless otherwise stated above: > > >> > > >> IBM United Kingdom Limited > > >> Registered in England and Wales with number 741598 > > >> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 > 3AU > > >> > > > > >
Re: [DISCUSS] FLIP-451: Refactor Async sink API
> > (1) Implicitly point a public API change is not enough, Could you add a > section Public Interfaces to enumerate all Public APIs that you proposed > and you changed? > It’s a standard part of a FLIP template[1]. > yes this is updated in the FLIP now. > (2) About the proposed public interface ResultHandler, > Could you explain or show how to use the methods #completeExceptionally and > #retryForEntries? I didn’t find > detail explanation or Usage example code to understand them. > Added to the FLIP now. > (3) Could you add necessary java documents for all public API changes like > new method AsyncSinkWriterConfiguration#setRequestTimeoutMs ? The java doc > of [2] is a good example. > sure, Added now. (4) Another minor reminder AsyncSinkBase is a @PublicEvolving interface > too, please correct it, and please ensure the backward compatibility has > been considered for all public interfaces the FLIP changed. > Done Best Regards Ahmed Hamdy On Wed, 22 May 2024 at 04:16, Leonard Xu wrote: > Thanks for your reply, Ahmed. > > > (2) The FLIP-451 aims to introduce a timeout configuration, but I didn’t > >> find the configuration in FLIP even I lookup some historical versions of > >> the FLIP. Did I miss some key informations? > >> > > > > Yes, I tried to implicitly point that it will be added to the existing > > AsyncSinkWriterConfiguration to not inflate the FLIP, but I get it might > be > > confusing. I have added the changes to the configuration classes in the > > FLIP to make it clearer. > > (1) Implicitly point a public API change is not enough, Could you add a > section Public Interfaces to enumerate all Public APIs that you proposed > and you changed? > It’s a standard part of a FLIP template[1]. > > (2) About the proposed public interface ResultHandler, > Could you explain or show how to use the methods #completeExceptionally and > #retryForEntries? I didn’t find > detail explanation or Usage example code to understand them. > > (3) Could you add necessary java documents for all public API changes like > new method AsyncSinkWriterConfiguration#setRequestTimeoutMs ? The java doc > of [2] is a good example. > > (4) Another minor reminder AsyncSinkBase is a @PublicEvolving interface > too, please correct it, and please ensure the backward compatibility has > been considered for all public interfaces the FLIP changed. > > > Best, > Leonard > [1]https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template > [2] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink > > > > > > > > On Tue, 21 May 2024 at 14:56, Leonard Xu wrote: > > > >> Thanks Ahmed for kicking off this discussion, sorry for jumping the > >> discussion late. > >> > >> （1）I’m confused about the discuss thread name ‘FLIP-451: Refactor Async > >> sink API’ and FLIP title/vote thread name ' > >> FLIP-451: Introduce timeout configuration to AsyncSink API < > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API > >’, > >> they are different for me. Could you help explain the change history? > >> > >> (2) The FLIP-451 aims to introduce a timeout configuration, but I didn’t > >> find the configuration in FLIP even I lookup some historical versions of > >> the FLIP. Did I miss some key informations? > >> > >> (3) About the code change part, there’re some un-complete pieces in > >> AsyncSinkWriter for example `submitRequestEntries(List > >> requestEntries,);` is incorrect and `sendTime` variable I didn’t > >> find the place we define it and where we use it. > >> > >> Sorry for jumping the discussion thread during vote phase again. > >> > >> Best, > >> Leonard > >> > >> > >>> 2024年5月21日 下午3:49，Ahmed Hamdy 写道： > >>> > >>> Hi Hong, > >>> Thanks for pointing that out, no we are not > >>> deprecating getFatalExceptionCons(). I have updated the FLIP > >>> Best Regards > >>> Ahmed Hamdy > >>> > >>> > >>> On Mon, 20 May 2024 at 15:40, Hong Liang wrote: > >>> > >>>> Hi Ahmed, > >>>> Thanks for putting this together! Should we still be marking > >>>> getFatalExceptionCons() as @Deprecated in this FLIP, if we are not > >>>> providing a replacement? > >>>> > >>>> Regards, > >>>> Hong > >>>> > >>>> On Mon, May 13, 2024 at 7:58 PM Ahmed Hamdy >
Re: [DISCUSS] FLIP-451: Refactor Async sink API
Hi Leonard, Thanks for joining the discussion. > （1）I’m confused about the discuss thread name ‘FLIP-451: Refactor Async > sink API’ and FLIP title/vote thread name ' > FLIP-451: Introduce timeout configuration to AsyncSink API < > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API>’, > they are different for me. Could you help explain the change history? > The discussion started with a wider scope to refactor the API for handling results and exceptions as well as adding timeout configuration, however the scope was reduced after feedback above to focus on the timeout configuration since this is the most urgent and is not tightly coupled with the remaining suggestions. (2) The FLIP-451 aims to introduce a timeout configuration, but I didn’t > find the configuration in FLIP even I lookup some historical versions of > the FLIP. Did I miss some key informations? > Yes, I tried to implicitly point that it will be added to the existing AsyncSinkWriterConfiguration to not inflate the FLIP, but I get it might be confusing. I have added the changes to the configuration classes in the FLIP to make it clearer. (3) About the code change part, there’re some un-complete pieces in > AsyncSinkWriter for example `submitRequestEntries(List > requestEntries,);` is incorrect and `sendTime` variable I didn’t > find the place we define it and where we use it. > Thanks for catching, I fixed the incomplete methods, This should clarify how the new method is going to be integrated with the rest of the writer. Regarding the sendTime variable, I have replaced it with requestTimestamp; this is an unchanged part of the code to ensure the existing completeRequest method is unchanged. Best Regards Ahmed Hamdy On Tue, 21 May 2024 at 14:56, Leonard Xu wrote: > Thanks Ahmed for kicking off this discussion, sorry for jumping the > discussion late. > > （1）I’m confused about the discuss thread name ‘FLIP-451: Refactor Async > sink API’ and FLIP title/vote thread name ' > FLIP-451: Introduce timeout configuration to AsyncSink API < > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API>’, > they are different for me. Could you help explain the change history? > > (2) The FLIP-451 aims to introduce a timeout configuration, but I didn’t > find the configuration in FLIP even I lookup some historical versions of > the FLIP. Did I miss some key informations? > > (3) About the code change part, there’re some un-complete pieces in > AsyncSinkWriter for example `submitRequestEntries(List > requestEntries,);` is incorrect and `sendTime` variable I didn’t > find the place we define it and where we use it. > > Sorry for jumping the discussion thread during vote phase again. > > Best, > Leonard > > > > 2024年5月21日 下午3:49，Ahmed Hamdy 写道： > > > > Hi Hong, > > Thanks for pointing that out, no we are not > > deprecating getFatalExceptionCons(). I have updated the FLIP > > Best Regards > > Ahmed Hamdy > > > > > > On Mon, 20 May 2024 at 15:40, Hong Liang wrote: > > > >> Hi Ahmed, > >> Thanks for putting this together! Should we still be marking > >> getFatalExceptionCons() as @Deprecated in this FLIP, if we are not > >> providing a replacement? > >> > >> Regards, > >> Hong > >> > >> On Mon, May 13, 2024 at 7:58 PM Ahmed Hamdy > wrote: > >> > >>> Hi David, > >>> yes there error classification was initially left to sink implementers > to > >>> handle while we provided utilities to classify[1] and bubble up[2] > fatal > >>> exceptions to avoid retrying them. > >>> Additionally some sink implementations provide an option to short > circuit > >>> the failures by exposing a `failOnError` flag as in > >> KinesisStreamsSink[3], > >>> however this FLIP scope doesn't include any changes for retry > mechanisms. > >>> > >>> 1- > >>> > >>> > >> > https://github.com/apache/flink/blob/015867803ff0c128b1c67064c41f37ca0731ed86/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/throwable/FatalExceptionClassifier.java#L32 > >>> 2- > >>> > >>> > >> > https://github.com/apache/flink/blob/015867803ff0c128b1c67064c41f37ca0731ed86/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L533 > >>> 3- > >>> > >>> > >> > https://github.com/apache/flink-connector-aws/blob/c6e0abb65a0e51b40dd218b890a111886fbf797f/f
Re: [DISCUSS] FLIP-451: Refactor Async sink API
Hi Hong, Thanks for pointing that out, no we are not deprecating getFatalExceptionCons(). I have updated the FLIP Best Regards Ahmed Hamdy On Mon, 20 May 2024 at 15:40, Hong Liang wrote: > Hi Ahmed, > Thanks for putting this together! Should we still be marking > getFatalExceptionCons() as @Deprecated in this FLIP, if we are not > providing a replacement? > > Regards, > Hong > > On Mon, May 13, 2024 at 7:58 PM Ahmed Hamdy wrote: > > > Hi David, > > yes there error classification was initially left to sink implementers to > > handle while we provided utilities to classify[1] and bubble up[2] fatal > > exceptions to avoid retrying them. > > Additionally some sink implementations provide an option to short circuit > > the failures by exposing a `failOnError` flag as in > KinesisStreamsSink[3], > > however this FLIP scope doesn't include any changes for retry mechanisms. > > > > 1- > > > > > https://github.com/apache/flink/blob/015867803ff0c128b1c67064c41f37ca0731ed86/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/throwable/FatalExceptionClassifier.java#L32 > > 2- > > > > > https://github.com/apache/flink/blob/015867803ff0c128b1c67064c41f37ca0731ed86/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L533 > > 3- > > > > > https://github.com/apache/flink-connector-aws/blob/c6e0abb65a0e51b40dd218b890a111886fbf797f/flink-connector-aws/flink-connector-aws-kinesis-streams/src/main/java/org/apache/flink/connector/kinesis/sink/KinesisStreamsSinkWriter.java#L106 > > > > Best Regards > > Ahmed Hamdy > > > > > > On Mon, 13 May 2024 at 16:20, David Radley > > wrote: > > > > > Hi, > > > I wonder if the way that the async request fails could be a retriable > or > > > non-retriable error, so it would retry only for retriable (transient) > > > errors (like IOExceptions) . I see some talk on the internet around > > > retriable SQL errors. > > > If this was the case then we may need configuration to limit the > number > > > of retries of retriable errors. > > > Kind regards, David > > > > > > > > > From: Muhammet Orazov > > > Date: Monday, 13 May 2024 at 10:30 > > > To: dev@flink.apache.org > > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-451: Refactor Async sink API > > > Great, thanks for clarifying! > > > > > > Best, > > > Muhammet > > > > > > > > > On 2024-05-06 13:40, Ahmed Hamdy wrote: > > > > Hi Muhammet, > > > > Thanks for the feedback. > > > > > > > >> Could you please add more here why it is harder? Would the > > > >> `completeExceptionally` > > > >> method be related to it? Maybe you can add usage example for it > also. > > > >> > > > > > > > > this is mainly due to the current implementation of fatal exception > > > > failures which depends on base `getFatalExceptionConsumer` method > that > > > > is > > > > decoupled from the actual called method `submitRequestEntries`, Since > > > > this > > > > is now not the primary concern of the FLIP, I have removed it from > the > > > > motivation so that the scope is defined around introducing the > timeout > > > > configuration. > > > > > > > >> Should we add a list of possible connectors that this FLIP would > > > >> improve? > > > > > > > > Good call, I have added under migration plan. > > > > > > > > Best Regards > > > > Ahmed Hamdy > > > > > > > > > > > > On Mon, 6 May 2024 at 08:49, Muhammet Orazov > > > > wrote: > > > > > > > >> Hey Ahmed, > > > >> > > > >> Thanks for the FLIP! +1 (non-binding) > > > >> > > > >> > Additionally the current interface for passing fatal exceptions > and > > > >> > retrying records relies on java consumers which makes it harder to > > > >> > understand. > > > >> > > > >> Could you please add more here why it is harder? Would the > > > >> `completeExceptionally` > > > >> method be related to it? Maybe you can add usage example for it > also. > > > >> > > > >> > we should proceed by adding support in all supporting connector > > repos. > >
[jira] [Created] (FLINK-35401) Add SQS Table API support
Ahmed Hamdy created FLINK-35401: --- Summary: Add SQS Table API support Key: FLINK-35401 URL: https://issues.apache.org/jira/browse/FLINK-35401 Project: Flink Issue Type: New Feature Components: Connectors / AWS Reporter: Ahmed Hamdy This is an umbrella task for FLIP-438. FLIP-438: https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] Apache Flink CDC 3.1.0 released
Congratulations, Great Work! Best Regards Ahmed Hamdy On Fri, 17 May 2024 at 10:56, ConradJam wrote: > Congratulations! > > Thanks for the great work. > > Hang Ruan 于2024年5月17日周五 17:48写道： > > > Congratulations! > > > > Thanks for the great work. > > > > Best, > > Hang > > > > Qingsheng Ren 于2024年5月17日周五 17:33写道： > > > > > The Apache Flink community is very happy to announce the release of > > > Apache Flink CDC 3.1.0. > > > > > > Apache Flink CDC is a distributed data integration tool for real time > > > data and batch data, bringing the simplicity and elegance of data > > > integration via YAML to describe the data movement and transformation > > > in a data pipeline. > > > > > > Please check out the release blog post for an overview of the release: > > > > > > > > > https://flink.apache.org/2024/05/17/apache-flink-cdc-3.1.0-release-announcement/ > > > > > > The release is available for download at: > > > https://flink.apache.org/downloads.html > > > > > > Maven artifacts for Flink CDC can be found at: > > > https://search.maven.org/search?q=g:org.apache.flink%20cdc > > > > > > The full release notes are available in Jira: > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354387 > > > > > > We would like to thank all contributors of the Apache Flink community > > > who made this release possible! > > > > > > Regards, > > > Qingsheng Ren > > > > > > > > -- > Best > > ConradJam >
Re: [VOTE] FLIP-449: Reorganization of flink-connector-jdbc
Hi all, +1 (non-binding) Best Regards Ahmed Hamdy On Fri, 17 May 2024 at 02:13, Jiabao Sun wrote: > Thanks for driving this proposal! > > +1 (binding) > > Best, > Jiabao > > > On 2024/05/10 22:18:04 Jeyhun Karimov wrote: > > Thanks for driving this! > > > > +1 (non-binding) > > > > Regards, > > Jeyhun > > > > On Fri, May 10, 2024 at 12:50 PM Muhammet Orazov > > wrote: > > > > > Thanks João for your efforts and driving this! > > > > > > +1 (non-binding) > > > > > > Best, > > > Muhammet > > > > > > On 2024-05-09 12:01, Joao Boto wrote: > > > > Hi everyone, > > > > > > > > Thanks for all the feedback, I'd like to start a vote on the > FLIP-449: > > > > Reorganization of flink-connector-jdbc [1]. > > > > The discussion thread is here [2]. > > > > > > > > The vote will be open for at least 72 hours unless there is an > > > > objection or > > > > insufficient votes. > > > > > > > > [1] > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc > > > > [2] https://lists.apache.org/thread/jc1yvvo35xwqzlxl5mj77qw3hq6f5sgr > > > > > > > > Best > > > > Joao Boto > > > > > >
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Hi Lorenzo, yes it is pretty useful to start tackling the ones that still don't implement the new interface, happy so sync offline to close this task, additionally a volunteer to review the migrations would be highly appreciated. Best Regards Ahmed Hamdy On Wed, 15 May 2024 at 10:01, wrote: > Thank you Martijn for the detailed FLIP. > Will save this for the future as an example for these kind of FLIPs :) > > This also shakes the adoption of the new interface for downstream > connectors. > Volunteered for the pubsub port (table API), and trying to assess the > current status of pending PRs. > > I give you my +1 (non-bingind) :) > On May 13, 2024 at 11:52 +0200, Ahmed Hamdy , wrote: > > Thanks for the clarification. > > +1 to starting the vote. > > Best Regards > > Ahmed Hamdy > > > > > > On Mon, 13 May 2024 at 09:10, Martijn Visser > > wrote: > > > > > Hi Ahmed, > > > > > > There's no reason to refrain from releases for Flink 1.* versions: > these > > > connector implementations are still supported in the Flink 1.* series. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Sun, May 12, 2024 at 5:55 PM Ahmed Hamdy > wrote: > > > > > > > > Thanks Martijn > > > > > I believe you missed my question, > > > > > > > > > > Should this change take place in 1.20, what are the planned > release steps > > > > > > > for connectors that only offer a deprecated interface in this > case > > > (i.e. > > > > > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain > from > > > > > releases > > > > > > > till the blockers are implemented? > > > > > > > > > > > > > > > > > Could you please clarify? > > > > > > > > > > Best Regards > > > > > Ahmed Hamdy > > > > > > > > > > > > > > > On Sun, 12 May 2024 at 14:07, Martijn Visser < > martijnvis...@apache.org> > > > > > wrote: > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > If there are no more considerations, I'll open up a vote in > the next > > > > > couple > > > > > > > of days. > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Martijn > > > > > > > > > > > > > > On Wed, May 8, 2024 at 4:08 AM Hongshun Wang < > loserwang1...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Martijn, Thanks for the proposal +1 from me.Some sinks > still use > > > > > > > > > sinkfunction; it's time to take a step forward. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Hongshun > > > > > > > > > > > > > > > > > > On Mon, May 6, 2024 at 5:44 PM Leonard Xu < > xbjt...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > +1 from my side, thanks Martijn for the effort. > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > > > > > > > 2024年5月4日 下午7:41，Ahmed Hamdy > 写道： > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Martijn > > > > > > > > > > > > > Thanks for the proposal +1 from me. > > > > > > > > > > > > > Should this change take place in 1.20, what are > the planned > > > release > > > > > > > > > steps > > > > > > > > > > > > > for connectors that only offer a deprecated > interface in this > > > case > > > > > > > > > (i.e. > > > > > > > > > > > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going > to refrain from > > > > > > > > > > > releases > > > > > > > > > > > > > that support 1.20+ till the blockers are > implemented? >
Re: [DISCUSS] FLIP-XXX: Improve JDBC connector extensibility for Table API
Hi Lorenzo, This seems like a very useful addition. +1 (non-binding) from my side. I echo Jeyhun's question about backward compatibility as it is not mentioned in the FLIP. Best Regards Ahmed Hamdy On Wed, 15 May 2024 at 08:12, wrote: > Hello Muhammet and Jeyhun! > Thanks for your comments! > > @Jeyhun: > > > Could you please elaborate more on how the new approach will be backwards > compatible? > > In the FLIP I provide how the current Factories in JDBC would be changed > with this refactor, do you mean something different? Can you be more > specific with your request? > On May 14, 2024 at 12:32 +0200, Jeyhun Karimov , > wrote: > > Hi Lorenzo, > > > > Thanks for driving this FLIP. +1 for it. > > > > Could you please elaborate more on how the new approach will be backwards > > compatible? > > > > Regards, > > Jeyhun > > > > On Tue, May 14, 2024 at 10:00 AM Muhammet Orazov > > wrote: > > > > > Hey Lorenzo, > > > > > > Thanks for driving this FLIP! +1 > > > > > > It will improve the user experience of using JDBC based > > > connectors and help developers to build with different drivers. > > > > > > Best, > > > Muhammet > > > > > > On 2024-05-13 10:20, lorenzo.affe...@ververica.com.INVALID wrote: > > > > > Hello dev! > > > > > > > > > > I want to share a draft of my FLIP to refactor the JDBC connector > to > > > > > improve its extensibility [1]. > > > > > The goal is to allow implementers to write new connectors on top > of the > > > > > JDBC one for Table API with clean and maintainable code. > > > > > > > > > > Any feedback from the community is more and welcome. > > > > > > > > > > [1] > > > > > > > > > https://docs.google.com/document/d/1kl_AikMlqPUI-LNiPBraAFVZDRg1LF4bn6uiNtR4dlY/edit?usp=sharing > > > >
Re: [VOTE] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
+1 (non-binding). Thanks Martijn! Best Regards Ahmed Hamdy On Wed, 15 May 2024 at 14:11, Jing Ge wrote: > +1(binding) Thanks Martijn! > > Best regards, > Jing > > On Wed, May 15, 2024 at 7:00 PM Muhammet Orazov > wrote: > > > Thanks Martijn driving this! +1 (non-binding) > > > > Best, > > Muhammet > > > > On 2024-05-14 06:43, Martijn Visser wrote: > > > Hi everyone, > > > > > > With no more discussions being open in the thread [1] I would like to > > > start > > > a vote on FLIP-453: Promote Unified Sink API V2 to Public and Deprecate > > > SinkFunction [2] > > > > > > The vote will be open for at least 72 hours unless there is an > > > objection or > > > insufficient votes. > > > > > > Best regards, > > > > > > Martijn > > > > > > [1] https://lists.apache.org/thread/hod6bg421bzwhbfv60lwsck7r81dvo59 > > > [2] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-453%3A+Promote+Unified+Sink+API+V2+to+Public+and+Deprecate+SinkFunction > > >
Re: [VOTE] FLIP-451: Introduce timeout configuration to AsyncSink
Thanks David, I have replied to your question in the discussion thread. Best Regards Ahmed Hamdy On Mon, 13 May 2024 at 16:21, David Radley wrote: > Hi, > I raised a question on the discussion thread, around retriable errors, as > a possible alternative, > Kind regards, David. > > > From: Aleksandr Pilipenko > Date: Monday, 13 May 2024 at 16:07 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [VOTE] FLIP-451: Introduce timeout configuration > to AsyncSink > Thanks for driving this! > > +1 (non-binding) > > Thanks, > Aleksandr > > On Mon, 13 May 2024 at 14:08, > wrote: > > > Thanks Ahmed! > > > > +1 non binding > > On May 13, 2024 at 12:40 +0200, Jeyhun Karimov , > > wrote: > > > Thanks for driving this Ahmed. > > > > > > +1 (non-binding) > > > > > > Regards, > > > Jeyhun > > > > > > On Mon, May 13, 2024 at 12:37 PM Muhammet Orazov > > > wrote: > > > > > > > Thanks Ahmed, +1 (non-binding) > > > > > > > > Best, > > > > Muhammet > > > > > > > > On 2024-05-13 09:50, Ahmed Hamdy wrote: > > > > > > Hi all, > > > > > > > > > > > > Thanks for the feedback on the discussion thread[1], I would like > > to > > > > > > start > > > > > > a vote on FLIP-451[2]: Introduce timeout configuration to > AsyncSink > > > > > > > > > > > > The vote will be open for at least 72 hours unless there is an > > > > > > objection or > > > > > > insufficient votes. > > > > > > > > > > > > 1- > https://lists.apache.org/thread/ft7wcw7kyftvww25n5fm4l925tlgdfg0 > > > > > > 2- > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API > > > > > > Best Regards > > > > > > Ahmed Hamdy > > > > > > > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >
Re: [DISCUSS] FLIP-451: Refactor Async sink API
Hi David, yes there error classification was initially left to sink implementers to handle while we provided utilities to classify[1] and bubble up[2] fatal exceptions to avoid retrying them. Additionally some sink implementations provide an option to short circuit the failures by exposing a `failOnError` flag as in KinesisStreamsSink[3], however this FLIP scope doesn't include any changes for retry mechanisms. 1- https://github.com/apache/flink/blob/015867803ff0c128b1c67064c41f37ca0731ed86/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/throwable/FatalExceptionClassifier.java#L32 2- https://github.com/apache/flink/blob/015867803ff0c128b1c67064c41f37ca0731ed86/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/sink/writer/AsyncSinkWriter.java#L533 3- https://github.com/apache/flink-connector-aws/blob/c6e0abb65a0e51b40dd218b890a111886fbf797f/flink-connector-aws/flink-connector-aws-kinesis-streams/src/main/java/org/apache/flink/connector/kinesis/sink/KinesisStreamsSinkWriter.java#L106 Best Regards Ahmed Hamdy On Mon, 13 May 2024 at 16:20, David Radley wrote: > Hi, > I wonder if the way that the async request fails could be a retriable or > non-retriable error, so it would retry only for retriable (transient) > errors (like IOExceptions) . I see some talk on the internet around > retriable SQL errors. > If this was the case then we may need configuration to limit the number > of retries of retriable errors. > Kind regards, David > > > From: Muhammet Orazov > Date: Monday, 13 May 2024 at 10:30 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-451: Refactor Async sink API > Great, thanks for clarifying! > > Best, > Muhammet > > > On 2024-05-06 13:40, Ahmed Hamdy wrote: > > Hi Muhammet, > > Thanks for the feedback. > > > >> Could you please add more here why it is harder? Would the > >> `completeExceptionally` > >> method be related to it? Maybe you can add usage example for it also. > >> > > > > this is mainly due to the current implementation of fatal exception > > failures which depends on base `getFatalExceptionConsumer` method that > > is > > decoupled from the actual called method `submitRequestEntries`, Since > > this > > is now not the primary concern of the FLIP, I have removed it from the > > motivation so that the scope is defined around introducing the timeout > > configuration. > > > >> Should we add a list of possible connectors that this FLIP would > >> improve? > > > > Good call, I have added under migration plan. > > > > Best Regards > > Ahmed Hamdy > > > > > > On Mon, 6 May 2024 at 08:49, Muhammet Orazov > > wrote: > > > >> Hey Ahmed, > >> > >> Thanks for the FLIP! +1 (non-binding) > >> > >> > Additionally the current interface for passing fatal exceptions and > >> > retrying records relies on java consumers which makes it harder to > >> > understand. > >> > >> Could you please add more here why it is harder? Would the > >> `completeExceptionally` > >> method be related to it? Maybe you can add usage example for it also. > >> > >> > we should proceed by adding support in all supporting connector repos. > >> > >> Should we add list of possible connectors that this FLIP would > >> improve? > >> > >> Best, > >> Muhammet > >> > >> > >> On 2024-04-29 14:08, Ahmed Hamdy wrote: > >> > Hi all, > >> > I would like to start a discussion on FLIP-451[1] > >> > The proposal comes on encountering a couple of issues while working > >> > with > >> > implementers for Async Sink. > >> > The FLIP mainly proposes a new API similar to AsyncFunction and > >> > ResultFuture as well as introducing timeout handling for AsyncSink > >> > requests. > >> > The FLIP targets 1.20 with backward compatible changes and we should > >> > proceed by adding support in all supporting connector repos. > >> > > >> > 1- > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Refactor+Async+Sink+API > >> > Best Regards > >> > Ahmed Hamdy > >> > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Thanks for the clarification. +1 to starting the vote. Best Regards Ahmed Hamdy On Mon, 13 May 2024 at 09:10, Martijn Visser wrote: > Hi Ahmed, > > There's no reason to refrain from releases for Flink 1.* versions: these > connector implementations are still supported in the Flink 1.* series. > > Best regards, > > Martijn > > On Sun, May 12, 2024 at 5:55 PM Ahmed Hamdy wrote: > > > Thanks Martijn > > I believe you missed my question, > > > > Should this change take place in 1.20, what are the planned release steps > > > for connectors that only offer a deprecated interface in this case > (i.e. > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain from > > releases > > > till the blockers are implemented? > > > > > > > Could you please clarify? > > > > Best Regards > > Ahmed Hamdy > > > > > > On Sun, 12 May 2024 at 14:07, Martijn Visser > > wrote: > > > > > Hi all, > > > > > > If there are no more considerations, I'll open up a vote in the next > > couple > > > of days. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Wed, May 8, 2024 at 4:08 AM Hongshun Wang > > > wrote: > > > > > > > Hi Martijn, Thanks for the proposal +1 from me.Some sinks still use > > > > sinkfunction; it's time to take a step forward. > > > > > > > > Best, > > > > Hongshun > > > > > > > > On Mon, May 6, 2024 at 5:44 PM Leonard Xu wrote: > > > > > > > > > +1 from my side, thanks Martijn for the effort. > > > > > > > > > > Best, > > > > > Leonard > > > > > > > > > > > 2024年5月4日 下午7:41，Ahmed Hamdy 写道： > > > > > > > > > > > > Hi Martijn > > > > > > Thanks for the proposal +1 from me. > > > > > > Should this change take place in 1.20, what are the planned > release > > > > steps > > > > > > for connectors that only offer a deprecated interface in this > case > > > > (i.e. > > > > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain from > > > > > releases > > > > > > that support 1.20+ till the blockers are implemented? > > > > > > Best Regards > > > > > > Ahmed Hamdy > > > > > > > > > > > > > > > > > > On Fri, 3 May 2024 at 14:32, Péter Váry < > > peter.vary.apa...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > >>> With regards to FLINK-35149, the fix version indicates a change > > at > > > > > Flink > > > > > >> CDC; is that indeed correct, or does it require a change in the > > > SinkV2 > > > > > >> interface? > > > > > >> > > > > > >> The fix doesn't need change in SinkV2, so we are good there. > > > > > >> The issue is that the new SinkV2 > > > > > SupportsCommitter/SupportsPreWriteTopology > > > > > >> doesn't work with the CDC yet. > > > > > >> > > > > > >> Martijn Visser ezt írta (időpont: > > 2024. > > > > máj. > > > > > >> 3., > > > > > >> P, 14:06): > > > > > >> > > > > > >>> Hi Ferenc, > > > > > >>> > > > > > >>> You're right, 1.20 it is :) > > > > > >>> > > > > > >>> I've assigned the HBase one to you! > > > > > >>> > > > > > >>> Thanks, > > > > > >>> > > > > > >>> Martijn > > > > > >>> > > > > > >>> On Fri, May 3, 2024 at 1:55 PM Ferenc Csaky > > > > > > > > > > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Hi Martijn, > > > > > >>>> > > > > > >>>> +1 for the proposal. > > > > > >>>> > > > > > >>>>> targeted for Flink 1.19 > > > > > >>>> > > > > > >>>> I guess you meant Flink 1.20 here. > > > > > >>>>
[VOTE] FLIP-451: Introduce timeout configuration to AsyncSink
Hi all, Thanks for the feedback on the discussion thread[1], I would like to start a vote on FLIP-451[2]: Introduce timeout configuration to AsyncSink The vote will be open for at least 72 hours unless there is an objection or insufficient votes. 1-https://lists.apache.org/thread/ft7wcw7kyftvww25n5fm4l925tlgdfg0 2- https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Introduce+timeout+configuration+to+AsyncSink+API Best Regards Ahmed Hamdy
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Thanks Martijn I believe you missed my question, Should this change take place in 1.20, what are the planned release steps > for connectors that only offer a deprecated interface in this case (i.e. > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain from releases > till the blockers are implemented? > Could you please clarify? Best Regards Ahmed Hamdy On Sun, 12 May 2024 at 14:07, Martijn Visser wrote: > Hi all, > > If there are no more considerations, I'll open up a vote in the next couple > of days. > > Best regards, > > Martijn > > On Wed, May 8, 2024 at 4:08 AM Hongshun Wang > wrote: > > > Hi Martijn, Thanks for the proposal +1 from me.Some sinks still use > > sinkfunction; it's time to take a step forward. > > > > Best, > > Hongshun > > > > On Mon, May 6, 2024 at 5:44 PM Leonard Xu wrote: > > > > > +1 from my side, thanks Martijn for the effort. > > > > > > Best, > > > Leonard > > > > > > > 2024年5月4日 下午7:41，Ahmed Hamdy 写道： > > > > > > > > Hi Martijn > > > > Thanks for the proposal +1 from me. > > > > Should this change take place in 1.20, what are the planned release > > steps > > > > for connectors that only offer a deprecated interface in this case > > (i.e. > > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain from > > > releases > > > > that support 1.20+ till the blockers are implemented? > > > > Best Regards > > > > Ahmed Hamdy > > > > > > > > > > > > On Fri, 3 May 2024 at 14:32, Péter Váry > > > > wrote: > > > > > > > >>> With regards to FLINK-35149, the fix version indicates a change at > > > Flink > > > >> CDC; is that indeed correct, or does it require a change in the > SinkV2 > > > >> interface? > > > >> > > > >> The fix doesn't need change in SinkV2, so we are good there. > > > >> The issue is that the new SinkV2 > > > SupportsCommitter/SupportsPreWriteTopology > > > >> doesn't work with the CDC yet. > > > >> > > > >> Martijn Visser ezt írta (időpont: 2024. > > máj. > > > >> 3., > > > >> P, 14:06): > > > >> > > > >>> Hi Ferenc, > > > >>> > > > >>> You're right, 1.20 it is :) > > > >>> > > > >>> I've assigned the HBase one to you! > > > >>> > > > >>> Thanks, > > > >>> > > > >>> Martijn > > > >>> > > > >>> On Fri, May 3, 2024 at 1:55 PM Ferenc Csaky > > > > > > > > >>> wrote: > > > >>> > > > >>>> Hi Martijn, > > > >>>> > > > >>>> +1 for the proposal. > > > >>>> > > > >>>>> targeted for Flink 1.19 > > > >>>> > > > >>>> I guess you meant Flink 1.20 here. > > > >>>> > > > >>>> Also, I volunteer to take updating the HBase sink, feel free to > > assign > > > >>>> that task to me. > > > >>>> > > > >>>> Best, > > > >>>> Ferenc > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Friday, May 3rd, 2024 at 10:20, Martijn Visser < > > > >>>> martijnvis...@apache.org> wrote: > > > >>>> > > > >>>>> > > > >>>>> > > > >>>>> Hi Peter, > > > >>>>> > > > >>>>> I'll add it for completeness, thanks! > > > >>>>> With regards to FLINK-35149, the fix version indicates a change > at > > > >>> Flink > > > >>>>> CDC; is that indeed correct, or does it require a change in the > > > >> SinkV2 > > > >>>>> interface? > > > >>>>> > > > >>>>> Best regards, > > > >>>>> > > > >>>>> Martijn > > > >>>>> > > > >>>>> > > > >>>>> On Fri, May 3, 2024 at 7:47 AM Péter Váry > > > >> peter.vary.apa...@gmail.com > > > >>>>> > > > >&g
Re: Re: [VOTE] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table
+1 (non-binding) Best Regards Ahmed Hamdy On Sat, 11 May 2024 at 16:26, Keith Lee wrote: > +1 (non-binding) > > Best regards > Keith Lee > > > On Sat, 11 May 2024 at 04:58, Shengkai Fang wrote: > > > +1 (binding) > > > > Best, > > Shengkai > > > > Ron Liu 于2024年5月10日周五 12:07写道： > > > > > +1(binding) > > > > > > Best, > > > Ron > > > > > > Jark Wu 于2024年5月10日周五 09:51写道： > > > > > > > +1 (binding) > > > > > > > > Best, > > > > Jark > > > > > > > > On Thu, 9 May 2024 at 21:27, Lincoln Lee > > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > Best, > > > > > Lincoln Lee > > > > > > > > > > > > > > > Feng Jin 于2024年5月9日周四 19:45写道： > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > Best, > > > > > > Feng > > > > > > > > > > > > > > > > > > On Thu, May 9, 2024 at 7:37 PM Xuyang > wrote: > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Best！ > > > > > > > Xuyang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2024-05-09 13:57:07, "Ron Liu" wrote: > > > > > > > >Sorry for the re-post, just to format this email content. > > > > > > > > > > > > > > > >Hi Dev > > > > > > > > > > > > > > > >Thank you to everyone for the feedback on FLIP-448: Introduce > > > > > Pluggable > > > > > > > >Workflow Scheduler Interface for Materialized Table[1][2]. > > > > > > > >I'd like to start a vote for it. The vote will be open for at > > > least > > > > 72 > > > > > > > >hours unless there is an objection or not enough votes. > > > > > > > > > > > > > > > >[1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-448%3A+Introduce+Pluggable+Workflow+Scheduler+Interface+for+Materialized+Table > > > > > > > > > > > > > > > >[2] > > > > https://lists.apache.org/thread/57xfo6p25rbrhcg01dhyok46zt6jc5q1 > > > > > > > > > > > > > > > >Best, > > > > > > > >Ron > > > > > > > > > > > > > > > >Ron Liu 于2024年5月9日周四 13:52写道： > > > > > > > > > > > > > > > >> Hi Dev, Thank you to everyone for the feedback on FLIP-448: > > > > > Introduce > > > > > > > >> Pluggable Workflow Scheduler Interface for Materialized > > > > Table[1][2]. > > > > > > I'd > > > > > > > >> like to start a vote for it. The vote will be open for at > > least > > > 72 > > > > > > hours > > > > > > > >> unless there is an objection or not enough votes. [1] > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-448%3A+Introduce+Pluggable+Workflow+Scheduler+Interface+for+Materialized+Table > > > > > > > >> > > > > > > > >> [2] > > > > > https://lists.apache.org/thread/57xfo6p25rbrhcg01dhyok46zt6jc5q1 > > > > > > > >> Best, Ron > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
[jira] [Created] (FLINK-35322) PubSub Connector Weekly build fails
Ahmed Hamdy created FLINK-35322: --- Summary: PubSub Connector Weekly build fails Key: FLINK-35322 URL: https://issues.apache.org/jira/browse/FLINK-35322 Project: Flink Issue Type: Bug Components: Connectors / Google Cloud PubSub Affects Versions: 3.1.0 Reporter: Ahmed Hamdy Weekly builds for GCP pubSub connector is failing for 1.19 due to compilation error in tests. https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8768752932/job/24063472769 https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8863605354 https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8954270618 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0
Thanks for the effort and the data provided. +1 (non-binding) Best Regards Ahmed Hamdy On Thu, 9 May 2024 at 08:20, Keith Lee wrote: > +1 (non-binding) > > Great work on the benchmark data. > > Keith > > On Thu, May 9, 2024 at 4:53 AM 周仁祥 wrote: > > > +1 (non-binding) > > > > ConradJam 于2024年5月7日周二 11:35写道： > > > > > +1 (no-binding) > > > > > > Piotr Nowojski 于2024年5月6日周一 20:17写道： > > > > > > > +1 (binding) > > > > > > > > Piotrek > > > > > > > > pon., 6 maj 2024 o 12:35 Roman Khachatryan > > > napisał(a): > > > > > > > > > +1 (binding) > > > > > > > > > > Regards, > > > > > Roman > > > > > > > > > > > > > > > On Mon, May 6, 2024 at 11:56 AM gongzhongqiang < > > > > gongzhongqi...@apache.org> > > > > > wrote: > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > Best, > > > > > > Zhongqiang Gong > > > > > > > > > > > > yue ma 于2024年5月6日周一 10:54写道： > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > Thanks for all the feedback, I'd like to start a vote on the > > > > FLIP-447: > > > > > > > Upgrade FRocksDB from 6.20.3 to 8.10.0 [1]. The discussion > thread > > > is > > > > > here > > > > > > > [2]. > > > > > > > > > > > > > > The vote will be open for at least 72 hours unless there is an > > > > > objection > > > > > > or > > > > > > > insufficient votes. > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-447%3A+Upgrade+FRocksDB+from+6.20.3++to+8.10.0 > > > > > > > [2] > > > https://lists.apache.org/thread/lrxjfpjjwlq4sjzm1oolx58n1n8r48hw > > > > > > > > > > > > > > -- > > > > > > > Best, > > > > > > > Yue > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best > > > > > > ConradJam > > > > > > > > > -- > > Best, > > renxiang > > >
[jira] [Created] (FLINK-35320) RabbitMQ Source fails to consume from quorum queues when prefetch Count is set
Ahmed Hamdy created FLINK-35320:
---
Summary: RabbitMQ Source fails to consume from quorum queues when
prefetch Count is set
Key: FLINK-35320
URL: https://issues.apache.org/jira/browse/FLINK-35320
Project: Flink
Issue Type: Bug
Components: Connectors/ RabbitMQ
Affects Versions: 1.16.3, 1.16.2, 1.16.0
Reporter: Ahmed Hamdy
Fix For: 3.1.0
h2. Description
{{RMQSource}} currently sets prefetch Count with [global QoS
|https://github.com/apache/flink-connector-rabbitmq/blob/66e323a3e79befc08ae03f2789a8aa94b343d504/flink-connector-rabbitmq/src/main/java/org/apache/flink/streaming/connectors/rabbitmq/RMQSource.java#L223]
which is incompatible with [Quorum
queues|https://www.rabbitmq.com/docs/quorum-queues#global-qos].
h2. Consideration
Currently the {{RMQSource}} implements {{SourceFunction}} which is deprecated
from 1.18, the current RabbitMQ connector is compatible with 1.16 which is out
of support, another approach would be migrating the source to the new API for
the next connector release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [RESULT][VOTE] FLIP-454: New Apicurio Avro format
Hi David, thanks for the update We should wait to reach threshold of "binding" votes as per the process[1] 1- https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals Best Regards Ahmed Hamdy On Wed, 8 May 2024 at 16:06, David Radley wrote: > Hi everyone, > I am happy to say that FLIP-454: New Apicurio Avro format [1] has been > accepted and voted through this thread [2]. > > The proposal has been accepted with 4 approving votes and there > are no vetos: > > - Ahmed Hamdy (non-binding) > - Jeyhun Karimov (non-binding) > - Mark Nuttall (non-binding) > - Nic Townsend (non-binding) > > Martijn: > Please could you update the Flip with: > - the voting thread link > - the accepted status > - the Jira number (https://issues.apache.org/jira/browse/FLINK-35311). > As the involved committer, are you willing to assign me the Jira to work > on and merge once you approve the changes? > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format > [2] https://lists.apache.org/list?dev@flink.apache.org:lte=1M:apicurio > > Thanks to all involved. > > Kind regards, > David > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU >
Re: [DISCUSS] FLIP-444: Native file copy support
Hi Piotr +1 for the proposal, it seems to have a lot of gains. Best Regards Ahmed Hamdy On Mon, 6 May 2024 at 12:06, Zakelly Lan wrote: > Hi Piotrek, > > Thanks for your answers! > > Good question. The intention and use case behind `DuplicatingFileSystem` is > > different. It marks if `FileSystem` can quickly copy/duplicate files > > in the remote `FileSystem`. For example an equivalent of a hard link or > > bumping a reference count in the remote system. That's a bit different > > to copy paths between remote and local file systems. > > > > However, it could arguably be unified under one interface where we would > > re-use or re-name `canFastDuplicate(Path, Path)` to > > `canFastCopy(Path, Path)` with the following use cases: > > - `canFastCopy(remoteA, remoteB)` returns true - current equivalent of > > `DuplicatingFileSystem` - quickly duplicate/hard link remote path > > - `canFastCopy(local, remote)` returns true - FS can natively upload > local > > file to a remote location > > - `canFastCopy(remote, local)` returns true - FS can natively download > > local file from a remote location > > > > Maybe indeed that's a better solution vs having two separate interfaces > for > > copying and duplicating? > > > > I'd prefer a unified one interface, `canFastCopy(Path, Path)` looks good to > me. This also resolves my question 1 about the destination. > > > Best, > Zakelly > > On Mon, May 6, 2024 at 6:36 PM Piotr Nowojski > wrote: > > > Hi All! > > > > Thanks for your comments. > > > > Muhammet and Hong, about the config options. > > > > > Could you please also add the configuration property for this? An > example > > showing how users would set this parameter would be helpful. > > > > > 1/ Configure the implementation of PathsCopyingFileSystem used > > > 2/ Configure the location of the s5cmd binary (version control etc.) > > > > Ops, sorry I added the config options that I had in mind to the FLIP. I > > don't know why I have omitted this. Basically I suggest that in order to > > use native file copying: > > 1. `FileSystem` must support it via implementing `PathsCopyingFileSystem` > > interface > > 2. That `FileSystem` would have to be configured to actually use it. For > > example S3 file system would return `true` that it can copy paths > > only if `s3.s5cmd.path` has been specified. > > > > > Would this affect any filesystem connectors that use FileSystem[1][2] > > dependencies? > > > > Definitely not out of the box. Any place in Flink that is currently > > uploading/downloading files from a FileSystem could use this feature, but > > it > > would have to be implemented. The same way this FLIP will implement > native > > files copying when downloading state during recovery, > > but the old code path will be still used for uploading state files > during a > > checkpoint. > > > > > How adding a s5cmd will affect memory footprint? Since this is a native > > binary, memory consumption will not be controlled by JVM or Flink. > > > > As you mentioned the memory usage of `s5cmd` will not be controlled, so > the > > memory footprint will grow. S5cmd integration with Flink > > has been tested quite extensively on our production environment already, > > and we haven't observed any issues so far despite the fact we > > are using quite small pods. But of course if your setup is working on the > > edge of OOM, this could tip you over that edge. > > > > Zakelly: > > > > > 1. What is the semantic of `canCopyPath`? Should it be associated with > a > > > specific destination path? e.g. It can be copied to local, but not to > the > > > remote FS. > > > > For the S3 (both for SDKv2 and s5cmd implementations), the copying > > direction (upload/download) doesn't matter. I don't know about other > > file systems, I haven't investigated anything besides S3. Nevertheless I > > wouldn't worry too much about it, since we can start with the simple > > `canCopyPath` that handles both directions. If this will become important > > in the future, adding directional `canDownloadPath` or `canUploadPath` > > would be a backward compatible change, so we can safely extend it in the > > future if needed. > > > > > 2. Is the existing interface `DuplicatingFileSystem` feasible/enough > for > > this case? > > > > Good question. The intention and use case behind `DuplicatingFileSystem` > is > > different. It marks if `FileSystem` can quickly copy/duplicate fil
Re: [DISCUSS] Flink CDC 3.2 Release Planning
Thanks Qinsheng for driving, +1 for the feature freeze date. I am happy to assist with any release duties. Best Regards Ahmed Hamdy On Wed, 8 May 2024 at 10:50, Jiabao Sun wrote: > Thanks Qingsheng, > > Improving stability is crucial for Flink CDC, looking forward to this > release. > If assistance is needed, I am happy to help with it. > > Best, > Jiabao > > Qingsheng Ren 于2024年5月8日周三 14:22写道： > > > Hi devs, > > > > As we are in the midst of the release voting process for Flink CDC > 3.1.0, I > > think it's a good time to kick off the upcoming Flink CDC 3.2 release > > cycle. > > > > In this release cycle I would like to focus on the stability of Flink > CDC, > > especially for the newly introduced YAML-based data integration > > framework. To ensure we can iterate and improve swiftly, I propose to > make > > 3.2 a relatively short release cycle, targeting a feature freeze by May > 24, > > 2024. > > > > For developers that are interested in participating and contributing new > > features in this release cycle, please feel free to list your planning > > features in the wiki page [1]. > > > > I'm happy to volunteer as a release manager and of course open to work > > together with someone on this. > > > > What do you think? > > > > Best, > > Qingsheng > > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/Flink+CDC+3.2+Release > > >
Re: [DISCUSS] FLIP-451: Refactor Async sink API
Hi Muhammet, Thanks for the feedback. > Could you please add more here why it is harder? Would the > `completeExceptionally` > method be related to it? Maybe you can add usage example for it also. > this is mainly due to the current implementation of fatal exception failures which depends on base `getFatalExceptionConsumer` method that is decoupled from the actual called method `submitRequestEntries`, Since this is now not the primary concern of the FLIP, I have removed it from the motivation so that the scope is defined around introducing the timeout configuration. > Should we add a list of possible connectors that this FLIP would improve? Good call, I have added under migration plan. Best Regards Ahmed Hamdy On Mon, 6 May 2024 at 08:49, Muhammet Orazov wrote: > Hey Ahmed, > > Thanks for the FLIP! +1 (non-binding) > > > Additionally the current interface for passing fatal exceptions and > > retrying records relies on java consumers which makes it harder to > > understand. > > Could you please add more here why it is harder? Would the > `completeExceptionally` > method be related to it? Maybe you can add usage example for it also. > > > we should proceed by adding support in all supporting connector repos. > > Should we add list of possible connectors that this FLIP would improve? > > Best, > Muhammet > > > On 2024-04-29 14:08, Ahmed Hamdy wrote: > > Hi all, > > I would like to start a discussion on FLIP-451[1] > > The proposal comes on encountering a couple of issues while working > > with > > implementers for Async Sink. > > The FLIP mainly proposes a new API similar to AsyncFunction and > > ResultFuture as well as introducing timeout handling for AsyncSink > > requests. > > The FLIP targets 1.20 with backward compatible changes and we should > > proceed by adding support in all supporting connector repos. > > > > 1- > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Refactor+Async+Sink+API > > Best Regards > > Ahmed Hamdy >
Re: [DISCUSS] FLIP-451: Refactor Async sink API
Hi Jeyhun, I have changed the scope of FLIP to exclude problems addressed by FLIP-284 and redefined scope to introduce timeout configuration. Best Regards Ahmed Hamdy On Mon, 29 Apr 2024 at 20:57, Ahmed Hamdy wrote: > Hi Jeyhun, > Thanks for your feedback. I agree the phrasing is a bit confusing, the > main scope for FLIP-451 is limited to introducing timeout configuration and > the new "ResultHandler" to Async Sink API. > I will remove reference to FLIP-284 from the FLIP to disambiguate. > Best Regards > Ahmed Hamdy > > > On Mon, 29 Apr 2024 at 17:59, Jeyhun Karimov wrote: > >> Hi Ahmed, >> >> Thanks a lot for the FLIP. +1 for it. >> My main concern is that the boundary/scope of the two FLIPs (451 and 284) >> and their differentiation/overlap is unclear for me from the FLIP >> document. >> Could you please elaborate more on this? >> >> Regards, >> Jeyhun >> >> >> On Mon, Apr 29, 2024 at 4:13 PM Ahmed Hamdy wrote: >> >> > Hi all, >> > I would like to start a discussion on FLIP-451[1] >> > The proposal comes on encountering a couple of issues while working with >> > implementers for Async Sink. >> > The FLIP mainly proposes a new API similar to AsyncFunction and >> > ResultFuture as well as introducing timeout handling for AsyncSink >> > requests. >> > The FLIP targets 1.20 with backward compatible changes and we should >> > proceed by adding support in all supporting connector repos. >> > >> > 1- >> > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Refactor+Async+Sink+API >> > Best Regards >> > Ahmed Hamdy >> > >> >
Re: [Vote] FLIP-438: Amazon SQS Sink Connector
Hi Priya, It seems that we have acquired the threshold of binding votes, Should we close the vote, announce the result and create Jiras? Best Regards Ahmed Hamdy On Wed, 17 Apr 2024 at 03:24, Leonard Xu wrote: > +1 (binding) > > Best, > Leonard > > > 2024年4月17日 上午2:25，Robert Metzger 写道： > > > > +1 binding > > > > On Tue, Apr 16, 2024 at 2:05 PM Jeyhun Karimov > wrote: > > > >> Thanks Priya for driving the FLIP. > >> > >> +1 (non-binding) > >> > >> Regards, > >> Jeyhun > >> > >> On Tue, Apr 16, 2024 at 12:37 PM Hong Liang wrote: > >> > >>> +1 (binding) > >>> > >>> Thanks Priya for driving this! This has been a requested feature for a > >>> while now, and will benefit the community :) > >>> > >>> Hong > >>> > >>> On Tue, Apr 16, 2024 at 3:23 AM Muhammet Orazov > >>> wrote: > >>> > >>>> +1 (non-binding) > >>>> > >>>> Thanks Priya for the FLIP and driving it! > >>>> > >>>> Best, > >>>> Muhammet > >>>> > >>>> On 2024-04-12 21:56, Dhingra, Priya wrote: > >>>>> Hi devs, > >>>>> > >>>>> > >>>>> > >>>>> Thank you to everyone for the feedback on FLIP-438: Amazon SQS Sink > >>>>> Connector< > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> I would like to start a vote for it. The vote will be open for at > >> least > >>>>> 72 > >>>>> > >>>>> hours unless there is an objection or not enough votes. > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector > >>>>> > >>>>> Regards > >>>>> Priya > >>>> > >>> > >> > >
Re: [VOTE] FLIP-436: Introduce Catalog-related Syntax
+1 (non-binding)
Best Regards
Ahmed Hamdy
On Sat, 27 Apr 2024 at 02:46, Muhammet Orazov
wrote:
>
> Hey Yubin, +1 (non-binding)
>
> Looks good, thanks!
>
> Best,
> Muhammet
>
> On 2024-04-24 08:56, Yubin Li wrote:
> > Hi everyone,
> >
> > During the implementation of the "describe catalog" syntax, it was
> > found that the original output style needed to be improved.
> > ```
> > desc catalog extended cat2;
> >
> +--+-+
> > | catalog_description_item |
> > catalog_description_value |
> >
> +--+-+
> > | Name |
> > cat2 |
> > | Type |
> > generic_in_memory |
> > | Comment |
> > |
> > | Properties | ('default-database','db'),
> > ('type','generic_in_memory') |
> >
> +--+-+
> > 4 rows in set
> > ```
> > After offline discussions with Jane Chan and Jark Wu, we suggest
> > improving it to the following form:
> > ```
> > desc catalog extended cat2;
> > +-+---+
> > | info name |info value |
> > +-+---+
> > |name | cat2 |
> > |type | generic_in_memory |
> > | comment | |
> > | option:default-database |db |
> > +-+---+
> > 4 rows in set
> > ```
> >
> > For the following reasons:
> > 1. The title should be consistent with engines such as Databricks for
> > easy understanding, and it should also be consistent with Flink's own
> > naming style. Therefore, the title adopts "info name", "info value",
> > and the key name should be unified in lowercase, so "Name" is replaced
> > by "name".
> > Note: Databricks output style [1] as follows:
> > ```
> >> DESCRIBE CATALOG main;
> > info_name info_value
> >
> > Catalog Name main
> > Comment Main catalog (auto-created)
> > Owner metastore-admin-users
> > Catalog Type Regular
> > ```
> > 2. There may be many attributes of the catalog, and it is very poor in
> > readability when displayed in one line. It should be expanded into
> > multiple lines, and the key name is prefixed with "option:" to
> > identify that this is an attribute row. And since `type` is an
> > important information of the catalog, even if `extended` is not
> > specified, it should also be displayed, and correspondingly,
> > "option:type" should be removed to avoid redundancy.
> >
> > WDYT? Looking forward to your reply!
> >
> > [1]
> >
> https://learn.microsoft.com/zh-tw/azure/databricks/sql/language-manual/sql-ref-syntax-aux-describe-catalog
> >
> > Best,
> > Yubin
> >
> > On Wed, Mar 20, 2024 at 2:15 PM Benchao Li
> > wrote:
> >>
> >> +1 (binding)
> >>
> >> gongzhongqiang 于2024年3月20日周三 11:40写道：
> >> >
> >> > +1 (non-binding)
> >> >
> >> > Best,
> >> > Zhongqiang Gong
> >> >
> >> > Yubin Li 于2024年3月19日周二 18:03写道：
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > Thanks for all the feedback, I'd like to start a vote on the
> FLIP-436:
> >> > > Introduce Catalog-related Syntax [1]. The discussion thread is here
> >> > > [2].
> >> > >
> >> > > The vote will be open for at least 72 hours unless there is an
> >> > > objection or insufficient votes.
> >> > >
> >> > > [1]
> >> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-436%3A+Introduce+Catalog-related+Syntax
> >> > > [2]
> https://lists.apache.org/thread/10k1bjb4sngyjwhmfqfky28lyoo7sv0z
> >> > >
> >> > > Best regards,
> >> > > Yubin
> >> > >
> >>
> >>
> >>
> >> --
> >>
> >> Best,
> >> Benchao Li
>
Re: [VOTE] FLIP-446: Kubernetes Operator State Snapshot CRD
+1 (non-binding) Best Regards Ahmed Hamdy On Sat, 27 Apr 2024 at 05:33, ConradJam wrote: > +1 (non-binding) > > Biao Geng 于2024年4月26日周五 18:45写道： > > > +1 (non-binding) > > > > Best, > > Biao Geng > > > > > > gongzhongqiang 于2024年4月26日周五 00:41写道： > > > > > +1( (non-binding) > > > > > > > > > Best, > > > Zhongqiang Gong > > > > > > Mate Czagany 于2024年4月24日周三 16:06写道： > > > > > > > Hi everyone, > > > > > > > > I'd like to start a vote on the FLIP-446: Kubernetes Operator State > > > > Snapshot CRD [1]. The discussion thread is here [2]. > > > > > > > > The vote will be open for at least 72 hours unless there is an > > objection > > > or > > > > insufficient votes. > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-446%3A+Kubernetes+Operator+State+Snapshot+CRD > > > > [2] https://lists.apache.org/thread/q5dzjwj0qk34rbg2sczyypfhokxoc3q7 > > > > > > > > Regards, > > > > Mate > > > > > > > > > > > > -- > Best > > ConradJam >
Re: [VOTE] FLIP-454: New Apicurio Avro format
+1 (non-binding) Best Regards Ahmed Hamdy On Fri, 3 May 2024 at 15:16, Jeyhun Karimov wrote: > +1 (non binding) > > Thanks for driving this FLIP David. > > Regards, > Jeyhun > > On Fri, May 3, 2024 at 2:21 PM Mark Nuttall wrote: > > > +1, I would also like to see first class support for Avro and Apicurio > > > > -- Mark Nuttall, mnutt...@apache.org > > Senior Software Engineer, IBM Event Automation > > > > On 2024/05/02 09:41:09 David Radley wrote: > > > Hi everyone, > > > > > > I'd like to start a vote on the FLIP-454: New Apicurio Avro format > > > [1]. The discussion thread is here [2]. > > > > > > The vote will be open for at least 72 hours unless there is an > > > objection > > > or > > > insufficient votes. > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-454%3A+New+Apicurio+Avro+format > > > [2] https://lists.apache.org/thread/wtkl4yn847tdd0wrqm5xgv9wc0cb0kr8 > > > > > > > > > Kind regards, David. > > > > > > Unless otherwise stated above: > > > > > > IBM United Kingdom Limited > > > Registered in England and Wales with number 741598 > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU > > > > > >
Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #1
Hi Qisheng, +1 (non-binding) - Verified checksums and hashes - Verified signatures - Verified github tag exists - Verified no binaries in source - build source Best Regards Ahmed Hamdy On Fri, 3 May 2024 at 23:03, Jeyhun Karimov wrote: > Hi Qinsheng, > > Thanks for driving the release. > +1 (non-binding) > > - No binaries in source > - Verified Signatures > - Github tag exists > - Build source > > Regards, > Jeyhun > > On Thu, May 2, 2024 at 10:52 PM Muhammet Orazov > wrote: > > > Hey Qingsheng, > > > > Thanks a lot! +1 (non-binding) > > > > - Checked sha512sum hash > > - Checked GPG signature > > - Reviewed release notes > > - Reviewed GitHub web pr (added minor suggestions) > > - Built the source with JDK 11 & 8 > > - Checked that src doesn't contain binary files > > > > Best, > > Muhammet > > > > On 2024-04-30 05:11, Qingsheng Ren wrote: > > > Hi everyone, > > > > > > Please review and vote on the release candidate #1 for the version > > > 3.1.0 of > > > Apache Flink CDC, as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > **Release Overview** > > > > > > As an overview, the release consists of the following: > > > a) Flink CDC source release to be deployed to dist.apache.org > > > b) Maven artifacts to be deployed to the Maven Central Repository > > > > > > **Staging Areas to Review** > > > > > > The staging areas containing the above mentioned artifacts are as > > > follows, > > > for your review: > > > * All artifacts for a) can be found in the corresponding dev repository > > > at > > > dist.apache.org [1], which are signed with the key with fingerprint > > > A1BD477F79D036D2C30CA7DBCA8AEEC2F6EB040B [2] > > > * All artifacts for b) can be found at the Apache Nexus Repository [3] > > > > > > Other links for your review: > > > * JIRA release notes [4] > > > * Source code tag "release-3.1.0-rc1" with commit hash > > > 63b42cb937d481f558209ab3c8547959cf039643 [5] > > > * PR for release announcement blog post of Flink CDC 3.1.0 in flink-web > > > [6] > > > > > > **Vote Duration** > > > > > > The voting time will run for at least 72 hours, adopted by majority > > > approval with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Qingsheng Ren > > > > > > [1] https://dist.apache.org/repos/dist/dev/flink/flink-cdc-3.1.0-rc1/ > > > [2] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [3] > > > https://repository.apache.org/content/repositories/orgapacheflink-1731 > > > [4] > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354387 > > > [5] https://github.com/apache/flink-cdc/releases/tag/release-3.1.0-rc1 > > > [6] https://github.com/apache/flink-web/pull/739 > > >
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Hi Martijn Thanks for the proposal +1 from me. Should this change take place in 1.20, what are the planned release steps for connectors that only offer a deprecated interface in this case (i.e. RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain from releases that support 1.20+ till the blockers are implemented? Best Regards Ahmed Hamdy On Fri, 3 May 2024 at 14:32, Péter Váry wrote: > > With regards to FLINK-35149, the fix version indicates a change at Flink > CDC; is that indeed correct, or does it require a change in the SinkV2 > interface? > > The fix doesn't need change in SinkV2, so we are good there. > The issue is that the new SinkV2 SupportsCommitter/SupportsPreWriteTopology > doesn't work with the CDC yet. > > Martijn Visser ezt írta (időpont: 2024. máj. > 3., > P, 14:06): > > > Hi Ferenc, > > > > You're right, 1.20 it is :) > > > > I've assigned the HBase one to you! > > > > Thanks, > > > > Martijn > > > > On Fri, May 3, 2024 at 1:55 PM Ferenc Csaky > > wrote: > > > > > Hi Martijn, > > > > > > +1 for the proposal. > > > > > > > targeted for Flink 1.19 > > > > > > I guess you meant Flink 1.20 here. > > > > > > Also, I volunteer to take updating the HBase sink, feel free to assign > > > that task to me. > > > > > > Best, > > > Ferenc > > > > > > > > > > > > > > > On Friday, May 3rd, 2024 at 10:20, Martijn Visser < > > > martijnvis...@apache.org> wrote: > > > > > > > > > > > > > > > Hi Peter, > > > > > > > > I'll add it for completeness, thanks! > > > > With regards to FLINK-35149, the fix version indicates a change at > > Flink > > > > CDC; is that indeed correct, or does it require a change in the > SinkV2 > > > > interface? > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > > > > > On Fri, May 3, 2024 at 7:47 AM Péter Váry > peter.vary.apa...@gmail.com > > > > > > > > wrote: > > > > > > > > > Hi Martijn, > > > > > > > > > > We might want to add FLIP-371 [1] to the list. (Or we aim only for > > > higher > > > > > level FLIPs?) > > > > > > > > > > We are in the process of using the new API in Iceberg connector > [2] - > > > so > > > > > far, so good. > > > > > > > > > > I know of one minor known issue about the sink [3], which should be > > > ready > > > > > for the release. > > > > > > > > > > All-in-all, I think we are in good shape, and we could move forward > > > with > > > > > the promotion. > > > > > > > > > > Thanks, > > > > > Peter > > > > > > > > > > [1] - > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=263430387 > > > > > [2] - https://github.com/apache/iceberg/pull/10179 > > > > > [3] - https://issues.apache.org/jira/browse/FLINK-35149 > > > > > > > > > > On Thu, May 2, 2024, 09:47 Muhammet Orazov > > > mor+fl...@morazow.com.invalid > > > > > wrote: > > > > > > > > > > > Got it, thanks! > > > > > > > > > > > > On 2024-05-02 06:53, Martijn Visser wrote: > > > > > > > > > > > > > Hi Muhammet, > > > > > > > > > > > > > > Thanks for joining the discussion! The changes in this FLIP > would > > > be > > > > > > > targeted for Flink 1.19, since it's only a matter of changing > the > > > > > > > annotation. > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Martijn > > > > > > > > > > > > > > On Thu, May 2, 2024 at 7:26 AM Muhammet Orazov > > > mor+fl...@morazow.com > > > > > > > wrote: > > > > > > > > > > > > > > > Hello Martijn, > > > > > > > > > > > > > > > > Thanks for the FLIP and detailed history of changes, +1. > > > > > > > > > > > > > > > > Would FLIP changes target for 2.0? I think it would be good > > > > > > > > to have clear APIs on 2.0 release. > > > > > > > > > > > > > > > > Best, > > > > > > > > Muhammet > > > > > > > > > > > > > > > > On 2024-05-01 15:30, Martijn Visser wrote: > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > I would like to start a discussion on FLIP-453: Promote > > > Unified Sink > > > > > > > > > API V2 > > > > > > > > > to Public and Deprecate SinkFunction > > > > > > > > > https://cwiki.apache.org/confluence/x/rIobEg > > > > > > > > > > > > > > > > > > This FLIP proposes to promote the Unified Sink API V2 from > > > > > > > > > PublicEvolving > > > > > > > > > to Public and to mark the SinkFunction as Deprecated. > > > > > > > > > > > > > > > > > > I'm looking forward to your thoughts. > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > Martijn > > > > > >
Re: [DISCUSS] FLIP-451: Refactor Async sink API
Hi Jeyhun, Thanks for your feedback. I agree the phrasing is a bit confusing, the main scope for FLIP-451 is limited to introducing timeout configuration and the new "ResultHandler" to Async Sink API. I will remove reference to FLIP-284 from the FLIP to disambiguate. Best Regards Ahmed Hamdy On Mon, 29 Apr 2024 at 17:59, Jeyhun Karimov wrote: > Hi Ahmed, > > Thanks a lot for the FLIP. +1 for it. > My main concern is that the boundary/scope of the two FLIPs (451 and 284) > and their differentiation/overlap is unclear for me from the FLIP document. > Could you please elaborate more on this? > > Regards, > Jeyhun > > > On Mon, Apr 29, 2024 at 4:13 PM Ahmed Hamdy wrote: > > > Hi all, > > I would like to start a discussion on FLIP-451[1] > > The proposal comes on encountering a couple of issues while working with > > implementers for Async Sink. > > The FLIP mainly proposes a new API similar to AsyncFunction and > > ResultFuture as well as introducing timeout handling for AsyncSink > > requests. > > The FLIP targets 1.20 with backward compatible changes and we should > > proceed by adding support in all supporting connector repos. > > > > 1- > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Refactor+Async+Sink+API > > Best Regards > > Ahmed Hamdy > > >
[DISCUSS] FLIP-451: Refactor Async sink API
Hi all, I would like to start a discussion on FLIP-451[1] The proposal comes on encountering a couple of issues while working with implementers for Async Sink. The FLIP mainly proposes a new API similar to AsyncFunction and ResultFuture as well as introducing timeout handling for AsyncSink requests. The FLIP targets 1.20 with backward compatible changes and we should proceed by adding support in all supporting connector repos. 1- https://cwiki.apache.org/confluence/display/FLINK/FLIP-451%3A+Refactor+Async+Sink+API Best Regards Ahmed Hamdy
Re: [ DISCUSS ] FLIP-XXX : [Plugin] Enhancing Flink Failure Management in Kubernetes with Dynamic Termination Log Integration
Hi, I agree with the Martijn, We can reformulate the FLIP to introduce termination log as supported pluggable enricher. If you believe the scope of work is a subset (Further implementation) we can just add a Jira ticket for it. IMO this will also help with implementation taking the existing enrichers into reference. Best Regards Ahmed Hamdy On Tue, 23 Apr 2024 at 15:23, Martijn Visser wrote: > From a procedural point of view, we shouldn't make FLIPs sub-tasks for > existing FLIPs that have been voted/are released. That will only cause > confusion down the line. A new FLIP should take existing functionality > (like FLIP-304) into account, and propose how to improve on what that > original FLIP has introduced or how you're going to leverage what's already > there. > > On Tue, Apr 23, 2024 at 11:42 AM ramkrishna vasudevan < > ramvasu.fl...@gmail.com> wrote: > > > Hi Gyula and Ahmed, > > > > I totally agree that there is an interlap in the final goal that both the > > FLIPs are achieving here and infact FLIP-304 is more comprehensive for > job > > failures. > > > > But as a proposal to move forward can we make Swathi's FLIP/JIRA as a sub > > task for FLIP-304 and continue with the PR since the main aim is to get > the > > cluster failure pushed to the termination log for K8s based deployments. > > And once it is completed we can work to make FLIP-304 to support job > > failure propagation to termination log? > > > > Regards > > Ram > > > > On Thu, Apr 18, 2024 at 10:07 PM Swathi C > > wrote: > > > > > Hi Gyula and Ahmed, > > > > > > Thanks for reviewing this. > > > > > > @gyula.f...@gmail.com , currently since our aim > > as > > > part of this FLIP was only to fail the cluster when job manager/flink > has > > > issues such that the cluster would no longer be usable, hence, we > > proposed > > > only related to that. > > > Your right, that it covers only job main class errors, job manager run > > time > > > failures, if the Job manager wants to write any metadata to any other > > > system ( ABFS, S3 , ... ) and the job failures will not be covered. > > > > > > FLIP-304 is mainly used to provide Failure enrichers for job failures. > > > Since, this FLIP is mainly for flink Job manager failures, let us know > if > > > we can leverage the goodness of both and try to extend FLIP-304 and add > > our > > > plugin implementation to cover the job level issues ( propagate this > info > > > to the /dev/termination-log such that, the container status reports it > > for > > > flink on K8S by implementing Failure Enricher interface and > > > processFailure() to do this ) and use this FLIP proposal for generic > > flink > > > cluster (Job manager/cluster ) failures. > > > > > > Regards, > > > Swathi C > > > > > > On Thu, Apr 18, 2024 at 7:36 PM Ahmed Hamdy > > wrote: > > > > > > > Hi Swathi! > > > > Thanks for the proposal. > > > > Could you please elaborate what this FLIP offers more than > Flip-304[1]? > > > > Flip 304 proposes a Pluggable mechanism for enriching Job failures, > If > > I > > > am > > > > not mistaken this proposal looks like a subset of it. > > > > > > > > 1- > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-304%3A+Pluggable+Failure+Enrichers > > > > > > > > Best Regards > > > > Ahmed Hamdy > > > > > > > > > > > > On Thu, 18 Apr 2024 at 08:23, Gyula Fóra > wrote: > > > > > > > > > Hi Swathi! > > > > > > > > > > Thank you for creating this proposal. I really like the general > idea > > of > > > > > increasing the K8s native observability of Flink job errors. > > > > > > > > > > I took a quick look at your reference PR, the termination log > related > > > > logic > > > > > is contained completely in the ClusterEntrypoint. What type of > errors > > > > will > > > > > this actually cover? > > > > > > > > > > To me this seems to cover only: > > > > > - Job main class errors (ie startup errors) > > > > > - JobManager failures > > > > > > > > > > Would regular job errors (that cause only job failover but not JM > > > errors) > > > > > be
Re: [VOTE] Release flink-connector-cassandra v3.2.0, release candidate #1
Thanks Danny, +1 (non-binding) - Verified Checksums and hashes - Verified Signatures - Reviewed web PR - github tag exists - Build source Best Regards Ahmed Hamdy On Tue, 23 Apr 2024 at 03:57, Muhammet Orazov wrote: > Thanks! +1 (non-binding) > > - Checked 512 hash > - Checked gpg signature > - Reviewed web pr & release notes > - Built the source with JDK 11 & 8 > > Best, > Muhammet > > On 2024-04-22 13:04, Danny Cranmer wrote: > > Hi everyone, > > > > Please review and vote on release candidate #1 for > > flink-connector-cassandra v3.2.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This release supports Flink 1.18 and 1.19. > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > > [2], > > which are signed with the key with fingerprint 125FD8DB [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag v3.2.0-rc1 [5], > > * website pull request listing the new release [6]. > > * CI build of the tag [7]. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Danny > > > > [1] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353148 > > [2] > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-cassandra-3.2.0-rc1 > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1722 > > [5] > > > https://github.com/apache/flink-connector-cassandra/releases/tag/v3.2.0-rc1 > > [6] https://github.com/apache/flink-web/pull/737 > > [7] > > > https://github.com/apache/flink-connector-cassandra/actions/runs/8784310241 >
Re: [VOTE] Release flink-connector-kafka v3.2.0, release candidate #1
Thanks Danny, +1 (non-binding) - Verified Checksums and hashes - Verified Signatures - Reviewed web PR - github tag exists - Build source Best Regards Ahmed Hamdy On Tue, 23 Apr 2024 at 03:47, Muhammet Orazov wrote: > Thanks Danny, +1 (non-binding) > > - Checked 512 hash > - Checked gpg signature > - Reviewed pr > - Built the source with JDK 11 & 8 > > Best, > Muhammet > > On 2024-04-22 13:55, Danny Cranmer wrote: > > Hi everyone, > > > > Please review and vote on release candidate #1 for > > flink-connector-kafka > > v3.2.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This release supports Flink 1.18 and 1.19. > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > > [2], > > which are signed with the key with fingerprint 125FD8DB [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag v3.2.0-rc1 [5], > > * website pull request listing the new release [6]. > > * CI build of the tag [7]. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Danny > > > > [1] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354209 > > [2] > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-kafka-3.2.0-rc1 > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1723 > > [5] > > https://github.com/apache/flink-connector-kafka/releases/tag/v3.2.0-rc1 > > [6] https://github.com/apache/flink-web/pull/738 > > [7] https://github.com/apache/flink-connector-kafka >
Re: [VOTE] FLIP-436: Introduce Catalog-related Syntax
Hi, +1 (non-binding)
Best Regards
Ahmed Hamdy
On Wed, 24 Apr 2024 at 09:58, Yubin Li wrote:
> Hi everyone,
>
> During the implementation of the "describe catalog" syntax, it was
> found that the original output style needed to be improved.
> ```
> desc catalog extended cat2;
>
> +--+-+
> | catalog_description_item |
> catalog_description_value |
>
> +--+-+
> | Name |
> cat2 |
> | Type |
> generic_in_memory |
> | Comment |
> |
> | Properties | ('default-database','db'),
> ('type','generic_in_memory') |
>
> +--+-+
> 4 rows in set
> ```
> After offline discussions with Jane Chan and Jark Wu, we suggest
> improving it to the following form:
> ```
> desc catalog extended cat2;
> +-+---+
> | info name |info value |
> +-+---+
> |name | cat2 |
> |type | generic_in_memory |
> | comment | |
> | option:default-database |db |
> +-+---+
> 4 rows in set
> ```
>
> For the following reasons:
> 1. The title should be consistent with engines such as Databricks for
> easy understanding, and it should also be consistent with Flink's own
> naming style. Therefore, the title adopts "info name", "info value",
> and the key name should be unified in lowercase, so "Name" is replaced
> by "name".
> Note: Databricks output style [1] as follows:
> ```
> > DESCRIBE CATALOG main;
> info_name info_value
>
> Catalog Name main
> Comment Main catalog (auto-created)
> Owner metastore-admin-users
> Catalog Type Regular
> ```
> 2. There may be many attributes of the catalog, and it is very poor in
> readability when displayed in one line. It should be expanded into
> multiple lines, and the key name is prefixed with "option:" to
> identify that this is an attribute row. And since `type` is an
> important information of the catalog, even if `extended` is not
> specified, it should also be displayed, and correspondingly,
> "option:type" should be removed to avoid redundancy.
>
> WDYT? Looking forward to your reply!
>
> [1]
> https://learn.microsoft.com/zh-tw/azure/databricks/sql/language-manual/sql-ref-syntax-aux-describe-catalog
>
> Best,
> Yubin
>
> On Wed, Mar 20, 2024 at 2:15 PM Benchao Li wrote:
> >
> > +1 (binding)
> >
> > gongzhongqiang 于2024年3月20日周三 11:40写道：
> > >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Zhongqiang Gong
> > >
> > > Yubin Li 于2024年3月19日周二 18:03写道：
> > >
> > > > Hi everyone,
> > > >
> > > > Thanks for all the feedback, I'd like to start a vote on the
> FLIP-436:
> > > > Introduce Catalog-related Syntax [1]. The discussion thread is here
> > > > [2].
> > > >
> > > > The vote will be open for at least 72 hours unless there is an
> > > > objection or insufficient votes.
> > > >
> > > > [1]
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-436%3A+Introduce+Catalog-related+Syntax
> > > > [2] https://lists.apache.org/thread/10k1bjb4sngyjwhmfqfky28lyoo7sv0z
> > > >
> > > > Best regards,
> > > > Yubin
> > > >
> >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
>
Re: [VOTE] Release flink-connector-aws v4.3.0, release candidate #2
Thanks Danny, +1 (non-binding) - Verified Checksums - Verified Signatures - No binaries exists in source archive - Built source - Reviewed Web PR - Run basic Kinesis example Best Regards Ahmed Hamdy On Sun, 21 Apr 2024 at 14:25, Hang Ruan wrote: > +1 (non-binding) > > - Validated checksum hash > - Verified signature > - Verified that no binaries exist in the source archive > - Build the source with Maven and jdk8 > - Verified web PR > - Check that the jar is built by jdk8 > > Best, > Hang > > Danny Cranmer 于2024年4月19日周五 18:08写道： > > > Hi everyone, > > > > Please review and vote on release candidate #2 for flink-connector-aws > > v4.3.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This version supports Flink 1.18 and 1.19. > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > > [2], > > which are signed with the key with fingerprint 125FD8DB [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag v4.3.0-rc2 [5], > > * website pull request listing the new release [6]. > > * CI build of the tag [7]. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Release Manager > > > > [1] > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353793 > > [2] > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-4.3.0-rc2 > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1721/ > > [5] > https://github.com/apache/flink-connector-aws/releases/tag/v4.3.0-rc2 > > [6] https://github.com/apache/flink-web/pull/733 > > [7] > https://github.com/apache/flink-connector-aws/actions/runs/8751694197 > > >
Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0
+1 (non-binding) Great work! Best Regards Ahmed Hamdy On Wed, 17 Apr 2024 at 14:36, Jeyhun Karimov wrote: > +1 (non binding) > > Regards, > Jeyhun > > On Wed, Apr 17, 2024 at 2:22 PM Zhu Zhu wrote: > > > +1 (binding) > > > > Thanks, > > Zhu > > > > Yuxin Tan 于2024年4月17日周三 18:36写道： > > > > > +1 (non-binding) > > > > > > Best, > > > Yuxin > > > > > > > > > Zakelly Lan 于2024年4月17日周三 16:51写道： > > > > > > > +1 binding > > > > > > > > > > > > Best, > > > > Zakelly > > > > > > > > On Wed, Apr 17, 2024 at 2:05 PM Rui Fan <1996fan...@gmail.com> > wrote: > > > > > > > > > +1(binding) > > > > > > > > > > Best, > > > > > Rui > > > > > > > > > > On Wed, Apr 17, 2024 at 1:02 PM Xuannan Su > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Thanks for all the feedback about the FLIP-442: General > Improvement > > > to > > > > > > Configuration for Flink 2.0 [1] [2]. > > > > > > > > > > > > I'd like to start a vote for it. The vote will be open for at > least > > > 72 > > > > > > hours(excluding weekends,until APR 22, 12:00AM GMT) unless there > is > > > an > > > > > > objection or an insufficient number of votes. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-442%3A+General+Improvement+to+Configuration+for+Flink+2.0 > > > > > > [2] > > https://lists.apache.org/thread/15k0stwyoknhxvd643ctwjw3fd17pqwk > > > > > > > > > > > > > > > > > > Best regards, > > > > > > Xuannan > > > > > > > > > > > > > > > > > > > > >
Re: [ DISCUSS ] FLIP-XXX : [Plugin] Enhancing Flink Failure Management in Kubernetes with Dynamic Termination Log Integration
Hi Swathi! Thanks for the proposal. Could you please elaborate what this FLIP offers more than Flip-304[1]? Flip 304 proposes a Pluggable mechanism for enriching Job failures, If I am not mistaken this proposal looks like a subset of it. 1- https://cwiki.apache.org/confluence/display/FLINK/FLIP-304%3A+Pluggable+Failure+Enrichers Best Regards Ahmed Hamdy On Thu, 18 Apr 2024 at 08:23, Gyula Fóra wrote: > Hi Swathi! > > Thank you for creating this proposal. I really like the general idea of > increasing the K8s native observability of Flink job errors. > > I took a quick look at your reference PR, the termination log related logic > is contained completely in the ClusterEntrypoint. What type of errors will > this actually cover? > > To me this seems to cover only: > - Job main class errors (ie startup errors) > - JobManager failures > > Would regular job errors (that cause only job failover but not JM errors) > be reported somehow with this plugin? > > Thanks > Gyula > > On Tue, Apr 16, 2024 at 8:21 AM Swathi C > wrote: > > > Hi All, > > > > I would like to start a discussion on FLIP-XXX : [Plugin] Enhancing Flink > > Failure Management in Kubernetes with Dynamic Termination Log > Integration. > > > > > > > https://docs.google.com/document/d/1tWR0Fi3w7VQeD_9VUORh8EEOva3q-V0XhymTkNaXHOc/edit?usp=sharing > > > > > > This FLIP proposes an improvement plugin and focuses mainly on Flink on > > K8S but can be used as a generic plugin and add further enhancements. > > > > Looking forward to everyone's feedback and suggestions. Thank you !! > > > > Best Regards, > > Swathi Chandrashekar > > >
Re: [VOTE] Release flink-connector-mongodb v1.2.0, release candidate #2
+1 (non-binding) - verified hashes and checksums - verified signature - verified source contains no binaries - tag exists in github - reviewed web PR Best Regards Ahmed Hamdy On Thu, 18 Apr 2024 at 11:21, Danny Cranmer wrote: > Hi everyone, > > Please review and vote on the release candidate #2 for v1.2.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > This release supports Flink 1.18 and 1.19. > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], > which are signed with the key with fingerprint 125FD8DB [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v1.2.0-rc2 [5], > * website pull request listing the new release [6]. > * CI build of tag [7]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Danny > > [1] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354192 > [2] > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-mongodb-1.2.0-rc2 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] > https://repository.apache.org/content/repositories/orgapacheflink-1719/ > [5] > https://github.com/apache/flink-connector-mongodb/releases/tag/v1.2.0-rc2 > [6] https://github.com/apache/flink-web/pull/735 > [7] > https://github.com/apache/flink-connector-mongodb/actions/runs/8735987710 >
Re: [VOTE] Release flink-connector-jdbc v3.2.0, release candidate #2
+1 (non-binding) - Verified Checksums and hashes - Verified Signatures - No binaries in source - Build source - Github tag exists - Reviewed Web PR Best Regards Ahmed Hamdy On Thu, 18 Apr 2024 at 11:22, Danny Cranmer wrote: > Sorry for typos: > > > Please review and vote on the release candidate #1 for the version 3.2.0, > as follows: > Should be "release candidate #2" > > > * source code tag v3.2.0-rc1 [5], > Should be "source code tag v3.2.0-rc2" > > Thanks, > Danny > > On Thu, Apr 18, 2024 at 11:19 AM Danny Cranmer > wrote: > > > Hi everyone, > > > > Please review and vote on the release candidate #1 for the version 3.2.0, > > as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This release supports Flink 1.18 and 1.19. > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > > [2], which are signed with the key with fingerprint 125FD8DB [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag v3.2.0-rc1 [5], > > * website pull request listing the new release [6]. > > * CI run of tag [7]. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Danny > > > > [1] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353143 > > [2] > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-jdbc-3.2.0-rc2 > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1718/ > > [5] > https://github.com/apache/flink-connector-jdbc/releases/tag/v3.2.0-rc2 > > [6] https://github.com/apache/flink-web/pull/734 > > [7] > https://github.com/apache/flink-connector-jdbc/actions/runs/8736019099 > > >
Re: [DISCUSS] FLIP-445: Support dynamic parallelism inference for HiveSource
Hi Xia, I have read through the FLIP and discussion and the new version of the FLIP looks better. +1 for the proposal. Best Regards Ahmed Hamdy On Thu, 18 Apr 2024 at 12:21, Ron Liu wrote: > Hi, Xia > > Thanks for updating, looks good to me. > > Best, > Ron > > Xia Sun 于2024年4月18日周四 19:11写道： > > > Hi Ron, > > Yes, presenting it in a table might be more intuitive. I have already > added > > the table in the "Public Interfaces | New Config Option" chapter of FLIP. > > PTAL~ > > > > Ron Liu 于2024年4月18日周四 18:10写道： > > > > > Hi, Xia > > > > > > Thanks for your reply. > > > > > > > That means, in terms > > > of priority, `table.exec.hive.infer-source-parallelism` > > > > `table.exec.hive.infer-source-parallelism.mode`. > > > > > > I still have some confusion, if the > > > `table.exec.hive.infer-source-parallelism` > > > >`table.exec.hive.infer-source-parallelism.mode`, currently > > > `table.exec.hive.infer-source-parallelism` default value is true, that > > > means always static parallelism inference work? Or perhaps after this > > FLIP, > > > we changed the default behavior of > > > `table.exec.hive.infer-source-parallelism` to indicate dynamic > > parallelism > > > inference when enabled. > > > I think you should list the various behaviors of these two options that > > > coexist in FLIP by a table, only then users can know how the dynamic > and > > > static parallelism inference work. > > > > > > Best, > > > Ron > > > > > > Xia Sun 于2024年4月18日周四 16:33写道： > > > > > > > Hi Ron and Lijie, > > > > Thanks for joining the discussion and sharing your suggestions. > > > > > > > > > the InferMode class should also be introduced in the Public > > Interfaces > > > > > section! > > > > > > > > > > > > Thanks for the reminder, I have now added the InferMode class to the > > > Public > > > > Interfaces section as well. > > > > > > > > > `table.exec.hive.infer-source-parallelism.max` is 1024, I checked > > > through > > > > > the code that the default value is 1000? > > > > > > > > > > > > I have checked and the default value of > > > > `table.exec.hive.infer-source-parallelism.max` is indeed 1000. This > has > > > > been corrected in the FLIP. > > > > > > > > > how are`table.exec.hive.infer-source-parallelism` and > > > > > `table.exec.hive.infer-source-parallelism.mode` compatible? > > > > > > > > > > > > This is indeed a critical point. The current plan is to deprecate > > > > `table.exec.hive.infer-source-parallelism` but still utilize it as > the > > > main > > > > switch for enabling automatic parallelism inference. That means, in > > terms > > > > of priority, `table.exec.hive.infer-source-parallelism` > > > > > `table.exec.hive.infer-source-parallelism.mode`. In future versions, > if > > > > `table.exec.hive.infer-source-parallelism` is removed, this logic > will > > > also > > > > need to be revised, leaving only > > > > `table.exec.hive.infer-source-parallelism.mode` as the basis for > > deciding > > > > whether to enable parallelism inference. I have also added this > > > description > > > > to the FLIP. > > > > > > > > > > > > > In FLIP-367 it is supported to be able to set the Source's > > parallelism > > > > > individually, if in the future HiveSource also supports this > feature, > > > > > however, the default value of > > > > > `table.exec.hive.infer-source-parallelism.mode` is > > `InferMode.DYNAMIC`, > > > > at > > > > > this point will the parallelism be dynamically derived or will the > > > > manually > > > > > set parallelism take effect, and who has the higher priority? > > > > > > > > > > > > From my understanding, 'manually set parallelism' has the higher > > > priority, > > > > just like one of the preconditions for the effectiveness of dynamic > > > > parallelism inference in the AdaptiveBatchScheduler is that the > > vertex's > > > > parallelism isn't set. I believe whether it's static inference or > > dynamic > > > > inference, the ma
Re: [VOTE] Release flink-connector-gcp-pubsub v3.1.0, release candidate #1
Hi Danny, +1 (non-binding) - verified hashes and checksums - verified signature - verified source contains no binaries - tag exists in github - reviewed web PR Best Regards Ahmed Hamdy On Thu, 18 Apr 2024 at 11:32, Danny Cranmer wrote: > Hi everyone, > > Please review and vote on release candidate #1 for > flink-connector-gcp-pubsub v3.1.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > This release supports Flink 1.18 and 1.19. > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], > which are signed with the key with fingerprint 125FD8DB [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v3.1.0-rc1 [5], > * website pull request listing the new release [6]. > * CI build of the tag [7]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Danny > > [1] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353813 > [2] > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-gcp-pubsub-3.1.0-rc1 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapacheflink-1720 > [5] > > https://github.com/apache/flink-connector-gcp-pubsub/releases/tag/v3.1.0-rc1 > [6] https://github.com/apache/flink-web/pull/736/files > [7] > > https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/8735952883 >
Re: [VOTE] Release flink-connector-aws v4.3.0, release candidate #1
Hi Danny, Thanks for driving this. +1 (non-binding) - Checksum verified - Signature and Keys match - Licenses checked - Build from source - Run Kinesis example - Reviewed web PR Best Regards Ahmed Hamdy On Tue, 16 Apr 2024 at 13:01, Jeyhun Karimov wrote: > +1 (non-binding) > > - Verified tags > - Verified Lisence > - Reviewed web pr > > Regards, > Jeyhun > > On Tue, Apr 16, 2024 at 12:59 PM Danny Cranmer > wrote: > > > Hi everyone, > > > > Please review and vote on the release candidate #1 for > flink-connector-aws > > v4.3.0, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This release supports Apache Flink 1.18 and 1.19. > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > > [2], > > which are signed with the key with fingerprint 125FD8DB [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag v4.3.0-rc1 [5], > > * website pull request listing the new release [6]. > > * CI build of the tag against Flink 1.18.1 and 1.19.0 [7]. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Danny > > > > [1] > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353793 > > [2] > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-4.3.0-rc1 > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1711/ > > [5] > https://github.com/apache/flink-connector-aws/releases/tag/v4.3.0-rc1 > > [6] https://github.com/apache/flink-web/pull/733 > > [7] > https://github.com/apache/flink-connector-aws/actions/runs/8703830985 > > >
Re: [VOTE] FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines
+ 1 (non-binding) Best Regards Ahmed Hamdy On Wed, 17 Apr 2024 at 08:28, Yuepeng Pan wrote: > +1(non-binding). > > > > > Best, > Yuepeng Pan > > At 2024-04-17 14:27:27, "Ron liu" wrote: > >Hi Dev, > > > >Thank you to everyone for the feedback on FLIP-435: Introduce a New > >Materialized Table for Simplifying Data Pipelines[1][2]. > > > >I'd like to start a vote for it. The vote will be open for at least 72 > >hours unless there is an objection or not enough votes. > > > >[1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-435%3A+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines > >[2] https://lists.apache.org/thread/c1gnn3bvbfs8v1trlf975t327s4rsffs > > > >Best, > >Ron >
Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan
Congratulations, Zakelly Best Regards Ahmed Hamdy On Mon, 15 Apr 2024 at 07:56, Hangxiang Yu wrote: > Congratulations, Zakelly! > > On Mon, Apr 15, 2024 at 1:58 PM Yun Tang wrote: > > > Congratulations, Zakelly! > > > > Best > > Yun Tang > > > > From: Yanquan Lv > > Sent: Monday, April 15, 2024 13:23 > > To: dev@flink.apache.org > > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan > > > > Congratulations, Zakelly! > > > > Best, > > YanQuan > > > > Yuan Mei 于2024年4月15日周一 10:51写道： > > > > > Hi everyone, > > > > > > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has > > become > > > a new Flink Committer! > > > > > > Zakelly has been continuously contributing to the Flink project since > > 2020, > > > with a focus area on Checkpointing, State as well as frocksdb (the > > default > > > on-disk state db). > > > > > > He leads several FLIPs to improve checkpoints and state APIs, including > > > File Merging for Checkpoints and configuration/API reorganizations. He > is > > > also one of the main contributors to the recent efforts of > "disaggregated > > > state management for Flink 2.0" and drives the entire discussion in the > > > mailing thread, demonstrating outstanding technical depth and breadth > of > > > knowledge. > > > > > > Beyond his technical contributions, Zakelly is passionate about helping > > the > > > community in numerous ways. He spent quite some time setting up the > Flink > > > Speed Center and rebuilding the benchmark pipeline after the original > one > > > was out of lease. He helps build frocksdb and tests for the upcoming > > > frocksdb release (bump rocksdb from 6.20.3->8.10). > > > > > > Please join me in congratulating Zakelly for becoming an Apache Flink > > > committer! > > > > > > Best, > > > Yuan (on behalf of the Flink PMC) > > > > > > > > -- > Best, > Hangxiang. >
Re: [Vote] FLIP-438: Amazon SQS Sink Connector
Hi Priya +1 (non-binding) Best Regards Ahmed Hamdy On Fri, 12 Apr 2024 at 22:56, Dhingra, Priya wrote: > Hi devs, > > > > Thank you to everyone for the feedback on FLIP-438: Amazon SQS Sink > Connector< > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector > > > > > > I would like to start a vote for it. The vote will be open for at least 72 > > hours unless there is an objection or not enough votes. > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector > > Regards > Priya >
Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee
Congratulations Lincoln! Best Regards Ahmed Hamdy On Fri, 12 Apr 2024 at 11:09, yuxia wrote: > Congratulations, Lincoln! > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Feng Jin" > 收件人: "dev" > 发送时间: 星期五, 2024年 4 月 12日 下午 5:23:40 > 主题: Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee > > Congratulations, Lincoln! > > Best, > Feng Jin > > On Fri, Apr 12, 2024 at 5:20 PM xiangyu feng wrote: > > > Congratulations, Lincoln! > > > > Best, > > Xiangyu Feng > > > > Feifan Wang 于2024年4月12日周五 17:19写道： > > > > > Congratulations, Lincoln! > > > > > > > > > —— > > > > > > Best regards, > > > > > > Feifan Wang > > > > > > > > > > > > > > > At 2024-04-12 15:59:00, "Jark Wu" wrote: > > > >Hi everyone, > > > > > > > >On behalf of the PMC, I'm very happy to announce that Lincoln Lee has > > > >joined the Flink PMC! > > > > > > > >Lincoln has been an active member of the Apache Flink community for > > > >many years. He mainly works on Flink SQL component and has driven > > > >/pushed many FLIPs around SQL, including FLIP-282/373/415/435 in > > > >the recent versions. He has a great technical vision of Flink SQL and > > > >participated in plenty of discussions in the dev mailing list. Besides > > > >that, > > > >he is community-minded, such as being the release manager of 1.19, > > > >verifying releases, managing release syncs, writing the release > > > >announcement etc. > > > > > > > >Congratulations and welcome Lincoln! > > > > > > > >Best, > > > >Jark (on behalf of the Flink PMC) > > > > > >
Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge
Congratulations, Jing! Best Regards Ahmed Hamdy On Fri, 12 Apr 2024 at 10:46, weijie guo wrote: > Congratulations, Jing! > > Best regards, > > Weijie > > > Feng Jin 于2024年4月12日周五 17:24写道： > > > Congratulations, Jing > > > > Best, > > Feng Jin > > > > On Fri, Apr 12, 2024 at 4:46 PM Samrat Deb > wrote: > > > > > Congratulations, Jing! > > > > > > > > > On Fri, 12 Apr 2024 at 2:15 PM, Jiabao Sun > wrote: > > > > > > > Congratulations, Jing! > > > > > > > > Best, > > > > Jiabao > > > > > > > > Sergey Nuyanzin 于2024年4月12日周五 16:41写道： > > > > > > > > > Congratulations, Jing! > > > > > > > > > > On Fri, Apr 12, 2024 at 10:29 AM Rui Fan <1996fan...@gmail.com> > > wrote: > > > > > > > > > > > Congratulations, Jing~ > > > > > > > > > > > > Best, > > > > > > Rui > > > > > > > > > > > > On Fri, Apr 12, 2024 at 4:28 PM Yun Tang > wrote: > > > > > > > > > > > > > Congratulations, Jing! > > > > > > > > > > > > > > Best > > > > > > > Yun Tang > > > > > > > > > > > > > > From: Jark Wu > > > > > > > Sent: Friday, April 12, 2024 16:02 > > > > > > > To: dev > > > > > > > Cc: gej...@gmail.com > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > On behalf of the PMC, I'm very happy to announce that Jing Ge > has > > > > > > > joined the Flink PMC! > > > > > > > > > > > > > > Jing has been contributing to Apache Flink for a long time. He > > > > > > continuously > > > > > > > works on SQL, connectors, Source, and Sink APIs, test, and > > document > > > > > > > modules while contributing lots of code and insightful > > discussions. > > > > He > > > > > is > > > > > > > one of the maintainers of Flink CI infra. He is also willing to > > > help > > > > a > > > > > > lot > > > > > > > in the > > > > > > > community work, such as being the release manager for both 1.18 > > and > > > > > 1.19, > > > > > > > verifying releases, and answering questions on the mailing > list. > > > > > Besides > > > > > > > that, > > > > > > > he is continuously helping with the expansion of the Flink > > > community > > > > > and > > > > > > > has > > > > > > > given several talks about Flink at many conferences, such as > > Flink > > > > > > Forward > > > > > > > 2022 and 2023. > > > > > > > > > > > > > > Congratulations and welcome Jing! > > > > > > > > > > > > > > Best, > > > > > > > Jark (on behalf of the Flink PMC) > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best regards, > > > > > Sergey > > > > > > > > > > > > > > >
Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector
Thanks Priya the FLIP now looks much better, I would move out some of the implementation details . Just highlight how the writer actually uses the client (the subitRequestEntries part) and how the Sink itself exposes its API to users (the Sink & builder part), other parts should not be part of the FLIP. Let's leave something for the coders ;). I would still vote to add Table API support, you really would have done most of the work already, do you believe there is a specific concern not to include it? I am happy with the FLIP once reformatted. Thanks for the effort. Best Regards Ahmed Hamdy On Tue, 9 Apr 2024 at 23:59, Dhingra, Priya wrote: > > > On 4/9/24, 3:57 PM, "Dhingra, Priya" dhipr...@amazon.com.inva>LID> wrote: > > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > > > > Hi Ahmed and Samrat, > > > Thanks a lot for all the feedbacks, this is my first ever contribution to > apache Flink, hence I was bit unaware about few things but updated all of > them as per your suggestions, thanks again for all the support here, much > appreciated!! > > > 1) I am not sure why we need to suppress warnings in the sink example in > the > FLIP? > > > Removed and updated the FLIP. > > > 2) You provided the sink example as it is the Public Interface, however the > actual AsyncSink logic is mostly in the writer, so would be helpful to > provide a brief of the writer or the "submitRequestEntries" > > > Added in FLIP > > > 3) I am not sure what customDimensions are or how are they going to be used > by the client, (that's why I thought the writer example should be helpful). > > > Removed. This is no more required, we have added in our code to support > some specific usecase, no more required for apache PR. > > > 4) Are we going to use the existing aws client providers to handle the > authentication and async client creation similar to Kinesis/Firehose and > DDB. I would strongly recommend we do. > > > Yes > > > 5) Given you suggest implementing this in "flink-connector-aws" repo, it > should probably follow the same versioning of AWS connectors, hence > targeting 4.3.0. Also I am not sure why we are targeting support for 1.16 > given that it is out of support and 4.2 supports 1.17+. > > > Sorry, was not aware about the versioning we should have it here. I tested > the sqs sink with flint 1.16 so thought of putting the same, but was not > aware about out of support. Updated now with 4.3.0 and 1.17+ > > > 6) Are we planning to support Table API & SQL as well? This should not be > much of an effort IMO so I think we should. > > > No, not putting that in scope for first iteration. We can take that as > fast follow up. > > > 7) It would be great if we also added an implementation of Element > converter given that SQS message bodies are mainly Strings if I remember > correctly. We can extend it to other types for MessageAttributeValue > augmentation,this should be more valuable on table API as well to use it as > default Element Converter. > > > Updated in FLIP > > > 8. Different connectors provide different types of fault > tolerant guarantees[1]. What type of fault tolerant sink guarantees > flink-connector-sqs will provide ? > Could you elaborate on the fault-tolerant capabilities that the > flink-connector-sqs will provide? > > > at-least-once > > > > > 9) Can you help with what the minimal configuration required for > instantiating the sink ? > > > SQSSink.builder() > .setSqsUrl(sqsUrl) > .setSqsClientProperties(getSQSClientProperties()) > .setSerializationSchema(serializationSchema) > .build(); > > > > > 10) Amazon SQS offers various data types [2]. Could you outline the types > of SQS data the sink plans to support? > > SendMessageBatchRequestEntry > > > > > > > > > > > > > > > Hi Priya, > > > > > Thank you for the FLIP. sqs connector would be a great addition to the > flink connector aws. > > > > > +1 for all the queries raised by Ahmed. > > > > > Adding to Ahmed's queries, I have a few more: > > > > > 1. Different connectors provide different types of fault > tolerant guarantees[1]. What type of fault tolerant sink guarantees > flink-connector-sqs will provide ? > Could you elaborate on the fault-tolerant capabilities that the > flink-connector-sqs will provide? > > > > > 2. Can you help with what the minimal configuration required for > instantiating the sin
Re: [VOTE] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI
+1 (non-binding) Best Regards Ahmed Hamdy On Wed, 10 Apr 2024 at 06:39, Zhu Zhu wrote: > +1 > > Thanks, > Zhu > > gongzhongqiang 于2024年4月10日周三 13:11写道： > > > +1 (non binding) > > > > > > Bests, > > > > Zhongqiang Gong > > > > Rui Fan <1996fan...@gmail.com> 于2024年4月10日周三 12:36写道： > > > > > Hi devs, > > > > > > Thank you to everyone for the feedback on FLIP-441: Show > > > the JobType and remove Execution Mode on Flink WebUI[1] > > > which has been discussed in this thread [2]. > > > > > > I would like to start a vote for it. The vote will be open for at least > > 72 > > > hours unless there is an objection or not enough votes. > > > > > > [1] https://cwiki.apache.org/confluence/x/agrPEQ > > > [2] https://lists.apache.org/thread/0s52w17w24x7m2zo6ogl18t1fy412vcd > > > > > > Best, > > > Rui > > > > > >
Re: Iceberg table maintenance
Thanks Peter for forwarding this.
Best Regards
Ahmed Hamdy
On Tue, 9 Apr 2024 at 05:51, Péter Váry wrote:
> Forwarding the invite for the discussion we plan to do with the Iceberg
> folks, as some of you might be interested in this.
>
> -- Forwarded message -
> From: Brian Olsen
> Date: Mon, Apr 8, 2024, 18:29
> Subject: Re: Flink table maintenance
> To:
>
>
> Hey Iceberg nation,
>
> I would like to share about the meeting this Wednesday to further discuss
> details of Péter's proposal on Flink Maintenance Tasks.
> Calendar Link: https://calendar.app.google/83HGYWXoQJ8zXuVCA
>
> List discussion:
> https://lists.apache.org/thread/10mdf9zo6pn0dfq791nf4w1m7jh9k3sl
> <
> https://www.google.com/url?q=https://lists.apache.org/thread/10mdf9zo6pn0dfq791nf4w1m7jh9k3sl=D=calendar=2=AOvVaw2-aePIRr6APFVHpRDipMgX
> >
>
> Design Doc: Flink table maintenance
> <
> https://www.google.com/url?q=https://docs.google.com/document/d/16g3vR18mVBy8jbFaLjf2JwAANuYOmIwr15yDDxovdnA/edit?usp%3Dsharing=D=calendar=2=AOvVaw1oLYQP76-G1ZEOW5pTxV1M
> >
>
>
>
> On Mon, Apr 1, 2024 at 8:52 PM Manu Zhang wrote:
>
> > Hi Peter,
> >
> > Are you proposing to create a user facing locking feature in Iceberg, or
> >> just something something for internal use?
> >>
> >
> > Since it's a general issue, I'm proposing to create a general user
> > interface first, while the implementation can be left to users. For
> > example, we use Airflow to schedule maintenance jobs and we can check
> > in-progress jobs with the Airflow API. Hive metastore lock might be
> another
> > option we can implement for users.
> >
> > Thanks,
> > Manu
> >
> > On Tue, Apr 2, 2024 at 5:26 AM Péter Váry
> > wrote:
> >
> >> Hi Ajantha,
> >>
> >> I thought about enabling post commit topology based compaction for sinks
> >> using options, like we use for the parametrization of streaming reads
> [1].
> >> I think it will be hard to do it in a user friendly way - because of the
> >> high number of parameters -, but I think it is a possible solution with
> >> sensible defaults.
> >>
> >> There is a batch-like solution for data file compaction already
> available
> >> [2], but I do not see how we could extend Flink SQL to be able to call
> it.
> >>
> >> Writing to a branch using Flink SQL should be another thread, but by my
> >> first guess, it shouldn't be hard to implement using options, like:
> >> /*+ OPTIONS('branch'='b1') */
> >> Since writing to branch i already working through the Java API [3].
> >>
> >> Thanks, Peter
> >>
> >> 1 -
> >>
> https://iceberg.apache.org/docs/latest/flink-queries/#flink-streaming-read
> >> 2 -
> >>
> https://github.com/apache/iceberg/blob/820fc3ceda386149f42db8b54e6db9171d1a3a6d/flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/actions/RewriteDataFilesAction.java
> >> 3 -
> >> https://iceberg.apache.org/docs/latest/flink-writes/#branch-writes
> >>
> >> On Mon, Apr 1, 2024, 16:30 Ajantha Bhat wrote:
> >>
> >>> Thanks for the proposal Peter.
> >>>
> >>> I just wanted to know do we have any plans for supporting SQL syntax
> for
> >>> table maintenance (like CALL procedure) for pure Flink SQL users?
> >>> I didn't see any custom SQL parser plugin support in Flink. I also saw
> >>> that Branch write doesn't have SQL support (only Branch reads use
> Option),
> >>> So I am not sure about the roadmap of Iceberg SQL support in Flink.
> >>> Was there any discussion before?
> >>>
> >>> - Ajantha
> >>>
> >>> On Mon, Apr 1, 2024 at 7:51 PM Péter Váry >
> >>> wrote:
> >>>
> >>>> Hi Manu,
> >>>>
> >>>> Just to clarify:
> >>>> - Are you proposing to create a user facing locking feature in
> Iceberg,
> >>>> or just something something for internal use?
> >>>>
> >>>> I think we shouldn't add locking to Iceberg's user facing scope in
> this
> >>>> stage. A fully featured locking system has many more features that we
> need
> >>>> (priorities, fairness, timeouts etc). I could be tempted when we are
> >>>> talking about the REST catalog, but I think that should be further
> down the
> >>>> road, if ever...
> >>>>
> >>>> About using the tags:
> >>>> - I whole-heartedl
Re: [VOTE] FLIP-399: Flink Connector Doris
Hi Wudi, +1 (non-binding). Best Regards Ahmed Hamdy On Tue, 9 Apr 2024 at 09:21, Yuepeng Pan wrote: > Hi, Di. > > Thank you for driving it ! > > +1 (non-binding). > > Best, > Yuepeng Pan > > On 2024/04/09 02:47:55 wudi wrote: > > Hi devs, > > > > I would like to start a vote about FLIP-399 [1]. The FLIP is about > contributing the Flink Doris Connector[2] to the Flink community. > Discussion thread [3]. > > > > The vote will be open for at least 72 hours unless there is an objection > or > > insufficient votes. > > > > > > Thanks, > > Di.Wu > > > > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-399%3A+Flink+Connector+Doris > > [2] https://github.com/apache/doris-flink-connector > > [3] https://lists.apache.org/thread/p3z4wsw3ftdyfs9p2wd7bbr2gfyl3xnh > > > > >
Re: [DISCUSS] Externalized Google Cloud Connectors
Hi Claire, I am in favor of Martijn and Leonard's points, it is better to follow aws connectors and keep it under ASF. Do you have other suggestions for connectors than pub/sub at the moment? Best Regards Ahmed Hamdy On Thu, 4 Apr 2024 at 08:38, Martijn Visser wrote: > Hi Lorenzo, > > Bahir is retired, see the homepage. It plays no role (anymore). > > > This, unfortunately, is the tradeoff for developing the connectors > outside of Apache in exchange for development velocity. > > I understand that. It can be considered to develop the connectors outside > of the Flink project, in order to achieve development velocity. We've seen > a similar thing happen with the CDC connectors, before that was ultimately > donated to the Flink project. However, there are no guarantees that > external contributions are considered when evaluating committers, because > there's no visibility for the PMC on these external contributions. > > Best regards, > > Martijn > > On Wed, Apr 3, 2024 at 3:26 PM > wrote: > > > @Leonard @Martijn > > Following up on @Claire question, what is the role of Bahir ( > > https://bahir.apache.org/) in this scenario? > > > > I am also trying to understand how connectors fir in the Flink project > > scenario :) > > > > Thank you, > > Lorenzo > > On Apr 2, 2024 at 06:13 +0200, Leonard Xu , wrote: > > > Hey, Claire > > > > > > Thanks starting this discussion, all flink external connector repos are > > sub-projects of Apache Flink, including > > https://github.com/apache/flink-connector-aws. > > > > > > Creating a flink external connector repo named flink-connectors-gcp as > > sub-project of Apache Beam is not a good idea from my side. > > > > > > > Currently, we have no Flink committers on our team. We are actively > > > > involved in the Apache Beam community and have a number of ASF > members > > on > > > > the team. > > > > > > Not having Flink committer should not be a strong reason in this case, > > Flink community welcome contributors to contribute and maintain the > > connectors, as a contributor, through continuous connector development > and > > maintenance work in the community, you will also have the opportunity to > > become a Committer. > > > > > > Best, > > > Leonard > > > > > > > > > > 2024年2月14日 上午12:24，Claire McCarthy .INVALID> > > 写道： > > > > > > > > Hi Devs! > > > > > > > > I’d like to kick off a discussion on setting up a repo for a new > fleet > > of > > > > Google Cloud connectors. > > > > > > > > A bit of context: > > > > > > > > - > > > > > > > > We have a team of Google engineers who are looking to build/maintain > > > > 5-10 GCP connectors for Flink. > > > > - > > > > > > > > We are wondering if it would make sense to host our connectors under > > the > > > > ASF umbrella following a similar repo structure as AWS ( > > > > https://github.com/apache/flink-connector-aws). In our case: > > > > apache/flink-connectors-gcp. > > > > - > > > > > > > > Currently, we have no Flink committers on our team. We are actively > > > > involved in the Apache Beam community and have a number of ASF > members > > on > > > > the team. > > > > > > > > > > > > We saw that one of the original motivations for externalizing > > connectors > > > > was to encourage more activity and contributions around connectors by > > > > easing the contribution overhead. We understand that the decision was > > > > ultimately made to host the externalized connector repos under the > ASF > > > > organization. For the same reasons (release infra, quality assurance, > > > > integration with the community, etc.), we would like all GCP > > connectors to > > > > live under the ASF organization. > > > > > > > > We want to ask the Flink community what you all think of this idea, > and > > > > what would be the best way for us to go about contributing something > > like > > > > this. We are excited to contribute and want to learn and follow your > > > > practices. > > > > > > > > A specific issue we know of is that our changes need approval from > > Flink > > > > committers. Do you have a suggestion for how best to go about a new > > > > contribution like ours from a team that does not have committers? Is > it > > > > possible, for example, to partner with a committer (or a small > cohort) > > for > > > > tight engagement? We also know about ASF voting and release process, > > but > > > > that doesn't seem to be as much of a potential hurdle. > > > > > > > > Huge thanks in advance for sharing your thoughts! > > > > > > > > > > > > Claire > > > > > >
Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI
Acknowledged, +1 to start a vote. Best Regards Ahmed Hamdy On Mon, 8 Apr 2024 at 12:04, Rui Fan <1996fan...@gmail.com> wrote: > Sorry, it's a typo. It should be FLINK-32558[1]. > > [1] https://issues.apache.org/jira/browse/FLINK-32558 > > Best, > Rui > > On Mon, Apr 8, 2024 at 6:44 PM Ahmed Hamdy wrote: > > > Hi Rui, > > Thanks for the proposal. > > Is the deprecation Jira mentioned (FLINK-32258) correct? > > Best Regards > > Ahmed Hamdy > > > > > > On Sun, 7 Apr 2024 at 03:37, Rui Fan <1996fan...@gmail.com> wrote: > > > > > If there are no extra comments, I will start voting in three days, > thank > > > you~ > > > > > > Best, > > > Rui > > > > > > On Thu, Mar 28, 2024 at 4:46 PM Muhammet Orazov > > > wrote: > > > > > > > Hey Rui, > > > > > > > > Thanks for the detailed explanation and updating the FLIP! > > > > > > > > It is much clearer definitely, thanks for the proposal. > > > > > > > > Best, > > > > Muhammet > > > > > > > > On 2024-03-28 07:37, Rui Fan wrote: > > > > > Hi Muhammet, > > > > > > > > > > Thanks for your reply! > > > > > > > > > >> The execution mode is also used for the DataStream API [1], > > > > >> would that also affect/hide the DataStream execution mode > > > > >> if we remove it from the WebUI? > > > > > > > > > > Sorry, I didn't describe it clearly in FLIP-441[2], I have updated > > it. > > > > > Let me clarify the Execution Mode here: > > > > > > > > > > 1. Flink 1.19 website[3] also mentions the Execution mode, but it > > > > > actually matches the JobType[4] in the Flink code. Both of them > > > > > have 2 types: STREAMING and BATCH. > > > > > 2. execution.runtime-mode can be set to 3 types: STREAMING, > > > > > BATCH and AUTOMATIC. But the jobType will be inferred as > > > > > STREAMING or BATCH when execution.runtime-mode is set > > > > > to AUTOMATIC. > > > > > 3. The ExecutionMode I describe is: code link[5] , as we can > > > > > see, ExecutionMode has 4 enums: PIPELINED, > > > > > PIPELINED_FORCED, BATCH and BATCH_FORCED. > > > > > And we can see a flink streaming job from Flink WebUI, > > > > > the Execution mode is PIPELINE instead of STREAMING. > > > > > I attached a screenshot to the FLIP doc[2], you can see it there. > > > > > 4. What this proposal wants to do is to remove the ExecutionMode > > > > > with four enumerations on Flink WebUI and introduce the > > > > > JobType with two enumerations (STREAMING or BATCH). > > > > > STREAMING or BATCH is clearer and more accurate for users. > > > > > > > > > > Please let me know if it's not clear or anything is wrong, thanks a > > > > > lot! > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/execution_mode/ > > > > > [2] https://cwiki.apache.org/confluence/x/agrPEQ > > > > > [3] > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/ > > > > > [4] > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobType.java#L22 > > > > > [5] > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-core/src/main/java/org/apache/flink/api/common/ExecutionMode.java#L54 > > > > > > > > > > Best, > > > > > Rui > > > > > > > > > > On Thu, Mar 28, 2024 at 1:33 AM Venkatakrishnan Sowrirajan > > > > > > > > > > wrote: > > > > > > > > > >> Rui, > > > > >> > > > > >> I assume the current proposal would also handle the case of mixed > > mode > > > > >> (BATCH + STREAMING within the same app) in the future, right? > > > > >> > > > > >> Regards > > > > &
Re: [VOTE] Release flink-connector-rabbitmq, v3.0.2 release candidate #1
Hi Martijn, Sorry for joining late, +1 (non-binding) - Verified Checksums and Signatures - Verified no binaries in archive - Built source successfully - Checked tag correctly in github - reviewed web PR Best Regards Ahmed Hamdy On Mon, 15 Jan 2024 at 13:54, Danny Cranmer wrote: > +1 (binding) > > - Release notes look good > - Signatures match for binary/source archive > - Checksums match for binary/source archive > - Contents of Maven dist look complete > - Verified there are no binaries in the source archive > - Built src/tests pass running Maven locally > - Tag is present in Github > - CI built tag successfully [1] > - Approved docs PR > > Thanks, > Danny > > [1] > https://github.com/apache/flink-connector-rabbitmq/actions/runs/7502278783 > > On Mon, Jan 15, 2024 at 5:01 AM Zhongqiang Gong > > wrote: > > > +1 (non-binding) > > > > - Build with JDK 11 on ubuntu 22.04 > > - Gpg sign is correct > > - No binary files in the source release > > - Reviewed web pr (need to rebase) > > > > Best, > > Zhongqiang Gong > > > > On 2024/01/12 12:50:53 Martijn Visser wrote: > > > Hi everyone, > > > Please review and vote on the release candidate #1 for the version > > > 3.0.2, as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > This version is compatible with Flink 1.16.x, 1.17.x and 1.18.x. > > > > > > The complete staging area is available for your review, which includes: > > > * JIRA release notes [1], > > > * the official Apache source release to be deployed to dist.apache.org > > > [2], which are signed with the key with fingerprint > > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3], > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > * source code tag v3.0.2-rc1 [5], > > > * website pull request listing the new release [6]. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Release Manager > > > > > > [1] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353145 > > > [2] > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-rabbitmq-3.0.2-rc1 > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1697 > > > [5] > > > https://github.com/apache/flink-connector-rabbitmq/releases/tag/v3.0.2-rc1 > > > [6] https://github.com/apache/flink-web/pull/712 > > > > > >
Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI
Hi Rui, Thanks for the proposal. Is the deprecation Jira mentioned (FLINK-32258) correct? Best Regards Ahmed Hamdy On Sun, 7 Apr 2024 at 03:37, Rui Fan <1996fan...@gmail.com> wrote: > If there are no extra comments, I will start voting in three days, thank > you~ > > Best, > Rui > > On Thu, Mar 28, 2024 at 4:46 PM Muhammet Orazov > wrote: > > > Hey Rui, > > > > Thanks for the detailed explanation and updating the FLIP! > > > > It is much clearer definitely, thanks for the proposal. > > > > Best, > > Muhammet > > > > On 2024-03-28 07:37, Rui Fan wrote: > > > Hi Muhammet, > > > > > > Thanks for your reply! > > > > > >> The execution mode is also used for the DataStream API [1], > > >> would that also affect/hide the DataStream execution mode > > >> if we remove it from the WebUI? > > > > > > Sorry, I didn't describe it clearly in FLIP-441[2], I have updated it. > > > Let me clarify the Execution Mode here: > > > > > > 1. Flink 1.19 website[3] also mentions the Execution mode, but it > > > actually matches the JobType[4] in the Flink code. Both of them > > > have 2 types: STREAMING and BATCH. > > > 2. execution.runtime-mode can be set to 3 types: STREAMING, > > > BATCH and AUTOMATIC. But the jobType will be inferred as > > > STREAMING or BATCH when execution.runtime-mode is set > > > to AUTOMATIC. > > > 3. The ExecutionMode I describe is: code link[5] , as we can > > > see, ExecutionMode has 4 enums: PIPELINED, > > > PIPELINED_FORCED, BATCH and BATCH_FORCED. > > > And we can see a flink streaming job from Flink WebUI, > > > the Execution mode is PIPELINE instead of STREAMING. > > > I attached a screenshot to the FLIP doc[2], you can see it there. > > > 4. What this proposal wants to do is to remove the ExecutionMode > > > with four enumerations on Flink WebUI and introduce the > > > JobType with two enumerations (STREAMING or BATCH). > > > STREAMING or BATCH is clearer and more accurate for users. > > > > > > Please let me know if it's not clear or anything is wrong, thanks a > > > lot! > > > > > > [1] > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/datastream/execution_mode/ > > > [2] https://cwiki.apache.org/confluence/x/agrPEQ > > > [3] > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/datastream/execution_mode/ > > > [4] > > > > > > https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobType.java#L22 > > > [5] > > > > > > https://github.com/apache/flink/blob/f31c128bfc457b64dd7734f71123b74faa2958ba/flink-core/src/main/java/org/apache/flink/api/common/ExecutionMode.java#L54 > > > > > > Best, > > > Rui > > > > > > On Thu, Mar 28, 2024 at 1:33 AM Venkatakrishnan Sowrirajan > > > > > > wrote: > > > > > >> Rui, > > >> > > >> I assume the current proposal would also handle the case of mixed mode > > >> (BATCH + STREAMING within the same app) in the future, right? > > >> > > >> Regards > > >> Venkat > > >> > > >> On Wed, Mar 27, 2024 at 10:15 AM Venkatakrishnan Sowrirajan < > > >> vsowr...@asu.edu> wrote: > > >> > > >>> This will be a very useful addition to Flink UI. Thanks Rui for > > >>> starting > > >>> a FLIP for this improvement. > > >>> > > >>> Regards > > >>> Venkata krishnan > > >>> > > >>> > > >>> On Wed, Mar 27, 2024 at 4:49 AM Muhammet Orazov > > >>> wrote: > > >>> > > >>>> Hello Rui, > > >>>> > > >>>> Thanks for the proposal! It looks good! > > >>>> > > >>>> I have minor clarification from my side: > > >>>> > > >>>> The execution mode is also used for the DataStream API [1], > > >>>> would that also affect/hide the DataStream execution mode > > >>>> if we remove it from the WebUI? > > >>>> > > >>>> Best, > > >>>> Muhammet > > >>>> > > >>>> [1]: > > >>>> > > >>>> > > > https://urldefense.co
[jira] [Created] (FLINK-35050) Remove Lazy Initialization of DynamoDbBeanElementConverter
Ahmed Hamdy created FLINK-35050:
---
Summary: Remove Lazy Initialization of DynamoDbBeanElementConverter
Key: FLINK-35050
URL: https://issues.apache.org/jira/browse/FLINK-35050
Project: Flink
Issue Type: Technical Debt
Components: Connectors / DynamoDB
Affects Versions: aws-connector-4.3.0
Reporter: Ahmed Hamdy
h2. Description
{{ DynamoDbBeanElementConverter }} implements {{ ElementConverter }} which now
supports open method as of FLINK-29938, we need to remove lazy initialization
given that it is now unblocked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35022) Add TypeInformed Element Converter for DynamoDbSink
Ahmed Hamdy created FLINK-35022:
---
Summary: Add TypeInformed Element Converter for DynamoDbSink
Key: FLINK-35022
URL: https://issues.apache.org/jira/browse/FLINK-35022
Project: Flink
Issue Type: Improvement
Components: Connectors / DynamoDB
Affects Versions: aws-connector-4.3.0
Reporter: Ahmed Hamdy
h2. Context
{{DynamoDbSink}} as an extentsion of {{AsyncSinkBase}} depends on
{{org.apache.flink.connector.base.sink.writer.ElementConverter}} to convert
Flink stream objects to DynamoDb write requests, where item is represented as
{{Map}}.
{{AttributeValue}} is the wrapper for the DynamoDb comprehendable Object in a
format similar with type identification properties as in
{M": {"Name" : {"S": Joe }, "Age" : {"N": 35 }}}.
Since TypeInformation is already natively supported in Flink, many
implementations of the DynamoDb ElementConverted is just a boiler plate.
For example
{code:title="Simple POJO Element Conversion"}
public class Order {
String id;
int quantity;
double total;
}
{code}
The implementation of the converter must be
{code:title="Simple POJO DDB Element Converter"}
public static class SimplePojoElementConverter implements
ElementConverter {
@Override
public DynamoDbWriteRequest apply(Order order, SinkWriter.Context
context) {
Map itemMap = new HashMap<>();
itemMap.put("id", AttributeValue.builder().s(order.id).build());
itemMap.put("quantity",
AttributeValue.builder().n(String.valueOf(order.quantity)).build());
itemMap.put("total",
AttributeValue.builder().n(String.valueOf(order.total)).build());
return DynamoDbWriteRequest.builder()
.setType(DynamoDbWriteRequestType.PUT)
.setItem(itemMap)
.build();
}
@Override
public void open(Sink.InitContext context) {
}
}
{code}
while this might not be too much of work, however it is a fairly common case in
Flink and this implementation requires some fair knowledge of DDB model for new
users.
h2. Proposal
Introduce {{ DynamoDbTypeInformedElementConverter}} as follows:
{code:title="TypeInformedElementconverter"}
public class DynamoDbTypeInformedElementConverter implements
ElementConverter {
DynamoDbTypeInformedElementConverter(CompositeType typeInfo);
public DynamoDbWriteRequest convertElement(input) {
switch this.typeInfo{
case: BasicTypeInfo.STRING_TYPE_INFO: return input ->
AttributeValue.fromS(o.toString())
case: BasicTypeInfo.SHORT_TYPE_INFO:
case: BasicTypeInfo.INTEGER_TYPE_INFO: input ->
AttributeValue.fromN(o.toString())
case: TupleTypeInfo: input -> AttributeValue.fromL(converTuple(input))
.
}
}
}
// User Code
public static void main(String []args) {
DynamoDbTypeInformedElementConverter elementConverter = new
DynamoDbTypeInformedElementConverter(TypeInformation.of(Order.class));
DdbSink.setElementConverter(elementConverter);
}
{code}
We will start by supporting all Pojo/ basic/ Tuple/ Array typeInfo which should
be enough to cover all DDB supported types (s,n,bool,b,ss,ns,bs,bools,m,l)
1-
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/dynamodb/model/AttributeValue.html
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] FLIP-438: Amazon SQS Sink Connector
Hi Dhingra, thanks for raising the FLIP I am in favour of this addition in general. I have a couple of comments/questions on the FLIP. - I am not sure why we need to suppress warnings in the sink example in the FLIP? - You provided the sink example as it is the Public Interface, however the actual AsyncSink logic is mostly in the writer, so would be helpful to provide a brief of the writer or the "submitRequestEntries" - I am not sure what customDimensions are or how are they going to be used by the client, (that's why I thought the writer example should be helpful). - Are we going to use the existing aws client providers to handle the authentication and async client creation similar to Kinesis/Firehose and DDB. I would strongly recommend we do. - Given you suggest implementing this in "flink-connector-aws" repo, it should probably follow the same versioning of AWS connectors, hence targeting 4.3.0. Also I am not sure why we are targeting support for 1.16 given that it is out of support and 4.2 supports 1.17+. - Are we planning to support Table API & SQL as well? This should not be much of an effort IMO so I think we should. - It would be great if we also added an implementation of Element converter given that SQS message bodies are mainly Strings if I remember correctly. We can extend it to other types for MessageAttributeValue augmentation, this should be more valuable on table API as well to use it as default Element Converter. I would love to assist with the implementation and reviews if this FLIP was accepted. Best Regards Ahmed Hamdy On Fri, 5 Apr 2024 at 19:19, Dhingra, Priya wrote: > Hi Dev, > > I would like to start a discussion about FLIP-438: Amazon SQS Sink > Connector< > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector>. > This FLIP is proposing to add support for AWS SQS sink in > flink-connector-aws repo. > > For more details, see FLIP-438. Looking forward to your feedback. > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-438%3A+Amazon+SQS+Sink+Connector > > Thanks, > Priya >
Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project
Congratulations! Best Regards Ahmed Hamdy On Thu, 28 Mar 2024 at 08:16, Paul Lam wrote: > Congrats! > > Best, > Paul Lam > > > 2024年3月28日 16:03，Rui Fan <1996fan...@gmail.com> 写道： > > > > Congratulations~ > > > > Best, > > Rui > > > > On Thu, Mar 28, 2024 at 3:55 PM Yu Li wrote: > > > >> CC the Flink user and dev mailing list. > >> > >> Paimon originated within the Flink community, initially known as Flink > >> Table Store, and all our incubating mentors are members of the Flink > >> Project Management Committee. I am confident that the bonds of > >> enduring friendship and close collaboration will continue to unite the > >> two communities. > >> > >> And congratulations all! > >> > >> Best Regards, > >> Yu > >> > >> On Wed, 27 Mar 2024 at 20:35, Guojun Li > wrote: > >>> > >>> Congratulations! > >>> > >>> Best, > >>> Guojun > >>> > >>> On Wed, Mar 27, 2024 at 5:24 PM wulin wrote: > >>> > >>>> Congratulations~ > >>>> > >>>>> 2024年3月27日 15:54，王刚 写道： > >>>>> > >>>>> Congratulations~ > >>>>> > >>>>>> 2024年3月26日 10:25，Jingsong Li 写道： > >>>>>> > >>>>>> Hi Paimon community, > >>>>>> > >>>>>> I’m glad to announce that the ASF board has approved a resolution to > >>>>>> graduate Paimon into a full Top Level Project. Thanks to everyone > >> for > >>>>>> your help to get to this point. > >>>>>> > >>>>>> I just created an issue to track the things we need to modify [2], > >>>>>> please comment on it if you feel that something is missing. You can > >>>>>> refer to apache documentation [1] too. > >>>>>> > >>>>>> And, we already completed the GitHub repo migration [3], please > >> update > >>>>>> your local git repo to track the new repo [4]. > >>>>>> > >>>>>> You can run the following command to complete the remote repo > >> tracking > >>>>>> migration. > >>>>>> > >>>>>> git remote set-url origin https://github.com/apache/paimon.git > >>>>>> > >>>>>> If you have a different name, please change the 'origin' to your > >> remote > >>>> name. > >>>>>> > >>>>>> Please join me in celebrating! > >>>>>> > >>>>>> [1] > >>>> > >> > https://incubator.apache.org/guides/transferring.html#life_after_graduation > >>>>>> [2] https://github.com/apache/paimon/issues/3091 > >>>>>> [3] https://issues.apache.org/jira/browse/INFRA-25630 > >>>>>> [4] https://github.com/apache/paimon > >>>>>> > >>>>>> Best, > >>>>>> Jingsong Lee > >>>> > >>>> > >> > >
Re: [DISCUSS] Migrate Kinesis Table API source to KDS client
Hi Aleks, Thanks for the valuable feedback, yeah I agree it might be confusing for some users. Best Regards Ahmed Hamdy On Sun, 24 Mar 2024 at 18:40, Aleksandr Pilipenko wrote: > Hi Ahmed, thank you for starting this discussion. > > In general, I am more inclined toward proposal 2 - having logic that > dynamically selects implementation under the hood might introduce > unexpected behaviors for users, considering that one of the options is an > experimental connector with an unstable API. > > Best regards, > Aleksandr Pilipenko > > On Sun, 24 Mar 2024 at 15:19, Ahmed Hamdy wrote: > > > Hi devs, > > With the delivery of FLINK-31813 > > <https://issues.apache.org/jira/browse/FLINK-31813>[1] that offers a new > > Datastream Kinesis Streams source paving the way to deprecate the KCL > > backed `FlinkKinesisConsumer`, I would like to start working on > migrating > > Kinesis Source Table API to the new source (FLINK-31987 > > <https://issues.apache.org/jira/browse/FLINK-31987>[2]) in a > > similar approach to what I did with the sink migration. This work however > > is blocked on migration of additional features like EFO support and > > deaggregation which seems to be idle for some time now. I want to discuss > > an alternative approach for migration ensuring backward compatibility. > > > > Proposal 1: Given the pre-existing dependency from > > `flink-connector-kinesis` to `flink-connector-aws-kinesis-streams` I > > suggest we implement a `KinesisStreamsDynamicSource` backed by the new > > `KinesisStreamsSource` and add a `KinesisSourceSelector` to the existing > > source factory to dynamically select a source according to features > enabled > > from the DDL, i.e shall the unsupported features like EFO or aggregation > be > > enabled the legacy dynamic source would be provided, otherwise the new > > dynamic source is to be provided. Once all other features are implemented > > we migrate the factory as well. > > Proposal 2: We do nothing and wait for all features to be implemented. > > Let me know your thoughts. > > > > 1- https://issues.apache.org/jira/browse/FLINK-31813 > > 2- https://issues.apache.org/jira/browse/FLINK-31987 > > Best Regards > > Ahmed Hamdy > > >
[DISCUSS] Migrate Kinesis Table API source to KDS client
Hi devs, With the delivery of FLINK-31813 <https://issues.apache.org/jira/browse/FLINK-31813>[1] that offers a new Datastream Kinesis Streams source paving the way to deprecate the KCL backed `FlinkKinesisConsumer`, I would like to start working on migrating Kinesis Source Table API to the new source (FLINK-31987 <https://issues.apache.org/jira/browse/FLINK-31987>[2]) in a similar approach to what I did with the sink migration. This work however is blocked on migration of additional features like EFO support and deaggregation which seems to be idle for some time now. I want to discuss an alternative approach for migration ensuring backward compatibility. Proposal 1: Given the pre-existing dependency from `flink-connector-kinesis` to `flink-connector-aws-kinesis-streams` I suggest we implement a `KinesisStreamsDynamicSource` backed by the new `KinesisStreamsSource` and add a `KinesisSourceSelector` to the existing source factory to dynamically select a source according to features enabled from the DDL, i.e shall the unsupported features like EFO or aggregation be enabled the legacy dynamic source would be provided, otherwise the new dynamic source is to be provided. Once all other features are implemented we migrate the factory as well. Proposal 2: We do nothing and wait for all features to be implemented. Let me know your thoughts. 1- https://issues.apache.org/jira/browse/FLINK-31813 2- https://issues.apache.org/jira/browse/FLINK-31987 Best Regards Ahmed Hamdy
Re: [VOTE] FLIP-433: State Access on DataStream API V2
+1 (non-binding) Best Regards Ahmed Hamdy On Thu, 21 Mar 2024 at 10:45, Jinzhong Li wrote: > +1 (non-binding) > > Best, > Jinzhong > > On Thu, Mar 21, 2024 at 6:15 PM Zakelly Lan wrote: > > > +1 non-binding > > > > > > Best, > > Zakelly > > > > On Thu, Mar 21, 2024 at 5:34 PM Gyula Fóra wrote: > > > > > +1 (binding) > > > > > > Gyula > > > > > > On Thu, Mar 21, 2024 at 3:33 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > > +1(binding) > > > > > > > > Thanks to Weijie for driving this proposal, which solves the problem > > > that I > > > > raised in FLIP-359. > > > > > > > > Best, > > > > Rui > > > > > > > > On Thu, Mar 21, 2024 at 10:10 AM Hangxiang Yu > > > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > On Thu, Mar 21, 2024 at 10:04 AM Xintong Song < > tonysong...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Best, > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 20, 2024 at 8:30 PM weijie guo < > > > guoweijieres...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > > > > Thanks for all the feedback about the FLIP-433: State Access on > > > > > > > DataStream API V2 [1]. The discussion thread is here [2]. > > > > > > > > > > > > > > > > > > > > > The vote will be open for at least 72 hours unless there is an > > > > > > > objection or insufficient votes. > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Weijie > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-433%3A+State+Access+on+DataStream+API+V2 > > > > > > > > > > > > > > [2] > > > https://lists.apache.org/thread/8ghzqkvt99p4k7hjqxzwhqny7zb7xrwo > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best, > > > > > Hangxiang. > > > > > > > > > > > > > > >
Re: [DISCUSS] Planning Flink 1.20
+1 for the proposed timeline and release managers. Best Regards Ahmed Hamdy On Fri, 22 Mar 2024 at 07:41, Xintong Song wrote: > +1 for the proposed timeline and Weijie & Rui as the release managers. > > I think it would be welcomed if another 1-2 volunteers join as the release > managers, but that's not a must. We used to have only 1-2 release managers > for each release, > > Best, > > Xintong > > > > On Fri, Mar 22, 2024 at 2:55 PM Jark Wu wrote: > > > Thanks for kicking this off. > > > > +1 for the volunteered release managers (Weijie Guo, Rui Fan) and the > > targeting date (feature freeze: June 15). > > > > Best, > > Jark > > > > > > > > > > > > On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com> wrote: > > > > > Thanks Leonard for this feedback and help! > > > > > > Best, > > > Rui > > > > > > On Fri, Mar 22, 2024 at 12:36 PM weijie guo > > > > wrote: > > > > > > > Thanks Leonard! > > > > > > > > > I'd like to help you if you need some help like permissions from > PMC > > > > side, please feel free to ping me. > > > > > > > > Nice to know. It'll help a lot! > > > > > > > > Best regards, > > > > > > > > Weijie > > > > > > > > > > > > Leonard Xu 于2024年3月22日周五 12:09写道： > > > > > > > >> +1 for the proposed release managers (Weijie Guo, Rui Fan), both the > > two > > > >> candidates are pretty active committers thus I believe they know the > > > >> community development process well. The recent releases have four > > > release > > > >> managers, and I am also looking forward to having other volunteers > > > >> join the management of Flink 1.20. > > > >> > > > >> +1 for targeting date (feature freeze: June 15, 2024), referring to > > the > > > >> release cycle of recent versions, release cycle of 4 months makes > > sense > > > to > > > >> me. > > > >> > > > >> > > > >> I'd like to help you if you need some help like permissions from PMC > > > >> side, please feel free to ping me. > > > >> > > > >> Best, > > > >> Leonard > > > >> > > > >> > > > >> > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道： > > > >> > > > > >> > Hi Weijie, > > > >> > > > > >> > Thanks for kicking off 1.20! I'd like to join you and participate > in > > > the > > > >> > 1.20 release. > > > >> > > > > >> > Best, > > > >> > Rui > > > >> > > > > >> > On Tue, Mar 19, 2024 at 5:30 PM weijie guo < > > guoweijieres...@gmail.com > > > > > > > >> > wrote: > > > >> > > > > >> >> Hi everyone, > > > >> >> > > > >> >> With the release announcement of Flink 1.19, it's a good time to > > kick > > > >> off > > > >> >> discussion of the next release 1.20. > > > >> >> > > > >> >> > > > >> >> - Release managers > > > >> >> > > > >> >> > > > >> >> I'd like to volunteer as one of the release managers this time. > It > > > has > > > >> been > > > >> >> good practice to have a team of release managers from different > > > >> >> backgrounds, so please raise you hand if you'd like to volunteer > > and > > > >> get > > > >> >> involved. > > > >> >> > > > >> >> > > > >> >> > > > >> >> - Timeline > > > >> >> > > > >> >> > > > >> >> Flink 1.19 has been released. With a target release cycle of 4 > > > months, > > > >> >> we propose a feature freeze date of *June 15, 2024*. > > > >> >> > > > >> >> > > > >> >> > > > >> >> - Collecting features > > > >> >> > > > >> >> > > > >> >> As usual, we've created a wiki page[1] for collecting new > features > > in > > > >> 1.20. > > > >> >> > > > >> >> > > > >> >> In addition, we already have a number of FLIPs that have been > voted > > > or > > > >> are > > > >> >> in the process, including pre-works for version 2.0. > > > >> >> > > > >> >> > > > >> >> In the meantime, the release management team will be finalized in > > the > > > >> next > > > >> >> few days, and we'll continue to create Jira Boards and Sync > > meetings > > > >> >> to make it easy > > > >> >> for everyone to get an overview and track progress. > > > >> >> > > > >> >> > > > >> >> > > > >> >> Best regards, > > > >> >> > > > >> >> Weijie > > > >> >> > > > >> >> > > > >> >> > > > >> >> [1] > https://cwiki.apache.org/confluence/display/FLINK/1.20+Release > > > >> >> > > > >> > > > >> > > > > > >
Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL
Hi everyone, +1 for this proposal, I believe it is very useful to the minimum, It would be great even having "ML_PREDICT" and "ML_EVALUATE" as built-in PTFs in this FLIP as discussed. IIUC this will be included in the 1.20 roadmap? Best Regards Ahmed Hamdy On Fri, 22 Mar 2024 at 23:54, Hao Li wrote: > Hi Timo and Jark, > > I agree Oracle's syntax seems concise and more descriptive. For the > built-in `ML_PREDICT` and `ML_EVALUATE` functions I agree with Jark we can > support them as built-in PTF using `SqlTableFunction` for this FLIP. We can > have a different FLIP discussing user defined PTF and adopt that later for > model functions later. To summarize, the current proposed syntax is > > SELECT f1, f2, label FROM TABLE(ML_PREDICT(TABLE `my_data`, > `classifier_model`, f1, f2)) > > SELECT * FROM TABLE(ML_EVALUATE(TABLE `eval_data`, `classifier_model`, f1, > f2)) > > Is `DESCRIPTOR` a must in the syntax? If so, it becomes > > SELECT f1, f2, label FROM TABLE(ML_PREDICT(TABLE `my_data`, > `classifier_model`, DESCRIPTOR(f1), DESCRIPTOR(f2))) > > SELECT * FROM TABLE(ML_EVALUATE(TABLE `eval_data`, `classifier_model`, > DESCRIPTOR(f1), DESCRIPTOR(f2))) > > If Calcite supports dropping outer table keyword, it becomes > > SELECT f1, f2, label FROM ML_PREDICT(TABLE `my_data`, `classifier_model`, > DESCRIPTOR(f1), DESCRIPTOR(f2)) > > SELECT * FROM ML_EVALUATE(TABLE `eval_data`, `classifier_model`, > DESCRIPTOR( > f1), DESCRIPTOR(f2)) > > Thanks, > Hao > > > > On Fri, Mar 22, 2024 at 9:16 AM Jark Wu wrote: > > > Sorry, I mean we can bump the Calcite version if needed in Flink 1.20. > > > > On Fri, 22 Mar 2024 at 22:19, Jark Wu wrote: > > > > > Hi Timo, > > > > > > Introducing user-defined PTF is very useful in Flink, I'm +1 for this. > > > But I think the ML model FLIP is not blocked by this, because we > > > can introduce ML_PREDICT and ML_EVALUATE as built-in PTFs > > > just like TUMBLE/HOP. And support user-defined ML functions as > > > a future FLIP. > > > > > > Regarding the simplified PTF syntax which reduces the outer TABLE() > > > keyword, > > > it seems it was just supported[1] by the Calcite community last month > and > > > will be > > > released in the next version (v1.37). The Calcite community is > preparing > > > the > > > 1.37 release, so we can bump the version if needed in Flink 1.19. > > > > > > Best, > > > Jark > > > > > > [1]: https://issues.apache.org/jira/browse/CALCITE-6254 > > > > > > On Fri, 22 Mar 2024 at 21:46, Timo Walther wrote: > > > > > >> Hi everyone, > > >> > > >> this is a very important change to the Flink SQL syntax but we can't > > >> wait until the SQL standard is ready for this. So I'm +1 on > introducing > > >> the MODEL concept as a first class citizen in Flink. > > >> > > >> For your information: Over the past months I have already spent a > > >> significant amount of time thinking about how we can introduce PTFs in > > >> Flink. I reserved FLIP-440[1] for this purpose and I will share a > > >> version of this in the next 1-2 weeks. > > >> > > >> For a good implementation of FLIP-440 and also FLIP-437, we should > > >> evolve the PTF syntax in collaboration with Apache Calcite. > > >> > > >> There are different syntax versions out there: > > >> > > >> 1) Flink > > >> > > >> SELECT * FROM > > >>TABLE(TUMBLE(TABLE Bid, DESCRIPTOR(bidtime), INTERVAL '10' > MINUTES)); > > >> > > >> 2) SQL standard > > >> > > >> SELECT * FROM > > >>TABLE(TUMBLE(TABLE(Bid), DESCRIPTOR(bidtime), INTERVAL '10' > > MINUTES)); > > >> > > >> 3) Oracle > > >> > > >> SELECT * FROM > > >> TUMBLE(Bid, COLUMNS(bidtime), INTERVAL '10' MINUTES)); > > >> > > >> As you can see above, Flink does not follow the standard correctly as > it > > >> would need to use `TABLE()` but this is not provided by Calcite yet. > > >> > > >> I really like the Oracle syntax[2][3] a lot. It reduces necessary > > >> keywords to a minimum. Personally, I would like to discuss this syntax > > >> in a separate FLIP and hope I will find supporters for: > > >> > > >> > > >> SELECT * FROM > > >>TUMBLE(TABLE Bid, DESCRIPTOR(bidtime), INTERVAL '10'
Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines
Hi Ron, Sorry for joining the discussion late, thanks for the effort. I think the base idea is great, however I have a couple of comments: - I want to iterate on Timo's comments regarding the confusion between "Dynamic Table" and current Flink "Table". Should the refactoring of the system happen in 2.0, should we rename it in this Flip ( as the suggestions in the thread ) and address the holistic changes in a separate Flip for 2.0? - I feel confused with how it is further with other components, the examples provided feel like a standalone ETL job, could you provide in the FLIP an example where the table is further used in subsequent queries (specially in batch mode). - I really like the standard of keeping the unified batch and streaming approach Best Regards Ahmed Hamdy On Fri, 22 Mar 2024 at 12:07, Lincoln Lee wrote: > Hi Timo, > > Thanks for your thoughtful inputs! > > Yes, expanding the MATERIALIZED VIEW(MV) could achieve the same function, > but our primary concern is that by using a view, we might limit future > opportunities > to optimize queries through automatic materialization rewriting [1], > leveraging > the support for MV by physical storage. This is because we would be > breaking > the intuitive semantics of a materialized view (a materialized view > represents > the result of a query) by allowing data modifications, thus losing the > potential > for such optimizations. > > With these considerations in mind, we were inspired by Google Looker's > Persistent > Derived Table [2]. PDT is designed for building Looker's automated > modeling, > aligning with our purpose for the stream-batch automatic pipeline. > Therefore, > we are considering another candidate, Derived Table, the term 'derive' > suggests a > query, and 'table' retains modifiability. This approach would not disrupt > our current > concept of a dynamic table, preserving the future utility of MVs. > > Conceptually, a Derived Table is a Dynamic Table + Continuous Query. By > introducing > a new concept Derived Table for this FLIP, this makes all concepts to play > together nicely. > > What do you think about this? > > [1] https://calcite.apache.org/docs/materialized_views.html > [2] > > https://cloud.google.com/looker/docs/derived-tables#persistent_derived_tables > > > Best, > Lincoln Lee > > > Timo Walther 于2024年3月22日周五 17:54写道： > > > Hi Ron, > > > > thanks for the detailed answer. Sorry, for my late reply, we had a > > conference that kept me busy. > > > > > In the current concept[1], it actually includes: Dynamic Tables & > > > & Continuous Query. Dynamic Table is just an abstract logical concept > > > > This explanation makes sense to me. But the docs also say "A continuous > > query is evaluated on the dynamic table yielding a new dynamic table.". > > So even our regular CREATE TABLEs are considered dynamic tables. This > > can also be seen in the diagram "Dynamic Table -> Continuous Query -> > > Dynamic Table". Currently, Flink queries can only be executed on Dynamic > > Tables. > > > > > In essence, a materialized view represents the result of a query. > > > > Isn't that what your proposal does as well? > > > > > the object of the suspend operation is the refresh task of the > > dynamic table > > > > I understand that Snowflake uses the term [1] to merge their concepts of > > STREAM, TASK, and TABLE into one piece of concept. But Flink has no > > concept of a "refresh task". Also, they already introduced MATERIALIZED > > VIEW. Flink is in the convenient position that the concept of > > materialized views is not taken (reserved maybe for exactly this use > > case?). And SQL standard concept could be "slightly adapted" to our > > needs. Looking at other vendors like Postgres[2], they also use > > `REFRESH` commands so why not adding additional commands such as DELETE > > or UPDATE. Oracle supports "ON PREBUILT TABLE clause tells the database > > to use an existing table segment"[3] which comes closer to what we want > > as well. > > > > > it is not intended to support data modification > > > > This is an argument that I understand. But we as Flink could allow data > > modifications. This way we are only extending the standard and don't > > introduce new concepts. > > > > If we can't agree on using MATERIALIZED VIEW concept. We should fix our > > syntax in a Flink 2.0 effort. Making regular tables bounded and dynamic > > tables unbounded. We would be closer to the SQL standard with this and > > pave the way for the
Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed
Congratulations, great work and great news. Best Regards Ahmed Hamdy On Thu, 21 Mar 2024 at 11:41, Benchao Li wrote: > Congratulations, and thanks for the great work! > > Yuan Mei 于2024年3月21日周四 18:31写道： > > > > Thanks for driving these efforts! > > > > Congratulations > > > > Best > > Yuan > > > > On Thu, Mar 21, 2024 at 4:35 PM Yu Li wrote: > > > > > Congratulations and look forward to its further development! > > > > > > Best Regards, > > > Yu > > > > > > On Thu, 21 Mar 2024 at 15:54, ConradJam wrote: > > > > > > > > Congrattulations! > > > > > > > > Leonard Xu 于2024年3月20日周三 21:36写道： > > > > > > > > > Hi devs and users, > > > > > > > > > > We are thrilled to announce that the donation of Flink CDC as a > > > > > sub-project of Apache Flink has completed. We invite you to explore > > > the new > > > > > resources available: > > > > > > > > > > - GitHub Repository: https://github.com/apache/flink-cdc > > > > > - Flink CDC Documentation: > > > > > https://nightlies.apache.org/flink/flink-cdc-docs-stable > > > > > > > > > > After Flink community accepted this donation[1], we have completed > > > > > software copyright signing, code repo migration, code cleanup, > website > > > > > migration, CI migration and github issues migration etc. > > > > > Here I am particularly grateful to Hang Ruan, Zhongqaing Gong, > > > Qingsheng > > > > > Ren, Jiabao Sun, LvYanquan, loserwang1024 and other contributors > for > > > their > > > > > contributions and help during this process！ > > > > > > > > > > > > > > > For all previous contributors: The contribution process has > slightly > > > > > changed to align with the main Flink project. To report bugs or > > > suggest new > > > > > features, please open tickets > > > > > Apache Jira (https://issues.apache.org/jira). Note that we will > no > > > > > longer accept GitHub issues for these purposes. > > > > > > > > > > > > > > > Welcome to explore the new repository and documentation. Your > feedback > > > and > > > > > contributions are invaluable as we continue to improve Flink CDC. > > > > > > > > > > Thanks everyone for your support and happy exploring Flink CDC! > > > > > > > > > > Best， > > > > > Leonard > > > > > [1] > https://lists.apache.org/thread/cw29fhsp99243yfo95xrkw82s5s418ob > > > > > > > > > > > > > > > > > > -- > > > > Best > > > > > > > > ConradJam > > > > > > > -- > > Best, > Benchao Li >
Re: [ANNOUNCE] Apache Flink 1.19.0 released
Congratulations! Best Regards Ahmed Hamdy On Mon, 18 Mar 2024 at 12:30, Xintong Song wrote: > Congratulations~! > > Best, > > Xintong > > > > On Mon, Mar 18, 2024 at 7:02 PM Feng Jin wrote: > > > Congratulations! > > > > Best, > > Feng > > > > On Mon, Mar 18, 2024 at 6:18 PM Yuepeng Pan > wrote: > > > > > Congratulations! > > > > > > > > > Thanks to release managers and everyone involved. > > > > > > > > > > > > > > > Best, > > > Yuepeng Pan > > > > > > > > > > > > > > > > > > > > > > > > > > > At 2024-03-18 18:09:45, "Yubin Li" wrote: > > > >Congratulations! > > > > > > > >Thanks to release managers and everyone involved. > > > > > > > >On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu > > wrote: > > > >> > > > >> Congratulations! > > > >> Thanks release managers and all involved! > > > >> > > > >> On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan > > > wrote: > > > >> > > > >> > Congratulations! > > > >> > > > > >> > Best, > > > >> > Hang > > > >> > > > > >> > Paul Lam 于2024年3月18日周一 17:18写道： > > > >> > > > > >> > > Congrats! Thanks to everyone involved! > > > >> > > > > > >> > > Best, > > > >> > > Paul Lam > > > >> > > > > > >> > > > 2024年3月18日 16:37，Samrat Deb 写道： > > > >> > > > > > > >> > > > Congratulations ! > > > >> > > > > > > >> > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li < > > > jingsongl...@gmail.com> > > > >> > > wrote: > > > >> > > > > > > >> > > >> Congratulations! > > > >> > > >> > > > >> > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan < > 1996fan...@gmail.com> > > > wrote: > > > >> > > >>> > > > >> > > >>> Congratulations, thanks for the great work! > > > >> > > >>> > > > >> > > >>> Best, > > > >> > > >>> Rui > > > >> > > >>> > > > >> > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee < > > > lincoln.8...@gmail.com> > > > >> > > >> wrote: > > > >> > > >>>> > > > >> > > >>>> The Apache Flink community is very happy to announce the > > > release of > > > >> > > >> Apache Flink 1.19.0, which is the fisrt release for the > Apache > > > Flink > > > >> > > 1.19 > > > >> > > >> series. > > > >> > > >>>> > > > >> > > >>>> Apache Flink® is an open-source stream processing framework > > for > > > >> > > >> distributed, high-performing, always-available, and accurate > > data > > > >> > > streaming > > > >> > > >> applications. > > > >> > > >>>> > > > >> > > >>>> The release is available for download at: > > > >> > > >>>> https://flink.apache.org/downloads.html > > > >> > > >>>> > > > >> > > >>>> Please check out the release blog post for an overview of > the > > > >> > > >> improvements for this bugfix release: > > > >> > > >>>> > > > >> > > >> > > > >> > > > > > >> > > > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > >> > > >>>> > > > >> > > >>>> The full release notes are available in Jira: > > > >> > > >>>> > > > >> > > >> > > > >> > > > > > >> > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > >> > > >>>> > > > >> > > >>>> We would like to thank all contributors of the Apache Flink > > > >> > community > > > >> > > >> who made this release possible! > > > >> > > >>>> > > > >> > > >>>> > > > >> > > >>>> Best, > > > >> > > >>>> Yun, Jing, Martijn and Lincoln > > > >> > > >> > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> -- > > > >> Best, > > > >> Hangxiang. > > > > > >
Re: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes
Hi Jinzhong, Thanks for driving this, +1 (non-binding) Best Regards Ahmed Hamdy On Wed, 13 Mar 2024 at 01:33, Jing Ge wrote: > +1 (binding) Thanks! > > Best regards, > Jing > > On Sun, Mar 10, 2024 at 5:32 PM Jing Ge wrote: > > > Hi Jinzhong, > > > > Thanks for driving this topic and sorry for just joining the discussion > > now. I replied in your discussion thread. Would you like to take a look > > and let's keep the discussion there? I will come back to this thread and > > vote once the discussion is done. Thanks! > > > > Best regards, > > Jing > > > > On Thu, Mar 7, 2024 at 4:39 AM Zakelly Lan > wrote: > > > >> +1 non-binding > >> > >> Thanks for proposing this. > >> > >> Best, > >> Zakelly > >> > >> On Thu, Mar 7, 2024 at 10:13 AM Yanfei Lei wrote: > >> > >> > +1(binding) for this vote. > >> > > >> > Hangxiang Yu 于2024年3月7日周四 09:54写道： > >> > > > >> > > +1 (binding) > >> > > > >> > > On Thu, Mar 7, 2024 at 9:34 AM Yun Tang wrote: > >> > > > >> > > > > +1 for this FLIP. > >> > > > Sorry for not being clear in my previous reply, it's a binding > vote. > >> > > > > >> > > > Best > >> > > > Yun Tang > >> > > > > >> > > > From: Jeyhun Karimov > >> > > > Sent: Thursday, March 7, 2024 4:40 > >> > > > To: dev@flink.apache.org > >> > > > Subject: Re: [VOTE] FLIP-420: Add API annotations for RocksDB > >> > StateBackend > >> > > > user-facing classes > >> > > > > >> > > > Hi Jinzhong, > >> > > > > >> > > > Thanks for the FLIP. > >> > > > > >> > > > +1 (non-binding) > >> > > > > >> > > > Regards, > >> > > > Jeyhun > >> > > > > >> > > > On Wed, Mar 6, 2024 at 5:09 PM Yun Tang wrote: > >> > > > > >> > > > > +1 for this FLIP. > >> > > > > > >> > > > > > >> > > > > > >> > > > > Best > >> > > > > Yun Tang > >> > > > > > >> > > > > From: Jinzhong Li > >> > > > > Sent: Wednesday, March 6, 2024 20:29 > >> > > > > To: dev@flink.apache.org > >> > > > > Subject: [VOTE] FLIP-420: Add API annotations for RocksDB > >> > StateBackend > >> > > > > user-facing classes > >> > > > > > >> > > > > Hi All, > >> > > > > > >> > > > > I'd like to start a vote on the FLIP-420: Add API annotations > for > >> > RocksDB > >> > > > > StateBackend user-facing classes[1]. > >> > > > > The discussion thread is here [2]. > >> > > > > > >> > > > > The vote will be open for at least 72 hours unless there is an > >> > objection > >> > > > or > >> > > > > not enough votes. > >> > > > > > >> > > > > > >> > > > > [1]https://cwiki.apache.org/confluence/x/JQs4EQ > >> > > > > [2] > >> https://lists.apache.org/thread/4t71lz2j2ft8hf90ylvtomynhr2qthoo > >> > > > > > >> > > > > > >> > > > > Best, > >> > > > > Jinzhong Li > >> > > > > > >> > > > > >> > > > >> > > > >> > > -- > >> > > Best, > >> > > Hangxiang. > >> > > >> > > >> > > >> > -- > >> > Best, > >> > Yanfei > >> > > >> > > >
Re: [DISCUSS] FLIP-399: Flink Connector Doris
Hello, Thanks for the proposal, +1 for the FLIP. Best Regards Ahmed Hamdy On Mon, 11 Mar 2024 at 15:12, wudi <676366...@qq.com.invalid> wrote: > Hi, Leonard > Thank you for your suggestion. > I referred to other Connectors[1], modified the naming and types of > relevant parameters[2], and also updated FLIP. > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/overview/ > [1] > https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisConfigOptions.java > > Brs, > di.wu > > > 2024年3月7日 14:33，Leonard Xu 写道： > > > > Thanks wudi for the updating, the FLIP generally looks good to me, I > only left two minor suggestions: > > > > (1) The suffix `.s` in configoption doris.request.query.timeout.s looks > strange to me, could we change all time interval related option value type > to Duration ? > > > > (2) Could you check and improve all config options like > `doris.exec.mem.limit` to make them to follow flink config option naming > and value type? > > > > Best, > > Leonard > > > > > >> > >> > >>> 2024年3月6日 06:12，Jing Ge 写道： > >>> > >>> Hi Di, > >>> > >>> Thanks for your proposal. +1 for the contribution. I'd like to know > your > >>> thoughts about the following questions: > >>> > >>> 1. According to your clarification of the exactly-once, thanks for it > BTW, > >>> no PreCommitTopology is required. Does it make sense to let > DorisSink[1] > >>> implement SupportsCommitter, since the TwoPhaseCommittingSink is > >>> deprecated[2] before turning the Doris connector into a Flink > connector? > >>> 2. OLAP engines are commonly used as the tail/downstream of a data > pipeline > >>> to support further e.g. ad-hoc query or cube with feasible > pre-aggregation. > >>> Just out of curiosity, would you like to share some real use cases that > >>> will use OLAP engines as the source of a streaming data pipeline? Or it > >>> will only be used as the source for the batch? > >>> 3. The E2E test only covered sink[3], if I am not mistaken. Would you > like > >>> to test the source in E2E too? > >>> > >>> [1] > >>> > https://github.com/apache/doris-flink-connector/blob/43e0e5cf9b832854ea228fb093077872e3a311b6/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/DorisSink.java#L55 > >>> [2] > >>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Enhance+and+synchronize+Sink+API+to+match+the+Source+API > >>> [3] > >>> > https://github.com/apache/doris-flink-connector/blob/43e0e5cf9b832854ea228fb093077872e3a311b6/flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/MySQLDorisE2ECase.java#L96 > >>> > >>> Best regards, > >>> Jing > >>> > >>> On Tue, Mar 5, 2024 at 11:18 AM wudi <676366...@qq.com.invalid> wrote: > >>> > >>>> Hi, Jeyhun Karimov. > >>>> Thanks for your question. > >>>> > >>>> - How to ensure Exactly-Once? > >>>> 1. When the Checkpoint Barrier arrives, DorisSink will trigger the > >>>> precommit api of StreamLoad to complete the persistence of data in > Doris > >>>> (the data will not be visible at this time), and will also pass this > TxnID > >>>> to the Committer. > >>>> 2. When this Checkpoint of the entire Job is completed, the Committer > will > >>>> call the commit api of StreamLoad and commit TxnID to complete the > >>>> visibility of the transaction. > >>>> 3. When the task is restarted, the Txn with successful precommit and > >>>> failed commit will be aborted based on the label-prefix, and Doris' > abort > >>>> API will be called. (At the same time, Doris will also abort > transactions > >>>> that have not been committed for a long time) > >>>> > >>>> ps: At the same time, this part of the content has been updated in > FLIP > >>>> > >>>> - Because the default table model in Doris is Duplicate ( > >>>> https://doris.apache.org/docs/data-table/data-model/), which does not > >>>> have a primary key, batch writing may cause data duplication, but > UNIQ The > >>>> model has a primary key, which ensures the idempotence of writing, > thus > >>>> achievin
Re: [VOTE] Release 1.19.0, release candidate #2
Hi Lincoln +1 (non-binding) from me - Verified Checksums & Signatures - Verified Source dists don't contain binaries - Built source successfully - reviewed web PR Best Regards Ahmed Hamdy On Mon, 11 Mar 2024 at 15:18, Lincoln Lee wrote: > Hi Robin, > > Thanks for helping verifying the release note[1], FLINK-14879 should not > have been included, after confirming this > I moved all unresolved non-blocker issues left over from 1.19.0 to 1.20.0 > and reconfigured the release note [1]. > > Best, > Lincoln Lee > > [1] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > Robin Moffatt 于2024年3月11日周一 19:36写道： > > > Looking at the release notes [1] it lists `DESCRIBE DATABASE` > (FLINK-14879) > > and `DESCRIBE CATALOG` (FLINK-14690). > > When I try these in 1.19 RC2 the behaviour is as in 1.18.1, i.e. it is > not > > supported: > > > > ``` > > [INFO] Execute statement succeed. > > > > Flink SQL> show catalogs; > > +-+ > > |catalog name | > > +-+ > > | c_new | > > | default_catalog | > > +-+ > > 2 rows in set > > > > Flink SQL> DESCRIBE CATALOG c_new; > > [ERROR] Could not execute SQL statement. Reason: > > org.apache.calcite.sql.validate.SqlValidatorException: Column 'c_new' not > > found in any table > > > > Flink SQL> show databases; > > +--+ > > |database name | > > +--+ > > | default_database | > > +--+ > > 1 row in set > > > > Flink SQL> DESCRIBE DATABASE default_database; > > [ERROR] Could not execute SQL statement. Reason: > > org.apache.calcite.sql.validate.SqlValidatorException: Column > > 'default_database' not found in > > any table > > ``` > > > > Is this an error in the release notes, or my mistake in interpreting > them? > > > > thanks, Robin. > > > > > > [1] > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > > On Thu, 7 Mar 2024 at 10:01, Lincoln Lee wrote: > > > > > Hi everyone, > > > > > > Please review and vote on the release candidate #2 for the version > > 1.19.0, > > > as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > The complete staging area is available for your review, which includes: > > > > > > * JIRA release notes [1], and the pull request adding release note for > > > users [2] > > > * the official Apache source release and binary convenience releases to > > be > > > deployed to dist.apache.org [3], which are signed with the key with > > > fingerprint E57D30ABEE75CA06 [4], > > > * all artifacts to be deployed to the Maven Central Repository [5], > > > * source code tag "release-1.19.0-rc2" [6], > > > * website pull request listing the new release and adding announcement > > blog > > > post [7]. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > [1] > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353282 > > > [2] https://github.com/apache/flink/pull/24394 > > > [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.19.0-rc2/ > > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [5] > > https://repository.apache.org/content/repositories/orgapacheflink-1709 > > > [6] https://github.com/apache/flink/releases/tag/release-1.19.0-rc2 > > > [7] https://github.com/apache/flink-web/pull/721 > > > > > > > > > Best, > > > Yun, Jing, Martijn and Lincoln > > > > > >
Re: [DISCUSS] Externalized Python Connector Release/Dependency Process
Hi Danny +1 (non-binding) Best Regards Ahmed Hamdy On Tue, 9 Jan 2024 at 11:59, Xingbo Huang wrote: > Hi Danny, > > +1 > > Thanks a lot for investigating this. Let me share the current code > management and release situation of pyflink here. I hope it will be helpful > to you. > > Since Flink 1.13, release managers need to release two python packages to > pypi, apache-flink[1] and apache-flink-libraries[2]. apache-flink contains > all pyflink python code and apache-flink-libraries contains the jar package > corresponding to flink binary. The reason why the content of > apache-flink-libaries is not put into apache-flink is because starting from > Flink 1.11, pyflink provides different python versions and platform wheel > packages. If all wheel packages contain these jar packages, the space of > apache-flink will quickly increase, but pypi's project space is limited(I > applied to expand apache-flink twice), so in 1.13, I move the corresponding > jar package from apache-flink to an independent apache-flink-libraries > package. > > Since each apache-connector wheel package today is shared by the platform, > we do not need to publish a corresponding apache-libraries package for a > connector currently, but we still need to package the corresponding jar of > the connector into the corresponding apche-connector pypi package. > > [1] https://pypi.org/project/apache-flink/ > [2] https://pypi.org/project/apache-flink-libraries/ > > Best, > Xingbo > > Leonard Xu 于2024年1月9日周二 13:46写道： > > > +1 > > > > Thanks Danny for driving this. > > > > Best, > > Leonard > > > > > > > 2024年1月9日 上午2:01，Márton Balassi 写道： > > > > > > +1 > > > > > > Thanks, Danny - I really appreciate you taking the time for the > in-depth > > > investigation. Please proceed, looking forward to your experience. > > > > > > On Mon, Jan 8, 2024 at 6:04 PM Martijn Visser < > martijnvis...@apache.org> > > > wrote: > > > > > >> Thanks for investigating Danny. It looks like the best direction to go > > to > > >> :) > > >> > > >> On Mon, Jan 8, 2024 at 5:56 PM Péter Váry < > peter.vary.apa...@gmail.com> > > >> wrote: > > >>> > > >>> Thanks Danny for working on this! > > >>> > > >>> It would be good to do this in a way that the different connectors > > could > > >>> reuse as much code as possible, so if possible put most of the code > to > > >> the > > >>> flink connector shared utils repo [1] > > >>> > > >>> +1 from for the general direction (non-binding) > > >>> > > >>> Thanks, > > >>> Peter > > >>> > > >>> [1] https://github.com/apache/flink-connector-shared-utils > > >>> > > >>> > > >>> Danny Cranmer ezt írta (időpont: 2024. > jan. > > >> 8., > > >>> H, 17:31): > > >>> > > >>>> Hello all, > > >>>> > > >>>> I have been working with Péter and Marton on externalizing python > > >>>> connectors [1] from the main repo to the connector repositories. We > > >> have > > >>>> the code moved and the CI running tests for Kafka and AWS > Connectors. > > >> I am > > >>>> now looking into the release process. > > >>>> > > >>>> When we undertake a Flink release we perform the following steps > [2], > > >>>> regarding Python: 1/ run python build on CI, 2/ download Wheels > > >> artifacts, > > >>>> 3/ upload artifacts to the dist and 4/ deploy to pypi. The plan is > to > > >>>> follow the same steps for connectors, using Github actions instead > of > > >> Azure > > >>>> pipeline. > > >>>> > > >>>> Today we have a single pypi project for pyflink that contains all > the > > >> Flink > > >>>> libs, apache-flink [3]. I propose we create a new pypi project per > > >>>> connector using the existing connector version, and following naming > > >>>> convention: apache-, for example: > > >>>> apache-flink-connector-aws, apache-flink-connector-kafka. Therefore > to > > >> use > > >>>> a DataStream API connector in python, users would need to first > > >> install the > > >>>> lib, for example "python -m pip install apache-flink-connector-aws". > > >>>> > > >>>> Once we have consensus I will update the release process and > perform a > > >>>> release of the flink-connector-aws project to test it end-to-end. I > > >> look > > >>>> forward to any feedback. > > >>>> > > >>>> Thanks, > > >>>> Danny > > >>>> > > >>>> [1] https://issues.apache.org/jira/browse/FLINK-33528 > > >>>> [2] > > >>>> > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release > > >>>> [3] https://pypi.org/project/apache-flink/ > > >>>> > > >> > > > > >
Re: [VOTE] Release flink-connector-aws, v4.2.0 release candidate #1
Hi Danny +1 (non-binding) - Verified signatures and checksums - verified no binaries exists in archive - built source - reviewed web PR - Ran E2E example with Kinesis, Firehose & DynamoDB datastream connectors. Best Regards Ahmed Hamdy On Fri, 24 Nov 2023 at 08:38, Martijn Visser wrote: > Hi Danny, > > Thanks for driving this. > > +1 (binding) > > - Validated hashes > - Verified signature > - Verified that no binaries exist in the source archive > - Build the source with Maven > - Verified licenses > - Verified web PRs > > On Mon, Nov 20, 2023 at 12:29 PM Jiabao Sun > wrote: > > > > Thanks Danny for driving the release, > > > > +1 (non-binding) > > > > - built from source code succeeded > > - verified signatures > > - verified hashsums > > - checked release notes > > > > Best, > > Jiabao > > > > > > > 2023年11月20日 19:11，Danny Cranmer 写道： > > > > > > Hello all, > > > > > > +1 (binding). > > > > > > - Release notes look good > > > - Signatures and checksums match > > > - There are no binaries in the source archive > > > - pom versions are correct > > > - Tag is present in Github > > > - CI passes against FLink 1.17 and 1.18 > > > - Source build and tests pass > > > > > > Thanks, > > > Danny > > > > > > On Wed, Nov 1, 2023 at 1:15 AM mystic lama > wrote: > > > > > >> +1 (non-binding) > > >> > > >> - validated shasum > > >> - verified build > > >> - Java 8 - build good and all test cases pass > > >> - Java 11 - build good and all test cases pass > > >> > > >> Observations: got test failures with Java 17, something to look for in > > >> future > > >> > > >> On Tue, 31 Oct 2023 at 08:42, Danny Cranmer > > >> wrote: > > >> > > >>> Hi everyone, > > >>> > > >>> Please review and vote on release candidate #1 for the version > 4.2.0, as > > >>> follows: > > >>> [ ] +1, Approve the release > > >>> [ ] -1, Do not approve the release (please provide specific comments) > > >>> > > >>> The complete staging area is available for your review, which > includes: > > >>> * JIRA release notes [1], > > >>> * the official Apache source release to be deployed to > dist.apache.org > > >>> [2], > > >>> which are signed with the key with fingerprint > > >>> 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3], > > >>> * all artifacts to be deployed to the Maven Central Repository [4], > > >>> * source code tag v4.2.0-rc1 [5], > > >>> * website pull request listing the new release [6]. > > >>> * A link to the CI run on the release tag [7] > > >>> > > >>> The vote will be open for at least 72 hours. It is adopted by > majority > > >>> approval, with at least 3 PMC affirmative votes. > > >>> > > >>> Thanks, > > >>> Danny > > >>> > > >>> [1] > > >>> > > >>> > > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353011 > > >>> [2] > > >>> > > >> > https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-4.2.0-rc1 > > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > >>> [4] > > >>> > https://repository.apache.org/content/repositories/orgapacheflink-1665/ > > >>> [5] > > >> https://github.com/apache/flink-connector-aws/releases/tag/v4.2.0-rc1 > > >>> [6] https://github.com/apache/flink-web/pull/693 > > >>> [7] > > >> https://github.com/apache/flink-connector-aws/actions/runs/6707962074 > > >>> > > >> > > >
Re: [VOTE] Release flink-connector-gcp-pubsub v3.0.2, release candidate #1
+1(non-binding) -Checked release notes - Verified checksums and signatures - Verified no binaries in release - Build connector from source - Reviewed web PR. Best Regards Ahmed Hamdy On Fri, 3 Nov 2023 at 11:23, Danny Cranmer wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the version 3.0.2 of > flink-connector-pubsub, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], > which are signed with the key with fingerprint > 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v3.0.2-rc1 [5], > * website pull request listing the new release [6]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Danny > > [1] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353144 > [2] > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-gcp-pubsub-3.0.2-rc1 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] > https://repository.apache.org/content/repositories/orgapacheflink-1667/ > [5] https://github.com/apache/flink-connector-gcp-pubsub/tree/v3.0.2-rc1 > [6] https://github.com/apache/flink-web/pull/695 >
Re: [VOTE] Release flink-connector-opensearch v1.1.0, release candidate #1
Hi Danny, +1 (non-binding) - Checked release notes - Verified Checksums and Signature - Checked no binaries in release. - Reviewed web PR - Build Connector from source. Best Regards Ahmed Hamdy On Fri, 3 Nov 2023 at 13:05, Andriy Redko wrote: > +1 (non-binding), thanks a lot Danny! > > Best Regards, > Andriy Redko > > > Hi everyone, > > > Please review and vote on the release candidate #1 for the version 1.1.0 > of > > flink-connector-opensearch, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > [2], > > which are signed with the key with fingerprint > > 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag v1.1.0-rc1 [5], > > * website pull request listing the new release [6]. > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > Thanks, > > Danny > > > [1] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353141 > > [2] > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-1.1.0-rc1/ > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > https://repository.apache.org/content/repositories/orgapacheflink-1666/ > > [5] https://github.com/apache/flink-connector-opensearch/tree/v1.1.0-rc1 > > [6] https://github.com/apache/flink-web/pull/694 > >
Re: [VOTE] Apache Flink Kafka connector version 3.0.1, RC1
+1 (non-binding) - Verified Singatures - Verified Checksum - Build source successfully - Checked release tag exists - Reviewed the web PR Best Regards Ahmed Hamdy On Sun, 29 Oct 2023 at 08:02, Leonard Xu wrote: > +1 (binding) > > - Verified signatures > - Verified hashsums > - Checked Github release tag > - Built from source code succeeded > - Checked release notes > - Reviewed the web PR > > Best, > Leonard > > > > 2023年10月29日 上午11:34，mystic lama 写道： > > > > +1 (non-binding) > > > > - verified signatures > > - build with Java 8 and Java 11 - build success > > > > Minor observation > > - RAT check flagged that README.md is missing ASL > > > > On Fri, 27 Oct 2023 at 23:40, Xianxun Ye > wrote: > > > >> +1(non-binding) > >> > >> - Started a local Flink 1.18 cluster, read and wrote with Kafka and > Upsert > >> Kafka connector successfully to Kafka 2.2 cluster > >> > >> One minor question: should we update the dependency manual of these two > >> documentations[1][2]? > >> > >> [1] > >> > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/#dependencies > >> [2] > >> > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/upsert-kafka/#dependencies > >> > >> Best regards, > >> Xianxun > >> > >>> 2023年10月26日 16:12，Martijn Visser 写道： > >>> > >>> +1 (binding) > >>> > >>> - Validated hashes > >>> - Verified signature > >>> - Verified that no binaries exist in the source archive > >>> - Build the source with Maven via mvn clean install > >>> -Pcheck-convergence -Dflink.version=1.18.0 > >>> - Verified licenses > >>> - Verified web PR > >>> - Started a cluster and the Flink SQL client, successfully read and > >>> wrote with the Kafka connector to Confluent Cloud with AVRO and Schema > >>> Registry enabled > >>> > >>> On Thu, Oct 26, 2023 at 5:09 AM Qingsheng Ren > wrote: > >>>> > >>>> +1 (binding) > >>>> > >>>> - Verified signature and checksum > >>>> - Verified that no binary exists in the source archive > >>>> - Built from source with Java 8 using -Dflink.version=1.18 > >>>> - Started a local Flink 1.18 cluster, submitted jobs with SQL client > >>>> reading from and writing (with exactly-once) to Kafka 3.2.3 cluster > >>>> - Nothing suspicious in LICENSE and NOTICE file > >>>> - Reviewed web PR > >>>> > >>>> Thanks for the effort, Gordon! > >>>> > >>>> Best, > >>>> Qingsheng > >>>> > >>>> On Thu, Oct 26, 2023 at 5:13 AM Tzu-Li (Gordon) Tai < > >> tzuli...@apache.org> > >>>> wrote: > >>>> > >>>>> Hi everyone, > >>>>> > >>>>> Please review and vote on release candidate #1 for version 3.0.1 of > the > >>>>> Apache Flink Kafka Connector, as follows: > >>>>> [ ] +1, Approve the release > >>>>> [ ] -1, Do not approve the release (please provide specific comments) > >>>>> > >>>>> This release contains important changes for the following: > >>>>> - Supports Flink 1.18.x series > >>>>> - [FLINK-28303] EOS violation when using LATEST_OFFSETS startup mode > >>>>> - [FLINK-33231] Memory leak causing OOM when there are no offsets to > >> commit > >>>>> back to Kafka > >>>>> - [FLINK-28758] FlinkKafkaConsumer fails to stop with savepoint > >>>>> > >>>>> The release candidate contains the source release as well as JAR > >> artifacts > >>>>> to be released to Maven, built against Flink 1.17.1 and 1.18.0. > >>>>> > >>>>> The complete staging area is available for your review, which > includes: > >>>>> * JIRA release notes [1], > >>>>> * the official Apache source release to be deployed to > dist.apache.org > >>>>> [2], > >>>>> which are signed with the key with fingerprint > >>>>> 1C1E2394D3194E1944613488F320986D35C33D6A [3], > >>>>> * all artifacts to be deployed to the Maven Central Repository [4], > >>>>> * source code tag v3.0.1-rc1 [5], > >>>>> * website pull request listing the new release [6]. > >>>>> > >>>>> The vote will be open for at least 72 hours. It is adopted by > majority > >>>>> approval, with at least 3 PMC affirmative votes. > >>>>> > >>>>> Thanks, > >>>>> Gordon > >>>>> > >>>>> [1] > >>>>> > >>>>> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352910 > >>>>> [2] > >>>>> > >>>>> > >> > https://dist.apache.org/repos/dist/dev/flink/flink-connector-kafka-3.0.1-rc1/ > >>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > >>>>> [4] > >> https://repository.apache.org/content/repositories/orgapacheflink-1664 > >>>>> [5] > https://github.com/apache/flink-connector-kafka/commits/v3.0.1-rc1 > >>>>> [6] https://github.com/apache/flink-web/pull/692 > >>>>> > >> > >> > >
Re: [VOTE] FLIP-239: Port JDBC Connector to FLIP-27-143
+1(non-binding) Regards Ahmed On Fri, 13 Oct 2023, 10:49 Qingsheng Ren, wrote: > +1 (binding) > > Thanks for the work, Joao! > > Best, > Qingsheng > > > On Oct 12, 2023, at 19:32, Leonard Xu wrote: > > > > Thanks Joao for driving this work. > > > > +1 (binding) > > > > Best, > > Leonard > > > >> On Oct 12, 2023, at 7:09 PM, Sergey Nuyanzin > wrote: > >> > >> +1 (binding) > > > >
Re: [VOTE] FLIP-367: Support Setting Parallelism for Table/SQL Sources
+1 (non-binding) Best Regards Ahmed Hamdy On Mon, 9 Oct 2023 at 09:38, xiangyu feng wrote: > +1 (non-binding) > > Feng Jin 于2023年10月9日周一 16:00写道： > > > +1 (non-binding) > > > > Best, > > Feng > > > > On Mon, Oct 9, 2023 at 3:12 PM Yangze Guo wrote: > > > > > +1 (binding) > > > > > > Best, > > > Yangze Guo > > > > > > On Mon, Oct 9, 2023 at 2:46 PM Yun Tang wrote: > > > > > > > > +1 (binding) > > > > > > > > Best > > > > Yun Tang > > > > > > > > From: Weihua Hu > > > > Sent: Monday, October 9, 2023 12:03 > > > > To: dev@flink.apache.org > > > > Subject: Re: [VOTE] FLIP-367: Support Setting Parallelism for > Table/SQL > > > Sources > > > > > > > > +1 (binding) > > > > > > > > Best, > > > > Weihua > > > > > > > > > > > > On Mon, Oct 9, 2023 at 11:47 AM Shammon FY > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > On Mon, Oct 9, 2023 at 11:12 AM Benchao Li > > > wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Zhanghao Chen 于2023年10月9日周一 10:20写道： > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Thanks for all the feedback on FLIP-367: Support Setting > > > Parallelism > > > > > for > > > > > > Table/SQL Sources [1][2]. > > > > > > > > > > > > > > I'd like to start a vote for FLIP-367. The vote will be open > > until > > > Oct > > > > > > 12th 12:00 PM GMT) unless there is an objection or insufficient > > > votes. > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150 > > > > > > > [2] > > > https://lists.apache.org/thread/gtpswl42jzv0c9o3clwqskpllnw8rh87 > > > > > > > > > > > > > > Best, > > > > > > > Zhanghao Chen > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Best, > > > > > > Benchao Li > > > > > > > > > > > > > > > > >
Re: [VOTE] FLIP-327: Support switching from batch to stream mode to improve throughput when processing backlog data
+1(non binding) Best regards Ahmed Hamdy On Mon, 25 Sep 2023, 19:57 Venkatakrishnan Sowrirajan, wrote: > +1 (non-binding) > > On Sun, Sep 24, 2023, 6:49 PM Xintong Song wrote: > > > +1 (binding) > > > > Best, > > > > Xintong > > > > > > > > On Sat, Sep 23, 2023 at 10:16 PM Yuepeng Pan > > wrote: > > > > > +1(non-binding), thank you for driving this proposal. > > > > > > Best, > > > Yuepeng Pan. > > > At 2023-09-22 14:07:45, "Dong Lin" wrote: > > > >Hi all, > > > > > > > >We would like to start the vote for FLIP-327: Support switching from > > batch > > > >to stream mode to improve throughput when processing backlog data [1]. > > > This > > > >FLIP was discussed in this thread [2]. > > > > > > > >The vote will be open until at least Sep 27th (at least 72 > > > >hours), following the consensus voting process. > > > > > > > >Cheers, > > > >Xuannan and Dong > > > > > > > >[1] > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-327*3A*Support*switching*from*batch*to*stream*mode*to*improve*throughput*when*processing*backlog*data__;JSsrKysrKysrKysrKysr!!IKRxdwAv5BmarQ!ZP19r7-3IBaBI-kV0olZvaz5a2TFN3uxge2TJM7WQvovjfRbOl71NaC3SEh_UEJmH7Lssqu0bx4FKResPPc7$ > > > >[2] > > > https://urldefense.com/v3/__https://lists.apache.org/thread/29nvjt9sgnzvs90browb8r6ng31dcs3n__;!!IKRxdwAv5BmarQ!ZP19r7-3IBaBI-kV0olZvaz5a2TFN3uxge2TJM7WQvovjfRbOl71NaC3SEh_UEJmH7Lssqu0bx4FKZEAz9yp$ > > > > > >
Re: [VOTE] FLIP-362: Support minimum resource limitation
Thanks for the proposal +1(non-binding) Best regards Ahmed On Mon, 25 Sep 2023, 13:35 Yangze Guo, wrote: > +1 (binding) > > Best, > Yangze Guo > > On Mon, Sep 25, 2023 at 5:39 PM xiangyu feng wrote: > > > > Hi all, > > > > I would like to start the vote for FLIP-362: Support minimum resource > > limitation[1]. > > This FLIP was discussed in this thread [2]. > > > > The vote will be open for at least 72 hours unless there is an objection > or > > insufficient votes. > > > > Regards, > > Xiangyu > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-362%3A+Support+minimum+resource+limitation > > [2] https://lists.apache.org/thread/m2v9n4yynm97v8swhqj2o5k0sqlb5ym4 >
Re: [VOTE] FLIP-307: Flink Connector Redshift
+1 (non-binding) Best Regards Ahmed Hamdy On Mon, 18 Sept 2023 at 16:52, Jing Ge wrote: > +1(binding). Thanks! > > Best regards, > Jing > > On Mon, Sep 18, 2023 at 5:26 PM Danny Cranmer > wrote: > > > Thanks for driving this Samrat! > > > > +1 (binding) > > > > Thanks, > > Danny > > > > On Mon, Sep 18, 2023 at 4:17 AM Samrat Deb > wrote: > > > > > Hi All, > > > > > > Thanks for all the feedback on FLIP-307: Flink Connector Redshift > [1][2] > > > > > > I'd like to start a vote for FLIP-307. The vote will be open for at > least > > > 72 > > > hours unless there is an objection or insufficient votes. > > > > > > Bests, > > > Samrat > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift > > > [2] https://lists.apache.org/thread/wsz4jgdpnlyw1x781f9qpk7y416b45dj > > > > > >
Re: [DISCUSS] FLIP-229: Prometheus Sink Connector
Thanks Lorenzo, Looking forward to the PRs. Best Regards Ahmed Hamdy On Sat, 16 Sept 2023 at 06:27, Lorenzo Nicora wrote: > Hello > > (apologies if this is a duplicate reply) > > I was working with Karthi on this connector, and I have taken over the > development. > We have a working version we would like to submit to the community. > > The renumbered FLIP-312 is also updated with more details [1]. > Happy to answer any questions. > > Regards > Lorenzo > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector > > On Mon, 21 Aug 2023, 13:06 Ahmed Hamdy, wrote: > > > Hello Karthi > > Is this FLIP still in progress? I see the FLIP not updated & couldn't > find > > open JIRAs. > > I am happy to take over if you are no longer working on this. > > Best Regards > > Ahmed Hamdy > > > > > > On Mon, 22 May 2023 at 14:49, Martijn Visser > > wrote: > > > > > Hi all, > > > > > > > For example, a user might want to read in logs, perform some > > aggregations > > > and publish it into a metrics store for visualisation. This might be a > > > great use-case for reducing the cardinality of metrics! > > > > > > I can see that. What I would like to see in the FLIP is a proposal on > the > > > boundaries of the metrics reporter vs the Prometheus sink. I think it's > > > important that we make clear when to use a metric reporter and when > not. > > I > > > can imagine that there will be Flink users who think that they can get > > data > > > from the metric reporter, make aggregrations in Flink and then store it > > > using the Prometheus sink. > > > > > > Overall, I think more context must be added to the FLIP, especially on > > the > > > motivation. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Fri, May 19, 2023 at 4:28 PM Karthi Thyagarajan < > > kar...@karthitect.com> > > > wrote: > > > > > > > Hi Lijie > > > > > > > > Thank you for pointing this out. I've corrected it [1]. Also, this > page > > > > [2] still shows 178 and 229 as available, which is why I picked it > up. > > > > > > > > Thanks > > > > Karthi > > > > > > > > [1] > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-312%3A+Prometheus+Sink+Connector > > > > [2] > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > > > > > > > > On May 15, 2023, at 9:37 PM, Lijie Wang > > > wrote: > > > > > > > > > > > > Hi Karthi, > > > > > > > > I think you are using a wrong FLIP id, the FLIP-229 has already be > > > used[1]. > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job > > > > > > > > Best, > > > > Lijie > > > > > > > > Martijn Visser 于2023年5月16日周二 04:44写道： > > > > > > > > Hi Karthi, > > > > > > > > Thanks for the FLIP and opening up the discussion. My main question > is: > > > why > > > > should we create a separate connector and not use and/or improve the > > > > existing integrations with Prometheus? I would like to understand > more > > so > > > > that it can be added to the motivation of the FLIP. > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > On Mon, May 15, 2023 at 6:03 PM Karthi Thyagarajan < > > > kar...@karthitect.com> > > > > wrote: > > > > > > > > > Hello all, > > > > > > > > > > We would like to start a discussion thread on FLIP-229: Prometheus > > Sink > > > > > Connector [1] where we propose to provide a sink connector for > > > Prometheus > > > > > [2] based on the Async Sink [3]. Looking forward to comments and > > > > feedback. > > > > > Thank you. > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Prometheus+Sink+Connector > > > > > [2] https://prometheus.io/ > > > > > [3] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink > > > > > > > > > > > > > > > > > > > > > > >
Re: [DISSCUSS] Kubernetes Operator Flink Version Support Policy
Makes sense, Thanks for the clarification. Best Regards Ahmed Hamdy On Thu, 14 Sept 2023 at 14:07, Gyula Fóra wrote: > Hi Ahmed! > > As I mentioned in the first email, the Flink Operator explicitly aims to > make running Flink and Flink Platforms on Kubernetes easy. As most users > are platform teams supporting Flink inside a company or running a service > it's basically always required to support several Flink versions at the > same time. > > Enterprise users are in many cases using Flink versions that are older than > the last 2 minor releases (currently supported by the community). However > the operator itself is somewhat independent of Flink itself, and most > operator features work across several Flink versions at the same time. > > Based on this it's relatively easy for us to support deploying to previous > Flink minor versions (within some reasonable limit). This means that as > long as platform teams keep the operator up-to-date they get the latest > stability / deployment improvements but can still provide compatibility for > their users for older Flink job versions. > > Cheers, > Gyula > > On Thu, Sep 14, 2023 at 2:59 PM Ahmed Hamdy wrote: > > > Thanks Gyula, > > +1 for the proposal in general. > > May I ask why are we interested in supporting more than the ones > supported > > by the community? > > for example I understand all versions prior to 1.16 are now out of > support, > > why should we tie our compatibility 4 versions behind? > > Best Regards > > Ahmed Hamdy > > > > > > On Thu, 14 Sept 2023 at 12:18, ConradJam wrote: > > > > > +1 > > > > > > Yang Wang 于2023年9月14日周四 16:15写道： > > > > > > > Since the users could always use the old Flink Kubernetes Operator > > > version > > > > along with old Flink versions, I am totally in favor of this proposal > > to > > > > reduce maintenance burden. > > > > > > > > Best, > > > > Yang > > > > > > > > Biao Geng 于2023年9月6日周三 18:15写道： > > > > > > > > > +1 for the proposal. > > > > > > > > > > Best, > > > > > Biao Geng > > > > > > > > > > Gyula Fóra 于2023年9月6日周三 16:10写道： > > > > > > > > > > > @Zhanghao Chen: > > > > > > > > > > > > I am not completely sure at this point what this will mean for > 2.0 > > > > simply > > > > > > because I am also not sure what that will mean for the operator > as > > > well > > > > > :) > > > > > > I think this will depend on the compatibility guarantees we can > > > provide > > > > > > across Flink major versions in general. We have to look into that > > and > > > > > > tackle the question there independently. > > > > > > > > > > > > Gyula > > > > > > > > > > > > On Tue, Sep 5, 2023 at 6:12 PM Maximilian Michels < > m...@apache.org> > > > > > wrote: > > > > > > > > > > > > > +1 Sounds good! Four releases give a decent amount of time to > > > migrate > > > > > > > to the next Flink version. > > > > > > > > > > > > > > On Tue, Sep 5, 2023 at 5:33 PM Őrhidi Mátyás < > > > > matyas.orh...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > On Tue, Sep 5, 2023 at 8:03 AM Thomas Weise > > > > wrote: > > > > > > > > > > > > > > > > > +1, thanks for the proposal > > > > > > > > > > > > > > > > > > On Tue, Sep 5, 2023 at 8:13 AM Gyula Fóra < > > > gyula.f...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi All! > > > > > > > > > > > > > > > > > > > > @Maximilian Michels has raised the > > question > > > > of > > > > > > > Flink > > > > > > > > > > version support in the operator before the last release. > I > > > > would > > > > > > > like to > > > > > > > > > > open this discussion publicly so we can finalize this > > before
Re: [DISSCUSS] Kubernetes Operator Flink Version Support Policy
Thanks Gyula, +1 for the proposal in general. May I ask why are we interested in supporting more than the ones supported by the community? for example I understand all versions prior to 1.16 are now out of support, why should we tie our compatibility 4 versions behind? Best Regards Ahmed Hamdy On Thu, 14 Sept 2023 at 12:18, ConradJam wrote: > +1 > > Yang Wang 于2023年9月14日周四 16:15写道： > > > Since the users could always use the old Flink Kubernetes Operator > version > > along with old Flink versions, I am totally in favor of this proposal to > > reduce maintenance burden. > > > > Best, > > Yang > > > > Biao Geng 于2023年9月6日周三 18:15写道： > > > > > +1 for the proposal. > > > > > > Best, > > > Biao Geng > > > > > > Gyula Fóra 于2023年9月6日周三 16:10写道： > > > > > > > @Zhanghao Chen: > > > > > > > > I am not completely sure at this point what this will mean for 2.0 > > simply > > > > because I am also not sure what that will mean for the operator as > well > > > :) > > > > I think this will depend on the compatibility guarantees we can > provide > > > > across Flink major versions in general. We have to look into that and > > > > tackle the question there independently. > > > > > > > > Gyula > > > > > > > > On Tue, Sep 5, 2023 at 6:12 PM Maximilian Michels > > > wrote: > > > > > > > > > +1 Sounds good! Four releases give a decent amount of time to > migrate > > > > > to the next Flink version. > > > > > > > > > > On Tue, Sep 5, 2023 at 5:33 PM Őrhidi Mátyás < > > matyas.orh...@gmail.com> > > > > > wrote: > > > > > > > > > > > > +1 > > > > > > > > > > > > On Tue, Sep 5, 2023 at 8:03 AM Thomas Weise > > wrote: > > > > > > > > > > > > > +1, thanks for the proposal > > > > > > > > > > > > > > On Tue, Sep 5, 2023 at 8:13 AM Gyula Fóra < > gyula.f...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > > > Hi All! > > > > > > > > > > > > > > > > @Maximilian Michels has raised the question > > of > > > > > Flink > > > > > > > > version support in the operator before the last release. I > > would > > > > > like to > > > > > > > > open this discussion publicly so we can finalize this before > > the > > > > next > > > > > > > > release. > > > > > > > > > > > > > > > > Background: > > > > > > > > Currently the Flink Operator supports all Flink versions > since > > > > Flink > > > > > > > 1.13. > > > > > > > > While this is great for the users, it introduces a lot of > > > backward > > > > > > > > compatibility related code in the operator logic and also > adds > > > > > > > considerable > > > > > > > > time to the CI. We should strike a reasonable balance here > that > > > > > allows us > > > > > > > > to move forward and eliminate some of this tech debt. > > > > > > > > > > > > > > > > In the current model it is also impossible to support all > > > features > > > > > for > > > > > > > all > > > > > > > > Flink versions which leads to some confusion over time. > > > > > > > > > > > > > > > > Proposal: > > > > > > > > Since it's a key feature of the kubernetes operator to > support > > > > > several > > > > > > > > versions at the same time, I propose to support the last 4 > > stable > > > > > Flink > > > > > > > > minor versions. Currently this would mean to support Flink > > > > 1.14-1.17 > > > > > (and > > > > > > > > drop 1.13 support). When Flink 1.18 is released we would drop > > > 1.14 > > > > > > > support > > > > > > > > and so on. Given the Flink release cadence this means about 2 > > > year > > > > > > > support > > > > > > > > for each Flink version. > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > Cheers, > > > > > > > > Gyula > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
