Hi all,
Thanks for the feedback.
@Martijn
Yes that is an interesting question. I agree with Danny that we should have a
future discussion about the S3 filesystem, and we might treat it differently
given it’s a "core" feature.
It would naturally come together with the flink-s3-fs-hadoop and
flink-s3-fs-presto plugins.
In my opinion, the release cycle for plugins might not fit directly into our
connector release cycle at the moment, but I can definitely see the benefit of
having an independent release cycle (faster iteration, incrementing dependency
versions more easily). We could consider proposing a separate lifecycle
management for Flink plugins.
Regards,
Hong
On 24/01/2023, 14:46, "Danny Cranmer" <dannycran...@apache.org> wrote:
CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you can confirm the sender and know the
content is safe.
Hello Hong,
Thanks for starting this discussion. +1 from me, the GSR formats are still
technically "AWS connectors" so it makes sense to move them. Especially
since they have a common dependency on flink-connector-aws-base and we are
targeting this repository for the Native Glue SQL integration [1].
Martijn we have not yet considered the S3 filesystem for externalization.
It does make sense in the longer term but should have a future discussion
thread. Given this is more of a "core" feature I would be interested in
whether the connector lifecycle model still fits.
Thanks.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-277%3A+Native+GlueCatalog+Support+in+Flink
On Tue, Jan 24, 2023 at 12:26 PM Martijn Visser <martijnvis...@apache.org>
wrote:
> Hi Hong,
>
> I think it's a good idea, but it does make up for another interesting
> question since the AWS SDK is also used in flink-s3-fs-base. I'm assuming
> that will keep for now or do you also have plans for that in the future?
>
> Best regards,
>
> Martijn
>
> Op di 24 jan. 2023 om 12:52 schreef Jing Ge <j...@ververica.com.invalid>:
>
> > Hi,
> >
> > +1, I think, for those formats that are only used by one or few specific
> > connectors, it is a good idea to externalize them.
> >
> > Best regards,
> > Jing
> >
> > On Tue, Jan 24, 2023 at 9:44 AM Ahmed Hamdy <hamdy10...@gmail.com>
> wrote:
> >
> > > Hi Hong
> > > +1 for the proposal. Additionally I believe it serves the same value
> for
> > > the schema classes as it does for connectors.
> > >
> > > Best Regards,
> > > Ahmed Hamdy
> > >
> > > On Tue, 24 Jan 2023 at 07:51, Teoh, Hong <lian...@amazon.co.uk.invalid
> >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > As part of the effort to externalize connectors, I would like to
> > propose
> > > > that we also externalize flink-avro-glue-schema-registry and
> > > > flink-json-glue-schema-registry formats to the flink-connector-aws
> > > > repository.
> > > >
> > > > Benefits of this being:
> > > >
> > > > 1. We can unify and upgrade the AWS SDK versions more easily.
> > > > 2. We can now move flink-connector-aws-base to the external
> > repository
> > > > as well.
> > > >
> > > > Regards,
> > > > Hong
> > > >
> > > >
> > >
> >
>
