Re: [DISCUSS] Switch docker iamge base to Eclipse Temurin

[Xingbo Huang]

Thanks Chesnay for driving this. I found a problem with image name
change[1] and I have created a PR[2] to fix it.

Best,
Xingbo

[1] https://issues.apache.org/jira/browse/FLINK-29161
[2] https://github.com/apache/flink/pull/20726

Chesnay Schepler  于2022年8月31日周三 17:15写道：

> I will optimistically merge the PRs that make the switch so we can
> gather some e2e testing data.
>
> On 30/08/2022 14:51, Chesnay Schepler wrote:
> > yes, alpine would have similar issues as CentOS. As for usability,
> > from personal experience it has always been a bit of a drag to extend
> > or use manually because it is such a minimal image.
> >
> > On 30/08/2022 14:45, Matthias Pohl wrote:
> >> Thanks for bringing this up, Chesnay. Can you elaborate a bit more on
> >> what
> >> you mean when referring to Alpine as being "not as user-friendly"?
> >> Doesn't
> >> it come with the same issue that switching to CentOS comes with that we
> >> have to update our scripts (I'm thinking of apt in particular)? Or what
> >> else did you have in mind in terms of user-friendliness? I would imagine
> >> selecting the required packages would be a bit more tedious.
> >>
> >> I'm wondering whether we considered the security aspect. A more minimal
> >> Alpine base image might reduce the risk of running into CVEs. But then;
> >> it's also the question how fast those CVEs are actually fixed on average
> >> (now comparing Ubuntu and Alpine for instance). Or is this even a
> >> concern
> >> for us?
> >>
> >> I didn't find any clear answers around that topic with a quick Google
> >> search. [1] was kind of interesting to read.
> >>
> >> Anyway, I definitely see the benefits of just switching to Ubuntu due to
> >> the fact that it also relies on Debian's package management (reducing
> >> the
> >> migration effort) and that we're using it for our CI builds
> >> (consistency).
> >>
> >> +1 for going with Ubuntu if security is not a big concern
> >>
> >> Best,
> >> Matthias
> >>
> >> [1]
> >>
> https://jfrog.com/knowledge-base/why-use-ubuntu-as-a-docker-base-image-when-alpine-exists/
> >>
> >>
> >> On Tue, Aug 30, 2022 at 11:40 AM Chesnay Schepler 
> >> wrote:
> >>
> >>> Hello,
> >>>
> >>> during the release of the 1.15.2 images
> >>>  it was
> >>> noted that we use the openjdk:8/11 images, which have been deprecated
> >>>  and thus no
> >>> longer receive any updates.
> >>>
> >>> There are a number of alternatives, the most promising being Eclipse
> >>> Temurin , the successor of
> >>> AdoptOpenJDK, since it's vendor neutral.
> >>>
> >>> This would imply a switch of distros from Debian to most likely Ubuntu
> >>> 22.04 (Alpine isn't as user-friendly, and CentOS is likely incompatible
> >>> with existing images using our images as a base). We are also running
> >>> our CI on Ubuntu, so I don't expect any issues.
> >>>
> >>> Let me know what you think.
> >>>
> >>> The required changes on our side appear to be minimal; I have already
> >>> prepared a PR .
> >>>
> >
>
>

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

[Zheng Yu Chen]

Hi @Hong

As @Chesnay said, we just follow the same order that the CLI currently
enforces. @

Gyula and me agree this sugget，

It also makes sense to mention that this FLIP is not about the
prioritization of config options. so this feature maybe do this filp and
after discuss , not now

After the addition is complete, if there are no new comments, we will enter
the voting stage and start it ~

what do you think?

Teoh, Hong  于2022年8月27日周六 16:06写道：

> Hi Zheng Yu,
>
> Sorry for the late reply, I was on holiday last week.
>
> Could we propose the following config instead?
>
> rest.submit.job.override-config = true/false
>   - true: REST API will have priority over user code (i.e. Rest API /
> Flink CLI > User Code > Cluster Config)
>   - false (default): user code will have priority over REST API (i.e. User
> Code > Rest API / Flink CLI > Cluster Config)
>
> This way, the default behaviour will be according to the proposed FLIP,
> but we will have an additional toggle to ignore the configuration set in
> user code.
>
> >However, there will be a problem in doing this. If the user writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior
>
> Hmm, would it be better if we set it such that the
> "rest.submit.job.override-config" cannot be overwritten in user code
> (ignored), just like configuration  "jobmanager.rpc.port"?
>
> What do you think?
>
> Regards,
> Hong
>
>
>
> On 26/08/2022, 12:05, "Zheng Yu Chen"  wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> HI Hong,
>
> Maybe you forgot. I don’t know if the current FLIP still needs to be
> modified. If not, we will start with the current plan. If you have
> relevant ideas in the future, please continue to discuss. If not, we
> will open voting at a later date.
>
> Teoh, Hong  于2022年8月22日周一 16:02写道：
> >
> > Hi Zheng Yu,
> >
> > We would have to take the same "cluster configuration" (cannot be
> set on job submission) vs "job configuration" approach in the User code as
> well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config
> as a cluster configuration. That way, neither the REST API / User code can
> override this configuration, and it can only be set in cluster
> configuration on startup.
> >
> > There are other cluster specific configurations that are already
> treated this way (e.g. jobmanager.rpc.address,
> jobmanager.memory.process.size). As part of this work, I wonder if we can
> update the Flink docs on configuration (
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/)
> to specify which configuration is "cluster" and which is "job".
> >
> > Regards,
> > Hong
> >
> >
> >
> > On 20/08/2022, 09:16, "Zheng Yu Chen"  wrote:
> >
> > CAUTION: This email originated from outside of the organization.
> Do not click links or open attachments unless you can confirm the sender
> and know the content is safe.
> >
> >
> >
> >  @Gyula say that  add a config for this but not expose it on the
> rest
> > api is right
> > when user start up session cluster,we can config (eg
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false)
> to
> > controller this behavior
> >
> > Now we have an existing cluster, the admin configures
> checkpointing
> > interval by this session cluster
> >
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false
> >
> > - true : allow user reset new value config
> > - false(default) : not allow user set new config from ,throw exp
> to
> > tell user what properties is admin was set it
> >
> > However, there will be a problem in doing this. If the user
> writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior.
> >
> > what do you think ?
> >
> > @Hong @Gyula @Teoh @Danny
> >
> >
> > Gyula Fóra  于2022年8月18日周四 18:37写道：
> > >
> > > +1 for the proposal.
> > >
> > > @Hong : I feel that the inverted override flag should be a
> cluster setting
> > > and not something the user can override at will. I fear that
> this might
> > > defeat the purpose of the feature itself.
> > > So I think we should add a config for this but not expose it
> on the rest api
> > >
> > > Gyula
> > >
> > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> 
> > > wrote:
> > >
> > > 

[jira] [Created] (FLINK-29163) There is a problem when compiling Flink 1.15.rc-2 and flink-1.14.3-rc1 in Windows operations

[mindest liu (Jira)]

mindest liu created FLINK-29163:
---

 Summary: There is a problem when compiling Flink 1.15.rc-2 and 
flink-1.14.3-rc1 in Windows operations
 Key: FLINK-29163
 URL: https://issues.apache.org/jira/browse/FLINK-29163
 Project: Flink
  Issue Type: Bug
  Components: flink-contrib
Affects Versions: 1.15.2
Reporter: mindest liu


Compile Flink 1.15.2 on Windows platform and it appears

 

[ERROR]Failed to execute goal 
org.apache.maven.plugins:maven-assembly-plugin:2.4:single

(create-test-dependency-user-jar-depend) on project flink-clients_2.11:failed 
to create assembly:Error creating assembly archive 
test-user-classloader-job-lib-jar:You must set as least one file.->[help 1]

Error creating assembly archive Pack: you must set at least one file. Occurs

 

when compiling to Flink clients

 

The compiled JDK version is 11

 

Maven version is 3.2.5



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29162) missCounter and loadCount metrics also increase even if cache hint

[dalongliu (Jira)]

dalongliu created FLINK-29162:
-

 Summary: missCounter and loadCount metrics also increase even if 
cache hint
 Key: FLINK-29162
 URL: https://issues.apache.org/jira/browse/FLINK-29162
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Affects Versions: 1.16.0
Reporter: dalongliu
 Fix For: 1.16.0
 Attachments: image-2022-09-01-10-56-34-264.png

!image-2022-09-01-10-56-34-264.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29161) Run kubernetes application HA test failed with unable to prepare context: path "dev/test_kubernetes_application_ha-debian" not found

[Xingbo Huang (Jira)]

Xingbo Huang created FLINK-29161:


 Summary: Run kubernetes application HA test failed with unable to 
prepare context: path "dev/test_kubernetes_application_ha-debian" not found
 Key: FLINK-29161
 URL: https://issues.apache.org/jira/browse/FLINK-29161
 Project: Flink
  Issue Type: Technical Debt
  Components: Build System / Azure Pipelines
Affects Versions: 1.16.0
Reporter: Xingbo Huang
 Fix For: 1.16.0


{code:java}
2022-08-31T09:13:50.8050601Z Aug 31 09:13:50 Preparing Dockeriles
2022-08-31T09:13:50.8051483Z Aug 31 09:13:50 Executing command: git clone 
https://github.com/apache/flink-docker.git --branch dev-master --single-branch
2022-08-31T09:13:50.8069617Z Cloning into 'flink-docker'...
2022-08-31T09:13:51.4728169Z Generating Dockerfiles... done.
2022-08-31T09:13:51.4732889Z Aug 31 09:13:51 Building images
2022-08-31T09:13:51.5013231Z unable to prepare context: path 
"dev/test_kubernetes_application_ha-debian" not found
2022-08-31T09:13:51.5041357Z Aug 31 09:13:51 ~/work/1/s/cri-dockerd
2022-08-31T09:13:51.5042427Z Aug 31 09:13:51 Command: build_image 
test_kubernetes_application_ha 10.1.0.160 failed. Retrying...
2022-08-31T09:13:53.5058625Z Aug 31 09:13:53 Starting fileserver for Flink 
distribution
2022-08-31T09:13:53.5060253Z Aug 31 09:13:53 
~/work/1/s/flink-dist/target/flink-1.16-SNAPSHOT-bin ~/work/1/s/cri-dockerd
2022-08-31T09:14:13.8161820Z Aug 31 09:14:13 ~/work/1/s/cri-dockerd
2022-08-31T09:14:13.8164441Z Aug 31 09:14:13 
~/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-06753986230 
~/work/1/s/cri-dockerd
2022-08-31T09:14:13.8165256Z Aug 31 09:14:13 Preparing Dockeriles
2022-08-31T09:14:13.8166118Z Aug 31 09:14:13 Executing command: git clone 
https://github.com/apache/flink-docker.git --branch dev-master --single-branch
2022-08-31T09:14:13.8182037Z fatal: destination path 'flink-docker' already 
exists and is not an empty directory.
2022-08-31T09:14:13.8188942Z Aug 31 09:14:13 Retry 1/5 exited 128, retrying in 
1 seconds...
2022-08-31T09:14:14.1481612Z Traceback (most recent call last):
2022-08-31T09:14:14.1483171Z   File 
"/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/python3_fileserver.py",
 line 26, in 
2022-08-31T09:14:14.1484572Z httpd = socketserver.TCPServer(("", ), 
handler)
2022-08-31T09:14:14.1485179Z   File "/usr/lib/python3.8/socketserver.py", line 
452, in __init__
2022-08-31T09:14:14.1485724Z self.server_bind()
2022-08-31T09:14:14.1486247Z   File "/usr/lib/python3.8/socketserver.py", line 
466, in server_bind
2022-08-31T09:14:14.1486795Z self.socket.bind(self.server_address)
2022-08-31T09:14:14.1487783Z OSError: [Errno 98] Address already in use {code}
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=40556=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14]

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29160) 在windows 编译flink 1.15.rc-2时出现问题

[mindest liu (Jira)]

mindest liu created FLINK-29160:
---

 Summary: 在windows 编译flink 1.15.rc-2时出现问题
 Key: FLINK-29160
 URL: https://issues.apache.org/jira/browse/FLINK-29160
 Project: Flink
  Issue Type: Bug
  Components: flink-contrib
Affects Versions: 1.15.2
 Environment: 操作系统:windows 

jdk:11.

maven:3.2.5
Reporter: mindest liu


在windows 平台上编译flink 1.15.2出现

编译到flink-clients 时出现 Error creating assembly archive pack: You must set at 
least one file.问题

其中 编译的jdk 版本是11.

maven版本是3.2.5



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-254 Guarantee binary compatibility for Public/-Evolving APIs between patch releases

[Yu Li]

*bq. a list of all breaking changes between 1.15.0 and the latest 1.15.x
(or intermediate releases)*
Yes, this might help users to be more prepared before upgrading, if they
could know whether need to recompile their applications. Asking about
1.14/1.15 since they are the "in service" versions.

But it's totally fine if we're targeting for the future (smile).

Best Regards,
Yu


On Wed, 31 Aug 2022 at 21:41, Chesnay Schepler  wrote:

> Well then someone didnt update the FLIP number as they should!
> I'll increment the number of this FLIP to 255.
>
> On 31/08/2022 15:06, Matthias Pohl wrote:
> > +1 for bringing this into a consistent state. Thanks, Chesnay.
> >
> > nit: There's a conflict between this FLIP-254 and the FLIP-254 on the
> redis
> > streams connector.
> >
> > On Wed, Aug 31, 2022 at 2:52 PM Chesnay Schepler 
> wrote:
> >
> >> * backport to 1.14/1.15
> >>
> >> On 31/08/2022 14:45, Chesnay Schepler wrote:
> >>> @Yu I haven't really considered 1.14/1.15.
> >>>
> >>> What exactly are you interested in; a list of all breaking changes
> >>> between 1.15.0 and the latest 1.15.x (or intermediate releases),
> >>> or are you suggesting to also backport this whole thing to 1.16 (which
> >>> should be possible)?
> >>>
> >>> On 31/08/2022 13:31, Yu Li wrote:
>  +1 for the FLIP. Thanks for the efforts Chesnay.
> 
>  I believe ensuring binary compatibility for patch releases will also
>  benefit our end users besides the cloud service providers.
> 
>  I'm also wondering about the compatibility checking result after
>  enabling
>  japicmp for all modules with existing patch releases (1.14.x and
>  1.15.x).
>  Do you already have one with your local customized japicmp or do we
>  need to
>  wait until the works tracked in JIRA are done? (smile)
> 
>  Best Regards,
>  Yu
> 
> 
>  On Wed, 31 Aug 2022 at 18:50, Konstantin Knauf 
>  wrote:
> 
> > Hi Chesnay,
> >
> > thanks for bringing this up and for your research and fixes to
> japicmd.
> >
> > +1 for the proposal. For Immerok as an Apache Flink cloud service
> > provider
> > it is very valuable to know that our users don't need to upgrade
> > their Jobs
> > when the Flink patch version changes. I am sure the same is true for
> > internal platform teams as well as end users of Apache Flink.
> >
> > Cheers,
> >
> > Konstantin
> >
> >
> >
> >
> >
> > Am Mi., 31. Aug. 2022 um 12:31 Uhr schrieb Chesnay Schepler <
> > ches...@apache.org>:
> >
> >> Hello,
> >>
> >> I just published a FLIP to guarantee binary compatibility for patch
> >> releases. I don't think our current guarantees of
> source-compatibility
> >> are sufficient for patch releases.
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857
> >> Let me know what you think.
> >>
> >> Regards,
> >> Chesnay
> >>
> > --
> > https://twitter.com/snntrable
> > https://github.com/knaufk
> >
> >>
>
>

[jira] [Created] (FLINK-29159) Revisit/harden initial deployment logic

[Thomas Weise (Jira)]

Thomas Weise created FLINK-29159:


 Summary: Revisit/harden initial deployment logic
 Key: FLINK-29159
 URL: https://issues.apache.org/jira/browse/FLINK-29159
 Project: Flink
  Issue Type: Technical Debt
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.1.0
Reporter: Thomas Weise


Found isFirstDeployment logic not working as expected for a deployment that had 
never successfully deployed (image pull error).  We are probably also lacking 
test coverage for the initialSavepointPath field.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-254 Guarantee binary compatibility for Public/-Evolving APIs between patch releases

[Chesnay Schepler]

Well then someone didnt update the FLIP number as they should!
I'll increment the number of this FLIP to 255.

On 31/08/2022 15:06, Matthias Pohl wrote:

+1 for bringing this into a consistent state. Thanks, Chesnay.

nit: There's a conflict between this FLIP-254 and the FLIP-254 on the redis
streams connector.

On Wed, Aug 31, 2022 at 2:52 PM Chesnay Schepler  wrote:


* backport to 1.14/1.15

On 31/08/2022 14:45, Chesnay Schepler wrote:

@Yu I haven't really considered 1.14/1.15.

What exactly are you interested in; a list of all breaking changes
between 1.15.0 and the latest 1.15.x (or intermediate releases),
or are you suggesting to also backport this whole thing to 1.16 (which
should be possible)?

On 31/08/2022 13:31, Yu Li wrote:

+1 for the FLIP. Thanks for the efforts Chesnay.

I believe ensuring binary compatibility for patch releases will also
benefit our end users besides the cloud service providers.

I'm also wondering about the compatibility checking result after
enabling
japicmp for all modules with existing patch releases (1.14.x and
1.15.x).
Do you already have one with your local customized japicmp or do we
need to
wait until the works tracked in JIRA are done? (smile)

Best Regards,
Yu


On Wed, 31 Aug 2022 at 18:50, Konstantin Knauf 
wrote:


Hi Chesnay,

thanks for bringing this up and for your research and fixes to japicmd.

+1 for the proposal. For Immerok as an Apache Flink cloud service
provider
it is very valuable to know that our users don't need to upgrade
their Jobs
when the Flink patch version changes. I am sure the same is true for
internal platform teams as well as end users of Apache Flink.

Cheers,

Konstantin





Am Mi., 31. Aug. 2022 um 12:31 Uhr schrieb Chesnay Schepler <
ches...@apache.org>:


Hello,

I just published a FLIP to guarantee binary compatibility for patch
releases. I don't think our current guarantees of source-compatibility
are sufficient for patch releases.



https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857

Let me know what you think.

Regards,
Chesnay


--
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-254 Guarantee binary compatibility for Public/-Evolving APIs between patch releases

[Matthias Pohl]

+1 for bringing this into a consistent state. Thanks, Chesnay.

nit: There's a conflict between this FLIP-254 and the FLIP-254 on the redis
streams connector.

On Wed, Aug 31, 2022 at 2:52 PM Chesnay Schepler  wrote:

> * backport to 1.14/1.15
>
> On 31/08/2022 14:45, Chesnay Schepler wrote:
> > @Yu I haven't really considered 1.14/1.15.
> >
> > What exactly are you interested in; a list of all breaking changes
> > between 1.15.0 and the latest 1.15.x (or intermediate releases),
> > or are you suggesting to also backport this whole thing to 1.16 (which
> > should be possible)?
> >
> > On 31/08/2022 13:31, Yu Li wrote:
> >> +1 for the FLIP. Thanks for the efforts Chesnay.
> >>
> >> I believe ensuring binary compatibility for patch releases will also
> >> benefit our end users besides the cloud service providers.
> >>
> >> I'm also wondering about the compatibility checking result after
> >> enabling
> >> japicmp for all modules with existing patch releases (1.14.x and
> >> 1.15.x).
> >> Do you already have one with your local customized japicmp or do we
> >> need to
> >> wait until the works tracked in JIRA are done? (smile)
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Wed, 31 Aug 2022 at 18:50, Konstantin Knauf 
> >> wrote:
> >>
> >>> Hi Chesnay,
> >>>
> >>> thanks for bringing this up and for your research and fixes to japicmd.
> >>>
> >>> +1 for the proposal. For Immerok as an Apache Flink cloud service
> >>> provider
> >>> it is very valuable to know that our users don't need to upgrade
> >>> their Jobs
> >>> when the Flink patch version changes. I am sure the same is true for
> >>> internal platform teams as well as end users of Apache Flink.
> >>>
> >>> Cheers,
> >>>
> >>> Konstantin
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Am Mi., 31. Aug. 2022 um 12:31 Uhr schrieb Chesnay Schepler <
> >>> ches...@apache.org>:
> >>>
>  Hello,
> 
>  I just published a FLIP to guarantee binary compatibility for patch
>  releases. I don't think our current guarantees of source-compatibility
>  are sufficient for patch releases.
> 
> 
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857
> >>>
>  Let me know what you think.
> 
>  Regards,
>  Chesnay
> 
> >>>
> >>> --
> >>> https://twitter.com/snntrable
> >>> https://github.com/knaufk
> >>>
> >
>
>

Re: [DISCUSS] FLIP-254 Guarantee binary compatibility for Public/-Evolving APIs between patch releases

[Chesnay Schepler]

* backport to 1.14/1.15

On 31/08/2022 14:45, Chesnay Schepler wrote:

@Yu I haven't really considered 1.14/1.15.

What exactly are you interested in; a list of all breaking changes 
between 1.15.0 and the latest 1.15.x (or intermediate releases),
or are you suggesting to also backport this whole thing to 1.16 (which 
should be possible)?


On 31/08/2022 13:31, Yu Li wrote:

+1 for the FLIP. Thanks for the efforts Chesnay.

I believe ensuring binary compatibility for patch releases will also
benefit our end users besides the cloud service providers.

I'm also wondering about the compatibility checking result after 
enabling
japicmp for all modules with existing patch releases (1.14.x and 
1.15.x).
Do you already have one with your local customized japicmp or do we 
need to

wait until the works tracked in JIRA are done? (smile)

Best Regards,
Yu


On Wed, 31 Aug 2022 at 18:50, Konstantin Knauf  
wrote:



Hi Chesnay,

thanks for bringing this up and for your research and fixes to japicmd.

+1 for the proposal. For Immerok as an Apache Flink cloud service 
provider
it is very valuable to know that our users don't need to upgrade 
their Jobs

when the Flink patch version changes. I am sure the same is true for
internal platform teams as well as end users of Apache Flink.

Cheers,

Konstantin





Am Mi., 31. Aug. 2022 um 12:31 Uhr schrieb Chesnay Schepler <
ches...@apache.org>:


Hello,

I just published a FLIP to guarantee binary compatibility for patch
releases. I don't think our current guarantees of source-compatibility
are sufficient for patch releases.


https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857 


Let me know what you think.

Regards,
Chesnay



--
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-254 Guarantee binary compatibility for Public/-Evolving APIs between patch releases

[Chesnay Schepler]

@Yu I haven't really considered 1.14/1.15.

What exactly are you interested in; a list of all breaking changes 
between 1.15.0 and the latest 1.15.x (or intermediate releases),
or are you suggesting to also backport this whole thing to 1.16 (which 
should be possible)?


On 31/08/2022 13:31, Yu Li wrote:

+1 for the FLIP. Thanks for the efforts Chesnay.

I believe ensuring binary compatibility for patch releases will also
benefit our end users besides the cloud service providers.

I'm also wondering about the compatibility checking result after enabling
japicmp for all modules with existing patch releases (1.14.x and 1.15.x).
Do you already have one with your local customized japicmp or do we need to
wait until the works tracked in JIRA are done? (smile)

Best Regards,
Yu


On Wed, 31 Aug 2022 at 18:50, Konstantin Knauf  wrote:


Hi Chesnay,

thanks for bringing this up and for your research and fixes to japicmd.

+1 for the proposal. For Immerok as an Apache Flink cloud service provider
it is very valuable to know that our users don't need to upgrade their Jobs
when the Flink patch version changes. I am sure the same is true for
internal platform teams as well as end users of Apache Flink.

Cheers,

Konstantin





Am Mi., 31. Aug. 2022 um 12:31 Uhr schrieb Chesnay Schepler <
ches...@apache.org>:


Hello,

I just published a FLIP to guarantee binary compatibility for patch
releases. I don't think our current guarantees of source-compatibility
are sufficient for patch releases.



https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857

Let me know what you think.

Regards,
Chesnay



--
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-254 Redis Streams connector

[Konstantin Knauf]

Hi Martijn, Hi Mohammad,

thanks for creating this FLIP.

I am wondering, do you know how much Redis Streams are used compared to
other Redis data structures? At least as a sink and LookupTableSource
supporting regular Redis data structures seems quite useful.

I suppose such connectors would live in the same repository. Would be good
to consider when naming the Redis Streams connector.

Cheers,

Konstantin



Am Mi., 31. Aug. 2022 um 13:55 Uhr schrieb Martijn Visser <
martijnvis...@apache.org>:

> Hi everyone,
>
> Last call for comments, else I'll open up a VOTE thread later this week.
>
> Best regards,
>
> Martijn
>
> Op ma 1 aug. 2022 om 15:40 schreef Martijn Visser <
> martijnvis...@apache.org
> >:
>
> > Hi everyone,
> >
> > Together with Mohammad we would like to start a discussion thread on
> > FLIP-254 for a Redis Streams connector. This would also be a new
> connector
> > in its own external connector repository, being
> > https://www.github.com/apache/flink-connector-redis. This repository
> > already exists, because there was an open PR to add this to the Flink
> main
> > repository and they were planned to be one of the first movers to the
> > external connector repo. However, we never formally agreed that we want
> > this, therefore we're now creating this FLIP.
> >
> > Looking forward to any comments or feedback. If not, I'll open a VOTE
> > thread in the next couple of days.
> >
> > Best regards,
> >
> > Martijn
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] FLIP-254 Redis Streams connector

[Martijn Visser]

Hi everyone,

Last call for comments, else I'll open up a VOTE thread later this week.

Best regards,

Martijn

Op ma 1 aug. 2022 om 15:40 schreef Martijn Visser :

> Hi everyone,
>
> Together with Mohammad we would like to start a discussion thread on
> FLIP-254 for a Redis Streams connector. This would also be a new connector
> in its own external connector repository, being
> https://www.github.com/apache/flink-connector-redis. This repository
> already exists, because there was an open PR to add this to the Flink main
> repository and they were planned to be one of the first movers to the
> external connector repo. However, we never formally agreed that we want
> this, therefore we're now creating this FLIP.
>
> Looking forward to any comments or feedback. If not, I'll open a VOTE
> thread in the next couple of days.
>
> Best regards,
>
> Martijn
>

[GitHub] [flink-connector-rabbitmq] MartijnVisser commented on pull request #1: [FLINK-20628] RabbitMQ Connector using FLIP-27 Source API

[GitBox]

MartijnVisser commented on PR #1:
URL: 
https://github.com/apache/flink-connector-rabbitmq/pull/1#issuecomment-1232836643

   @pscls Weird. Could you push once more, since the logs are no longer 
available?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@flink.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Re: Re: Re: Re: [DISCUSS] FLIP-239: Port JDBC Connector Source to FLIP-27

[Martijn Visser]

It would be great if João could help finalize the PR. In case he doesn't
have the permission to update the FLIP, please reach out to me so we can
get that sorted.

Op wo 10 aug. 2022 om 04:41 schreef Roc Marshal :

> Hi,
> Martijn,  Boto.
> I just complete the design of the source and the skeleton design of the
> sink at present.   I think the current Flip is missing part of the sink
> design.
> @Boto Would you like to complete the sink part directly on the FLIP page?
> Looking forward to your reply.
>
> On 2022/08/01 13:42:48 Martijn Visser wrote:
> > Hi,
> >
> > There is currently already a PR submitted to port the JDBC interface to
> the
> > new interfaces. Can we make sure that this FLIP is being finalized, so
> that
> > you and other maintainers can work on getting the PRs correct and
> > eventually merged in?
> >
> > Best regards,
> >
> > Martijn
> >
> > Op ma 4 jul. 2022 om 16:38 schreef Martijn Visser <
> martijnvis...@apache.org
> > >:
> >
> > > Hi Roc,
> > >
> > > Thanks for the FLIP and opening the discussion. I have a couple of
> initial
> > > questions/remarks:
> > >
> > > * The FLIP contains information for both Source and Sink, but nothing
> > > explicitly on the Lookup functionality. I'm assuming we also want to
> have
> > > that implementation covered while porting this to the new interfaces.
> > > * The FLIP mentions porting to both the new Source and the new Sink
> API,
> > > but the FLIP only contains detailed information on the Source. Are you
> > > planning to add that to the FLIP before casting a vote? Because the
> > > discussion should definitely be resolved for both the Source and the
> Sink.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > Op za 2 jul. 2022 om 06:35 schreef Roc Marshal :
> > >
> > >> Hi, Weike.
> > >>
> > >> Thank you for your reply
> > >> As you said, too many splits stored in SourceEnumerator will increase
> the
> > >> load of JM.
> > >> What do you think if we introduce a capacity of splits in
> > >> SourceEnumerator to limit the total number, and introduce a reject or
> > >> callback mechanism with too many splits in the timely generation
> strategy
> > >> to solve this problem?
> > >> Looking forward to a better solution .
> > >>
> > >> Best regards,
> > >> Roc Marshal
> > >>
> > >> On 2022/07/01 07:58:22 Dong Weike wrote:
> > >> > Hi,
> > >> >
> > >> > Thank you for bringing this up, and I am +1 for this feature.
> > >> >
> > >> > IMO, one important thing that I would like to mention is that an
> > >> improperly-designed FLIP-27 connector could impose very severe memory
> > >> pressure on the JobManager, especially when there are enormous number
> of
> > >> splits for the source tables, e.g. there are billions of records to
> read.
> > >> Frankly speaking, we have been haunted by this problem for a long
> time when
> > >> using the Flink CDC Connectors to read large tables.
> > >> >
> > >> > Therefore, in order to prevent JobManager from experiencing frequent
> > >> OOM faults, JdbcSourceEnumerator should avoid saving too many
> > >> JdbcSourceSplits in the unassigned list. And it would be better if
> all the
> > >> splits would be computed on the fly.
> > >> >
> > >> > Best,
> > >> > Weike
> > >> >
> > >> > -邮件原件-
> > >> > 发件人: Lijie Wang 
> > >> > 发送时间: 2022年7月1日 上午 10:25
> > >> > 收件人: dev@flink.apache.org
> > >> > 主题: Re: Re: [DISCUSS] FLIP-239: Port JDBC Connector Source to
> FLIP-27
> > >> >
> > >> > Hi Roc,
> > >> >
> > >> > Thanks for driving the discussion.
> > >> >
> > >> > Could you describe in detail what the JdbcSourceSplit represents? It
> > >> looks like something wrong with the comments of JdbcSourceSplit in
> FLIP(it
> > >> describe as "A {@link SourceSplit} that represents a file, or a
> region of a
> > >> file").
> > >> >
> > >> > Best,
> > >> > Lijie
> > >> >
> > >> >
> > >> > Roc Marshal  于2022年6月30日周四 21:41写道：
> > >> >
> > >> > > Hi, Boto.
> > >> > > Thanks for your reply.
> > >> > >
> > >> > >+1 to me on watermark strategy definition in ‘streaming’ &
> table
> > >> > > source. I'm not sure if FLIP-202[1]  is suitable for a separate
> > >> > > discussion, but I think your proposal is very helpful to the new
> > >> > > source. It would be great if the new source could be compatible
> with
> > >> this abstraction.
> > >> > >
> > >> > >In addition, whether we need to support such a special bounded
> > >> > > scenario abstraction?
> > >> > >The number of JdbcSourceSplit is certain, but the time to
> generate
> > >> > > all JdbcSourceSplit completely is not certain in the user defined
> > >> > > implementation. When the condition that the JdbcSourceSplit
> > >> > > generate-process end is met, the JdbcSourceSplit will not be
> > >> generated.
> > >> > > After all JdbcSourceSplit processing is completed, the reader
> will be
> > >> > > notified that there are no more JdbcSourceSplit from
> > >> > > JdbcSourceSplitEnumerator.
> > >> > >
> > >> > > - [1]
> > >> > >
> > >>
> 

Re: [DISCUSS] FLIP-254 Guarantee binary compatibility for Public/-Evolving APIs between patch releases

[Yu Li]

+1 for the FLIP. Thanks for the efforts Chesnay.

I believe ensuring binary compatibility for patch releases will also
benefit our end users besides the cloud service providers.

I'm also wondering about the compatibility checking result after enabling
japicmp for all modules with existing patch releases (1.14.x and 1.15.x).
Do you already have one with your local customized japicmp or do we need to
wait until the works tracked in JIRA are done? (smile)

Best Regards,
Yu


On Wed, 31 Aug 2022 at 18:50, Konstantin Knauf  wrote:

> Hi Chesnay,
>
> thanks for bringing this up and for your research and fixes to japicmd.
>
> +1 for the proposal. For Immerok as an Apache Flink cloud service provider
> it is very valuable to know that our users don't need to upgrade their Jobs
> when the Flink patch version changes. I am sure the same is true for
> internal platform teams as well as end users of Apache Flink.
>
> Cheers,
>
> Konstantin
>
>
>
>
>
> Am Mi., 31. Aug. 2022 um 12:31 Uhr schrieb Chesnay Schepler <
> ches...@apache.org>:
>
> > Hello,
> >
> > I just published a FLIP to guarantee binary compatibility for patch
> > releases. I don't think our current guarantees of source-compatibility
> > are sufficient for patch releases.
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857
> >
> > Let me know what you think.
> >
> > Regards,
> > Chesnay
> >
>
>
> --
> https://twitter.com/snntrable
> https://github.com/knaufk
>

[jira] [Created] (FLINK-29158) Fix logging in DefaultCompletedCheckpointStore

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-29158:
-

 Summary: Fix logging in DefaultCompletedCheckpointStore
 Key: FLINK-29158
 URL: https://issues.apache.org/jira/browse/FLINK-29158
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Affects Versions: 1.15.2
Reporter: Roman Khachatryan
Assignee: Roman Khachatryan
 Fix For: 1.15.3


See [https://github.com/apache/flink/pull/16582#discussion_r949214456]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29157) Clarify the contract between CompletedCheckpointStore and SharedStateRegistry

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-29157:
-

 Summary: Clarify the contract between CompletedCheckpointStore and 
SharedStateRegistry
 Key: FLINK-29157
 URL: https://issues.apache.org/jira/browse/FLINK-29157
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Checkpointing
Affects Versions: 1.15.2, 1.16.0
Reporter: Roman Khachatryan
Assignee: Roman Khachatryan
 Fix For: 1.16.0, 1.15.3


After FLINK-24611, CompletedCheckpointStore is required to call 
SharedStateRegistry.unregisterUnusedState() on checkpoint subsumption and 
shutdown.

Although it's not clear whether CompletedCheckpointStore is internal there are 
in fact external implementations (which weren't updated accordingly).

 

After FLINK-25872, CompletedCheckpointStore also must call 
checkpointsCleaner.cleanSubsumedCheckpoints.

 

Another issue with a custom implementation was using different java objects for 
state for CheckpointStore and SharedStateRegistry (after FLINK-24086). 

 

So it makes sense to:
 * clarify the contract (different in 1.15 and 1.16)
 * require using the same checkpoint objects by SharedStateRegistryFactory and 
CompletedCheckpointStore
 * mark the interface(s) as PublicEvolving



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-254 Guarantee binary compatibility for Public/-Evolving APIs between patch releases

[Konstantin Knauf]

Hi Chesnay,

thanks for bringing this up and for your research and fixes to japicmd.

+1 for the proposal. For Immerok as an Apache Flink cloud service provider
it is very valuable to know that our users don't need to upgrade their Jobs
when the Flink patch version changes. I am sure the same is true for
internal platform teams as well as end users of Apache Flink.

Cheers,

Konstantin





Am Mi., 31. Aug. 2022 um 12:31 Uhr schrieb Chesnay Schepler <
ches...@apache.org>:

> Hello,
>
> I just published a FLIP to guarantee binary compatibility for patch
> releases. I don't think our current guarantees of source-compatibility
> are sufficient for patch releases.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857
>
> Let me know what you think.
>
> Regards,
> Chesnay
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

[DISCUSS] FLIP-254 Guarantee binary compatibility for Public/-Evolving APIs between patch releases

[Chesnay Schepler]

Hello,

I just published a FLIP to guarantee binary compatibility for patch 
releases. I don't think our current guarantees of source-compatibility 
are sufficient for patch releases.


https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857

Let me know what you think.

Regards,
Chesnay

Re: [DISCUSS] Switch docker iamge base to Eclipse Temurin

[Chesnay Schepler]

I will optimistically merge the PRs that make the switch so we can 
gather some e2e testing data.


On 30/08/2022 14:51, Chesnay Schepler wrote:
yes, alpine would have similar issues as CentOS. As for usability, 
from personal experience it has always been a bit of a drag to extend 
or use manually because it is such a minimal image.


On 30/08/2022 14:45, Matthias Pohl wrote:
Thanks for bringing this up, Chesnay. Can you elaborate a bit more on 
what
you mean when referring to Alpine as being "not as user-friendly"? 
Doesn't

it come with the same issue that switching to CentOS comes with that we
have to update our scripts (I'm thinking of apt in particular)? Or what
else did you have in mind in terms of user-friendliness? I would imagine
selecting the required packages would be a bit more tedious.

I'm wondering whether we considered the security aspect. A more minimal
Alpine base image might reduce the risk of running into CVEs. But then;
it's also the question how fast those CVEs are actually fixed on average
(now comparing Ubuntu and Alpine for instance). Or is this even a 
concern

for us?

I didn't find any clear answers around that topic with a quick Google
search. [1] was kind of interesting to read.

Anyway, I definitely see the benefits of just switching to Ubuntu due to
the fact that it also relies on Debian's package management (reducing 
the
migration effort) and that we're using it for our CI builds 
(consistency).


+1 for going with Ubuntu if security is not a big concern

Best,
Matthias

[1]
https://jfrog.com/knowledge-base/why-use-ubuntu-as-a-docker-base-image-when-alpine-exists/ 



On Tue, Aug 30, 2022 at 11:40 AM Chesnay Schepler 
wrote:


Hello,

during the release of the 1.15.2 images
 it was
noted that we use the openjdk:8/11 images, which have been deprecated
 and thus no
longer receive any updates.

There are a number of alternatives, the most promising being Eclipse
Temurin , the successor of
AdoptOpenJDK, since it's vendor neutral.

This would imply a switch of distros from Debian to most likely Ubuntu
22.04 (Alpine isn't as user-friendly, and CentOS is likely incompatible
with existing images using our images as a base). We are also running
our CI on Ubuntu, so I don't expect any issues.

Let me know what you think.

The required changes on our side appear to be minimal; I have already
prepared a PR .

[jira] [Created] (FLINK-29156) Support LISTAGG in the Table API

[zhangjingcun (Jira)]

zhangjingcun created FLINK-29156:


 Summary: Support LISTAGG in the Table API
 Key: FLINK-29156
 URL: https://issues.apache.org/jira/browse/FLINK-29156
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Reporter: zhangjingcun


Currently, LISTAGG  are not supported in Table API.
table.group_by(col("a"))
     .select(
         col("a"),
        call("LISTAGG", col("b"), ','))



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [Discuss] KafkaSink making low level components PublicEvolving

[Piotr Nowojski]

Hi Mason,

Thanks for the clarification and great that you have found a workaround :)
I would be personally ok with making it `public class` marked as
`@Internal` if needed.

Best,
Piotrek

śr., 31 sie 2022 o 01:35 Mason Chen  napisał(a):

> Hi Piotr,
>
> The reasoning makes sense to me. I probably meant to say `public` class
> visibility and `@Internal` in my proposal, similar to other
> connector interface components (e.g. KafkaSourceReader).
>
> However, I actually found a workaround using the Public API sink builder
> and factory classes to create the sink writer, so I will abandon this
> effort for now until it would be required. Thanks for the feedback!
>
> Best,
> Mason
>
> On Tue, Aug 30, 2022 at 7:00 AM Piotr Nowojski 
> wrote:
>
> > Hi Mason,
> >
> > I would be against making them part of the Public API. It would be
> against
> > our current conventions and it would make future changes in this
> connector
> > much more difficult. It's difficult to support Public APIs, even if they
> > are `@PublicEvolving`.
> >
> > Best,
> > Piotrek
> >
> > pon., 29 sie 2022 o 19:17 Mason Chen 
> napisał(a):
> >
> > > Hi all,
> > >
> > > I know everyone is busy with preparing the 1.16 release. Just want to
> > give
> > > this thread a gentle ping, please give your +1 if there are no
> concerns!
> > >
> > > Best,
> > > Mason
> > >
> > > On Thu, Aug 11, 2022 at 1:27 AM Mason Chen 
> > wrote:
> > >
> > > > I understand that the 1.16 feature freeze is in place so I propose
> for
> > > > these changes to go in the 1.17 release.
> > > >
> > > > Best,
> > > > Mason
> > > >
> > > > On Thu, Aug 11, 2022 at 1:20 AM Mason Chen 
> > > wrote:
> > > >
> > > >> Hi devs and community,
> > > >>
> > > >> I would like to discuss making the low level components of the
> > KafkaSink
> > > >> PublicEvolving. It looks like all of the classes are package private
> > > apart
> > > >> from the public facing API and classes were put in the same package
> to
> > > make
> > > >> this work. My desire to make at least the KafkaSinkWriter a public
> > > class,
> > > >> with no further changes to private/package-private methods. The
> > > class(es)
> > > >> would be annotated with PublicEvolving.
> > > >>
> > > >> I am in the process of implementing my own Sink that writes to Kafka
> > and
> > > >> I want to implement the SinkWriter by composing the functionality
> > > contained
> > > >> in the KafkaSinkWriter. There is significant work in this
> > implementation
> > > >> and I would like to use it and contribute back, bug fixes for
> example,
> > > if I
> > > >> find any in independent testing.
> > > >>
> > > >> Hopefully, that motivates the request. Please let me know your
> > thoughts!
> > > >>
> > > >> Best,
> > > >> Mason
> > > >>
> > > >
> > >
> >
>

Re: [VOTE] FLIP-243: Dedicated Opensearch connectors

[Danny Cranmer]

Thanks for putting this together Andy. I missed the discussion but would
like to propose you consider using the Async Sink (FLIP-171) for the sink
implementation on top of the sink v2 interfaces.

+1 (binding)

Thanks,
Danny

On Thu, Aug 18, 2022 at 8:06 AM Matthias Pohl  wrote:

> I'm gonna revive this one here. I guess, having a dedicated OpenSearch
> connector makes sense considering that the ElasticSearch project drifts
> away... Thanks for looking into it Andriy.
>
> +1 (binding)
>
> On 2022/07/28 08:18:20 Martijn Visser wrote:
> > Hi Andy,
> >
> > For future threads, it's best to start a separate email with the subject
> > starting with [VOTE]. I've just created one for awareness, linking back
> to
> > this thread.
> >
> > +1 (binding) for this.
> >
> > Best regards,
> >
> > Martijn
> >
> > Op do 28 jul. 2022 om 09:46 schreef Teoh, Hong
>  > >:
> >
> > > Hi all,
> > >
> > > I'd also like to add my non-binding +1 for this FLIP. I think it's a
> great
> > > idea to make onboarding easier for new data analysts/engineers, and IMO
> > > adding more dedicated connectors is a step in the right direction.
> > >
> > >
> > > Regards,
> > > Hong
> > >
> > >
> > >
> > > On 27/07/2022, 19:15, "Ryan Skraba"  wrote:
> > >
> > > CAUTION: This email originated from outside of the organization. Do
> > > not click links or open attachments unless you can confirm the sender
> and
> > > know the content is safe.
> > >
> > >
> > >
> > > Hello!
> > >
> > > I'd like to add my non-binding +1 for this FLIP.
> > >
> > > Full disclosure: as a colleague of Andriy, I sometimes hear the
> gory
> > > details of divergence between Elasticsearch and OpenSearch.
> Objectively,
> > > this is a good reason to create independent OpenSearch connectors.
> > >
> > > As a side comment, while Elasticsearch as a trademark and service
> mark
> > > never has an internal capital S, OpenSearch always does.
> > >
> > > All my best, Ryan
> > >
> > > On 2022/07/13 20:22:11 Andriy Redko wrote:
> > > > Hey Folks,
> > > >
> > > > Thanks a lot for all the feedback and comments so far. Based on
> the
> > > discussion [1],
> > > > it seems like there is a genuine interest in supporting
> OpenSearch
> > > [2] natively. With
> > > > that being said, I would like to start a vote on FLIP-243 [3].
> > > >
> > > > The vote will last for at least 72 hours unless there is an
> > > objection or
> > > > insufficient votes.
> > > >
> > > > Thank you!
> > > >
> > > > [1]
> https://lists.apache.org/thread/jls0vqc7jb84jp14j4jok1pqfgo2cl30
> > > > [2] https://opensearch.org/
> > > > [3]
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-243%3A+Dedicated+Opensearch+connectors
> > > >
> > > > Best Regards,
> > > > Andriy Redko
> > > >
> > > >
> > >
> > >
> >
>

[jira] [Created] (FLINK-29155) Improve default config of grpcServer in Process Mode

[Xingbo Huang (Jira)]

Xingbo Huang created FLINK-29155:


 Summary: Improve default config of grpcServer in Process Mode
 Key: FLINK-29155
 URL: https://issues.apache.org/jira/browse/FLINK-29155
 Project: Flink
  Issue Type: Improvement
  Components: API / Python
Affects Versions: 1.15.2, 1.14.5, 1.16.0
Reporter: Xingbo Huang


The existing grpcServer configuration may cause channel disconnection when 
there is a large amount of data. Some PyFlink users are very troubled by this 
problem.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29154) Support LookupTableSource for table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-29154:


 Summary: Support LookupTableSource for table store
 Key: FLINK-29154
 URL: https://issues.apache.org/jira/browse/FLINK-29154
 Project: Flink
  Issue Type: New Feature
  Components: Table Store
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: table-store-0.3.0


At present, the bottom layer of Table Store is LSM structure, and it has full 
and incremental reading, which can have certain lookup capability. We can 
unlock Table Store to implement LookupTableSource.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [flink-connector-elasticsearch] Supporting ElasticSearch 8 & Dropping ES 6 ?

[Niels Basjes]

Ok, that's a bit bigger than I thought.


On Tue, Aug 30, 2022 at 10:03 AM Martijn Visser 
wrote:

> Hi Niels,
>
> With regards to ES support, we should follow the supported version that ES
> supports [1]. So if we add support for ES8, we drop support for ES6.
>
> With regards to the used clients, we can't upgrade to the latest version of
> `RestHighLevelClient`. That's because Elasticsearch has changed the license
> from version 7.11 onwards to the Apache-incompatible SSPL license [2]. What
> needs to happen is that the ES connector is refactored to use the new
> Elasticsearch Java Client [3] which is still available under Apache 2.0
> license in order to add support for ES8 clusters.
>
> For now, we need to make Elasticsearch changes to both the Flink main repo
> and the external one. Hopefully we can drop the ES connector from Flink's
> repo with Flink 1.17.
>
> Best regards,
>
> Martijn
>
> [1] https://www.elastic.co/support/eol
> [2] https://www.elastic.co/pricing/faq/licensing
> [3] https://github.com/elastic/elasticsearch-java
>
> Op do 11 aug. 2022 om 16:45 schreef Niels Basjes :
>
> > Hi,
> >
> > I was building something that puts data in ES and found that the current
> > connector is not compatible with an ES 8 cluster.
> >
> > I found that
> > - The currently employed `RestHighLevelClient` has been deprecated by
> > Elastic .
> > - There is an `ApiCompatibilityMode` you can use in the latest ES 7
> > RestHighLevelClient to connect to an ES 8 cluster (works on my machine).
> > - Between the 7.10 (used now) and the 7.17 (latest) they have introduced
> in
> > addition to the ApiCompatibilityMode also several breaking API changes by
> > changing the packages of several classes we use.
> > - If you update to the latest 7x then the
> > `flink-connector-elasticsearch-base` can no longer be shared between ES 6
> > and ES 7 simply because of these package changes.
> > - In 2022 there was still an ES 6 release but according to
> > https://www.elastic.co/support/eol it is now EOL.
> >
> > So I now see several ways forward:
> > 1) We drop ES6, update ES7 and use this compatibility mode as a quick
> > solution.
> > 2) We keep ES6, duplicating the `flink-connector-elasticsearch-base`
> (into
> > a 6.x and 7.x version with only these few classes being different),
> update
> > ES7 and use this compatibility mode as a quick solution.
> >
> > Later on someone should then create a connector based on the new
> > ElasticSearch 8 Java client API.
> >
> > See also:
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-compatibility.html
> >
> >
> https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/master/migrate-hlrc.html
> >
> > My question is
> > - Which do we do?
> > - I vote for the "We drop ES6, update ES7 and use this compatibility
> > mode"
> > - Where do we do it?
> > - We have these connectors in both the 'flink' and
> > 'flink-connector-elasticsearch' repos. Do we update only the new one? Or
> > both?
> >
> >
> > --
> > Best regards / Met vriendelijke groeten,
> >
> > Niels Basjes
> >
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Created] (FLINK-29153) KafkaConsumerThread should catch WakeupException when committing offsets

[Qingsheng Ren (Jira)]

Qingsheng Ren created FLINK-29153:
-

 Summary: KafkaConsumerThread should catch WakeupException when 
committing offsets
 Key: FLINK-29153
 URL: https://issues.apache.org/jira/browse/FLINK-29153
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.16.0
Reporter: Qingsheng Ren
Assignee: Qingsheng Ren
 Fix For: 1.16.0


{{KafkaConsumerThread}} in the legacy {{FlinkKafkaConsumer}} makes a wakeup on 
the {{KafkaConsumer}} on offset commit to wakeup the potential blocking 
{{KafkaConsumer.poll()}}. However the wakeup might happen when the consumer is 
not polling. The wakeup will be remembered by the consumer and re-examined 
while committing the offset asynchronously, which leads to an unnecessary 
{{WakeupException}}. 

As the JavaDoc and method signature of KafkaConsumer does not show that the 
WakeupException could be thrown in KafkaConsumer#commitAsync, this could be 
considered as a bug on Kafka side (breaking the contract of public API), but we 
can still fix it by catching the WakeupException and retrying. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29152) Describe statement resutls is different from the Hive

[Shengkai Fang (Jira)]

Shengkai Fang created FLINK-29152:
-

 Summary: Describe statement resutls is different from the Hive 
 Key: FLINK-29152
 URL: https://issues.apache.org/jira/browse/FLINK-29152
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Affects Versions: 1.16.0
Reporter: Shengkai Fang


In hive, the results schema is 


{code:java}
+---++--+
| col_name  | data_type  | comment  |
+---++--+
| a | int|  |
| b | string |  |
+---++--+
{code}

but our implementation is 

{code:java}
0: jdbc:hive2://localhost:1/default> describe sink;
+---+---+---+---+-++
| name  | type  | null  |  key  | extras  | watermark  |
+---+---+---+---+-++
| a | INT   | true  | NULL  | NULL| NULL   |
+---+---+---+---+-++
{code}

BTW, it's better we can support {{DESCRIBE FORMATTED}} like hive does.

{code:java}
+---++---+
|   col_name| data_type 
 |comment|
+---++---+
| # col_name| data_type 
 | comment   |
|   | NULL  
 | NULL  |
| a | int   
 |   |
| b | string
 |   |
|   | NULL  
 | NULL  |
| # Detailed Table Information  | NULL  
 | NULL  |
| Database: | default   
 | NULL  |
| Owner:| null  
 | NULL  |
| CreateTime:   | Tue Aug 30 06:54:00 UTC 2022  
 | NULL  |
| LastAccessTime:   | UNKNOWN   
 | NULL  |
| Retention:| 0 
 | NULL  |
| Location: | hdfs://namenode:8020/user/hive/warehouse/sink 
 | NULL  |
| Table Type:   | MANAGED_TABLE 
 | NULL  |
| Table Parameters: | NULL  
 | NULL  |
|   | comment   
 |   |
|   | numFiles  
 | 0 |
|   | totalSize 
 | 0 |
|   | transient_lastDdlTime 
 | 1661842440|
|   | NULL  
 | NULL  |
| # Storage Information | NULL  
 | NULL  |
| SerDe Library:| 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL  |
| InputFormat:  | org.apache.hadoop.mapred.TextInputFormat  
 | NULL  |
| OutputFormat: | 
org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat | NULL  |
| Compressed:   | No
 | NULL  |
| Num Buckets:  | -1
 | NULL  |
| Bucket Columns:   | []
 | NULL  |
| Sort Columns: | []
 | NULL  |
| Storage Desc Params:  | NULL  
 | NULL  |
|   | serialization.format  
 | 1 |
+---++---+
{code}





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29151) SHOW CREATE TABLE doesn't work for Hive dialect

[Shengkai Fang (Jira)]

Shengkai Fang created FLINK-29151:
-

 Summary: SHOW CREATE TABLE doesn't work for Hive dialect
 Key: FLINK-29151
 URL: https://issues.apache.org/jira/browse/FLINK-29151
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Affects Versions: 1.16.0
Reporter: Shengkai Fang


{code:java}
0: jdbc:hive2://localhost:1/default> show create table sink;
Error: org.apache.flink.table.gateway.service.utils.SqlExecutionException: 
Failed to execute the operation 9b060771-34b8-453d-abf5-674c86b62921.
    at 
org.apache.flink.table.gateway.service.operation.OperationManager$Operation.processThrowable(OperationManager.java:389)
    at 
org.apache.flink.table.gateway.service.operation.OperationManager$Operation.lambda$run$0(OperationManager.java:248)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.table.api.ValidationException
    at 
org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.handleUnsupportedOperation(HiveParserDDLSemanticAnalyzer.java:2188)
    at 
org.apache.flink.table.planner.delegation.hive.parse.HiveParserDDLSemanticAnalyzer.convertToOperation(HiveParserDDLSemanticAnalyzer.java:414)
    at 
org.apache.flink.table.planner.delegation.hive.HiveParser.processCmd(HiveParser.java:334)
    at 
org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:213)
    at 
org.apache.flink.table.gateway.service.operation.OperationExecutor.executeStatement(OperationExecutor.java:90)
    at 
org.apache.flink.table.gateway.service.SqlGatewayServiceImpl.lambda$executeStatement$0(SqlGatewayServiceImpl.java:182)
    at 
org.apache.flink.table.gateway.service.operation.OperationManager.lambda$submitOperation$1(OperationManager.java:111)
    at 
org.apache.flink.table.gateway.service.operation.OperationManager$Operation.lambda$run$0(OperationManager.java:239)
    ... 7 more
Caused by: java.lang.UnsupportedOperationException: Unsupported operation: 
TOK_SHOW_CREATETABLE
    ... 15 more (state=,code=0) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29150) TIMESTAMPDIFF microsecond unit for table api

[jackylau (Jira)]

jackylau created FLINK-29150:


 Summary: TIMESTAMPDIFF  microsecond unit for table api
 Key: FLINK-29150
 URL: https://issues.apache.org/jira/browse/FLINK-29150
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.17.0
Reporter: jackylau
 Fix For: 1.17.0


like mysql https://www.geeksforgeeks.org/timestampdiff-function-in-mysql/



--
This message was sent by Atlassian Jira
(v8.20.10#820010)