Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

[Александр Смирнов]

Hi Qingsheng,

Thank you for considering my comments.

>  there might be custom logic before making retry, such as re-establish the 
> connection

Yes, I understand that. I meant that such logic can be placed in a
separate function, that can be implemented by connectors. Just moving
the retry logic would make connector's LookupFunction more concise +
avoid duplicate code. However, it's a minor change. The decision is up
to you.

> We decide not to provide common DDL options and let developers to define 
> their own options as we do now per connector.

What is the reason for that? One of the main goals of this FLIP was to
unify the configs, wasn't it? I understand that current cache design
doesn't depend on ConfigOptions, like was before. But still we can put
these options into the framework, so connectors can reuse them and
avoid code duplication, and, what is more significant, avoid possible
different options naming. This moment can be pointed out in
documentation for connector developers.

Best regards,
Alexander

вт, 17 мая 2022 г. в 17:11, Qingsheng Ren :
>
> Hi Alexander,
>
> Thanks for the review and glad to see we are on the same page! I think you 
> forgot to cc the dev mailing list so I’m also quoting your reply under this 
> email.
>
> >  We can add 'maxRetryTimes' option into this class
>
> In my opinion the retry logic should be implemented in lookup() instead of in 
> LookupFunction#eval(). Retrying is only meaningful under some specific 
> retriable failures, and there might be custom logic before making retry, such 
> as re-establish the connection (JdbcRowDataLookupFunction is an example), so 
> it's more handy to leave it to the connector.
>
> > I don't see DDL options, that were in previous version of FLIP. Do you have 
> > any special plans for them?
>
> We decide not to provide common DDL options and let developers to define 
> their own options as we do now per connector.
>
> The rest of comments sound great and I’ll update the FLIP. Hope we can 
> finalize our proposal soon!
>
> Best,
>
> Qingsheng
>
>
> > On May 17, 2022, at 13:46, Александр Смирнов  wrote:
> >
> > Hi Qingsheng and devs!
> >
> > I like the overall design of updated FLIP, however I have several
> > suggestions and questions.
> >
> > 1) Introducing LookupFunction as a subclass of TableFunction is a good
> > idea. We can add 'maxRetryTimes' option into this class. 'eval' method
> > of new LookupFunction is great for this purpose. The same is for
> > 'async' case.
> >
> > 2) There might be other configs in future, such as 'cacheMissingKey'
> > in LookupFunctionProvider or 'rescanInterval' in ScanRuntimeProvider.
> > Maybe use Builder pattern in LookupFunctionProvider and
> > RescanRuntimeProvider for more flexibility (use one 'build' method
> > instead of many 'of' methods in future)?
> >
> > 3) What are the plans for existing TableFunctionProvider and
> > AsyncTableFunctionProvider? I think they should be deprecated.
> >
> > 4) Am I right that the current design does not assume usage of
> > user-provided LookupCache in re-scanning? In this case, it is not very
> > clear why do we need methods such as 'invalidate' or 'putAll' in
> > LookupCache.
> >
> > 5) I don't see DDL options, that were in previous version of FLIP. Do
> > you have any special plans for them?
> >
> > If you don't mind, I would be glad to be able to make small
> > adjustments to the FLIP document too. I think it's worth mentioning
> > about what exactly optimizations are planning in the future.
> >
> > Best regards,
> > Smirnov Alexander
> >
> > пт, 13 мая 2022 г. в 20:27, Qingsheng Ren :
> >>
> >> Hi Alexander and devs,
> >>
> >> Thank you very much for the in-depth discussion! As Jark mentioned we were 
> >> inspired by Alexander's idea and made a refactor on our design. FLIP-221 
> >> [1] has been updated to reflect our design now and we are happy to hear 
> >> more suggestions from you!
> >>
> >> Compared to the previous design:
> >> 1. The lookup cache serves at table runtime level and is integrated as a 
> >> component of LookupJoinRunner as discussed previously.
> >> 2. Interfaces are renamed and re-designed to reflect the new design.
> >> 3. We separate the all-caching case individually and introduce a new 
> >> RescanRuntimeProvider to reuse the ability of scanning. We are planning to 
> >> support SourceFunction / InputFormat for now considering the complexity of 
> >> FLIP-27 Source API.
> >> 4. A new interface LookupFunction is introduced to make the semantic of 
> >> lookup more straightforward for developers.
> >>
> >> For replying to Alexander:
> >>> However I'm a little confused whether InputFormat is deprecated or not. 
> >>> Am I right that it will be so in the future, but currently it's not?
> >> Yes you are right. InputFormat is not deprecated for now. I think it will 
> >> be deprecated in the future but we don't have a clear plan for that.
> >>
> >> Thanks again for the discussion on this FLIP and looking forward to 
> >> 

Re: [VOTE] FLIP-229: Introduces Join Hint for Flink SQL Batch Job

[godfrey he]

Thanks Xuyang for driving this, +1(binding)

Best,
Godfrey

Xuyang  于2022年5月17日周二 10:21写道：
>
> Hi, everyone.
> Thanks for your feedback for FLIP-229: Introduces Join Hint for Flink SQL 
> Batch Job[1] on the discussion thread[2].
> I'd like to start a vote for it. The vote will be open for at least 72 hours 
> unless there is an objection or not enough votes.
>
> --
>
> Best！
> Xuyang
>
>
> [1] 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-229%3A+Introduces+Join+Hint+for+Flink+SQL+Batch+Job
> [2] https://lists.apache.org/thread/y668bxyjz66ggtjypfz9t571m0tyvv9h

[jira] [Created] (FLINK-27675) Improve manual savepoint tracking

[Gyula Fora (Jira)]

Gyula Fora created FLINK-27675:
--

 Summary: Improve manual savepoint tracking
 Key: FLINK-27675
 URL: https://issues.apache.org/jira/browse/FLINK-27675
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.0.0


There are 2 problems with the manual savpeoint result observing logic that can 
cause the reconciler to not make progress with the deployment (recoveries, 
upgrades etc).
 # Whenever the jobmanager deployment is not in READY state or the job itself 
is not RUNNING, the trigger info must be reset and we should not try to query 
it anymore. Flink will not retry the savepoint if the job fails, restarted 
anyways.
 # If there is a sensible error when fetching the savepoint status (such as: 
There is no savepoint operation with triggerId=xxx for job ) we should simply 
reset the trigger. These errors will never go away on their own and will simply 
cause the deployment to get stuck in observing/waiting for a savepoint to 
complete



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: Re: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)

[godfrey he]

Hi Mang,

Thanks for driving this FLIP.

Please follow the FLIP template[1] style, and the `Syntax ` is part of
the `Public API Changes` section.
‘Program research’ and 'Implementation Plan' are part of the `Proposed
Changes` section,
or move ‘Program research’ to the appendix.

> Providing methods that are used to execute CTAS for Table API users.
We should introduce `createTable` in `Table` instead of `TableEnvironment`.
Because all table operations are defined in `Table`, see: Table#executeInsert,
Table#insertInto, etc.
About the method name, I prefer to use `createTableAs`.

> TableSink needs to provide the CleanUp API, developers implement as needed.
I think it's hard for TableSink to implement a clean up operation. For
file system sink,
the data can be written to a temporary directory, but for key/value
sinks, it's hard to
remove the written keys, unless the sink records all written keys.

> Do not do drop table operations in the framework, drop table is implemented in
TableSink according to the needs of specific TableSink
The TM process may crash at any time, and the drop operation will not
be executed any more.

How about we do the drop table operation and cleanup data action in the catalog?
Where to execute the drop operation. one approach is in client, other is in JM.
1. in client: this requires the client to be alive until the job is
finished and failed.
2. in JM: this requires the JM could provide some interfaces/hooks
that the planner
implements the logic and the code will be executed in JM.
I prefer the approach two, but it requires more detail design with
runtime @gaoyunhaii, @kevin.yingjie


[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template

Best,
Godfrey


Mang Zhang  于2022年5月6日周五 11:24写道：

>
> Hi, Yuxia
> Thanks for your reply!
> About the question 1, we will not support, FLIP-218[1] is to simplify the 
> complexity of user DDL and make it easier for users to use. I have never 
> encountered this case in a big data.
> About the question 2, we will provide a public API like below public void 
> cleanUp();
>
>   Regarding the mechanism of cleanUp, people who are familiar with the 
> runtime module need to provide professional advice, which is what we need to 
> focus on.
>
>
>
>
>
>
>
>
>
>
> --
>
> Best regards,
> Mang Zhang
>
>
>
>
>
> At 2022-04-29 17:00:03, "yuxia"  wrote:
> >Thanks for for driving this work, it's to be a useful feature.
> >About the flip-218, I have some questions.
> >
> >1: Does our CTAS syntax support specify target table's schema including 
> >column name and data type? I think it maybe a useful fature in case we want 
> >to change the data types in target table instead of always copy the source 
> >table's schema. It'll be more flexible with this feature.
> >Btw, MySQL's "CREATE TABLE ... SELECT Statement"[1] support this feature.
> >
> >2: Seems it'll requre sink to implement an public interface to drop table, 
> >so what's the interface will look like?
> >
> >[1] https://dev.mysql.com/doc/refman/8.0/en/create-table-select.html
> >
> >Best regards,
> >Yuxia
> >
> >- 原始邮件 -
> >发件人: "Mang Zhang" 
> >收件人: "dev" 
> >发送时间: 星期四, 2022年 4 月 28日 下午 4:57:24
> >主题: [DISCUSS] FLIP-218: Support SELECT clause in CREATE TABLE(CTAS)
> >
> >Hi, everyone
> >
> >
> >I would like to open a discussion for support select clause in CREATE 
> >TABLE(CTAS),
> >With the development of business and the enhancement of flink sql 
> >capabilities, queries become more and more complex.
> >Now the user needs to use the Create Table statement to create the target 
> >table first, and then execute the insert statement.
> >However, the target table may have many columns, which will bring a lot of 
> >work outside the business logic to the user.
> >At the same time, ensure that the schema of the created target table is 
> >consistent with the schema of the query result.
> >Using a CTAS syntax like Hive/Spark can greatly facilitate the user.
> >
> >
> >
> >You can find more details in FLIP-218[1]. Looking forward to your feedback.
> >
> >
> >
> >[1] 
> >https://cwiki.apache.org/confluence/display/FLINK/FLIP-218%3A+Support+SELECT+clause+in+CREATE+TABLE(CTAS)
> >
> >
> >
> >
> >--
> >
> >Best regards,
> >Mang Zhang
>
>

Re:Metrics in Flink UI

[JasonLee]

Hi Zain


Do you have only one operator or all of the operators are chained together? 
Maybe you can break the operator chain and check it out.


Best
JasonLee


 Replied Message 
| From | Zain Haider Nemati |
| Date | 05/18/2022 03:28 |
| To | user ,
 |
| Subject | Metrics in Flink UI |
Hi,
I'm running a job on a local flink cluster but metrics are showing as Bytes
received,records received,bytes sent,backpressure all 0 in the flink UI
even though I'm receiving data in the sink.
Do I need to additionally configure something to see these metrics work in
real time?

[jira] [Created] (FLINK-27674) Elasticsearch6SinkE2ECase test hangs

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-27674:


 Summary: Elasticsearch6SinkE2ECase test hangs
 Key: FLINK-27674
 URL: https://issues.apache.org/jira/browse/FLINK-27674
 Project: Flink
  Issue Type: Bug
  Components: Connectors / ElasticSearch
Affects Versions: 1.16.0
Reporter: Huang Xingbo



{code:java}
2022-05-17T14:23:20.2220667Z May 17 14:23:20 [INFO] Running 
org.apache.flink.streaming.tests.Elasticsearch6SinkE2ECase
2022-05-17T16:46:42.2757438Z 
==
2022-05-17T16:46:42.2769104Z === WARNING: This task took already 95% of the 
available time budget of 284 minutes ===
2022-05-17T16:46:42.2769752Z 
==
2022-05-17T16:46:42.2777415Z 
==
2022-05-17T16:46:42.2778082Z The following Java processes are running (JPS)
2022-05-17T16:46:42.2778711Z 
==
2022-05-17T16:46:42.4370838Z 366642 surefirebooter7296810110127514313.jar
2022-05-17T16:46:42.4371962Z 323278 Launcher
2022-05-17T16:46:42.4378950Z 384312 Jps
2022-05-17T16:46:42.4452836Z 
==
2022-05-17T16:46:42.4453843Z Printing stack trace of Java process 366642
2022-05-17T16:46:42.4454796Z 
==
2022-05-17T16:46:42.9967200Z 2022-05-17 16:46:42
2022-05-17T16:46:42.9968158Z Full thread dump OpenJDK 64-Bit Server VM 
(25.332-b09 mixed mode):
2022-05-17T16:46:42.9968402Z 
2022-05-17T16:46:42.9968953Z "Attach Listener" #6017 daemon prio=9 os_prio=0 
tid=0x7f4cf0007000 nid=0x5dd53 waiting on condition [0x]
2022-05-17T16:46:42.9969533Zjava.lang.Thread.State: RUNNABLE
2022-05-17T16:46:42.9969714Z 
2022-05-17T16:46:42.9970494Z "ForkJoinPool-1-worker-0" #6011 daemon prio=5 
os_prio=0 tid=0x7f4b4ed92800 nid=0x5cfea waiting on condition 
[0x7f4b35b9b000]
2022-05-17T16:46:42.9971118Zjava.lang.Thread.State: TIMED_WAITING (parking)
2022-05-17T16:46:42.9971752Zat sun.misc.Unsafe.park(Native Method)
2022-05-17T16:46:42.9972494Z- parking to wait for  <0x939d6cd0> (a 
java.util.concurrent.ForkJoinPool)
2022-05-17T16:46:42.9973134Zat 
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1824)
2022-05-17T16:46:42.9973801Zat 
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1693)
2022-05-17T16:46:43.0113835Zat 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2022-05-17T16:46:43.0114433Z 
2022-05-17T16:46:43.0118656Z "flink-rest-client-netty-thread-1" #6006 daemon 
prio=5 os_prio=0 tid=0x7f4b4ed8f000 nid=0x5bb03 runnable 
[0x7f4b34181000]
2022-05-17T16:46:43.0119406Zjava.lang.Thread.State: RUNNABLE
2022-05-17T16:46:43.0120063Zat 
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
2022-05-17T16:46:43.0120921Zat 
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
2022-05-17T16:46:43.0121666Zat 
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
2022-05-17T16:46:43.0122507Zat 
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
2022-05-17T16:46:43.0123735Z- locked <0xe4e90c80> (a 
org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySet)
2022-05-17T16:46:43.0124873Z- locked <0xe4e90a90> (a 
java.util.Collections$UnmodifiableSet)
2022-05-17T16:46:43.0125863Z- locked <0xe4e909b8> (a 
sun.nio.ch.EPollSelectorImpl)
2022-05-17T16:46:43.0126648Zat 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
2022-05-17T16:46:43.0127370Zat 
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
2022-05-17T16:46:43.0128211Zat 
org.apache.flink.shaded.netty4.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)
2022-05-17T16:46:43.0129260Zat 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:810)
2022-05-17T16:46:43.0130119Zat 
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457)
2022-05-17T16:46:43.0131134Zat 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
2022-05-17T16:46:43.0137869Zat 
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2022-05-17T16:46:43.0138683Zat java.lang.Thread.run(Thread.java:750)
{code}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35749=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a



--
This message was sent by Atlassian Jira

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

[Lijie Wang]

Hi Weihua,
thanks for feedback.

1. Yes, only *Manually* is supported in this FLIP, but it's the first step
towards auto-detection.
2. We wii print the blocked nodes in logs. Maybe also put it into the
exception of insufficient resources.
3. No. This FLIP won't change the WebUI. The blocklist information can be
obtained through REST API and metrics.

Best,
Lijie

Weihua Hu  于2022年5月17日周二 21:41写道：

> Hi,
> Thanks for creating this FLIP.
> We have implemented an automatic blocklist detection mechanism internally,
> which is indeed very effective for handling node failures.
> Due to the large number of nodes, although SREs already support automatic
> offline failure nodes, the detection is not 100% accurate and there is some
> delay.
> So the blocklist mechanism can make flink job recover from failure much
> faster.
>
> Here are some of my thoughts:
> 1. In this FLIP, it needs users to locate machine failure manually, there
> is a certain cost of use
> 2. What happens if too many nodes are blocked, resulting in insufficient
> resources? Will there be a special Exception for the user?
> 3. Will we display the blocklist information in the WebUI? The blocklist
> is for cluster level, and if multiple users share a cluster, some users may
> be a little confused when resources are not enough, or when resources are
> applied for more.
>
> Also, Looking forward to the next FLIP on auto-detection.
>
> Best,
> Weihua
>
> > 2022年5月16日 下午11:22，Lijie Wang  写道：
> >
> > Hi Konstantin,
> >
> > Maybe change it to the following:
> >
> > 1. POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}
> > Merge is not allowed. If the {id} already exists, return error.
> Otherwise,
> > create a new item.
> >
> > 2. POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge
> > Merge is allowed. If the {id} already exists, merge. Otherwise, create a
> > new item.
> >
> > WDYT?
> >
> > Best,
> > Lijie
> >
> > Konstantin Knauf  于2022年5月16日周一 20:07写道：
> >
> >> Hi Lijie,
> >>
> >> hm, maybe the following is more appropriate in that case
> >>
> >> POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge
> >>
> >> Best,
> >>
> >> Konstantin
> >>
> >> Am Mo., 16. Mai 2022 um 07:05 Uhr schrieb Lijie Wang <
> >> wangdachui9...@gmail.com>:
> >>
> >>> Hi Konstantin,
> >>> thanks for your feedback.
> >>>
> >>> From what I understand, PUT should be idempotent. However, we have a
> >>> *timeout* field in the request. This means that initiating the same
> >> request
> >>> at two different times will lead to different resource status
> (timestamps
> >>> of the items to be removed will be different).
> >>>
> >>> Should we use PUT in this case? WDYT?
> >>>
> >>> Best,
> >>> Lijie
> >>>
> >>> Konstantin Knauf  于2022年5月13日周五 17:20写道：
> >>>
>  Hi Lijie,
> 
>  wouldn't the REST API-idiomatic way for an update/replace be a PUT on
> >> the
>  resource?
> 
>  PUT: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}
> 
>  Best,
> 
>  Konstantin
> 
> 
> 
>  Am Fr., 13. Mai 2022 um 11:01 Uhr schrieb Lijie Wang <
>  wangdachui9...@gmail.com>:
> 
> > Hi everyone,
> >
> > I've had an offline discussion with Becket Qin and Zhu Zhu, and made
> >>> the
> > following changes on REST API:
> > 1. To avoid ambiguity, *timeout* and *endTimestamp* can only choose
> >>> one.
>  If
> > both are specified, will return error.
> > 2.  If the specified item is already there, the *ADD* operation has
> >> two
> > behaviors:  *return error*(default value) or *merge/update*, and we
> >>> add a
> > flag to the request body to control it. You can find more details
> >>> "Public
> > Interface" section.
> >
> > If there is no more feedback, we will start the vote thread next
> >> week.
> >
> > Best,
> > Lijie
> >
> > Lijie Wang  于2022年5月10日周二 17:14写道：
> >
> >> Hi Becket Qin,
> >>
> >> Thanks for your suggestions.  I have moved the description of
> >> configurations, metrics and REST API into "Public Interface"
> >> section,
>  and
> >> made a few updates according to your suggestion.  And in this FLIP,
>  there
> >> no public java Interfaces or pluggables that users need to
> >> implement
> >>> by
> >> themselves.
> >>
> >> Answers for you questions:
> >> 1. Yes, there 2 block actions: MARK_BLOCKED and.
> >> MARK_BLOCKED_AND_EVACUATE_TASKS (has renamed). Currently, block
> >> items
>  can
> >> only be added through the REST API, so these 2 action are mentioned
> >>> in
> > the
> >> REST API part (The REST API part has beed moved to public interface
>  now).
> >> 2. I agree with you. I have changed the "Cause" field to String,
> >> and
> > allow
> >> users to specify it via REST API.
> >> 3. Yes, it is useful to allow different timeouts. As mentioned
> >> above,
>  we
> >> will introduce 2 fields : *timeout* and 

Memory configuration for Queue

[Zain Haider Nemati]

Hi,
I am using a kafka source with a kinesis sink and the speed of data coming
in is not the same as data flowing out hence the need to configure a
relatively larger queue to hold the data before backpressuring. Which
memory configuration corresponds to this that I'll need to configure?

Metrics in Flink UI

[Zain Haider Nemati]

Hi,
I'm running a job on a local flink cluster but metrics are showing as Bytes
received,records received,bytes sent,backpressure all 0 in the flink UI
even though I'm receiving data in the sink.
Do I need to additionally configure something to see these metrics work in
real time?

[jira] [Created] (FLINK-27673) [JUnit5 Migration] Module: flink-table-api-scala

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-27673:
---

 Summary: [JUnit5 Migration] Module: flink-table-api-scala
 Key: FLINK-27673
 URL: https://issues.apache.org/jira/browse/FLINK-27673
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API, Tests
Reporter: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Data Engineering Track at ApacheCon (October 3-6, New Orleans) - CFP ends 23/05

[Ismaël Mejía]

Hello,

ApacheCon North America is back in person this year in October.
https://apachecon.com/acna2022/

Together with Jarek Potiuk, we are organizing for the first time a Data
Engineering Track as part of ApacheCon.

You might be wondering why a different track if we already have the Big Data
track. Simple, this new track covers the ‘other’ open-source projects we use to
clean data, orchestrate workloads, do observability, visualization, governance,
data lineage and many other tasks that are part of data engineering and that are
usually not covered by the data processing / database tracks.

If you are curious you can find more details here:
https://s.apache.org/apacheconna-2022-dataeng-track

So why are you getting this message? Well it could be that (1) you are
already a contributor to a project in the data engineering space and you
might be interested in sending your proposal, or (2) you are interested in
integrations of these tools with your existing data tools/projects.

If you are interested you can submit a proposal using the CfP link below.
Don’t forget to choose the Data Engineering Track.
https://apachecon.com/acna2022/cfp.html

The Call for Presentations (CfP) closes in less than one week on May 23th,
2022.

We are looking forward to receiving your submissions and hopefully seeing you in
New Orleans in October.

Thanks,
Ismaël and Jarek

ps. Excuses if you already received this email by a different channel/ML

[jira] [Created] (FLINK-27672) [JUnit5 Migration] Module: flink-table-common

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-27672:
---

 Summary: [JUnit5 Migration] Module: flink-table-common
 Key: FLINK-27672
 URL: https://issues.apache.org/jira/browse/FLINK-27672
 Project: Flink
  Issue Type: Sub-task
Reporter: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] Creating an Apache Flink slack workspace

[Gyula Fóra]

+1 (binding)

On Tue, 17 May 2022 at 19:52, Yufei Zhang  wrote:

> +1 (nonbinding)
>
> On Tue, May 17, 2022 at 5:29 PM Márton Balassi 
> wrote:
>
> > +1 (binding)
> >
> > On Tue, May 17, 2022 at 11:00 AM Jingsong Li 
> > wrote:
> >
> > > Thank Xintong for driving this work.
> > >
> > > +1
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Tue, May 17, 2022 at 4:49 PM Martijn Visser <
> martijnvis...@apache.org
> > >
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > On Tue, 17 May 2022 at 10:38, Yu Li  wrote:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > Thanks Xintong for driving this!
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Tue, 17 May 2022 at 16:32, Robert Metzger 
> > > > wrote:
> > > > >
> > > > > > Thanks for starting the VOTE!
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, May 17, 2022 at 10:29 AM Jark Wu 
> wrote:
> > > > > >
> > > > > > > Thank Xintong for driving this work.
> > > > > > >
> > > > > > > +1 from my side (binding)
> > > > > > >
> > > > > > > Best,
> > > > > > > Jark
> > > > > > >
> > > > > > > On Tue, 17 May 2022 at 16:24, Xintong Song <
> > tonysong...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > As previously discussed in [1], I would like to open a vote
> on
> > > > > creating
> > > > > > > an
> > > > > > > > Apache Flink slack workspace channel.
> > > > > > > >
> > > > > > > > The proposed actions include:
> > > > > > > > - Creating a dedicated slack workspace with the name Apache
> > Flink
> > > > > that
> > > > > > is
> > > > > > > > controlled and maintained by the Apache Flink PMC
> > > > > > > > - Updating the Flink website about rules for using various
> > > > > > communication
> > > > > > > > channels
> > > > > > > > - Setting up an Archive for the Apache Flink slack
> > > > > > > > - Revisiting this initiative by the end of 2022
> > > > > > > >
> > > > > > > > The vote will last for at least 72 hours, and will be
> accepted
> > > by a
> > > > > > > > consensus of active PMC members.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Xintong
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Creating an Apache Flink slack workspace

[Yufei Zhang]

+1 (nonbinding)

On Tue, May 17, 2022 at 5:29 PM Márton Balassi 
wrote:

> +1 (binding)
>
> On Tue, May 17, 2022 at 11:00 AM Jingsong Li 
> wrote:
>
> > Thank Xintong for driving this work.
> >
> > +1
> >
> > Best,
> > Jingsong
> >
> > On Tue, May 17, 2022 at 4:49 PM Martijn Visser  >
> > wrote:
> >
> > > +1 (binding)
> > >
> > > On Tue, 17 May 2022 at 10:38, Yu Li  wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks Xintong for driving this!
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Tue, 17 May 2022 at 16:32, Robert Metzger 
> > > wrote:
> > > >
> > > > > Thanks for starting the VOTE!
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 17, 2022 at 10:29 AM Jark Wu  wrote:
> > > > >
> > > > > > Thank Xintong for driving this work.
> > > > > >
> > > > > > +1 from my side (binding)
> > > > > >
> > > > > > Best,
> > > > > > Jark
> > > > > >
> > > > > > On Tue, 17 May 2022 at 16:24, Xintong Song <
> tonysong...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > As previously discussed in [1], I would like to open a vote on
> > > > creating
> > > > > > an
> > > > > > > Apache Flink slack workspace channel.
> > > > > > >
> > > > > > > The proposed actions include:
> > > > > > > - Creating a dedicated slack workspace with the name Apache
> Flink
> > > > that
> > > > > is
> > > > > > > controlled and maintained by the Apache Flink PMC
> > > > > > > - Updating the Flink website about rules for using various
> > > > > communication
> > > > > > > channels
> > > > > > > - Setting up an Archive for the Apache Flink slack
> > > > > > > - Revisiting this initiative by the end of 2022
> > > > > > >
> > > > > > > The vote will last for at least 72 hours, and will be accepted
> > by a
> > > > > > > consensus of active PMC members.
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Xintong
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] Kubernetes Operator release-1.0 branch cut

[Márton Balassi]

Thanks Gyula and Yang. Awesome!

On Tue, May 17, 2022 at 4:46 PM Gyula Fóra  wrote:

> Hi Flink devs!
>
> The release-1.0 branch has been forked from main and version numbers have
> been upgraded accordingly.
>
> https://github.com/apache/flink-kubernetes-operator/tree/release-1.0
>
> The version on the main branch has been updated to 1.1-SNAPSHOT.
>
> From now on, for PRs that should be presented in 1.0.0, please make sure:
> - Merge the PRs first to main, then backport to release-1.0 branch
> - The JIRA ticket should be closed with the correct fix-versions
>
> There are still a few outstanding tickets, mainly docs/minor fixes but no
> currently known blocker issues.
>
> We are working together with Yang to prepare the first RC by next monday.
>
> Cheers,
> Gyula
>

[jira] [Created] (FLINK-27671) Publish SNAPSHOT docker images

[Alexander Fedulov (Jira)]

Alexander Fedulov created FLINK-27671:
-

 Summary: Publish SNAPSHOT docker images
 Key: FLINK-27671
 URL: https://issues.apache.org/jira/browse/FLINK-27671
 Project: Flink
  Issue Type: Improvement
Reporter: Alexander Fedulov
Assignee: Alexander Fedulov


A discussion on the Flink-Dev mailing list [1] concluded that it is desirable 
to add the ability to publish SNAPSHOT Docker images of Apache Flink to GHCR. 


[1] [https://lists.apache.org/thread/0h60qz8vrw980n1vscz3pxcsv3c3h24m]



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] FLIP-233: Introduce HTTP Connector

[Austin Cawley-Edwards]

Hey Jeremy,

Thanks for kicking off this discussion. As a Flink user, I too struggled
with the lack of HTTP support and rolled my own with AsyncIO. Reading
through the FLIP, I just have a few general questions and comments.

* It is not clear to me if multiple HTTP methods are supported or not? It's
listed in "Limitations" that only POSTs are allowed, but the constructor
accepts a "method" argument.
* More of a nit, the FLIP contains a lot of code, making it feel a bit more
like a PR already. It would be easier to understand the proposed interfaces
alone, and keep the implementation POC as a separate link IMO.

Since there are so many different types of HTTP APIs, and many different
ways of using them, I think the proposal would benefit from taking a more
general approach to both request building and response handling. For
instance, some APIs may return hints for retry that are not contained in
the status code alone (e.g., a "retry-after" header or such + a 429 status
code). Can we already think about how to more generally expose these two
concepts? For the retries, it might be too idealistic, but standardizing on
a retry interface like the one proposed in FLIP-232[1] would make these
aysnc/http APIs feel much more aligned.

wdyt?

Austin

[1]:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211883963

On Tue, May 17, 2022 at 10:21 AM Ber, Jeremy 
wrote:

> Hi there, We would like to start a discussion thread on FLIP-233:
> Introduce HTTP Connector<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-233%3A+Introduce+HTTP+Connector>
> where we propose to create a connector for delivering arbitrary data
> packets from Apache Flink to an HTTP Endpoint.  This connector will give
> users flexibility to deliver data to any destination which supports REST
> endpoints. This includes REST APIs, Amazon Lambda, users internal or
> external consumers, among other options.
>
> Looking forward to your feedback.
>
> Thank you,
> Jeremy Ber
>
>

[ANNOUNCE] Kubernetes Operator release-1.0 branch cut

[Gyula Fóra]

Hi Flink devs!

The release-1.0 branch has been forked from main and version numbers have
been upgraded accordingly.

https://github.com/apache/flink-kubernetes-operator/tree/release-1.0

The version on the main branch has been updated to 1.1-SNAPSHOT.

>From now on, for PRs that should be presented in 1.0.0, please make sure:
- Merge the PRs first to main, then backport to release-1.0 branch
- The JIRA ticket should be closed with the correct fix-versions

There are still a few outstanding tickets, mainly docs/minor fixes but no
currently known blocker issues.

We are working together with Yang to prepare the first RC by next monday.

Cheers,
Gyula

[jira] [Created] (FLINK-27670) Python wrappers for Kinesis Sinks

[Zichen Liu (Jira)]

Zichen Liu created FLINK-27670:
--

 Summary: Python wrappers for Kinesis Sinks
 Key: FLINK-27670
 URL: https://issues.apache.org/jira/browse/FLINK-27670
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kinesis
Reporter: Zichen Liu
 Fix For: 1.15.1


Create Python Wrappers for the new Kinesis Streams sink and the Kinesis 
Firehose sink.

An example implementation may be found here 
[https://github.com/apache/flink/pull/15491/files] for the old Kinesis sink.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[DISCUSS] FLIP-233: Introduce HTTP Connector

[Ber, Jeremy]

Hi there, We would like to start a discussion thread on FLIP-233: Introduce 
HTTP 
Connector
 where we propose to create a connector for delivering arbitrary data packets 
from Apache Flink to an HTTP Endpoint.  This connector will give users 
flexibility to deliver data to any destination which supports REST endpoints. 
This includes REST APIs, Amazon Lambda, users internal or external consumers, 
among other options.

Looking forward to your feedback.

Thank you,
Jeremy Ber

[jira] [Created] (FLINK-27669) [JUnit5 Migration] Migrate flink-file-sink-common

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-27669:
---

 Summary: [JUnit5 Migration] Migrate flink-file-sink-common
 Key: FLINK-27669
 URL: https://issues.apache.org/jira/browse/FLINK-27669
 Project: Flink
  Issue Type: Sub-task
Reporter: Sergey Nuyanzin






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] FLIP-224: Blacklist Mechanism

[Weihua Hu]

Hi,
Thanks for creating this FLIP.
We have implemented an automatic blocklist detection mechanism internally, 
which is indeed very effective for handling node failures. 
Due to the large number of nodes, although SREs already support automatic 
offline failure nodes, the detection is not 100% accurate and there is some 
delay.
So the blocklist mechanism can make flink job recover from failure much faster.

Here are some of my thoughts:
1. In this FLIP, it needs users to locate machine failure manually, there is a 
certain cost of use
2. What happens if too many nodes are blocked, resulting in insufficient 
resources? Will there be a special Exception for the user?
3. Will we display the blocklist information in the WebUI? The blocklist is for 
cluster level, and if multiple users share a cluster, some users may be a 
little confused when resources are not enough, or when resources are applied 
for more.

Also, Looking forward to the next FLIP on auto-detection.

Best,
Weihua

> 2022年5月16日 下午11:22，Lijie Wang  写道：
> 
> Hi Konstantin,
> 
> Maybe change it to the following:
> 
> 1. POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}
> Merge is not allowed. If the {id} already exists, return error. Otherwise,
> create a new item.
> 
> 2. POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge
> Merge is allowed. If the {id} already exists, merge. Otherwise, create a
> new item.
> 
> WDYT?
> 
> Best,
> Lijie
> 
> Konstantin Knauf  于2022年5月16日周一 20:07写道：
> 
>> Hi Lijie,
>> 
>> hm, maybe the following is more appropriate in that case
>> 
>> POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge
>> 
>> Best,
>> 
>> Konstantin
>> 
>> Am Mo., 16. Mai 2022 um 07:05 Uhr schrieb Lijie Wang <
>> wangdachui9...@gmail.com>:
>> 
>>> Hi Konstantin,
>>> thanks for your feedback.
>>> 
>>> From what I understand, PUT should be idempotent. However, we have a
>>> *timeout* field in the request. This means that initiating the same
>> request
>>> at two different times will lead to different resource status (timestamps
>>> of the items to be removed will be different).
>>> 
>>> Should we use PUT in this case? WDYT?
>>> 
>>> Best,
>>> Lijie
>>> 
>>> Konstantin Knauf  于2022年5月13日周五 17:20写道：
>>> 
 Hi Lijie,
 
 wouldn't the REST API-idiomatic way for an update/replace be a PUT on
>> the
 resource?
 
 PUT: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}
 
 Best,
 
 Konstantin
 
 
 
 Am Fr., 13. Mai 2022 um 11:01 Uhr schrieb Lijie Wang <
 wangdachui9...@gmail.com>:
 
> Hi everyone,
> 
> I've had an offline discussion with Becket Qin and Zhu Zhu, and made
>>> the
> following changes on REST API:
> 1. To avoid ambiguity, *timeout* and *endTimestamp* can only choose
>>> one.
 If
> both are specified, will return error.
> 2.  If the specified item is already there, the *ADD* operation has
>> two
> behaviors:  *return error*(default value) or *merge/update*, and we
>>> add a
> flag to the request body to control it. You can find more details
>>> "Public
> Interface" section.
> 
> If there is no more feedback, we will start the vote thread next
>> week.
> 
> Best,
> Lijie
> 
> Lijie Wang  于2022年5月10日周二 17:14写道：
> 
>> Hi Becket Qin,
>> 
>> Thanks for your suggestions.  I have moved the description of
>> configurations, metrics and REST API into "Public Interface"
>> section,
 and
>> made a few updates according to your suggestion.  And in this FLIP,
 there
>> no public java Interfaces or pluggables that users need to
>> implement
>>> by
>> themselves.
>> 
>> Answers for you questions:
>> 1. Yes, there 2 block actions: MARK_BLOCKED and.
>> MARK_BLOCKED_AND_EVACUATE_TASKS (has renamed). Currently, block
>> items
 can
>> only be added through the REST API, so these 2 action are mentioned
>>> in
> the
>> REST API part (The REST API part has beed moved to public interface
 now).
>> 2. I agree with you. I have changed the "Cause" field to String,
>> and
> allow
>> users to specify it via REST API.
>> 3. Yes, it is useful to allow different timeouts. As mentioned
>> above,
 we
>> will introduce 2 fields : *timeout* and *endTimestamp* into the ADD
 REST
>> API to specify when to remove the blocked item. These 2 fields are
>> optional, if neither is specified, it means that the blocked item
>> is
>> permanent and will not be removed. If both are specified, the
>> minimum
 of
>> *currentTimestamp+tiemout *and* endTimestamp* will be used as the
>>> time
 to
>> remove the blocked item. To keep the configurations more minimal,
>> we
 have
>> removed the *cluster.resource-blocklist.item.timeout* configuration
>> option.
>> 4. Yes, the block item will be overridden if the specified item
>>> already
>> exists. The ADD 

[jira] [Created] (FLINK-27668) Document dynamic operator configuration

[Gyula Fora (Jira)]

Gyula Fora created FLINK-27668:
--

 Summary: Document dynamic operator configuration
 Key: FLINK-27668
 URL: https://issues.apache.org/jira/browse/FLINK-27668
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.0.0


The Kubernetes operator now support dynamic config changes through the operator 
configmap.

This feature is not documented properly and it should be added to the 
operations/configuration page



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

[Jark Wu]

Thank Qingsheng for updating the design.

I just have a minor comment.
Changing RescanRuntimeProvider into the builder pattern seems not necessary.
The builder pattern is usually used when there are several optional
parameters.
However, ScanRuntimeProvider and rescan interval are both required when
defining RescanRuntimeProvider. So I think the "of" method is more concise.

Best,
Jark

On Tue, 17 May 2022 at 20:16, Qingsheng Ren  wrote:

> Hi devs,
>
> I just updated FLIP-221 [1] according to discussions addressed above. This
> version made minor changes to make interfaces clearer:
>
> 1. The argument type of LookupFunction#lookup and
> AsyncLookupFunction#asyncLookup are changed from Object... to RowData, in
> order to be symmetric with output type and be more descriptive.
> 2. “set” is removed from method names in LookupCacheMetricGroup to align
> with methods in MetricGroup.
> 3. Add statements to deprecate TableFunctionProvider and
> AsyncTableFunctionProvider
> 4. Add builder class for LookupFunctionProvider,
> AsyncLookupFunctionProvider and RescanRuntimeProvider.
> 5. “invalidateAll” and “putAll” methods are removed from LookupCache
>
> Looking forward to your ideas!
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221+Abstraction+for+lookup+source+cache+and+metric
>
> Best,
>
> Qingsheng
>
> > On May 17, 2022, at 18:10, Qingsheng Ren  wrote:
> >
> > Hi Alexander,
> >
> > Thanks for the review and glad to see we are on the same page! I think
> you forgot to cc the dev mailing list so I’m also quoting your reply under
> this email.
> >
> >> We can add 'maxRetryTimes' option into this class
> >
> > In my opinion the retry logic should be implemented in lookup() instead
> of in LookupFunction#eval(). Retrying is only meaningful under some
> specific retriable failures, and there might be custom logic before making
> retry, such as re-establish the connection (JdbcRowDataLookupFunction is an
> example), so it's more handy to leave it to the connector.
> >
> >> I don't see DDL options, that were in previous version of FLIP. Do you
> have any special plans for them?
> >
> > We decide not to provide common DDL options and let developers to define
> their own options as we do now per connector.
> >
> > The rest of comments sound great and I’ll update the FLIP. Hope we can
> finalize our proposal soon!
> >
> > Best,
> >
> > Qingsheng
> >
> >
> >> On May 17, 2022, at 13:46, Александр Смирнов 
> wrote:
> >>
> >> Hi Qingsheng and devs!
> >>
> >> I like the overall design of updated FLIP, however I have several
> >> suggestions and questions.
> >>
> >> 1) Introducing LookupFunction as a subclass of TableFunction is a good
> >> idea. We can add 'maxRetryTimes' option into this class. 'eval' method
> >> of new LookupFunction is great for this purpose. The same is for
> >> 'async' case.
> >>
> >> 2) There might be other configs in future, such as 'cacheMissingKey'
> >> in LookupFunctionProvider or 'rescanInterval' in ScanRuntimeProvider.
> >> Maybe use Builder pattern in LookupFunctionProvider and
> >> RescanRuntimeProvider for more flexibility (use one 'build' method
> >> instead of many 'of' methods in future)?
> >>
> >> 3) What are the plans for existing TableFunctionProvider and
> >> AsyncTableFunctionProvider? I think they should be deprecated.
> >>
> >> 4) Am I right that the current design does not assume usage of
> >> user-provided LookupCache in re-scanning? In this case, it is not very
> >> clear why do we need methods such as 'invalidate' or 'putAll' in
> >> LookupCache.
> >>
> >> 5) I don't see DDL options, that were in previous version of FLIP. Do
> >> you have any special plans for them?
> >>
> >> If you don't mind, I would be glad to be able to make small
> >> adjustments to the FLIP document too. I think it's worth mentioning
> >> about what exactly optimizations are planning in the future.
> >>
> >> Best regards,
> >> Smirnov Alexander
> >>
> >> пт, 13 мая 2022 г. в 20:27, Qingsheng Ren :
> >>>
> >>> Hi Alexander and devs,
> >>>
> >>> Thank you very much for the in-depth discussion! As Jark mentioned we
> were inspired by Alexander's idea and made a refactor on our design.
> FLIP-221 [1] has been updated to reflect our design now and we are happy to
> hear more suggestions from you!
> >>>
> >>> Compared to the previous design:
> >>> 1. The lookup cache serves at table runtime level and is integrated as
> a component of LookupJoinRunner as discussed previously.
> >>> 2. Interfaces are renamed and re-designed to reflect the new design.
> >>> 3. We separate the all-caching case individually and introduce a new
> RescanRuntimeProvider to reuse the ability of scanning. We are planning to
> support SourceFunction / InputFormat for now considering the complexity of
> FLIP-27 Source API.
> >>> 4. A new interface LookupFunction is introduced to make the semantic
> of lookup more straightforward for developers.
> >>>
> >>> For replying to Alexander:
>  However I'm a little confused 

Re: Channel became inactive while submitting job

[Weihua Hu]

Hi,

Which version of Flink are you using?  And what is the start cmd?

Best,
Weihua

> 2022年5月17日 下午6:33，Zain Haider Nemati  写道：
> 
>  main method caused an error: Failed to execute job 'Tracer Processor'.
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at 
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
> at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
> at 
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
> at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at 
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)

Re: [DISCUSS] Release first version of Elasticsearch connector from external repository

[Jing Ge]

Thanks Chesnay for the notes.

The 1.16-SNAPSHOT dependency is only a temporary solution. Once all related
commits w.r.t. architecture tests have been cp from the master to 1.15, the
dependency will be changed to 1.15. It is on the todo list.

On Tue, May 17, 2022 at 10:04 AM Chesnay Schepler 
wrote:

> some notes:
>
> - the ES repo currently depends on 1.16-SNAPSHOT; so we'd have to go
> back to 1.15 to do such a release as Konstantin proposed. This may have
> repercussions on the architecture tests.
> - the version scheme in the repo does not yet match what was discussed
> in the original proposal (which I think should be 3.0-SNAPSHOT)
> - we aren't able to publish the javadocs for the ES connector on the
> website. Are we fine with that?
> - there is no tooling in the ES repo for actually doing the release.
> - there is currently no e2e test coverage for the moved connector. The
> e2e tests living on the Flink side run against the connector version in
> Flink. Will we just change the version after the release (implying that
> technically the connector is released without e2e coverage)?
>
> On 11/05/2022 21:35, Konstantin Knauf wrote:
> > Hi Martijn,
> >
> > +1 to do a release which is compatible with Flink 1.5.x. With the
> release,
> > we should add something like a compatibility matrix to the documentation,
> > but I am sure that's already on your list.
> >
> > Cheers,
> >
> > Konstantin
> >
> >
> >
> >
> > Am Mi., 11. Mai 2022 um 20:10 Uhr schrieb Martijn Visser <
> > martijnvis...@apache.org>:
> >
> >> Hi everyone,
> >>
> >> As mentioned in the previous update [1] we were working on the final
> step
> >> for moving out the Elasticsearch connector, which was related to the
> >> documentation integration. This has now been completed: the
> documentation
> >> you see on
> >>
> >>
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/
> >> is generated via the documentation in the Elasticsearch external
> >> repository.
> >>
> >> In our plan we outlined that when we moved things out, we would create a
> >> first release out of this repository [2]. I would like to take that
> step to
> >> complete this first connector and move on to the next one.
> >>
> >> If there's any feedback or questions, do let me know. Else I'll reach
> out
> >> to some PMCs to help facilitate this release and we'll open a VOTE
> thread
> >> shortly.
> >>
> >> Best regards,
> >>
> >> Martijn Visser
> >> https://twitter.com/MartijnVisser82
> >> https://github.com/MartijnVisser
> >>
> >> [1] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h
> >> [2] https://lists.apache.org/thread/vqbjoo94wwqcvo32c80dkmp7r5gmy68r
> >>
> >
>
>

Re: [DISCUSS] FLIP-222: Support full query lifecycle statements in SQL client

[godfrey he]

Hi Paul,

Thanks for driving this, LGTM overall.

I have a few minor comments:

>SHOW QUERIES
I want to clear the scope the command, does the command show the
queries submitted
via SqlClient, or all queries in current cluster (submitted via other CLI)?
History queries are included? What's the behavior for per-job cluster?

The result should contain 'finish_time' field, which is more friendly
for batch job.

>DROP QUERY ''
What's the behavior for batch jobs and the non-running jobs?

>SAVEPOINT ''
+1 to align with the SQL standard.
What's the behavior for batch jobs?

SHOW SAVEPOINTS is missing.

* Table API
+1 to introduce the API in Table API

Best,
Godfrey

Paul Lam  于2022年5月11日周三 19:20写道：
>
> Hi Jark,
>
> Thanks a lot for your opinions and suggestions! Please see my replies inline.
>
> > 1) the display of savepoint_path
>
>
> Agreed. Adding it to the FLIP.
>
> > 2) Please make a decision on multiple options in the FLIP.
>
> Okay. I’ll keep one and move the other to the rejected alternatives section.
>
> > 4) +1 SHOW QUERIES
> > Btw, the displayed column "address" is a little confusing to me.
> > At the first glance, I'm not sure what address it is, JM RPC address? JM 
> > REST address? Gateway address?
> > If this is a link to the job's web UI URL, how about calling it "web_url" 
> > and display in
> > "http://:" format?
> > Besides, how about displaying "startTime" or "uptime" as well?
>
> I’m good with these changes. Updating the FLIP according to your suggestions.
>
> > 5) STOP/CANCEL QUERY vs DROP QUERY
> > I'm +1 to DROP, because it's more compliant with SQL standard naming, i.e., 
> > "SHOW/CREATE/DROP".
> > Separating STOP and CANCEL confuses users a lot what are the differences 
> > between them.
> > I'm +1 to add the "PURGE" keyword to the DROP QUERY statement, which 
> > indicates to stop query without savepoint.
> > Note that, PURGE doesn't mean stop with --drain flag. The drain flag will 
> > flush all the registered timers
> > and windows which could lead to incorrect results when the job is resumed. 
> > I think the drain flag is rarely used
> > (please correct me if I'm wrong), therefore, I suggest moving this feature 
> > into future work when the needs are clear.
>
> I’m +1 to represent ungrateful cancel by PURGE. I think —drain flag is not 
> used very often as you said, and we
> could just add a table config option to enable that flag.
>
> > 7)  and  should be quoted
> > All the  and  should be string literal, otherwise 
> > it's hard to parse them.
> > For example, STOP QUERY '’.
>
> Good point! Adding it to the FLIP.
>
> > 8) Examples
> > Could you add an example that consists of all the statements to show how to 
> > manage the full lifecycle of queries?
> > Including show queries, create savepoint, remove savepoint, stop query with 
> > a savepoint, and restart query with savepoint.
>
> Agreed. Adding it to the FLIP as well.
>
> Best,
> Paul Lam
>
> > 2022年5月7日 18:22，Jark Wu  写道：
> >
> > Hi Paul,
> >
> > I think this FLIP has already in a good shape. I just left some additional 
> > thoughts:
> >
> > 1) the display of savepoint_path
> > Could the displayed savepoint_path include the scheme part?
> > E.g. `hdfs:///flink-savepoints/savepoint-cca7bc-bb1e257f0dab`
> > IIUC, the scheme part is omitted when it's a local filesystem.
> > But the behavior would be clearer if including the scheme part in the 
> > design doc.
> >
> > 2) Please make a decision on multiple options in the FLIP.
> > It might give the impression that we will support all the options.
> >
> > 3) +1 SAVEPOINT and RELEASE SAVEPOINT
> > Personally, I also prefer "SAVEPOINT " and "RELEASE SAVEPOINT 
> > "
> > to "CREATE/DROP SAVEPOINT", as they have been used in mature databases.
> >
> > 4) +1 SHOW QUERIES
> > Btw, the displayed column "address" is a little confusing to me.
> > At the first glance, I'm not sure what address it is, JM RPC address? JM 
> > REST address? Gateway address?
> > If this is a link to the job's web UI URL, how about calling it "web_url" 
> > and display in
> > "http://:" format?
> > Besides, how about displaying "startTime" or "uptime" as well?
> >
> > 5) STOP/CANCEL QUERY vs DROP QUERY
> > I'm +1 to DROP, because it's more compliant with SQL standard naming, i.e., 
> > "SHOW/CREATE/DROP".
> > Separating STOP and CANCEL confuses users a lot what are the differences 
> > between them.
> > I'm +1 to add the "PURGE" keyword to the DROP QUERY statement, which 
> > indicates to stop query without savepoint.
> > Note that, PURGE doesn't mean stop with --drain flag. The drain flag will 
> > flush all the registered timers
> > and windows which could lead to incorrect results when the job is resumed. 
> > I think the drain flag is rarely used
> > (please correct me if I'm wrong), therefore, I suggest moving this feature 
> > into future work when the needs are clear.
> >
> > 6) Table API
> > I think it makes sense to support the new statements in Table API.
> > We should try to make the Gateway and 

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

[Qingsheng Ren]

Hi devs, 

I just updated FLIP-221 [1] according to discussions addressed above. This 
version made minor changes to make interfaces clearer:

1. The argument type of LookupFunction#lookup and 
AsyncLookupFunction#asyncLookup are changed from Object... to RowData, in order 
to be symmetric with output type and be more descriptive.
2. “set” is removed from method names in LookupCacheMetricGroup to align with 
methods in MetricGroup.
3. Add statements to deprecate TableFunctionProvider and 
AsyncTableFunctionProvider
4. Add builder class for LookupFunctionProvider, AsyncLookupFunctionProvider 
and RescanRuntimeProvider.
5. “invalidateAll” and “putAll” methods are removed from LookupCache

Looking forward to your ideas!

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-221+Abstraction+for+lookup+source+cache+and+metric

Best,

Qingsheng

> On May 17, 2022, at 18:10, Qingsheng Ren  wrote:
> 
> Hi Alexander, 
> 
> Thanks for the review and glad to see we are on the same page! I think you 
> forgot to cc the dev mailing list so I’m also quoting your reply under this 
> email. 
> 
>> We can add 'maxRetryTimes' option into this class
> 
> In my opinion the retry logic should be implemented in lookup() instead of in 
> LookupFunction#eval(). Retrying is only meaningful under some specific 
> retriable failures, and there might be custom logic before making retry, such 
> as re-establish the connection (JdbcRowDataLookupFunction is an example), so 
> it's more handy to leave it to the connector.
> 
>> I don't see DDL options, that were in previous version of FLIP. Do you have 
>> any special plans for them?
> 
> We decide not to provide common DDL options and let developers to define 
> their own options as we do now per connector. 
> 
> The rest of comments sound great and I’ll update the FLIP. Hope we can 
> finalize our proposal soon!
> 
> Best, 
> 
> Qingsheng
> 
> 
>> On May 17, 2022, at 13:46, Александр Смирнов  wrote:
>> 
>> Hi Qingsheng and devs!
>> 
>> I like the overall design of updated FLIP, however I have several
>> suggestions and questions.
>> 
>> 1) Introducing LookupFunction as a subclass of TableFunction is a good
>> idea. We can add 'maxRetryTimes' option into this class. 'eval' method
>> of new LookupFunction is great for this purpose. The same is for
>> 'async' case.
>> 
>> 2) There might be other configs in future, such as 'cacheMissingKey'
>> in LookupFunctionProvider or 'rescanInterval' in ScanRuntimeProvider.
>> Maybe use Builder pattern in LookupFunctionProvider and
>> RescanRuntimeProvider for more flexibility (use one 'build' method
>> instead of many 'of' methods in future)?
>> 
>> 3) What are the plans for existing TableFunctionProvider and
>> AsyncTableFunctionProvider? I think they should be deprecated.
>> 
>> 4) Am I right that the current design does not assume usage of
>> user-provided LookupCache in re-scanning? In this case, it is not very
>> clear why do we need methods such as 'invalidate' or 'putAll' in
>> LookupCache.
>> 
>> 5) I don't see DDL options, that were in previous version of FLIP. Do
>> you have any special plans for them?
>> 
>> If you don't mind, I would be glad to be able to make small
>> adjustments to the FLIP document too. I think it's worth mentioning
>> about what exactly optimizations are planning in the future.
>> 
>> Best regards,
>> Smirnov Alexander
>> 
>> пт, 13 мая 2022 г. в 20:27, Qingsheng Ren :
>>> 
>>> Hi Alexander and devs,
>>> 
>>> Thank you very much for the in-depth discussion! As Jark mentioned we were 
>>> inspired by Alexander's idea and made a refactor on our design. FLIP-221 
>>> [1] has been updated to reflect our design now and we are happy to hear 
>>> more suggestions from you!
>>> 
>>> Compared to the previous design:
>>> 1. The lookup cache serves at table runtime level and is integrated as a 
>>> component of LookupJoinRunner as discussed previously.
>>> 2. Interfaces are renamed and re-designed to reflect the new design.
>>> 3. We separate the all-caching case individually and introduce a new 
>>> RescanRuntimeProvider to reuse the ability of scanning. We are planning to 
>>> support SourceFunction / InputFormat for now considering the complexity of 
>>> FLIP-27 Source API.
>>> 4. A new interface LookupFunction is introduced to make the semantic of 
>>> lookup more straightforward for developers.
>>> 
>>> For replying to Alexander:
 However I'm a little confused whether InputFormat is deprecated or not. Am 
 I right that it will be so in the future, but currently it's not?
>>> Yes you are right. InputFormat is not deprecated for now. I think it will 
>>> be deprecated in the future but we don't have a clear plan for that.
>>> 
>>> Thanks again for the discussion on this FLIP and looking forward to 
>>> cooperating with you after we finalize the design and interfaces!
>>> 
>>> [1] 
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221+Abstraction+for+lookup+source+cache+and+metric
>>> 
>>> Best regards,

Re:Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

[zst...@163.com]

Hi Qingsheng, Alexander,



Thanks for your reply.
> Can you give an example of such upper - level Cache usage? It's not clear for 
> me currently. I think it's unnecessary to have such high level abstraction, 
> if nowhere in the code we won't operate with objects as instances of Cache. 
> But maybe there are other opinions on this.

I have't find any other usage yet. Maybe it can be used in DataStream API. If 
you all think it's unnecessary, we can ignore it.




> I think there won't be many problems with supporting metrics in ALL cache. 
> Moreover, some of proposed metrics are most useful especially in ALL case, 
> for example, 'latestLoadTimeGauge' or 'numCachedRecords', so necessary 
> metrics definitely should be supported in this cache strategy.
Sorry for my mistake. There is no problem with it.


Best regards,
Yuan


At 2022-05-17 17:15:20, "Qingsheng Ren"  wrote:
>Hi Yuan,
>
>Thanks for the review! Basically I’m with Alexander opinion. We’d like to 
>limit the scope in lookup scenario so we didn’t extend the cache to a generic 
>one. And as for the metric I think the existing metric definitions are also 
>applicable for all-cache case. 
>
>Cheers, 
>
>Qingsheng
>
>
>> On May 15, 2022, at 21:17, zst...@163.com wrote:
>> 
>> Hi Qingsheng and devs,
>> 
>> Thanks for your heated discussion and redesign to optmize this feature. I 
>> just have two comments:
>> 1. How about abtract the LookupCache to a higher level with a common Cache? 
>> It will be convenient for devs to use in other place.
>> 
>> 2. Does it have any metrics, such as NumCachedRecords for the AllCache?
>> Best regards,
>> Yuan
>> 
>> At 2022-05-13 20:27:44, "Qingsheng Ren"  wrote:
>> >Hi Alexander and devs,
>> >
>> >Thank you very much for the in-depth discussion! As Jark mentioned we were
>> >inspired by Alexander's idea and made a refactor on our design. FLIP-221
>> >[1] has been updated to reflect our design now and we are happy to hear
>> >more suggestions from you!
>> >
>> >Compared to the previous design:
>> >1. The lookup cache serves at table runtime level and is integrated as a
>> >component of LookupJoinRunner as discussed previously.
>> >2. Interfaces are renamed and re-designed to reflect the new design.
>> >3. We separate the all-caching case individually and introduce a new
>> >RescanRuntimeProvider to reuse the ability of scanning. We are planning to
>> >support SourceFunction / InputFormat for now considering the complexity of
>> >FLIP-27 Source API.
>> >4. A new interface LookupFunction is introduced to make the semantic of
>> >lookup more straightforward for developers.
>> >
>> >For replying to Alexander:
>> >> However I'm a little confused whether InputFormat is deprecated or not.
>> >Am I right that it will be so in the future, but currently it's not?
>> >Yes you are right. InputFormat is not deprecated for now. I think it will
>> >be deprecated in the future but we don't have a clear plan for that.
>> >
>> >Thanks again for the discussion on this FLIP and looking forward to
>> >cooperating with you after we finalize the design and interfaces!
>> >
>> >[1]
>> >https://cwiki.apache.org/confluence/display/FLINK/FLIP-221+Abstraction+for+lookup+source+cache+and+metric
>> >
>> >Best regards,
>> >
>> >Qingsheng
>> >
>> >
>> >On Fri, May 13, 2022 at 12:12 AM Александр Смирнов 
>> >wrote:
>> >
>> >> Hi Jark, Qingsheng and Leonard!
>> >>
>> >> Glad to see that we came to a consensus on almost all points!
>> >>
>> >> However I'm a little confused whether InputFormat is deprecated or
>> >> not. Am I right that it will be so in the future, but currently it's
>> >> not? Actually I also think that for the first version it's OK to use
>> >> InputFormat in ALL cache realization, because supporting rescan
>> >> ability seems like a very distant prospect. But for this decision we
>> >> need a consensus among all discussion participants.
>> >>
>> >> In general, I don't have something to argue with your statements. All
>> >> of them correspond my ideas. Looking ahead, it would be nice to work
>> >> on this FLIP cooperatively. I've already done a lot of work on lookup
>> >> join caching with realization very close to the one we are discussing,
>> >> and want to share the results of this work. Anyway looking forward for
>> >> the FLIP update!
>> >>
>> >> Best regards,
>> >> Smirnov Alexander
>> >>
>> >> чт, 12 мая 2022 г. в 17:38, Jark Wu :
>> >> >
>> >> > Hi Alex,
>> >> >
>> >> > Thanks for summarizing your points.
>> >> >
>> >> > In the past week, Qingsheng, Leonard, and I have discussed it several
>> >> times
>> >> > and we have totally refactored the design.
>> >> > I'm glad to say we have reached a consensus on many of your points!
>> >> > Qingsheng is still working on updating the design docs and maybe can be
>> >> > available in the next few days.
>> >> > I will share some conclusions from our discussions:
>> >> >
>> >> > 1) we have refactored the design towards to "cache in framework" way.
>> >> >
>> >> > 2) a "LookupCache" 

[jira] [Created] (FLINK-27667) YARNHighAvailabilityITCase fails with "Failed to delete temp directory /tmp/junit1681"

[Martijn Visser (Jira)]

Martijn Visser created FLINK-27667:
--

 Summary: YARNHighAvailabilityITCase fails with "Failed to delete 
temp directory /tmp/junit1681"
 Key: FLINK-27667
 URL: https://issues.apache.org/jira/browse/FLINK-27667
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.16.0
Reporter: Martijn Visser


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35733=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=29208
 
{code:bash}
May 17 08:36:22 [INFO] Results: 
May 17 08:36:22 [INFO] 
May 17 08:36:22 [ERROR] Errors: 
May 17 08:36:22 [ERROR] YARNHighAvailabilityITCase » IO Failed to delete temp 
directory /tmp/junit1681... 
May 17 08:36:22 [INFO] 
May 17 08:36:22 [ERROR] Tests run: 28, Failures: 0, Errors: 1, Skipped: 0 
May 17 08:36:22 [INFO] 
{code}
 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Channel became inactive while submitting job

[Zain Haider Nemati]

Hi,
I am trying to run a job in my local cluster and facing this issue.


org.apache.flink.client.program.ProgramInvocationException: The main method
caused an error: Failed to execute job 'Tracer Processor'.
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
at
org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at
org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
*Caused by: org.apache.flink.util.FlinkException: Failed to execute job
'Tracer Processor'.*
at
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1970)
at
org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:135)
at
org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:76)
at
*org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1834)*
at basicpackage.StreamingJob.main(StreamingJob.java:284)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
... 11 more
*Caused by: org.apache.flink.runtime.client.JobSubmissionException: Failed
to submit JobGraph.*
at
org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$9(RestClusterClient.java:405)
at
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
at
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$9(FutureUtils.java:390)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
org.apache.flink.runtime.rest.RestClient$ClientHandler.channelInactive(RestClient.java:588)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at
org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
at
org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at
org.apache.flink.shaded.netty4.io.netty.handler.stream.ChunkedWriteHandler.channelInactive(ChunkedWriteHandler.java:138)
at

[jira] [Created] (FLINK-27666) Cover manual savepoint triggering in E2E tests

[Gyula Fora (Jira)]

Gyula Fora created FLINK-27666:
--

 Summary: Cover manual savepoint triggering in E2E tests
 Key: FLINK-27666
 URL: https://issues.apache.org/jira/browse/FLINK-27666
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora


We should extend our e2e tests so that they cover manual savepoint triggering 
using the savepointTriggerNonce.

We should verify that savepoint was triggered, recorded in the status and 
trigger information cleared afterwards.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

[Qingsheng Ren]

Hi Yuan,

Thanks for the review! Basically I’m with Alexander opinion. We’d like to limit 
the scope in lookup scenario so we didn’t extend the cache to a generic one. 
And as for the metric I think the existing metric definitions are also 
applicable for all-cache case. 

Cheers, 

Qingsheng


> On May 15, 2022, at 21:17, zst...@163.com wrote:
> 
> Hi Qingsheng and devs,
> 
> Thanks for your heated discussion and redesign to optmize this feature. I 
> just have two comments:
> 1. How about abtract the LookupCache to a higher level with a common Cache? 
> It will be convenient for devs to use in other place.
> 
> 2. Does it have any metrics, such as NumCachedRecords for the AllCache?
> Best regards,
> Yuan
> 
> At 2022-05-13 20:27:44, "Qingsheng Ren"  wrote:
> >Hi Alexander and devs,
> >
> >Thank you very much for the in-depth discussion! As Jark mentioned we were
> >inspired by Alexander's idea and made a refactor on our design. FLIP-221
> >[1] has been updated to reflect our design now and we are happy to hear
> >more suggestions from you!
> >
> >Compared to the previous design:
> >1. The lookup cache serves at table runtime level and is integrated as a
> >component of LookupJoinRunner as discussed previously.
> >2. Interfaces are renamed and re-designed to reflect the new design.
> >3. We separate the all-caching case individually and introduce a new
> >RescanRuntimeProvider to reuse the ability of scanning. We are planning to
> >support SourceFunction / InputFormat for now considering the complexity of
> >FLIP-27 Source API.
> >4. A new interface LookupFunction is introduced to make the semantic of
> >lookup more straightforward for developers.
> >
> >For replying to Alexander:
> >> However I'm a little confused whether InputFormat is deprecated or not.
> >Am I right that it will be so in the future, but currently it's not?
> >Yes you are right. InputFormat is not deprecated for now. I think it will
> >be deprecated in the future but we don't have a clear plan for that.
> >
> >Thanks again for the discussion on this FLIP and looking forward to
> >cooperating with you after we finalize the design and interfaces!
> >
> >[1]
> >https://cwiki.apache.org/confluence/display/FLINK/FLIP-221+Abstraction+for+lookup+source+cache+and+metric
> >
> >Best regards,
> >
> >Qingsheng
> >
> >
> >On Fri, May 13, 2022 at 12:12 AM Александр Смирнов 
> >wrote:
> >
> >> Hi Jark, Qingsheng and Leonard!
> >>
> >> Glad to see that we came to a consensus on almost all points!
> >>
> >> However I'm a little confused whether InputFormat is deprecated or
> >> not. Am I right that it will be so in the future, but currently it's
> >> not? Actually I also think that for the first version it's OK to use
> >> InputFormat in ALL cache realization, because supporting rescan
> >> ability seems like a very distant prospect. But for this decision we
> >> need a consensus among all discussion participants.
> >>
> >> In general, I don't have something to argue with your statements. All
> >> of them correspond my ideas. Looking ahead, it would be nice to work
> >> on this FLIP cooperatively. I've already done a lot of work on lookup
> >> join caching with realization very close to the one we are discussing,
> >> and want to share the results of this work. Anyway looking forward for
> >> the FLIP update!
> >>
> >> Best regards,
> >> Smirnov Alexander
> >>
> >> чт, 12 мая 2022 г. в 17:38, Jark Wu :
> >> >
> >> > Hi Alex,
> >> >
> >> > Thanks for summarizing your points.
> >> >
> >> > In the past week, Qingsheng, Leonard, and I have discussed it several
> >> times
> >> > and we have totally refactored the design.
> >> > I'm glad to say we have reached a consensus on many of your points!
> >> > Qingsheng is still working on updating the design docs and maybe can be
> >> > available in the next few days.
> >> > I will share some conclusions from our discussions:
> >> >
> >> > 1) we have refactored the design towards to "cache in framework" way.
> >> >
> >> > 2) a "LookupCache" interface for users to customize and a default
> >> > implementation with builder for users to easy-use.
> >> > This can both make it possible to both have flexibility and conciseness.
> >> >
> >> > 3) Filter pushdown is important for ALL and LRU lookup cache, esp
> >> reducing
> >> > IO.
> >> > Filter pushdown should be the final state and the unified way to both
> >> > support pruning ALL cache and LRU cache,
> >> > so I think we should make effort in this direction. If we need to support
> >> > filter pushdown for ALL cache anyway, why not use
> >> > it for LRU cache as well? Either way, as we decide to implement the cache
> >> > in the framework, we have the chance to support
> >> > filter on cache anytime. This is an optimization and it doesn't affect
> >> the
> >> > public API. I think we can create a JIRA issue to
> >> > discuss it when the FLIP is accepted.
> >> >
> >> > 4) The idea to support ALL cache is similar to your proposal.
> >> > In the first version, we will 

[jira] [Created] (FLINK-27665) Optimise event triggering on DeploymentFailedExceptions

[Matyas Orhidi (Jira)]

Matyas Orhidi created FLINK-27665:
-

 Summary: Optimise event triggering on DeploymentFailedExceptions
 Key: FLINK-27665
 URL: https://issues.apache.org/jira/browse/FLINK-27665
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-0.1.0
Reporter: Matyas Orhidi
 Fix For: kubernetes-operator-1.0.0
 Attachments: image-2022-05-17-12-08-42-597.png, 
image-2022-05-17-12-13-19-489.png

Use `EventUtils` when handling `DeploymentFailedExceptions` to avoid appending 
new events on every reconcile loop:

!image-2022-05-17-12-13-19-489.png!

 

 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

[Qingsheng Ren]

Hi Alexander, 

Thanks for the review and glad to see we are on the same page! I think you 
forgot to cc the dev mailing list so I’m also quoting your reply under this 
email. 

>  We can add 'maxRetryTimes' option into this class

In my opinion the retry logic should be implemented in lookup() instead of in 
LookupFunction#eval(). Retrying is only meaningful under some specific 
retriable failures, and there might be custom logic before making retry, such 
as re-establish the connection (JdbcRowDataLookupFunction is an example), so 
it's more handy to leave it to the connector.

> I don't see DDL options, that were in previous version of FLIP. Do you have 
> any special plans for them?

We decide not to provide common DDL options and let developers to define their 
own options as we do now per connector. 

The rest of comments sound great and I’ll update the FLIP. Hope we can finalize 
our proposal soon!

Best, 

Qingsheng


> On May 17, 2022, at 13:46, Александр Смирнов  wrote:
> 
> Hi Qingsheng and devs!
> 
> I like the overall design of updated FLIP, however I have several
> suggestions and questions.
> 
> 1) Introducing LookupFunction as a subclass of TableFunction is a good
> idea. We can add 'maxRetryTimes' option into this class. 'eval' method
> of new LookupFunction is great for this purpose. The same is for
> 'async' case.
> 
> 2) There might be other configs in future, such as 'cacheMissingKey'
> in LookupFunctionProvider or 'rescanInterval' in ScanRuntimeProvider.
> Maybe use Builder pattern in LookupFunctionProvider and
> RescanRuntimeProvider for more flexibility (use one 'build' method
> instead of many 'of' methods in future)?
> 
> 3) What are the plans for existing TableFunctionProvider and
> AsyncTableFunctionProvider? I think they should be deprecated.
> 
> 4) Am I right that the current design does not assume usage of
> user-provided LookupCache in re-scanning? In this case, it is not very
> clear why do we need methods such as 'invalidate' or 'putAll' in
> LookupCache.
> 
> 5) I don't see DDL options, that were in previous version of FLIP. Do
> you have any special plans for them?
> 
> If you don't mind, I would be glad to be able to make small
> adjustments to the FLIP document too. I think it's worth mentioning
> about what exactly optimizations are planning in the future.
> 
> Best regards,
> Smirnov Alexander
> 
> пт, 13 мая 2022 г. в 20:27, Qingsheng Ren :
>> 
>> Hi Alexander and devs,
>> 
>> Thank you very much for the in-depth discussion! As Jark mentioned we were 
>> inspired by Alexander's idea and made a refactor on our design. FLIP-221 [1] 
>> has been updated to reflect our design now and we are happy to hear more 
>> suggestions from you!
>> 
>> Compared to the previous design:
>> 1. The lookup cache serves at table runtime level and is integrated as a 
>> component of LookupJoinRunner as discussed previously.
>> 2. Interfaces are renamed and re-designed to reflect the new design.
>> 3. We separate the all-caching case individually and introduce a new 
>> RescanRuntimeProvider to reuse the ability of scanning. We are planning to 
>> support SourceFunction / InputFormat for now considering the complexity of 
>> FLIP-27 Source API.
>> 4. A new interface LookupFunction is introduced to make the semantic of 
>> lookup more straightforward for developers.
>> 
>> For replying to Alexander:
>>> However I'm a little confused whether InputFormat is deprecated or not. Am 
>>> I right that it will be so in the future, but currently it's not?
>> Yes you are right. InputFormat is not deprecated for now. I think it will be 
>> deprecated in the future but we don't have a clear plan for that.
>> 
>> Thanks again for the discussion on this FLIP and looking forward to 
>> cooperating with you after we finalize the design and interfaces!
>> 
>> [1] 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221+Abstraction+for+lookup+source+cache+and+metric
>> 
>> Best regards,
>> 
>> Qingsheng
>> 
>> 
>> On Fri, May 13, 2022 at 12:12 AM Александр Смирнов  
>> wrote:
>>> 
>>> Hi Jark, Qingsheng and Leonard!
>>> 
>>> Glad to see that we came to a consensus on almost all points!
>>> 
>>> However I'm a little confused whether InputFormat is deprecated or
>>> not. Am I right that it will be so in the future, but currently it's
>>> not? Actually I also think that for the first version it's OK to use
>>> InputFormat in ALL cache realization, because supporting rescan
>>> ability seems like a very distant prospect. But for this decision we
>>> need a consensus among all discussion participants.
>>> 
>>> In general, I don't have something to argue with your statements. All
>>> of them correspond my ideas. Looking ahead, it would be nice to work
>>> on this FLIP cooperatively. I've already done a lot of work on lookup
>>> join caching with realization very close to the one we are discussing,
>>> and want to share the results of this work. Anyway looking forward for
>>> the FLIP update!
>>> 
>>> Best 

[jira] [Created] (FLINK-27664) cron_snapshot_deployment_maven fails due to JavaDoc building error

[Martijn Visser (Jira)]

Martijn Visser created FLINK-27664:
--

 Summary: cron_snapshot_deployment_maven fails due to JavaDoc 
building error
 Key: FLINK-27664
 URL: https://issues.apache.org/jira/browse/FLINK-27664
 Project: Flink
  Issue Type: Bug
  Components: Build System
Affects Versions: 1.15.0
Reporter: Martijn Visser


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35684=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7=14026

{code:bash}
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-javadoc-plugin:2.9.1:jar (attach-javadocs) on 
project flink-architecture-tests-production: MavenReportException: Error while 
creating archive:
[ERROR] Exit code: 1 - javadoc: error - class file for 
org.junit.platform.commons.annotation.Testable not found
[ERROR] 
[ERROR] Command line was: /usr/lib/jvm/java-8-openjdk-amd64/jre/../bin/javadoc 
-Xdoclint:none @options @packages
[ERROR] 
[ERROR] Refer to the generated Javadoc files in 
'/__w/1/s/flink-architecture-tests/flink-architecture-tests-production/target/apidocs'
 dir.
[ERROR] -> [Help 1]
{code}




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] Creating an Apache Flink slack workspace

[Márton Balassi]

+1 (binding)

On Tue, May 17, 2022 at 11:00 AM Jingsong Li  wrote:

> Thank Xintong for driving this work.
>
> +1
>
> Best,
> Jingsong
>
> On Tue, May 17, 2022 at 4:49 PM Martijn Visser 
> wrote:
>
> > +1 (binding)
> >
> > On Tue, 17 May 2022 at 10:38, Yu Li  wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks Xintong for driving this!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Tue, 17 May 2022 at 16:32, Robert Metzger 
> > wrote:
> > >
> > > > Thanks for starting the VOTE!
> > > >
> > > > +1 (binding)
> > > >
> > > >
> > > >
> > > > On Tue, May 17, 2022 at 10:29 AM Jark Wu  wrote:
> > > >
> > > > > Thank Xintong for driving this work.
> > > > >
> > > > > +1 from my side (binding)
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > > On Tue, 17 May 2022 at 16:24, Xintong Song 
> > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > As previously discussed in [1], I would like to open a vote on
> > > creating
> > > > > an
> > > > > > Apache Flink slack workspace channel.
> > > > > >
> > > > > > The proposed actions include:
> > > > > > - Creating a dedicated slack workspace with the name Apache Flink
> > > that
> > > > is
> > > > > > controlled and maintained by the Apache Flink PMC
> > > > > > - Updating the Flink website about rules for using various
> > > > communication
> > > > > > channels
> > > > > > - Setting up an Archive for the Apache Flink slack
> > > > > > - Revisiting this initiative by the end of 2022
> > > > > >
> > > > > > The vote will last for at least 72 hours, and will be accepted
> by a
> > > > > > consensus of active PMC members.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Creating an Apache Flink slack workspace

[Yang Wang]

+1 (binding)

And thanks Xintong for driving this work.


Best,
Yang

Jingsong Li  于2022年5月17日周二 17:00写道：

> Thank Xintong for driving this work.
>
> +1
>
> Best,
> Jingsong
>
> On Tue, May 17, 2022 at 4:49 PM Martijn Visser 
> wrote:
>
> > +1 (binding)
> >
> > On Tue, 17 May 2022 at 10:38, Yu Li  wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks Xintong for driving this!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Tue, 17 May 2022 at 16:32, Robert Metzger 
> > wrote:
> > >
> > > > Thanks for starting the VOTE!
> > > >
> > > > +1 (binding)
> > > >
> > > >
> > > >
> > > > On Tue, May 17, 2022 at 10:29 AM Jark Wu  wrote:
> > > >
> > > > > Thank Xintong for driving this work.
> > > > >
> > > > > +1 from my side (binding)
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > > On Tue, 17 May 2022 at 16:24, Xintong Song 
> > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > As previously discussed in [1], I would like to open a vote on
> > > creating
> > > > > an
> > > > > > Apache Flink slack workspace channel.
> > > > > >
> > > > > > The proposed actions include:
> > > > > > - Creating a dedicated slack workspace with the name Apache Flink
> > > that
> > > > is
> > > > > > controlled and maintained by the Apache Flink PMC
> > > > > > - Updating the Flink website about rules for using various
> > > > communication
> > > > > > channels
> > > > > > - Setting up an Archive for the Apache Flink slack
> > > > > > - Revisiting this initiative by the end of 2022
> > > > > >
> > > > > > The vote will last for at least 72 hours, and will be accepted
> by a
> > > > > > consensus of active PMC members.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-27663) upsert-kafka can't process delete message from upsert-kafka sink

[Zhiwen Sun (Jira)]

Zhiwen Sun created FLINK-27663:
--

 Summary: upsert-kafka can't process delete message from 
upsert-kafka sink
 Key: FLINK-27663
 URL: https://issues.apache.org/jira/browse/FLINK-27663
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.14.4, 1.13.6, 1.15.0
Reporter: Zhiwen Sun


upsert-kafka write DELETE data as Kafka messages with null values (indicate 
tombstone for the key).

But when use upsert-kafka as a source table to consumer kafka messages write by 
upsert-kafka sink, DELETE messages will be ignored.

 

related sql :

 

 
{code:java}
create table order_system_log(
  id bigint,
  PRIMARY KEY (id) NOT ENFORCED
) WITH (
 'connector' = 'upsert-kafka',
 'topic' = 'test_use',
 'properties.bootstrap.servers' = 'your broker',
 'properties.group.id' = 'your group id',
 'value.json.fail-on-missing-field' = 'false',
 'value.json.ignore-parse-errors' = 'true',
 'key.json.fail-on-missing-field' = 'false',
 'key.json.ignore-parse-errors' = 'true',
 'key.format' = 'json',
 'value.format' = 'json'
);
select
*
from
order_system_log
;
{code}
 

 

The problem may be produced by DeserializationSchema#deserialize,

this method does not collect data while subclass's deserialize return null.

 

 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] Creating an Apache Flink slack workspace

[Jingsong Li]

Thank Xintong for driving this work.

+1

Best,
Jingsong

On Tue, May 17, 2022 at 4:49 PM Martijn Visser 
wrote:

> +1 (binding)
>
> On Tue, 17 May 2022 at 10:38, Yu Li  wrote:
>
> > +1 (binding)
> >
> > Thanks Xintong for driving this!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Tue, 17 May 2022 at 16:32, Robert Metzger 
> wrote:
> >
> > > Thanks for starting the VOTE!
> > >
> > > +1 (binding)
> > >
> > >
> > >
> > > On Tue, May 17, 2022 at 10:29 AM Jark Wu  wrote:
> > >
> > > > Thank Xintong for driving this work.
> > > >
> > > > +1 from my side (binding)
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Tue, 17 May 2022 at 16:24, Xintong Song 
> > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > As previously discussed in [1], I would like to open a vote on
> > creating
> > > > an
> > > > > Apache Flink slack workspace channel.
> > > > >
> > > > > The proposed actions include:
> > > > > - Creating a dedicated slack workspace with the name Apache Flink
> > that
> > > is
> > > > > controlled and maintained by the Apache Flink PMC
> > > > > - Updating the Flink website about rules for using various
> > > communication
> > > > > channels
> > > > > - Setting up an Archive for the Apache Flink slack
> > > > > - Revisiting this initiative by the end of 2022
> > > > >
> > > > > The vote will last for at least 72 hours, and will be accepted by a
> > > > > consensus of active PMC members.
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Creating an Apache Flink slack workspace

[Martijn Visser]

+1 (binding)

On Tue, 17 May 2022 at 10:38, Yu Li  wrote:

> +1 (binding)
>
> Thanks Xintong for driving this!
>
> Best Regards,
> Yu
>
>
> On Tue, 17 May 2022 at 16:32, Robert Metzger  wrote:
>
> > Thanks for starting the VOTE!
> >
> > +1 (binding)
> >
> >
> >
> > On Tue, May 17, 2022 at 10:29 AM Jark Wu  wrote:
> >
> > > Thank Xintong for driving this work.
> > >
> > > +1 from my side (binding)
> > >
> > > Best,
> > > Jark
> > >
> > > On Tue, 17 May 2022 at 16:24, Xintong Song 
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > As previously discussed in [1], I would like to open a vote on
> creating
> > > an
> > > > Apache Flink slack workspace channel.
> > > >
> > > > The proposed actions include:
> > > > - Creating a dedicated slack workspace with the name Apache Flink
> that
> > is
> > > > controlled and maintained by the Apache Flink PMC
> > > > - Updating the Flink website about rules for using various
> > communication
> > > > channels
> > > > - Setting up an Archive for the Apache Flink slack
> > > > - Revisiting this initiative by the end of 2022
> > > >
> > > > The vote will last for at least 72 hours, and will be accepted by a
> > > > consensus of active PMC members.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > >
> >
>

Re: [VOTE] Creating an Apache Flink slack workspace

[Yu Li]

+1 (binding)

Thanks Xintong for driving this!

Best Regards,
Yu


On Tue, 17 May 2022 at 16:32, Robert Metzger  wrote:

> Thanks for starting the VOTE!
>
> +1 (binding)
>
>
>
> On Tue, May 17, 2022 at 10:29 AM Jark Wu  wrote:
>
> > Thank Xintong for driving this work.
> >
> > +1 from my side (binding)
> >
> > Best,
> > Jark
> >
> > On Tue, 17 May 2022 at 16:24, Xintong Song 
> wrote:
> >
> > > Hi everyone,
> > >
> > > As previously discussed in [1], I would like to open a vote on creating
> > an
> > > Apache Flink slack workspace channel.
> > >
> > > The proposed actions include:
> > > - Creating a dedicated slack workspace with the name Apache Flink that
> is
> > > controlled and maintained by the Apache Flink PMC
> > > - Updating the Flink website about rules for using various
> communication
> > > channels
> > > - Setting up an Archive for the Apache Flink slack
> > > - Revisiting this initiative by the end of 2022
> > >
> > > The vote will last for at least 72 hours, and will be accepted by a
> > > consensus of active PMC members.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> >
>

Re: [VOTE] Creating an Apache Flink slack workspace

[Robert Metzger]

Thanks for starting the VOTE!

+1 (binding)



On Tue, May 17, 2022 at 10:29 AM Jark Wu  wrote:

> Thank Xintong for driving this work.
>
> +1 from my side (binding)
>
> Best,
> Jark
>
> On Tue, 17 May 2022 at 16:24, Xintong Song  wrote:
>
> > Hi everyone,
> >
> > As previously discussed in [1], I would like to open a vote on creating
> an
> > Apache Flink slack workspace channel.
> >
> > The proposed actions include:
> > - Creating a dedicated slack workspace with the name Apache Flink that is
> > controlled and maintained by the Apache Flink PMC
> > - Updating the Flink website about rules for using various communication
> > channels
> > - Setting up an Archive for the Apache Flink slack
> > - Revisiting this initiative by the end of 2022
> >
> > The vote will last for at least 72 hours, and will be accepted by a
> > consensus of active PMC members.
> >
> > Best,
> >
> > Xintong
> >
>

Re: [VOTE] Creating an Apache Flink slack workspace

[Xintong Song]

Sorry, I forgot to attach the link to the discussion thread. [1]

Best,

Xintong


[1] https://lists.apache.org/thread/n43r4qmwprhdmzrj494dbbwr9w7bbdcv

On Tue, May 17, 2022 at 4:28 PM Jark Wu  wrote:

> Thank Xintong for driving this work.
>
> +1 from my side (binding)
>
> Best,
> Jark
>
> On Tue, 17 May 2022 at 16:24, Xintong Song  wrote:
>
> > Hi everyone,
> >
> > As previously discussed in [1], I would like to open a vote on creating
> an
> > Apache Flink slack workspace channel.
> >
> > The proposed actions include:
> > - Creating a dedicated slack workspace with the name Apache Flink that is
> > controlled and maintained by the Apache Flink PMC
> > - Updating the Flink website about rules for using various communication
> > channels
> > - Setting up an Archive for the Apache Flink slack
> > - Revisiting this initiative by the end of 2022
> >
> > The vote will last for at least 72 hours, and will be accepted by a
> > consensus of active PMC members.
> >
> > Best,
> >
> > Xintong
> >
>

Re: [VOTE] Creating an Apache Flink slack workspace

[Jark Wu]

Thank Xintong for driving this work.

+1 from my side (binding)

Best,
Jark

On Tue, 17 May 2022 at 16:24, Xintong Song  wrote:

> Hi everyone,
>
> As previously discussed in [1], I would like to open a vote on creating an
> Apache Flink slack workspace channel.
>
> The proposed actions include:
> - Creating a dedicated slack workspace with the name Apache Flink that is
> controlled and maintained by the Apache Flink PMC
> - Updating the Flink website about rules for using various communication
> channels
> - Setting up an Archive for the Apache Flink slack
> - Revisiting this initiative by the end of 2022
>
> The vote will last for at least 72 hours, and will be accepted by a
> consensus of active PMC members.
>
> Best,
>
> Xintong
>

[VOTE] Creating an Apache Flink slack workspace

[Xintong Song]

Hi everyone,

As previously discussed in [1], I would like to open a vote on creating an
Apache Flink slack workspace channel.

The proposed actions include:
- Creating a dedicated slack workspace with the name Apache Flink that is
controlled and maintained by the Apache Flink PMC
- Updating the Flink website about rules for using various communication
channels
- Setting up an Archive for the Apache Flink slack
- Revisiting this initiative by the end of 2022

The vote will last for at least 72 hours, and will be accepted by a
consensus of active PMC members.

Best,

Xintong

Re: [Discuss] Creating an Apache Flink slack workspace

[Kyle Bendickson]

Sorry for the delay in response.

> Nice, cool to hear Kyle! How do you all approach moderation?

Moderation has generally not been a problem so far. So unfortunately I
can’t answer that from experience. But if there’s an existing code of
conduct amongst Apache or within Flink that would probably be sufficient.
For dealing with any nefarious actors at the least.

>  Is there anything specific you feel like you've "gotten right"/ other
tips?

Keeping channels few, with only some breakout rooms.

For Flink, that might be larger than us (we’re probably at 7 or so).

But generally smaller has been better as it’s less to catch up on. We have
general + some channels for our priority 1 stuff (new Python library,
secondary indexing, etc).

- Cross posting important PRs and design docs from the mailing list.

Some people naturally gravitate towards slack. If it’s big and/or important
and you’d like many eyes (like a design doc), posting it in the channel of
relevant folks can help attract more interested folks who might otherwise
be lurkers

We admittedly get a lot of newer contributors this way; which is something
we welcome. Especially as the number of “peripheral” areas grow.

For the proposal, I’d be +1 (obviously non-binding) and think it’s a good
direction to head in. We then use Google groups to set up regular community
sync ups that people can join (not sure what you do there - if anything -
but we get a pretty good turn out)!

- Kyle

On Tue, May 17, 2022 at 12:33 AM Robert Metzger  wrote:

> Thanks a lot Kyle!
>
> What do you think of concluding this discussion and starting a VOTE about:
> 1. Setting up a PMC controlled Slack instance for the Flink community
> 2. Updating the Flink website about the various communication channels
> 3. Setting up an Archive for our Slack instance
> 4. Revisiting this initiative by the end of 2022.
>
> Xintong, do you want to start the VOTE on dev@?
>
> On Fri, May 13, 2022 at 9:41 PM Austin Cawley-Edwards <
> austin.caw...@gmail.com> wrote:
>
> > Nice, cool to hear Kyle! How do you all approach moderation? Is there
> > anything specific you feel like you've "gotten right"/ other tips?
> >
> > (as a side note, I also love slack).
> >
> > Austin
> >
> >
> > On Fri, May 13, 2022 at 2:27 PM Kyle Bendickson  wrote:
> >
> > > Hi all,
> > >
> > > Chiming in as I work in the Iceberg space and we have our own slack as
> > > well, that I am admittedly proud of.
> > >
> > > We don’t necessarily encounter issues with vendors, though of course we
> > do
> > > get some noise now and again.
> > >
> > > Overall, our slack workspace has been cited in multiple blogs and
> things
> > as
> > > one of the bigger benefits of using Iceberg.
> > >
> > > So I personally can’t recommend a slack workspace enough.
> > >
> > > Our slack workspace is also one major thing I feel boosts our ability
> to
> > > attract new contributors and even bug reports we’d otherwise not
> receive
> > as
> > > quickly.
> > >
> > > A lot of amazing devs / folks out there who maybe don’t see themselves
> as
> > > “prominent” enough but will speak up on slack.
> > >
> > > So +1 from your friends in Iceberg (at least me).
> > >
> > > Feel free to reach out if you have any questions!
> > >
> > > - Kyle
> > >
> > > On Fri, May 13, 2022 at 10:17 AM Austin Cawley-Edwards <
> > > austin.caw...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Would just like to share an interesting article from the dbt
> > > community[1],
> > > > which in part describes some of their challenges in managing Slack
> in a
> > > > large community. The biggest point it seems to make is that their
> Slack
> > > has
> > > > become a marketing tool for dbt/data vendors instead of a community
> > > space —
> > > > given the diversity of vendors in the Flink space, we may face
> similar
> > > > challenges. Perhaps their experience can help us with the initial
> > > > setup/guidelines.
> > > >
> > > > Cheers,
> > > > Austin
> > > >
> > > > [1]: https://pedram.substack.com/p/we-need-to-talk-about-dbt?s=r
> > > >
> > > > On Thu, May 12, 2022 at 6:04 AM Robert Metzger 
> > > > wrote:
> > > >
> > > > > +1 on setting up our own Slack instance (PMC owned)
> > > > > +1 for having a separate discussion about setting up a discussion
> > forum
> > > > (I
> > > > > like the idea of using GH discussions)
> > > > >
> > > > > Besides, we still need to investigate how
> > > > >> http://apache-airflow.slack-archives.org works, I think
> > > > >> a slack of our own can be easier to set up the archive.
> > > > >
> > > > >
> > > > > This is the code used by airflow:
> > https://github.com/ashb/slackarchive
> > > .
> > > > > I'm happy to look into setting up the archive for the community.
> > > > >
> > > > >
> > > > > On Thu, May 12, 2022 at 11:00 AM Jark Wu  wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> I would +1 to create Apache Flink Slack for the lower barriers to
> > > entry
> > > > >> as Jingsong mentioned.
> > > > >> Besides, we still need to 

Re: [Discuss] Creating an Apache Flink slack workspace

[Xintong Song]

Thanks everyone for the opinions. And thanks Robert for the summary.

I'll start a voting thread.

Best,

Xintong



On Tue, May 17, 2022 at 3:33 PM Robert Metzger  wrote:

> Thanks a lot Kyle!
>
> What do you think of concluding this discussion and starting a VOTE about:
> 1. Setting up a PMC controlled Slack instance for the Flink community
> 2. Updating the Flink website about the various communication channels
> 3. Setting up an Archive for our Slack instance
> 4. Revisiting this initiative by the end of 2022.
>
> Xintong, do you want to start the VOTE on dev@?
>
> On Fri, May 13, 2022 at 9:41 PM Austin Cawley-Edwards <
> austin.caw...@gmail.com> wrote:
>
> > Nice, cool to hear Kyle! How do you all approach moderation? Is there
> > anything specific you feel like you've "gotten right"/ other tips?
> >
> > (as a side note, I also love slack).
> >
> > Austin
> >
> >
> > On Fri, May 13, 2022 at 2:27 PM Kyle Bendickson  wrote:
> >
> > > Hi all,
> > >
> > > Chiming in as I work in the Iceberg space and we have our own slack as
> > > well, that I am admittedly proud of.
> > >
> > > We don’t necessarily encounter issues with vendors, though of course we
> > do
> > > get some noise now and again.
> > >
> > > Overall, our slack workspace has been cited in multiple blogs and
> things
> > as
> > > one of the bigger benefits of using Iceberg.
> > >
> > > So I personally can’t recommend a slack workspace enough.
> > >
> > > Our slack workspace is also one major thing I feel boosts our ability
> to
> > > attract new contributors and even bug reports we’d otherwise not
> receive
> > as
> > > quickly.
> > >
> > > A lot of amazing devs / folks out there who maybe don’t see themselves
> as
> > > “prominent” enough but will speak up on slack.
> > >
> > > So +1 from your friends in Iceberg (at least me).
> > >
> > > Feel free to reach out if you have any questions!
> > >
> > > - Kyle
> > >
> > > On Fri, May 13, 2022 at 10:17 AM Austin Cawley-Edwards <
> > > austin.caw...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Would just like to share an interesting article from the dbt
> > > community[1],
> > > > which in part describes some of their challenges in managing Slack
> in a
> > > > large community. The biggest point it seems to make is that their
> Slack
> > > has
> > > > become a marketing tool for dbt/data vendors instead of a community
> > > space —
> > > > given the diversity of vendors in the Flink space, we may face
> similar
> > > > challenges. Perhaps their experience can help us with the initial
> > > > setup/guidelines.
> > > >
> > > > Cheers,
> > > > Austin
> > > >
> > > > [1]: https://pedram.substack.com/p/we-need-to-talk-about-dbt?s=r
> > > >
> > > > On Thu, May 12, 2022 at 6:04 AM Robert Metzger 
> > > > wrote:
> > > >
> > > > > +1 on setting up our own Slack instance (PMC owned)
> > > > > +1 for having a separate discussion about setting up a discussion
> > forum
> > > > (I
> > > > > like the idea of using GH discussions)
> > > > >
> > > > > Besides, we still need to investigate how
> > > > >> http://apache-airflow.slack-archives.org works, I think
> > > > >> a slack of our own can be easier to set up the archive.
> > > > >
> > > > >
> > > > > This is the code used by airflow:
> > https://github.com/ashb/slackarchive
> > > .
> > > > > I'm happy to look into setting up the archive for the community.
> > > > >
> > > > >
> > > > > On Thu, May 12, 2022 at 11:00 AM Jark Wu  wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> I would +1 to create Apache Flink Slack for the lower barriers to
> > > entry
> > > > >> as Jingsong mentioned.
> > > > >> Besides, we still need to investigate how
> > > > >> http://apache-airflow.slack-archives.org works, I think
> > > > >> a slack of our own can be easier to set up the archive.
> > > > >>
> > > > >> Regarding Discourse vs Slack, I think they are not exclusive, but
> > > > >> complementary.
> > > > >> Someday in the future, we might be able to provide them both. But
> > what
> > > > we
> > > > >> are seeking today
> > > > >> is a tool that can provide real-time communication and ad-hoc
> > > questions
> > > > >> and interactions.
> > > > >> A forum is more similar to a mailing list. Forum is modern mailing
> > > list
> > > > >> but can't solve the problems
> > > > >> mentioned above. With slack-archives, the information and
> thoughtful
> > > > >> discussion in Slack can also be searchable.
> > > > >>
> > > > >> I think we can open another thread to discuss creating a forum for
> > > Flink
> > > > >> and keep this thread focused
> > > > >> on Slack. IMO, we can investigate more kinds of forums, for
> example
> > > > >> GitHub Discussion which is free, powerful
> > > > >>  and fully-managed. Airflow[1] and Next.JS also use GitHub
> > Discussion
> > > as
> > > > >> their forum.
> > > > >>
> > > > >> Best,
> > > > >> Jark
> > > > >>
> > > > >> [1]: https://github.com/apache/airflow/discussions
> > > > >> [2]: https://github.com/vercel/next.js/discussions
> > > 

Re: [DISCUSS] Release first version of Elasticsearch connector from external repository

[Chesnay Schepler]

some notes:

- the ES repo currently depends on 1.16-SNAPSHOT; so we'd have to go 
back to 1.15 to do such a release as Konstantin proposed. This may have 
repercussions on the architecture tests.
- the version scheme in the repo does not yet match what was discussed 
in the original proposal (which I think should be 3.0-SNAPSHOT)
- we aren't able to publish the javadocs for the ES connector on the 
website. Are we fine with that?

- there is no tooling in the ES repo for actually doing the release.
- there is currently no e2e test coverage for the moved connector. The 
e2e tests living on the Flink side run against the connector version in 
Flink. Will we just change the version after the release (implying that 
technically the connector is released without e2e coverage)?


On 11/05/2022 21:35, Konstantin Knauf wrote:

Hi Martijn,

+1 to do a release which is compatible with Flink 1.5.x. With the release,
we should add something like a compatibility matrix to the documentation,
but I am sure that's already on your list.

Cheers,

Konstantin




Am Mi., 11. Mai 2022 um 20:10 Uhr schrieb Martijn Visser <
martijnvis...@apache.org>:


Hi everyone,

As mentioned in the previous update [1] we were working on the final step
for moving out the Elasticsearch connector, which was related to the
documentation integration. This has now been completed: the documentation
you see on

https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/elasticsearch/
is generated via the documentation in the Elasticsearch external
repository.

In our plan we outlined that when we moved things out, we would create a
first release out of this repository [2]. I would like to take that step to
complete this first connector and move on to the next one.

If there's any feedback or questions, do let me know. Else I'll reach out
to some PMCs to help facilitate this release and we'll open a VOTE thread
shortly.

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82
https://github.com/MartijnVisser

[1] https://lists.apache.org/thread/8k1xonqt7hn0xldbky1cxfx3fzh6sj7h
[2] https://lists.apache.org/thread/vqbjoo94wwqcvo32c80dkmp7r5gmy68r

[jira] [Created] (FLINK-27662) [JUnit5 Migration] Migrate TypeInformationTestBase to Junit5

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-27662:
---

 Summary: [JUnit5 Migration] Migrate TypeInformationTestBase to 
Junit5
 Key: FLINK-27662
 URL: https://issues.apache.org/jira/browse/FLINK-27662
 Project: Flink
  Issue Type: Sub-task
Reporter: Sergey Nuyanzin


The task is a follow up for the feedback comment
https://github.com/apache/flink/pull/19716#discussion_r873685730



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [Discuss] Creating an Apache Flink slack workspace

[Robert Metzger]

Thanks a lot Kyle!

What do you think of concluding this discussion and starting a VOTE about:
1. Setting up a PMC controlled Slack instance for the Flink community
2. Updating the Flink website about the various communication channels
3. Setting up an Archive for our Slack instance
4. Revisiting this initiative by the end of 2022.

Xintong, do you want to start the VOTE on dev@?

On Fri, May 13, 2022 at 9:41 PM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> Nice, cool to hear Kyle! How do you all approach moderation? Is there
> anything specific you feel like you've "gotten right"/ other tips?
>
> (as a side note, I also love slack).
>
> Austin
>
>
> On Fri, May 13, 2022 at 2:27 PM Kyle Bendickson  wrote:
>
> > Hi all,
> >
> > Chiming in as I work in the Iceberg space and we have our own slack as
> > well, that I am admittedly proud of.
> >
> > We don’t necessarily encounter issues with vendors, though of course we
> do
> > get some noise now and again.
> >
> > Overall, our slack workspace has been cited in multiple blogs and things
> as
> > one of the bigger benefits of using Iceberg.
> >
> > So I personally can’t recommend a slack workspace enough.
> >
> > Our slack workspace is also one major thing I feel boosts our ability to
> > attract new contributors and even bug reports we’d otherwise not receive
> as
> > quickly.
> >
> > A lot of amazing devs / folks out there who maybe don’t see themselves as
> > “prominent” enough but will speak up on slack.
> >
> > So +1 from your friends in Iceberg (at least me).
> >
> > Feel free to reach out if you have any questions!
> >
> > - Kyle
> >
> > On Fri, May 13, 2022 at 10:17 AM Austin Cawley-Edwards <
> > austin.caw...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Would just like to share an interesting article from the dbt
> > community[1],
> > > which in part describes some of their challenges in managing Slack in a
> > > large community. The biggest point it seems to make is that their Slack
> > has
> > > become a marketing tool for dbt/data vendors instead of a community
> > space —
> > > given the diversity of vendors in the Flink space, we may face similar
> > > challenges. Perhaps their experience can help us with the initial
> > > setup/guidelines.
> > >
> > > Cheers,
> > > Austin
> > >
> > > [1]: https://pedram.substack.com/p/we-need-to-talk-about-dbt?s=r
> > >
> > > On Thu, May 12, 2022 at 6:04 AM Robert Metzger 
> > > wrote:
> > >
> > > > +1 on setting up our own Slack instance (PMC owned)
> > > > +1 for having a separate discussion about setting up a discussion
> forum
> > > (I
> > > > like the idea of using GH discussions)
> > > >
> > > > Besides, we still need to investigate how
> > > >> http://apache-airflow.slack-archives.org works, I think
> > > >> a slack of our own can be easier to set up the archive.
> > > >
> > > >
> > > > This is the code used by airflow:
> https://github.com/ashb/slackarchive
> > .
> > > > I'm happy to look into setting up the archive for the community.
> > > >
> > > >
> > > > On Thu, May 12, 2022 at 11:00 AM Jark Wu  wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I would +1 to create Apache Flink Slack for the lower barriers to
> > entry
> > > >> as Jingsong mentioned.
> > > >> Besides, we still need to investigate how
> > > >> http://apache-airflow.slack-archives.org works, I think
> > > >> a slack of our own can be easier to set up the archive.
> > > >>
> > > >> Regarding Discourse vs Slack, I think they are not exclusive, but
> > > >> complementary.
> > > >> Someday in the future, we might be able to provide them both. But
> what
> > > we
> > > >> are seeking today
> > > >> is a tool that can provide real-time communication and ad-hoc
> > questions
> > > >> and interactions.
> > > >> A forum is more similar to a mailing list. Forum is modern mailing
> > list
> > > >> but can't solve the problems
> > > >> mentioned above. With slack-archives, the information and thoughtful
> > > >> discussion in Slack can also be searchable.
> > > >>
> > > >> I think we can open another thread to discuss creating a forum for
> > Flink
> > > >> and keep this thread focused
> > > >> on Slack. IMO, we can investigate more kinds of forums, for example
> > > >> GitHub Discussion which is free, powerful
> > > >>  and fully-managed. Airflow[1] and Next.JS also use GitHub
> Discussion
> > as
> > > >> their forum.
> > > >>
> > > >> Best,
> > > >> Jark
> > > >>
> > > >> [1]: https://github.com/apache/airflow/discussions
> > > >> [2]: https://github.com/vercel/next.js/discussions
> > > >>
> > > >>
> > > >> On Thu, 12 May 2022 at 15:24, Martijn Visser  >
> > > >> wrote:
> > > >>
> > > >>> Hi,
> > > >>>
> > > >>> I would +1 setting up our own Slack. It will allow us to provide
> the
> > > best
> > > >>> experience for those in the community who want to use Slack.
> > > >>> More than happy to help with setting up community guidelines on how
> > to
> > > >>> use
> > > >>> Slack.
> > > >>>
> > > >>> Best regards,
> > > >>>
> > > >>> 

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang]

Hi, all.

After discussing with Becket Qin offline, I modified the FLIP a little. The
change is as follow:

1. Add /api_versions in the REST API.

The api is used by the client to know the current REST Endpoint supports
which version. The client is able to choose the version for later
communication.

2. SqlClient uses -e option to input the endpoint address and port.

Because the -h option has already been used by the SqlClient. We use the
-e, --endpoint for SqlClient to input the address:port of the endpoint.

3. Use './sql-client.h -e' to start the gateway mode rather than
'/sql-client.h gateway -e'.

If the user specifies the -e option, it definitely means to use the gateway
mode. Therefore, it is redundant to use another keyword to indicate the
mode.

Best,
Shengkai

Shengkai Fang  于2022年5月17日周二 14:13写道：

> Hi, Jark, Timo. Nice to have an agreement!
>
> Thanks for Jark's inputs about the multiple version Flink. I have already
> updated the FLIP in the rejected alternatives about details.
>
> 1. We should definitely just use LogicalTypeJsonSerializer and not a
> second JSON representation.
>
> Our concern is mainly that it's hard for users to use because of the
> flexible structure. The LogicalTypeJsonSerializer will serialize the
> VARCHAR to "VARCHAR()" or "{\"TYPE\": \"VARCHAR\", \"LENGTH\": 0}",
> which requires the end users to process the different situations. But in
> some cases, users just print the json to the terminal/web UI.  WDYT?
>
> > Serialize the RowData
>
> Sure. I will keep your advice in mind. I think the current serialization
> of the RowData will not use the column name as the Object key in the json.
> I am not sure whether I missed something. It would be nice if you can give
> me an example if I do something wrong.
>
> > Have you also thought about using Flink's state types from Flink
> tasks/jobs?
>
> Yes. But I still think we should use a new state machine. First of all,
> Operation in the FLIP is much different from the Job. Operations include
> DDL, DML and so on. So it's not suitable to use the small concept to
> replace the big concept. Actually some status in the JobStatus, e.g.
> RESTARTING/SUSPENDED/RECONCILING don't work in the DDL Operation.
>
> On the other hand, the Gateway allows users to submit jobs(DML) in
> sync/async mode. The running status in the Operation Status in the
> different mode has different meaning:
> - In the async mode, when the gateway submits the job, the state comes to
> the FINISHED state
> - In the sync mode, the running status in the Operation status includes
> submitting the job, running job. Even if a failover occurs, we still think
> that this Operation is in the RUNNING state. Unless the job is
> unrecoverable, we change the Operation status to ERROR.
>
> Therefore, I think these two concepts are not consistent and we should not
> reuse the JobStatus. I add a section in the rejected alternatives.
>
> > Options to configure the REST endpoint
>
> Yes. I have modified the FLIP about this.
>
> > Naming conversion
>
> Yes. I have modified the FLIP with your suggestions.
>
> > Another smaller shortcomings in the FLIP
>
> >> SQLGatewayService.getFunction / UserDefinedFunctionInfo
>
> After reviewing the java.sql.DatabaseMetaData#getFunctions's java doc, I
> find it will return the system and user functions available in the Catalog.
> I think you are right. Therefore, we'd better to rename to the
> listFunctions(SessionHandle sessionHandle, OperationHandle operationHandle,
> String catalog, String database, ShowFunctionsOperation.FunctionScope) and
> it returns FunctionInfo.
>
> >> SQLGatewayService.getGatewayInfo()/getSessionConfig
>
> The result of the SQLGatewayService.getGatewayInfo and getSessionConfig is
> not used by the endpoint. The endpoint just serializes the result and
> presents it to the users. If we use the ReadableConfig, it's hard for us to
> iterate all the key value pairs.
>
> > configure_session VS initialize_session
> >> If calling it initialize_session, should we limit it only being called
> once?
>
> If we limit it only being called once, it allows the input of the
> initialize_session script. But the current design in the Gateway is aligned
> with the TableEnvironment#executeSql. That is, the input of the statement
> is a single statement rather than the script. Considering the API in the
> FLIP is not as same as the initialization in the CLI, I think we can use
> the configure_session? What do you think, Timo?
>
> Best,
> Shengkai
>
>
>
>
>
>
>
> Timo Walther  于2022年5月16日周一 14:28写道：
>
>> Hi Shengkai, Hi Jark,
>>
>> thanks for the additional explanation and the update of the FLIP. This
>> will help us in the future for documenting our decisions. The arguments
>> why to include the Gateway into the main repo make a lot of sense to me.
>> Esp. also because both CLI and gateway need some parsing functionality
>> that is dependent on the current state of the SQL syntax.
>>
>> Here is my last set of feedback, other than that +1 

[jira] [Created] (FLINK-27661) [Metric]Flink-Metrics pushgateway support authentication

[jiangchunyang (Jira)]

jiangchunyang created FLINK-27661:
-

 Summary: [Metric]Flink-Metrics pushgateway support authentication
 Key: FLINK-27661
 URL: https://issues.apache.org/jira/browse/FLINK-27661
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Metrics
 Environment: Flink：1.13.0
Reporter: jiangchunyang
 Fix For: 1.13.0


We found that the native PushGateway does not support authentication. As a 
result, the metrics data in on YARN mode cannot be reported to pushGateway with 
authentication.  

Although we have some other solutions, such as landing files and others, we 
think pushGateway is the best solution.  

So I decided to do some implementation on my own, and will submit pr to the 
community later



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27660) Table API support create function using customed jar

[dalongliu (Jira)]

dalongliu created FLINK-27660:
-

 Summary: Table API support create function using customed jar
 Key: FLINK-27660
 URL: https://issues.apache.org/jira/browse/FLINK-27660
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Affects Versions: 1.16.0
Reporter: dalongliu
 Fix For: 1.16.0






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27659) Planner support to use jar which is registered by "USING JAR" syntax

[dalongliu (Jira)]

dalongliu created FLINK-27659:
-

 Summary: Planner support to use jar which is registered by "USING 
JAR" syntax
 Key: FLINK-27659
 URL: https://issues.apache.org/jira/browse/FLINK-27659
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API, Table SQL / Planner
Affects Versions: 1.16.0
Reporter: dalongliu
 Fix For: 1.16.0






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27658) Introduce MutableURLClassLoader allow to register and remove user jar dynamically

[dalongliu (Jira)]

dalongliu created FLINK-27658:
-

 Summary: Introduce MutableURLClassLoader allow to register and 
remove user jar dynamically
 Key: FLINK-27658
 URL: https://issues.apache.org/jira/browse/FLINK-27658
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 1.16.0
Reporter: dalongliu
 Fix For: 1.16.0






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

[Александр Смирнов]

Hi Yuan!

> How about abtract the LookupCache to a higher level with a common Cache?

Can you give an example of such upper - level Cache usage? It's not
clear for me currently. I think it's unnecessary to have such high
level abstraction, if nowhere in the code we won't operate with
objects as instances of Cache. But maybe there are other opinions on
this.

> Does it have any metrics, such as NumCachedRecords for the AllCache?

I think there won't be many problems with supporting metrics in ALL
cache.  Moreover, some of proposed metrics are most useful especially
in ALL case, for example, 'latestLoadTimeGauge' or 'numCachedRecords',
so necessary metrics definitely should be supported in this cache
strategy.

Best regards,
Alexander

вс, 15 мая 2022 г. в 20:17, zst...@163.com :
>
> Hi Qingsheng and devs,
>
>
>
>
> Thanks for your heated discussion and redesign to optmize this feature. I 
> just have two comments:
>
> 1. How about abtract the LookupCache to a higher level with a common Cache? 
> It will be convenient for devs to use in other place.
>
>
>
>
> 2. Does it have any metrics, such as NumCachedRecords for the AllCache?
>
> Best regards,
> Yuan
>
> At 2022-05-13 20:27:44, "Qingsheng Ren"  wrote:
> >Hi Alexander and devs,
> >
> >Thank you very much for the in-depth discussion! As Jark mentioned we were
> >inspired by Alexander's idea and made a refactor on our design. FLIP-221
> >[1] has been updated to reflect our design now and we are happy to hear
> >more suggestions from you!
> >
> >Compared to the previous design:
> >1. The lookup cache serves at table runtime level and is integrated as a
> >component of LookupJoinRunner as discussed previously.
> >2. Interfaces are renamed and re-designed to reflect the new design.
> >3. We separate the all-caching case individually and introduce a new
> >RescanRuntimeProvider to reuse the ability of scanning. We are planning to
> >support SourceFunction / InputFormat for now considering the complexity of
> >FLIP-27 Source API.
> >4. A new interface LookupFunction is introduced to make the semantic of
> >lookup more straightforward for developers.
> >
> >For replying to Alexander:
> >> However I'm a little confused whether InputFormat is deprecated or not.
> >Am I right that it will be so in the future, but currently it's not?
> >Yes you are right. InputFormat is not deprecated for now. I think it will
> >be deprecated in the future but we don't have a clear plan for that.
> >
> >Thanks again for the discussion on this FLIP and looking forward to
> >cooperating with you after we finalize the design and interfaces!
> >
> >[1]
> >https://cwiki.apache.org/confluence/display/FLINK/FLIP-221+Abstraction+for+lookup+source+cache+and+metric
> >
> >Best regards,
> >
> >Qingsheng
> >
> >
> >On Fri, May 13, 2022 at 12:12 AM Александр Смирнов 
> >wrote:
> >
> >> Hi Jark, Qingsheng and Leonard!
> >>
> >> Glad to see that we came to a consensus on almost all points!
> >>
> >> However I'm a little confused whether InputFormat is deprecated or
> >> not. Am I right that it will be so in the future, but currently it's
> >> not? Actually I also think that for the first version it's OK to use
> >> InputFormat in ALL cache realization, because supporting rescan
> >> ability seems like a very distant prospect. But for this decision we
> >> need a consensus among all discussion participants.
> >>
> >> In general, I don't have something to argue with your statements. All
> >> of them correspond my ideas. Looking ahead, it would be nice to work
> >> on this FLIP cooperatively. I've already done a lot of work on lookup
> >> join caching with realization very close to the one we are discussing,
> >> and want to share the results of this work. Anyway looking forward for
> >> the FLIP update!
> >>
> >> Best regards,
> >> Smirnov Alexander
> >>
> >> чт, 12 мая 2022 г. в 17:38, Jark Wu :
> >> >
> >> > Hi Alex,
> >> >
> >> > Thanks for summarizing your points.
> >> >
> >> > In the past week, Qingsheng, Leonard, and I have discussed it several
> >> times
> >> > and we have totally refactored the design.
> >> > I'm glad to say we have reached a consensus on many of your points!
> >> > Qingsheng is still working on updating the design docs and maybe can be
> >> > available in the next few days.
> >> > I will share some conclusions from our discussions:
> >> >
> >> > 1) we have refactored the design towards to "cache in framework" way.
> >> >
> >> > 2) a "LookupCache" interface for users to customize and a default
> >> > implementation with builder for users to easy-use.
> >> > This can both make it possible to both have flexibility and conciseness.
> >> >
> >> > 3) Filter pushdown is important for ALL and LRU lookup cache, esp
> >> reducing
> >> > IO.
> >> > Filter pushdown should be the final state and the unified way to both
> >> > support pruning ALL cache and LRU cache,
> >> > so I think we should make effort in this direction. If we need to support
> >> > filter pushdown for ALL cache 

Re: [DISCUSS] FLIP-91: Support SQL Client Gateway

[Shengkai Fang]

Hi, Jark, Timo. Nice to have an agreement!

Thanks for Jark's inputs about the multiple version Flink. I have already
updated the FLIP in the rejected alternatives about details.

1. We should definitely just use LogicalTypeJsonSerializer and not a second
JSON representation.

Our concern is mainly that it's hard for users to use because of the
flexible structure. The LogicalTypeJsonSerializer will serialize the
VARCHAR to "VARCHAR()" or "{\"TYPE\": \"VARCHAR\", \"LENGTH\": 0}",
which requires the end users to process the different situations. But in
some cases, users just print the json to the terminal/web UI.  WDYT?

> Serialize the RowData

Sure. I will keep your advice in mind. I think the current serialization of
the RowData will not use the column name as the Object key in the json. I
am not sure whether I missed something. It would be nice if you can give me
an example if I do something wrong.

> Have you also thought about using Flink's state types from Flink
tasks/jobs?

Yes. But I still think we should use a new state machine. First of all,
Operation in the FLIP is much different from the Job. Operations include
DDL, DML and so on. So it's not suitable to use the small concept to
replace the big concept. Actually some status in the JobStatus, e.g.
RESTARTING/SUSPENDED/RECONCILING don't work in the DDL Operation.

On the other hand, the Gateway allows users to submit jobs(DML) in
sync/async mode. The running status in the Operation Status in the
different mode has different meaning:
- In the async mode, when the gateway submits the job, the state comes to
the FINISHED state
- In the sync mode, the running status in the Operation status includes
submitting the job, running job. Even if a failover occurs, we still think
that this Operation is in the RUNNING state. Unless the job is
unrecoverable, we change the Operation status to ERROR.

Therefore, I think these two concepts are not consistent and we should not
reuse the JobStatus. I add a section in the rejected alternatives.

> Options to configure the REST endpoint

Yes. I have modified the FLIP about this.

> Naming conversion

Yes. I have modified the FLIP with your suggestions.

> Another smaller shortcomings in the FLIP

>> SQLGatewayService.getFunction / UserDefinedFunctionInfo

After reviewing the java.sql.DatabaseMetaData#getFunctions's java doc, I
find it will return the system and user functions available in the Catalog.
I think you are right. Therefore, we'd better to rename to the
listFunctions(SessionHandle sessionHandle, OperationHandle operationHandle,
String catalog, String database, ShowFunctionsOperation.FunctionScope) and
it returns FunctionInfo.

>> SQLGatewayService.getGatewayInfo()/getSessionConfig

The result of the SQLGatewayService.getGatewayInfo and getSessionConfig is
not used by the endpoint. The endpoint just serializes the result and
presents it to the users. If we use the ReadableConfig, it's hard for us to
iterate all the key value pairs.

> configure_session VS initialize_session
>> If calling it initialize_session, should we limit it only being called
once?

If we limit it only being called once, it allows the input of the
initialize_session script. But the current design in the Gateway is aligned
with the TableEnvironment#executeSql. That is, the input of the statement
is a single statement rather than the script. Considering the API in the
FLIP is not as same as the initialization in the CLI, I think we can use
the configure_session? What do you think, Timo?

Best,
Shengkai







Timo Walther  于2022年5月16日周一 14:28写道：

> Hi Shengkai, Hi Jark,
>
> thanks for the additional explanation and the update of the FLIP. This
> will help us in the future for documenting our decisions. The arguments
> why to include the Gateway into the main repo make a lot of sense to me.
> Esp. also because both CLI and gateway need some parsing functionality
> that is dependent on the current state of the SQL syntax.
>
> Here is my last set of feedback, other than that +1 for the proposal:
>
> Serialize the LogicalType
>
> The FLIP mentions LogicalTypeJsonSerializer but the shown JSON is
> different from the current master. We are using the serializable
> representation of LogicalType as much as possible nowadays. We should
> definitely just use LogicalTypeJsonSerializer and not a second JSON
> representation.
>
> 1) Serialize the RowData
>
> Side note for serializing ROWs: we should not use field names in JSON
> object keys. As e.g. `null` and other names with special characters
> cause issues in JSON.
>
> 2) We propose the state machine like HiveServer2
>
> Have you also thought about using Flink's state types from Flink
> tasks/jobs? If we were using Flink types directly, it would be easier to
> monitor the execution of a INSERT INTO job via the gateway without
> having to map state types. Monitoring jobs is the most important
> functionality and should be in sync with regular Flink job monitoring. A
> HiveServer2 endpoint can still 

[jira] [Created] (FLINK-27657) Implement remote operator state backend in PyFlink

[Juntao Hu (Jira)]

Juntao Hu created FLINK-27657:
-

 Summary: Implement remote operator state backend in PyFlink
 Key: FLINK-27657
 URL: https://issues.apache.org/jira/browse/FLINK-27657
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python
Affects Versions: 1.15.0
Reporter: Juntao Hu
 Fix For: 1.16.0


This is for supporting broadcast state, exisintg map state implementation and 
caching handler can be reused.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)