Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements

[Ran Tao]

thanks Lau.

The vote will last for at least 72 hours (03/09, 19:30 UTC+8).
It needs consensus approval, requiring 3 binding +1 votes and no
binding vetoes.


Best Regards,
Ran Tao


Jacky Lau  于2023年3月7日周二 15:11写道：

> Thanks Ran.
> +1 (non-binding)
>
> Regards,
> Jacky Lau
>
> Ran Tao  于2023年3月6日周一 19:32写道：
>
> > Hi Everyone,
> >
> >
> > I want to start the vote on FLIP-297: Improve Auxiliary Sql Statements
> [1].
> > The FLIP was discussed in this thread [2].
> >
> >
> > The goal of the FLIP is to improve flink auxiliary sql
> statements(compared
> > with sql standard or other mature engines).
> >
> > The vote will last for at least 72 hours (03/09, 19:30 UTC+8)
> > unless there is an objection or insufficient votes. Thank you all.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-297%3A+Improve+Auxiliary+Sql+Statements
> > [2] https://lists.apache.org/thread/54fyd27m8on1cf3hn6dz564zqmkobjyd
> >
> > Best Regards,
> > Ran Tao
> > https://github.com/chucheng92
> >
>

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

[Yangze Guo]

Hi Weihua,

Thanks for driving this. As Xintong mentioned, this was a technical
debt from FLIP-56.

The latest version of FLIP sounds good, +1 from my side. As a
contributor to this component, I'm willing to assist with the review
process. Feel free to reach me if you need help.

Best,
Yangze Guo

On Tue, Mar 7, 2023 at 1:47 PM Weihua Hu  wrote:
>
> Hi,
>
> @David @Matthias
> There are a few days after hearing your thoughts. I would like to know if
> there are any other concerns about this FLIP.
>
>
> Best,
> Weihua
>
>
> On Mon, Mar 6, 2023 at 7:53 PM Weihua Hu  wrote:
>
> >
> > Thanks Shammon,
> >
> > I've updated FLIP to add this redundant Task Manager limitation.
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Mon, Mar 6, 2023 at 5:07 PM Shammon FY  wrote:
> >
> >> Hi weihua
> >>
> >> Can you add content related to `heterogeneous resources` to this FLIP? We
> >> can record it and consider it in the future. It may be useful for some
> >> scenarios, such as the combination of streaming and ML.
> >>
> >> Best,
> >> Shammon
> >>
> >>
> >> On Mon, Mar 6, 2023 at 1:39 PM weijie guo 
> >> wrote:
> >>
> >> > Hi Weihua,
> >> >
> >> > Thanks for your clarification, SGTM.
> >> >
> >> > Best regards,
> >> >
> >> > Weijie
> >> >
> >> >
> >> > Weihua Hu  于2023年3月6日周一 11:43写道：
> >> >
> >> > > Thanks Weijie.
> >> > >
> >> > > Heterogeneous task managers will not be considered in this FLIP since
> >> > > it does not request heterogeneous resources as you said.
> >> > >
> >> > > My first thought is we can adjust the meaning of redundant
> >> configuration
> >> > > to redundant number of per resource type. These can be considered in
> >> > > detail when we decide to support heterogeneous task managers.
> >> > >
> >> > > Best,
> >> > > Weihua
> >> > >
> >> > >
> >> > > On Sat, Mar 4, 2023 at 1:13 AM weijie guo 
> >> > > wrote:
> >> > >
> >> > > > Thanks Weihua for preparing this FLIP.
> >> > > >
> >> > > > This FLIP overall looks reasonable to me after updating as
> >> suggested by
> >> > > > Matthias.
> >> > > >
> >> > > > I only have one small question about keeping some redundant task
> >> > > managers:
> >> > > > In the fine-grained resource management, theoretically, it can
> >> support
> >> > > > heterogeneous taskmanagers. When we complete the missing features
> >> for
> >> > > FGSM,
> >> > > > do we plan to take this into account?
> >> > > > Of course, if I remember correctly, FGSM will not request
> >> heterogeneous
> >> > > > resources at present, so it is also acceptable to me if there is no
> >> > > special
> >> > > > treatment now.
> >> > > >
> >> > > > +1 for this changes if we can ensure the test coverage.
> >> > > >
> >> > > > Best regards,
> >> > > >
> >> > > > Weijie
> >> > > >
> >> > > >
> >> > > > John Roesler  于2023年3月2日周四 12:53写道：
> >> > > >
> >> > > > > Thanks for the test plan, Weihua!
> >> > > > >
> >> > > > > Yes, it addresses my concerns.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > John
> >> > > > >
> >> > > > > On Wed, Mar 1, 2023, at 22:38, Weihua Hu wrote:
> >> > > > > > Hi, everyone,
> >> > > > > > Thanks for your suggestions and ideas.
> >> > > > > > Thanks Xintong for sharing the detailed backgrounds of
> >> SlotManager.
> >> > > > > >
> >> > > > > > *@Matthias
> >> > > > > >
> >> > > > > > 1. Did you do a proper test coverage analysis?
> >> > > > > >
> >> > > > > >
> >> > > > > > Just as Xintong said, we already have a CI stage for fine
> >> grained
> >> > > > > resource
> >> > > > > > managers.
> >> > > > > > And I will make sure FineGrainedSlotManager as the default
> >> > > SlotManager
> >> > > > > can
> >> > > > > > pass all the tests of CI.
> >> > > > > > In addition, I will review all unit tests of
> >> > > > DeclarativeSlotManager(DSM)
> >> > > > > to
> >> > > > > > ensure that there are no gaps in the
> >> > > > > > coverage provided by the FineGrainedSlotManager.
> >> > > > > > I also added the 'Test Plan' part to the FLIP.
> >> > > > > > @Matthias @John @Shammon Does this test plan address your
> >> concerns?
> >> > > > > >
> >> > > > > > 2.  DeclarativeSlotManager and FineGrainedSlotManager feel quite
> >> > big
> >> > > in
> >> > > > > >
> >> > > > > > terms of lines of code
> >> > > > > >
> >> > > > > >
> >> > > > > > IMO, the refactoring of SlotManager does not belong to this FLIP
> >> > > since
> >> > > > it
> >> > > > > > may lead to some unstable risks. For
> >> > > > > > FineGrainedSlotManager(FGSM), we already split some reasonable
> >> > > > > components.
> >> > > > > > They are:
> >> > > > > > * TaskManagerTracker: Track task managers and their resources.
> >> > > > > > * ResourceTracker: track requirements of jobs
> >> > > > > > * ResourceAllocationStrategy: Try to fulfill the resource
> >> > > requirements
> >> > > > > with
> >> > > > > > available/pending resources.
> >> > > > > > * SlotStatusSyncer: communicate with TaskManager, for
> >> > > > allocating/freeing
> >> > > > > > slot and reconciling the slot status
> >> > > > > > Maybe we can start a discussion 

[jira] [Created] (FLINK-31355) CommonDataStreamTests.test_execute_and_collect failed

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31355:
-

 Summary: CommonDataStreamTests.test_execute_and_collect failed
 Key: FLINK-31355
 URL: https://issues.apache.org/jira/browse/FLINK-31355
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.17.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46883=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=32651

{code}
Mar 07 07:20:04 ERRORroot:java_gateway.py:1055 Exception while sending 
command.
Mar 07 07:20:04 Traceback (most recent call last):
Mar 07 07:20:04   File 
"/__w/1/s/flink-python/.tox/py310-cython/lib/python3.10/site-packages/py4j/java_gateway.py",
 line 1224, in send_command
Mar 07 07:20:04 raise Py4JNetworkError("Answer from Java side is empty")
Mar 07 07:20:04 py4j.protocol.Py4JNetworkError: Answer from Java side is empty
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31354) NettyClientServerSslTest.testValidSslConnectionAdvanced timed out

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31354:
-

 Summary: NettyClientServerSslTest.testValidSslConnectionAdvanced 
timed out
 Key: FLINK-31354
 URL: https://issues.apache.org/jira/browse/FLINK-31354
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.17.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46883=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798=8242

{code}
Mar 07 05:15:21 [ERROR] NettyClientServerSslTest.testValidSslConnectionAdvanced 
 Time elapsed: 3.69 s  <<< ERROR!
Mar 07 05:15:21 
org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandshakeTimeoutException:
 handshake timed out after 1000ms
Mar 07 05:15:21 at 
org.apache.flink.shaded.netty4.io.netty.handler.ssl.SslHandler$7.run(SslHandler.java:2115)
Mar 07 05:15:21 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
Mar 07 05:15:21 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
Mar 07 05:15:21 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
Mar 07 05:15:21 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
Mar 07 05:15:21 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
Mar 07 05:15:21 at 
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
Mar 07 05:15:21 at 
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
Mar 07 05:15:21 at 
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Mar 07 05:15:21 at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31353) Merge SnapshotEnumerator into TableScan

[yuzelin (Jira)]

yuzelin created FLINK-31353:
---

 Summary: Merge SnapshotEnumerator into TableScan 
 Key: FLINK-31353
 URL: https://issues.apache.org/jira/browse/FLINK-31353
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Affects Versions: table-store-0.4.0
Reporter: yuzelin


The abilities of SnapshotEnumerator and TableScan are duplicated. Since 
configurations used to create SnapshotEnumerator also can be fetched in table, 
it's better to use TableScan to replace SnapshotEnumerator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31352) OverAggregateITCase.testWindowAggregationSumWithoutOrderBy times out on CI

[Jira]

David Morávek created FLINK-31352:
-

 Summary: 
OverAggregateITCase.testWindowAggregationSumWithoutOrderBy times out on CI
 Key: FLINK-31352
 URL: https://issues.apache.org/jira/browse/FLINK-31352
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: David Morávek


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46880=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=15913]

 
{code:java}
Mar 07 00:20:18 "main" #1 prio=5 os_prio=0 tid=0x7f805800b800 nid=0x3a9f 
waiting on condition [0x7f8060875000]
Mar 07 00:20:18java.lang.Thread.State: WAITING (parking)
Mar 07 00:20:18 at sun.misc.Unsafe.park(Native Method)
Mar 07 00:20:18 - parking to wait for  <0xa7481ba0> (a 
java.util.concurrent.CompletableFuture$Signaller)
Mar 07 00:20:18 at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
Mar 07 00:20:18 at 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
Mar 07 00:20:18 at 
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
Mar 07 00:20:18 at 
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
Mar 07 00:20:18 at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
Mar 07 00:20:18 at 
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:170)
Mar 07 00:20:18 at 
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:129)
Mar 07 00:20:18 at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
Mar 07 00:20:18 at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
Mar 07 00:20:18 at 
org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:222)
Mar 07 00:20:18 at 
java.util.Iterator.forEachRemaining(Iterator.java:115)
Mar 07 00:20:18 at 
org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:115)
Mar 07 00:20:18 at 
org.apache.flink.table.planner.runtime.utils.BatchTestBase.executeQuery(BatchTestBase.scala:308)
Mar 07 00:20:18 at 
org.apache.flink.table.planner.runtime.utils.BatchTestBase.check(BatchTestBase.scala:144)
Mar 07 00:20:18 at 
org.apache.flink.table.planner.runtime.utils.BatchTestBase.checkResult(BatchTestBase.scala:108)
Mar 07 00:20:18 at 
org.apache.flink.table.planner.runtime.batch.sql.OverAggregateITCase.testWindowAggregationSumWithoutOrderBy(OverAggregateITCase.scala:464)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[Aitozi]

> we can initiate corresponding support issues for
specific connectors to follow up on support after finalizing the API
changes

Make sense to me!

Best,
Aitozi.

Lincoln Lee  于2023年3月7日周二 15:05写道：

> Thanks Jingsong & Hang,
>
> Using Optional as the return value is a good idea. Previously, I
> hoped to keep the style of the LookupTableSource.LookupContext#getKeys as
> consistent as possible, but the getKeys is actually non-empty when used, so
> I support updating to Optional.  I'll update the flip doc and poc
> later tonight.
>
> Best,
> Lincoln Lee
>
>
> Lincoln Lee  于2023年3月7日周二 14:53写道：
>
> > Hi Aitozi,
> >
> > Thanks for your feedback!  Yes, including HBase and JDBC connector, they
> > can be considered for support in the next step (JDBC as as a standard
> > protocol supported not only in traditional databases, but also in more
> and
> > more new types of storage). Considering the ongoing externalizing of
> > connectors and the release cycles of the connectors are decoupled with
> the
> > release cycle of Flink, we can initiate corresponding support issues for
> > specific connectors to follow up on support after finalizing the API
> > changes, WDYT?
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Hang Ruan  于2023年3月7日周二 12:14写道：
> >
> >> Hi, Lincoln,
> >>
> >> Thanks for bringing this up. It looks good to me. I also agree with
> >> Jingsong's suggestion.
> >>
> >> Best,
> >> Hang
> >>
> >> Jingsong Li  于2023年3月7日周二 11:15写道：
> >>
> >> > Wow, we have 300 FLIPs...
> >> >
> >> > Thanks Lincoln,
> >> >
> >> > Have you considered returning an Optional?
> >> >
> >> > Empty array looks a little weird to me.
> >> >
> >> > Best,
> >> > Jingsong
> >> >
> >> > On Tue, Mar 7, 2023 at 10:32 AM Aitozi  wrote:
> >> > >
> >> > > Hi Lincoln,
> >> > > Thank you for sharing this FLIP. Overall, it looks good to me. I
> >> have
> >> > > one question: with the introduction of this interface,
> >> > > will any existing Flink connectors need to be updated in order to
> take
> >> > > advantage of its capabilities? For example, HBase.
> >> > >
> >> > > yuxia  于2023年3月7日周二 10:01写道：
> >> > >
> >> > > > Thanks. It makes sense to me.
> >> > > >
> >> > > > Best regards,
> >> > > > Yuxia
> >> > > >
> >> > > > - 原始邮件 -
> >> > > > 发件人: "Lincoln Lee" 
> >> > > > 收件人: "dev" 
> >> > > > 发送时间: 星期一, 2023年 3 月 06日 下午 10:26:26
> >> > > > 主题: Re: [DISCUSS] FLIP-300: Add targetColumns to
> >> > DynamicTableSink#Context
> >> > > > to solve the null overwrite problem of partial-insert
> >> > > >
> >> > > > hi yuxia,
> >> > > >
> >> > > > Thanks for your feedback and tracking the issue of update
> statement!
> >> > I've
> >> > > > updated the FLIP[1] and also the poc[2].
> >> > > > Since the bug and flip are orthogonal, we can focus on finalizing
> >> the
> >> > api
> >> > > > changes first, and then work on the flip implementation and bugfix
> >> > > > separately, WDYT?
> >> > > >
> >> > > > [1]
> >> > > >
> >> >
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> >> > > > [2] https://github.com/apache/flink/pull/22041
> >> > > >
> >> > > > Best,
> >> > > > Lincoln Lee
> >> > > >
> >> > > >
> >> > > > yuxia  于2023年3月6日周一 21:21写道：
> >> > > >
> >> > > > > Hi, Lincoln.
> >> > > > > Thanks for bringing this up. +1 for this FLIP, it's helpful for
> >> > external
> >> > > > > storage system to implement partial update.
> >> > > > > The FLIP looks good to me. I only want to add one comment,
> update
> >> > > > > statement also doesn't support updating nested column, I have
> >> created
> >> > > > > FLINK-31344[1] to track it.
> >> > > > > Maybe we also need to explain it in this FLIP.
> >> > > > >
> >> > > > > [1] https://issues.apache.org/jira/browse/FLINK-31344
> >> > > > >
> >> > > > > Best regards,
> >> > > > > Yuxia
> >> > > > >
> >> > > > > - 原始邮件 -
> >> > > > > 发件人: "Lincoln Lee" 
> >> > > > > 收件人: "dev" 
> >> > > > > 发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
> >> > > > > 主题: [DISCUSS] FLIP-300: Add targetColumns to
> >> > DynamicTableSink#Context to
> >> > > > > solve the null overwrite problem of partial-insert
> >> > > > >
> >> > > > > Hi everyone,
> >> > > > >
> >> > > > > This FLIP[1] aims to support connectors in avoiding overwriting
> >> > > > non-target
> >> > > > > columns with null values when processing partial column updates,
> >> we
> >> > > > propose
> >> > > > > adding information on the target column list to
> >> > DynamicTableSink#Context.
> >> > > > >
> >> > > > > FLINK-18726[2] supports inserting statements with specified
> column
> >> > list,
> >> > > > it
> >> > > > > fills null values (or potentially declared default values in the
> >> > future)
> >> > > > > for columns not appearing in the column list of insert statement
> >> to
> >> > the
> >> > > > > target table.
> >> > > > > But this behavior does not satisfy some partial column update
> >> > > > requirements
> >> > > > > of some storage systems which allow storing null values. The
> >> problem
> >> > is
> >> > > > > that 

Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements

[Jacky Lau]

Thanks Ran.
+1 (non-binding)

Regards,
Jacky Lau

Ran Tao  于2023年3月6日周一 19:32写道：

> Hi Everyone,
>
>
> I want to start the vote on FLIP-297: Improve Auxiliary Sql Statements [1].
> The FLIP was discussed in this thread [2].
>
>
> The goal of the FLIP is to improve flink auxiliary sql statements(compared
> with sql standard or other mature engines).
>
> The vote will last for at least 72 hours (03/09, 19:30 UTC+8)
> unless there is an objection or insufficient votes. Thank you all.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-297%3A+Improve+Auxiliary+Sql+Statements
> [2] https://lists.apache.org/thread/54fyd27m8on1cf3hn6dz564zqmkobjyd
>
> Best Regards,
> Ran Tao
> https://github.com/chucheng92
>

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[Lincoln Lee]

Thanks Jingsong & Hang,

Using Optional as the return value is a good idea. Previously, I
hoped to keep the style of the LookupTableSource.LookupContext#getKeys as
consistent as possible, but the getKeys is actually non-empty when used, so
I support updating to Optional.  I'll update the flip doc and poc
later tonight.

Best,
Lincoln Lee


Lincoln Lee  于2023年3月7日周二 14:53写道：

> Hi Aitozi,
>
> Thanks for your feedback!  Yes, including HBase and JDBC connector, they
> can be considered for support in the next step (JDBC as as a standard
> protocol supported not only in traditional databases, but also in more and
> more new types of storage). Considering the ongoing externalizing of
> connectors and the release cycles of the connectors are decoupled with the
> release cycle of Flink, we can initiate corresponding support issues for
> specific connectors to follow up on support after finalizing the API
> changes, WDYT?
>
> Best,
> Lincoln Lee
>
>
> Hang Ruan  于2023年3月7日周二 12:14写道：
>
>> Hi, Lincoln,
>>
>> Thanks for bringing this up. It looks good to me. I also agree with
>> Jingsong's suggestion.
>>
>> Best,
>> Hang
>>
>> Jingsong Li  于2023年3月7日周二 11:15写道：
>>
>> > Wow, we have 300 FLIPs...
>> >
>> > Thanks Lincoln,
>> >
>> > Have you considered returning an Optional?
>> >
>> > Empty array looks a little weird to me.
>> >
>> > Best,
>> > Jingsong
>> >
>> > On Tue, Mar 7, 2023 at 10:32 AM Aitozi  wrote:
>> > >
>> > > Hi Lincoln,
>> > > Thank you for sharing this FLIP. Overall, it looks good to me. I
>> have
>> > > one question: with the introduction of this interface,
>> > > will any existing Flink connectors need to be updated in order to take
>> > > advantage of its capabilities? For example, HBase.
>> > >
>> > > yuxia  于2023年3月7日周二 10:01写道：
>> > >
>> > > > Thanks. It makes sense to me.
>> > > >
>> > > > Best regards,
>> > > > Yuxia
>> > > >
>> > > > - 原始邮件 -
>> > > > 发件人: "Lincoln Lee" 
>> > > > 收件人: "dev" 
>> > > > 发送时间: 星期一, 2023年 3 月 06日 下午 10:26:26
>> > > > 主题: Re: [DISCUSS] FLIP-300: Add targetColumns to
>> > DynamicTableSink#Context
>> > > > to solve the null overwrite problem of partial-insert
>> > > >
>> > > > hi yuxia,
>> > > >
>> > > > Thanks for your feedback and tracking the issue of update statement!
>> > I've
>> > > > updated the FLIP[1] and also the poc[2].
>> > > > Since the bug and flip are orthogonal, we can focus on finalizing
>> the
>> > api
>> > > > changes first, and then work on the flip implementation and bugfix
>> > > > separately, WDYT?
>> > > >
>> > > > [1]
>> > > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
>> > > > [2] https://github.com/apache/flink/pull/22041
>> > > >
>> > > > Best,
>> > > > Lincoln Lee
>> > > >
>> > > >
>> > > > yuxia  于2023年3月6日周一 21:21写道：
>> > > >
>> > > > > Hi, Lincoln.
>> > > > > Thanks for bringing this up. +1 for this FLIP, it's helpful for
>> > external
>> > > > > storage system to implement partial update.
>> > > > > The FLIP looks good to me. I only want to add one comment, update
>> > > > > statement also doesn't support updating nested column, I have
>> created
>> > > > > FLINK-31344[1] to track it.
>> > > > > Maybe we also need to explain it in this FLIP.
>> > > > >
>> > > > > [1] https://issues.apache.org/jira/browse/FLINK-31344
>> > > > >
>> > > > > Best regards,
>> > > > > Yuxia
>> > > > >
>> > > > > - 原始邮件 -
>> > > > > 发件人: "Lincoln Lee" 
>> > > > > 收件人: "dev" 
>> > > > > 发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
>> > > > > 主题: [DISCUSS] FLIP-300: Add targetColumns to
>> > DynamicTableSink#Context to
>> > > > > solve the null overwrite problem of partial-insert
>> > > > >
>> > > > > Hi everyone,
>> > > > >
>> > > > > This FLIP[1] aims to support connectors in avoiding overwriting
>> > > > non-target
>> > > > > columns with null values when processing partial column updates,
>> we
>> > > > propose
>> > > > > adding information on the target column list to
>> > DynamicTableSink#Context.
>> > > > >
>> > > > > FLINK-18726[2] supports inserting statements with specified column
>> > list,
>> > > > it
>> > > > > fills null values (or potentially declared default values in the
>> > future)
>> > > > > for columns not appearing in the column list of insert statement
>> to
>> > the
>> > > > > target table.
>> > > > > But this behavior does not satisfy some partial column update
>> > > > requirements
>> > > > > of some storage systems which allow storing null values. The
>> problem
>> > is
>> > > > > that connectors cannot distinguish whether the null value of a
>> > column is
>> > > > > really from the user's data or whether it is a null value
>> populated
>> > > > because
>> > > > > of partial insert behavior.
>> > > > >
>> > > > > Looking forward to your comments or feedback.
>> > > > >
>> > > > > [1]
>> > > > >
>> > > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
>> > > > > [2] https://issues.apache.org/jira/browse/FLINK-18726
>> > > > >
>> > > > > Best,
>> > > 

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[Lincoln Lee]

Hi Aitozi,

Thanks for your feedback!  Yes, including HBase and JDBC connector, they
can be considered for support in the next step (JDBC as as a standard
protocol supported not only in traditional databases, but also in more and
more new types of storage). Considering the ongoing externalizing of
connectors and the release cycles of the connectors are decoupled with the
release cycle of Flink, we can initiate corresponding support issues for
specific connectors to follow up on support after finalizing the API
changes, WDYT?

Best,
Lincoln Lee


Hang Ruan  于2023年3月7日周二 12:14写道：

> Hi, Lincoln,
>
> Thanks for bringing this up. It looks good to me. I also agree with
> Jingsong's suggestion.
>
> Best,
> Hang
>
> Jingsong Li  于2023年3月7日周二 11:15写道：
>
> > Wow, we have 300 FLIPs...
> >
> > Thanks Lincoln,
> >
> > Have you considered returning an Optional?
> >
> > Empty array looks a little weird to me.
> >
> > Best,
> > Jingsong
> >
> > On Tue, Mar 7, 2023 at 10:32 AM Aitozi  wrote:
> > >
> > > Hi Lincoln,
> > > Thank you for sharing this FLIP. Overall, it looks good to me. I
> have
> > > one question: with the introduction of this interface,
> > > will any existing Flink connectors need to be updated in order to take
> > > advantage of its capabilities? For example, HBase.
> > >
> > > yuxia  于2023年3月7日周二 10:01写道：
> > >
> > > > Thanks. It makes sense to me.
> > > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > > - 原始邮件 -
> > > > 发件人: "Lincoln Lee" 
> > > > 收件人: "dev" 
> > > > 发送时间: 星期一, 2023年 3 月 06日 下午 10:26:26
> > > > 主题: Re: [DISCUSS] FLIP-300: Add targetColumns to
> > DynamicTableSink#Context
> > > > to solve the null overwrite problem of partial-insert
> > > >
> > > > hi yuxia,
> > > >
> > > > Thanks for your feedback and tracking the issue of update statement!
> > I've
> > > > updated the FLIP[1] and also the poc[2].
> > > > Since the bug and flip are orthogonal, we can focus on finalizing the
> > api
> > > > changes first, and then work on the flip implementation and bugfix
> > > > separately, WDYT?
> > > >
> > > > [1]
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> > > > [2] https://github.com/apache/flink/pull/22041
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > yuxia  于2023年3月6日周一 21:21写道：
> > > >
> > > > > Hi, Lincoln.
> > > > > Thanks for bringing this up. +1 for this FLIP, it's helpful for
> > external
> > > > > storage system to implement partial update.
> > > > > The FLIP looks good to me. I only want to add one comment, update
> > > > > statement also doesn't support updating nested column, I have
> created
> > > > > FLINK-31344[1] to track it.
> > > > > Maybe we also need to explain it in this FLIP.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/FLINK-31344
> > > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > > - 原始邮件 -
> > > > > 发件人: "Lincoln Lee" 
> > > > > 收件人: "dev" 
> > > > > 发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
> > > > > 主题: [DISCUSS] FLIP-300: Add targetColumns to
> > DynamicTableSink#Context to
> > > > > solve the null overwrite problem of partial-insert
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > This FLIP[1] aims to support connectors in avoiding overwriting
> > > > non-target
> > > > > columns with null values when processing partial column updates, we
> > > > propose
> > > > > adding information on the target column list to
> > DynamicTableSink#Context.
> > > > >
> > > > > FLINK-18726[2] supports inserting statements with specified column
> > list,
> > > > it
> > > > > fills null values (or potentially declared default values in the
> > future)
> > > > > for columns not appearing in the column list of insert statement to
> > the
> > > > > target table.
> > > > > But this behavior does not satisfy some partial column update
> > > > requirements
> > > > > of some storage systems which allow storing null values. The
> problem
> > is
> > > > > that connectors cannot distinguish whether the null value of a
> > column is
> > > > > really from the user's data or whether it is a null value populated
> > > > because
> > > > > of partial insert behavior.
> > > > >
> > > > > Looking forward to your comments or feedback.
> > > > >
> > > > > [1]
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> > > > > [2] https://issues.apache.org/jira/browse/FLINK-18726
> > > > >
> > > > > Best,
> > > > > Lincoln Lee
> > > > >
> > > >
> >
>

Re: Regarding new command to download jars in flink cluster

[Surendra Singh Lilhore]

Hi David,

Sorry I missed your reply.

>>>Since you're mentioning docker, I assume you're deploying your
application to k8s. Is that correct?

yes, I am using k8s.

>>>For handcrafted Kubernetes deployments, you can simply download the jar into
the user lib folder in an init container [1]. You can usually reuse existing
docker images to download the jar. For example, for S3, the AWS CLI image
will do the trick [2].

I believe that since Flink already supports different filesystems, we can
utilize the same code without depending on any other CLI. This would help
us save the time currently taken by the init container.

Not only can application mode clusters on Kubernetes utilize this, but
other clusters can also use it to download additional plugins before
starting the cluster.

Thanks
Surendra





On Sat, Mar 4, 2023 at 12:41 AM Surendra Singh Lilhore <
surendralilh...@apache.org> wrote:

> Hi Team,
>
>
>
> According to the Flink documentation, in the APP mode, the application jar
> should be bundled with the Flink image. However, building an image for each
> new application can be difficult. Can we introduce new commands that will
> help to download the required jar locally before starting Flink JM or TM
> containers? This should be a simple command that depends on the supported
> file system (S3, HDFS, ABFS) in Flink, and the command format should be
> something like this:
>
> *./flink fs-download  *
>
> Example:
>
> *./flink fs-download
> abfs://mycontai...@storageaccount.dfs.core.windows.net/jars /tmp/appjars
> *
>
> I have already tested this in my cluster, and it is working fine. Before
> raising a JIRA ticket, I would like to get suggestions from the community.
>
>
> Thanks and Regards
> Surendra
>

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

[Weihua Hu]

Hi,

@David @Matthias
There are a few days after hearing your thoughts. I would like to know if
there are any other concerns about this FLIP.


Best,
Weihua


On Mon, Mar 6, 2023 at 7:53 PM Weihua Hu  wrote:

>
> Thanks Shammon,
>
> I've updated FLIP to add this redundant Task Manager limitation.
>
>
> Best,
> Weihua
>
>
> On Mon, Mar 6, 2023 at 5:07 PM Shammon FY  wrote:
>
>> Hi weihua
>>
>> Can you add content related to `heterogeneous resources` to this FLIP? We
>> can record it and consider it in the future. It may be useful for some
>> scenarios, such as the combination of streaming and ML.
>>
>> Best,
>> Shammon
>>
>>
>> On Mon, Mar 6, 2023 at 1:39 PM weijie guo 
>> wrote:
>>
>> > Hi Weihua,
>> >
>> > Thanks for your clarification, SGTM.
>> >
>> > Best regards,
>> >
>> > Weijie
>> >
>> >
>> > Weihua Hu  于2023年3月6日周一 11:43写道：
>> >
>> > > Thanks Weijie.
>> > >
>> > > Heterogeneous task managers will not be considered in this FLIP since
>> > > it does not request heterogeneous resources as you said.
>> > >
>> > > My first thought is we can adjust the meaning of redundant
>> configuration
>> > > to redundant number of per resource type. These can be considered in
>> > > detail when we decide to support heterogeneous task managers.
>> > >
>> > > Best,
>> > > Weihua
>> > >
>> > >
>> > > On Sat, Mar 4, 2023 at 1:13 AM weijie guo 
>> > > wrote:
>> > >
>> > > > Thanks Weihua for preparing this FLIP.
>> > > >
>> > > > This FLIP overall looks reasonable to me after updating as
>> suggested by
>> > > > Matthias.
>> > > >
>> > > > I only have one small question about keeping some redundant task
>> > > managers:
>> > > > In the fine-grained resource management, theoretically, it can
>> support
>> > > > heterogeneous taskmanagers. When we complete the missing features
>> for
>> > > FGSM,
>> > > > do we plan to take this into account?
>> > > > Of course, if I remember correctly, FGSM will not request
>> heterogeneous
>> > > > resources at present, so it is also acceptable to me if there is no
>> > > special
>> > > > treatment now.
>> > > >
>> > > > +1 for this changes if we can ensure the test coverage.
>> > > >
>> > > > Best regards,
>> > > >
>> > > > Weijie
>> > > >
>> > > >
>> > > > John Roesler  于2023年3月2日周四 12:53写道：
>> > > >
>> > > > > Thanks for the test plan, Weihua!
>> > > > >
>> > > > > Yes, it addresses my concerns.
>> > > > >
>> > > > > Thanks,
>> > > > > John
>> > > > >
>> > > > > On Wed, Mar 1, 2023, at 22:38, Weihua Hu wrote:
>> > > > > > Hi, everyone,
>> > > > > > Thanks for your suggestions and ideas.
>> > > > > > Thanks Xintong for sharing the detailed backgrounds of
>> SlotManager.
>> > > > > >
>> > > > > > *@Matthias
>> > > > > >
>> > > > > > 1. Did you do a proper test coverage analysis?
>> > > > > >
>> > > > > >
>> > > > > > Just as Xintong said, we already have a CI stage for fine
>> grained
>> > > > > resource
>> > > > > > managers.
>> > > > > > And I will make sure FineGrainedSlotManager as the default
>> > > SlotManager
>> > > > > can
>> > > > > > pass all the tests of CI.
>> > > > > > In addition, I will review all unit tests of
>> > > > DeclarativeSlotManager(DSM)
>> > > > > to
>> > > > > > ensure that there are no gaps in the
>> > > > > > coverage provided by the FineGrainedSlotManager.
>> > > > > > I also added the 'Test Plan' part to the FLIP.
>> > > > > > @Matthias @John @Shammon Does this test plan address your
>> concerns?
>> > > > > >
>> > > > > > 2.  DeclarativeSlotManager and FineGrainedSlotManager feel quite
>> > big
>> > > in
>> > > > > >
>> > > > > > terms of lines of code
>> > > > > >
>> > > > > >
>> > > > > > IMO, the refactoring of SlotManager does not belong to this FLIP
>> > > since
>> > > > it
>> > > > > > may lead to some unstable risks. For
>> > > > > > FineGrainedSlotManager(FGSM), we already split some reasonable
>> > > > > components.
>> > > > > > They are:
>> > > > > > * TaskManagerTracker: Track task managers and their resources.
>> > > > > > * ResourceTracker: track requirements of jobs
>> > > > > > * ResourceAllocationStrategy: Try to fulfill the resource
>> > > requirements
>> > > > > with
>> > > > > > available/pending resources.
>> > > > > > * SlotStatusSyncer: communicate with TaskManager, for
>> > > > allocating/freeing
>> > > > > > slot and reconciling the slot status
>> > > > > > Maybe we can start a discussion about refactoring SlotManager in
>> > > > another
>> > > > > > FLIP if there are some good suggestions.
>> > > > > > WDYT
>> > > > > >
>> > > > > > 3. For me personally, having a more detailed summary comparing
>> the
>> > > > > >> subcomponents of both SlotManager implementations with where
>> > > > > >> their functionality matches and where they differ might help
>> > > > understand
>> > > > > the
>> > > > > >> consequences of the changes proposed in FLIP-298
>> > > > > >
>> > > > > > Good suggestion, I have updated the comparison in this FLIP.
>> > Looking
>> > > > > > forward to any suggestions/thoughts
>> > > > > > if they are not 

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

[Kui Yuan]

Hi Jing,


Thanks for the reminder. The aim of this flip is letting the sql users to
use those features in the Datastream API, we don't intend to extend
flip-217. In my opinion, the watermark alignment with only one source can
be configured by the options given in flip, and if the source connector
does not implement flip-217, the task will run with an error, reminding the
user to use `pipeline.watermark-alignment.allow- unaligned-source-splits`,
but maybe these behaviors are not understood by everyone, I will add some
tips about flip-217 in the flip to let users understand the behavior in the
case of source splits.


Best,

Kui Yuan

Jing Ge  于2023年3月7日周二 04:23写道：

> Hi Kui,
>
> Thanks for pointing that out. I knew FLIP-217 which was done by an
> engineer working in my team.  As far as I am concerned, your FLIP should
> answer the following questions:
>
> 1. How to enable the watermark alignment of a source splits with Flink SQL?
> e.g. which options should be used if only one source is used?
>
> 2. Default behaviour. i.e. Flink SQL users should be aware that watermark
> alignment of source split will only work for sources that implement
> FLIP-217 properly. Should users take care of
> `pipeline.watermark-alignment.allow-unaligned-source-splits`
> while using Flink SQL?
>
> Best regards,
> Jing
>
>
> On Fri, Mar 3, 2023 at 8:46 AM Kui Yuan  wrote:
>
> > Hi all,
> >
> > Thanks for all. There are more questions and I will answer one by one.
> >
> > @Jark Thanks for your tips. For the first question, I will add more
> details
> > in the flip, and give a POC[1] so that pepole can know how I'm currently
> > implementing these features.
> >
> > > IIRC, this is the first time we introduce the framework-level connector
> > > options that the option is not recognized and handled by connectors.
> > > The FLIP should cover how framework filters the watermark related
> options
> > > to avoid discover connector factory failed, and what happens if the
> > > connector already supported the conflict options
> >
> > For the second question, We know that the default strategy is
> 'on-periodic'
> > in SQL layer, and the default interval is 200ms. The reason for emiting
> > watermark periodically is that the time advancement of consecutive events
> > may be very small, we don't need to calculate watermark for each event.
> > Same for 'on-event' strategy, so my idea is that we can set a fixed gap
> for
> > 'on-event' strategy.
> >
> > > I'm not sure about the usage scenarios of event gap emit strategy. Do
> > > you have any specific use case of this strategy? I'm confused why no
> one
> > > requested this strategy before no matter in DataStream or SQL, but
> maybe
> > > I missed something. I'm not against to add this option, but just want
> to
> > be
> > > careful when adding new API because it's hard to remove in the future.
> >
> > As @Timo said, There is no default features like 'on-event-gap' in
> > DataStream API, but the users can achieve the 'on-event-gap' feature by
> > using `WatermarkGenerator` interface, just like the implemention in my
> > POC[1]. However, If we don't provide it  in SQL layer, there is no way
> for
> > users to use similar features.
> >
> > > Jark raised a very good point. I thought we only expose what is
> > > contained in DataStream API already. If this strategy is not part of
> > > DataStream API, would like to exclude it from the FLIP. We need to be
> > > careful which strategies we offer by default.
> >
> > @Jark @Timo I'm sorry, perhaps I don't understand what are your concerns
> > about CompiledPlan, maybe I missed something else, maybe you can look at
> my
> > POC first to see if there is somewhere to worry about.
> >
> > > Sorry, I forgot to remind you that Timo's concern about the changes to
> > the
> > > CompiledPlan looks like is still not covered in the FLIP.
> >
> > @Jing We could have more discussion about naming, but I prefer that the
> > naming should be consistent with the DataStream API.
> > About aligning splits/partitions/shards, maybe you missed FLIP-217[2]
> which
> > aims to support watermark alignment of source splits.
> >
> > > After reading the most up-to-date Flip, I didn't find any information
> if
> > > this solution will support aligning splits/partitions/shards [1]. Did I
> > > miss anything?
> >
> > Best
> > Kui Yuan
> >
> > [1] the POC:
> > https://github.com/yuchengxin/flink/tree/yuankui/watermark_params
> > [2] FLIP-217:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits
> >
> >
> > Jing Ge  于2023年3月3日周五 08:03写道：
> >
> > > Hi,
> > >
> > > Thanks Kui for driving this Flip and thanks all for the informative
> > > discussion.
> > >
> > > @Timo
> > >
> > > Your suggestion about the naming convention is excellent. Thanks! I was
> > > wondering why you, exceptionally, suggested 'scan.idle-timeout' instead
> > of
> > > 'scan.watermark.idle-timeout'. I must miss something here.
> > >
> > > There is one 

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[Hang Ruan]

Hi, Lincoln,

Thanks for bringing this up. It looks good to me. I also agree with
Jingsong's suggestion.

Best,
Hang

Jingsong Li  于2023年3月7日周二 11:15写道：

> Wow, we have 300 FLIPs...
>
> Thanks Lincoln,
>
> Have you considered returning an Optional?
>
> Empty array looks a little weird to me.
>
> Best,
> Jingsong
>
> On Tue, Mar 7, 2023 at 10:32 AM Aitozi  wrote:
> >
> > Hi Lincoln,
> > Thank you for sharing this FLIP. Overall, it looks good to me. I have
> > one question: with the introduction of this interface,
> > will any existing Flink connectors need to be updated in order to take
> > advantage of its capabilities? For example, HBase.
> >
> > yuxia  于2023年3月7日周二 10:01写道：
> >
> > > Thanks. It makes sense to me.
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "Lincoln Lee" 
> > > 收件人: "dev" 
> > > 发送时间: 星期一, 2023年 3 月 06日 下午 10:26:26
> > > 主题: Re: [DISCUSS] FLIP-300: Add targetColumns to
> DynamicTableSink#Context
> > > to solve the null overwrite problem of partial-insert
> > >
> > > hi yuxia,
> > >
> > > Thanks for your feedback and tracking the issue of update statement!
> I've
> > > updated the FLIP[1] and also the poc[2].
> > > Since the bug and flip are orthogonal, we can focus on finalizing the
> api
> > > changes first, and then work on the flip implementation and bugfix
> > > separately, WDYT?
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> > > [2] https://github.com/apache/flink/pull/22041
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > yuxia  于2023年3月6日周一 21:21写道：
> > >
> > > > Hi, Lincoln.
> > > > Thanks for bringing this up. +1 for this FLIP, it's helpful for
> external
> > > > storage system to implement partial update.
> > > > The FLIP looks good to me. I only want to add one comment, update
> > > > statement also doesn't support updating nested column, I have created
> > > > FLINK-31344[1] to track it.
> > > > Maybe we also need to explain it in this FLIP.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-31344
> > > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > > - 原始邮件 -
> > > > 发件人: "Lincoln Lee" 
> > > > 收件人: "dev" 
> > > > 发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
> > > > 主题: [DISCUSS] FLIP-300: Add targetColumns to
> DynamicTableSink#Context to
> > > > solve the null overwrite problem of partial-insert
> > > >
> > > > Hi everyone,
> > > >
> > > > This FLIP[1] aims to support connectors in avoiding overwriting
> > > non-target
> > > > columns with null values when processing partial column updates, we
> > > propose
> > > > adding information on the target column list to
> DynamicTableSink#Context.
> > > >
> > > > FLINK-18726[2] supports inserting statements with specified column
> list,
> > > it
> > > > fills null values (or potentially declared default values in the
> future)
> > > > for columns not appearing in the column list of insert statement to
> the
> > > > target table.
> > > > But this behavior does not satisfy some partial column update
> > > requirements
> > > > of some storage systems which allow storing null values. The problem
> is
> > > > that connectors cannot distinguish whether the null value of a
> column is
> > > > really from the user's data or whether it is a null value populated
> > > because
> > > > of partial insert behavior.
> > > >
> > > > Looking forward to your comments or feedback.
> > > >
> > > > [1]
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> > > > [2] https://issues.apache.org/jira/browse/FLINK-18726
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > >
>

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[Jingsong Li]

Wow, we have 300 FLIPs...

Thanks Lincoln,

Have you considered returning an Optional?

Empty array looks a little weird to me.

Best,
Jingsong

On Tue, Mar 7, 2023 at 10:32 AM Aitozi  wrote:
>
> Hi Lincoln,
> Thank you for sharing this FLIP. Overall, it looks good to me. I have
> one question: with the introduction of this interface,
> will any existing Flink connectors need to be updated in order to take
> advantage of its capabilities? For example, HBase.
>
> yuxia  于2023年3月7日周二 10:01写道：
>
> > Thanks. It makes sense to me.
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Lincoln Lee" 
> > 收件人: "dev" 
> > 发送时间: 星期一, 2023年 3 月 06日 下午 10:26:26
> > 主题: Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context
> > to solve the null overwrite problem of partial-insert
> >
> > hi yuxia,
> >
> > Thanks for your feedback and tracking the issue of update statement! I've
> > updated the FLIP[1] and also the poc[2].
> > Since the bug and flip are orthogonal, we can focus on finalizing the api
> > changes first, and then work on the flip implementation and bugfix
> > separately, WDYT?
> >
> > [1]
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> > [2] https://github.com/apache/flink/pull/22041
> >
> > Best,
> > Lincoln Lee
> >
> >
> > yuxia  于2023年3月6日周一 21:21写道：
> >
> > > Hi, Lincoln.
> > > Thanks for bringing this up. +1 for this FLIP, it's helpful for external
> > > storage system to implement partial update.
> > > The FLIP looks good to me. I only want to add one comment, update
> > > statement also doesn't support updating nested column, I have created
> > > FLINK-31344[1] to track it.
> > > Maybe we also need to explain it in this FLIP.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-31344
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "Lincoln Lee" 
> > > 收件人: "dev" 
> > > 发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
> > > 主题: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to
> > > solve the null overwrite problem of partial-insert
> > >
> > > Hi everyone,
> > >
> > > This FLIP[1] aims to support connectors in avoiding overwriting
> > non-target
> > > columns with null values when processing partial column updates, we
> > propose
> > > adding information on the target column list to DynamicTableSink#Context.
> > >
> > > FLINK-18726[2] supports inserting statements with specified column list,
> > it
> > > fills null values (or potentially declared default values in the future)
> > > for columns not appearing in the column list of insert statement to the
> > > target table.
> > > But this behavior does not satisfy some partial column update
> > requirements
> > > of some storage systems which allow storing null values. The problem is
> > > that connectors cannot distinguish whether the null value of a column is
> > > really from the user's data or whether it is a null value populated
> > because
> > > of partial insert behavior.
> > >
> > > Looking forward to your comments or feedback.
> > >
> > > [1]
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> > > [2] https://issues.apache.org/jira/browse/FLINK-18726
> > >
> > > Best,
> > > Lincoln Lee
> > >
> >

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[Aitozi]

Hi Lincoln,
Thank you for sharing this FLIP. Overall, it looks good to me. I have
one question: with the introduction of this interface,
will any existing Flink connectors need to be updated in order to take
advantage of its capabilities? For example, HBase.

yuxia  于2023年3月7日周二 10:01写道：

> Thanks. It makes sense to me.
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Lincoln Lee" 
> 收件人: "dev" 
> 发送时间: 星期一, 2023年 3 月 06日 下午 10:26:26
> 主题: Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context
> to solve the null overwrite problem of partial-insert
>
> hi yuxia,
>
> Thanks for your feedback and tracking the issue of update statement! I've
> updated the FLIP[1] and also the poc[2].
> Since the bug and flip are orthogonal, we can focus on finalizing the api
> changes first, and then work on the flip implementation and bugfix
> separately, WDYT?
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> [2] https://github.com/apache/flink/pull/22041
>
> Best,
> Lincoln Lee
>
>
> yuxia  于2023年3月6日周一 21:21写道：
>
> > Hi, Lincoln.
> > Thanks for bringing this up. +1 for this FLIP, it's helpful for external
> > storage system to implement partial update.
> > The FLIP looks good to me. I only want to add one comment, update
> > statement also doesn't support updating nested column, I have created
> > FLINK-31344[1] to track it.
> > Maybe we also need to explain it in this FLIP.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-31344
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "Lincoln Lee" 
> > 收件人: "dev" 
> > 发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
> > 主题: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to
> > solve the null overwrite problem of partial-insert
> >
> > Hi everyone,
> >
> > This FLIP[1] aims to support connectors in avoiding overwriting
> non-target
> > columns with null values when processing partial column updates, we
> propose
> > adding information on the target column list to DynamicTableSink#Context.
> >
> > FLINK-18726[2] supports inserting statements with specified column list,
> it
> > fills null values (or potentially declared default values in the future)
> > for columns not appearing in the column list of insert statement to the
> > target table.
> > But this behavior does not satisfy some partial column update
> requirements
> > of some storage systems which allow storing null values. The problem is
> > that connectors cannot distinguish whether the null value of a column is
> > really from the user's data or whether it is a null value populated
> because
> > of partial insert behavior.
> >
> > Looking forward to your comments or feedback.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> > [2] https://issues.apache.org/jira/browse/FLINK-18726
> >
> > Best,
> > Lincoln Lee
> >
>

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[yuxia]

Thanks. It makes sense to me.

Best regards,
Yuxia

- 原始邮件 -
发件人: "Lincoln Lee" 
收件人: "dev" 
发送时间: 星期一, 2023年 3 月 06日 下午 10:26:26
主题: Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to 
solve the null overwrite problem of partial-insert

hi yuxia,

Thanks for your feedback and tracking the issue of update statement! I've
updated the FLIP[1] and also the poc[2].
Since the bug and flip are orthogonal, we can focus on finalizing the api
changes first, and then work on the flip implementation and bugfix
separately, WDYT?

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
[2] https://github.com/apache/flink/pull/22041

Best,
Lincoln Lee


yuxia  于2023年3月6日周一 21:21写道：

> Hi, Lincoln.
> Thanks for bringing this up. +1 for this FLIP, it's helpful for external
> storage system to implement partial update.
> The FLIP looks good to me. I only want to add one comment, update
> statement also doesn't support updating nested column, I have created
> FLINK-31344[1] to track it.
> Maybe we also need to explain it in this FLIP.
>
> [1] https://issues.apache.org/jira/browse/FLINK-31344
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Lincoln Lee" 
> 收件人: "dev" 
> 发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
> 主题: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to
> solve the null overwrite problem of partial-insert
>
> Hi everyone,
>
> This FLIP[1] aims to support connectors in avoiding overwriting non-target
> columns with null values when processing partial column updates, we propose
> adding information on the target column list to DynamicTableSink#Context.
>
> FLINK-18726[2] supports inserting statements with specified column list, it
> fills null values (or potentially declared default values in the future)
> for columns not appearing in the column list of insert statement to the
> target table.
> But this behavior does not satisfy some partial column update requirements
> of some storage systems which allow storing null values. The problem is
> that connectors cannot distinguish whether the null value of a column is
> really from the user's data or whether it is a null value populated because
> of partial insert behavior.
>
> Looking forward to your comments or feedback.
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> [2] https://issues.apache.org/jira/browse/FLINK-18726
>
> Best,
> Lincoln Lee
>

Re: help: [FLINK-31321] @flinkbot run azure does not work?

[yuxia]

Thanks Martijn for clarification.  After I reabsed master, it works.

Best regards,
Yuxia

- 原始邮件 -
发件人: "Martijn Visser" 
收件人: "dev" 
发送时间: 星期一, 2023年 3 月 06日 下午 5:58:38
主题: Re: help: [FLINK-31321] @flinkbot run azure does not work?

Hi,

Please make sure that you have rebased on the latest changes. This was
already tracked and fixed with
https://issues.apache.org/jira/browse/FLINK-30972

Best regards,

Martijn

On Mon, Mar 6, 2023 at 10:12 AM yuxia  wrote:

> Hi, I have created FLINK-31334[1] to track it.
> [1] https://issues.apache.org/jira/browse/FLINK-31334
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "felixzh" 
> 收件人: "dev" 
> 发送时间: 星期一, 2023年 3 月 06日 上午 10:18:12
> 主题: Re:Re: help: [FLINK-31321] @flinkbot run azure does not work?
>
>
> http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.11_amd64.deb
> The above url exists.
>
> http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb
> The above url doex not exists.
> ubuntu5.10 -> ubuntu5.11 ?
>
>
>
>
>
>
>
> 在 2023-03-06 09:45:10，"yuxia"  写道：
> >I also encouter the same problem. I have no idea why it happens, but hope
> it can ne fixed assp.
> >
> >Best regards,
> >Yuxia
> >
> >- 原始邮件 -
> >发件人: "felixzh" 
> >收件人: "dev" 
> >发送时间: 星期一, 2023年 3 月 06日 上午 8:46:49
> >主题: help: [FLINK-31321] @flinkbot run azure does not work?
> >
> >2023-03-05T05:38:27.1951220Z libapr1 is already the newest version
> (1.6.5-1ubuntu1).
> >2023-03-05T05:38:27.1951745Z libapr1 set to manually installed.
> >2023-03-05T05:38:27.1952264Z 0 upgraded, 0 newly installed, 0 to remove
> and 13 not upgraded.
> >2023-03-05T05:38:27.1984256Z --2023-03-05 05:38:27--
> http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb
> >2023-03-05T05:38:27.2104330Z Resolving security.ubuntu.com (
> security.ubuntu.com)... 91.189.91.39, 91.189.91.38, 185.125.190.39, ...
> >2023-03-05T05:38:27.2904245Z Connecting to security.ubuntu.com (
> security.ubuntu.com)|91.189.91.39|:80... connected.
> >2023-03-05T05:38:27.3707348Z HTTP request sent, awaiting response... 404
> Not Found
> >2023-03-05T05:38:27.3708310Z 2023-03-05 05:38:27 ERROR 404: Not Found.
> >2023-03-05T05:38:27.3708467Z
> >2023-03-05T05:38:27.4023505Z
> >2023-03-05T05:38:27.4024204Z WARNING: apt does not have a stable CLI
> interface. Use with caution in scripts.
> >2023-03-05T05:38:27.4024423Z
> >2023-03-05T05:38:27.4566409Z Reading package lists...
> >2023-03-05T05:38:27.4595509Z E: Unsupported file
> ./libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb given on commandline
> >2023-03-05T05:38:27.4659700Z ##[error]Bash exited with code '100'.
> >2023-03-05T05:38:27.4677676Z ##[section]Finishing: Prepare E2E run
>

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

[Jing Ge]

Hi Kui,

Thanks for pointing that out. I knew FLIP-217 which was done by an
engineer working in my team.  As far as I am concerned, your FLIP should
answer the following questions:

1. How to enable the watermark alignment of a source splits with Flink SQL?
e.g. which options should be used if only one source is used?

2. Default behaviour. i.e. Flink SQL users should be aware that watermark
alignment of source split will only work for sources that implement
FLIP-217 properly. Should users take care of
`pipeline.watermark-alignment.allow-unaligned-source-splits`
while using Flink SQL?

Best regards,
Jing


On Fri, Mar 3, 2023 at 8:46 AM Kui Yuan  wrote:

> Hi all,
>
> Thanks for all. There are more questions and I will answer one by one.
>
> @Jark Thanks for your tips. For the first question, I will add more details
> in the flip, and give a POC[1] so that pepole can know how I'm currently
> implementing these features.
>
> > IIRC, this is the first time we introduce the framework-level connector
> > options that the option is not recognized and handled by connectors.
> > The FLIP should cover how framework filters the watermark related options
> > to avoid discover connector factory failed, and what happens if the
> > connector already supported the conflict options
>
> For the second question, We know that the default strategy is 'on-periodic'
> in SQL layer, and the default interval is 200ms. The reason for emiting
> watermark periodically is that the time advancement of consecutive events
> may be very small, we don't need to calculate watermark for each event.
> Same for 'on-event' strategy, so my idea is that we can set a fixed gap for
> 'on-event' strategy.
>
> > I'm not sure about the usage scenarios of event gap emit strategy. Do
> > you have any specific use case of this strategy? I'm confused why no one
> > requested this strategy before no matter in DataStream or SQL, but maybe
> > I missed something. I'm not against to add this option, but just want to
> be
> > careful when adding new API because it's hard to remove in the future.
>
> As @Timo said, There is no default features like 'on-event-gap' in
> DataStream API, but the users can achieve the 'on-event-gap' feature by
> using `WatermarkGenerator` interface, just like the implemention in my
> POC[1]. However, If we don't provide it  in SQL layer, there is no way for
> users to use similar features.
>
> > Jark raised a very good point. I thought we only expose what is
> > contained in DataStream API already. If this strategy is not part of
> > DataStream API, would like to exclude it from the FLIP. We need to be
> > careful which strategies we offer by default.
>
> @Jark @Timo I'm sorry, perhaps I don't understand what are your concerns
> about CompiledPlan, maybe I missed something else, maybe you can look at my
> POC first to see if there is somewhere to worry about.
>
> > Sorry, I forgot to remind you that Timo's concern about the changes to
> the
> > CompiledPlan looks like is still not covered in the FLIP.
>
> @Jing We could have more discussion about naming, but I prefer that the
> naming should be consistent with the DataStream API.
> About aligning splits/partitions/shards, maybe you missed FLIP-217[2] which
> aims to support watermark alignment of source splits.
>
> > After reading the most up-to-date Flip, I didn't find any information if
> > this solution will support aligning splits/partitions/shards [1]. Did I
> > miss anything?
>
> Best
> Kui Yuan
>
> [1] the POC:
> https://github.com/yuchengxin/flink/tree/yuankui/watermark_params
> [2] FLIP-217:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits
>
>
> Jing Ge  于2023年3月3日周五 08:03写道：
>
> > Hi,
> >
> > Thanks Kui for driving this Flip and thanks all for the informative
> > discussion.
> >
> > @Timo
> >
> > Your suggestion about the naming convention is excellent. Thanks! I was
> > wondering why you, exceptionally, suggested 'scan.idle-timeout' instead
> of
> > 'scan.watermark.idle-timeout'. I must miss something here.
> >
> > There is one more NIT. I am just aware that "drift" is used for the
> > watermark alignment. It seems to be fine while using DataStream API,
> > because we will not really see it. But with the OPTIONS in SQL, a much
> > bigger group of users (including SRE, tech support, etc) will see the
> word
> > "drift". Given that "drift" wasn't used widely yet and with all training
> > materials, Flink doc [1][2][3] (search with "lag"), "lag" has been used
> to
> > describe timestamp difference between watermark and its
> > corresponding event. Do we really need to introduce another term for the
> > same thing? How about using 'scan.watermark.alignment.max-lag'='1min' and
> > change the parameter name from maxAllowedWatermarkDrift to
> > maxAllowedWatermarkLag [4] because of naming consistency? Just my two
> cents
> > worth.
> >
> > @Kui
> >
> > After reading the most up-to-date Flip, I didn't find any 

[jira] [Created] (FLINK-31351) HiveServer2EndpointITCase.testExecuteStatementInSyncModeWithRuntimeException2 times out on CI

[Jira]

David Morávek created FLINK-31351:
-

 Summary: 
HiveServer2EndpointITCase.testExecuteStatementInSyncModeWithRuntimeException2 
times out on CI
 Key: FLINK-31351
 URL: https://issues.apache.org/jira/browse/FLINK-31351
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Reporter: David Morávek


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46872=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=24908]

 
{code:java}
Mar 06 18:28:56 "ForkJoinPool-1-worker-25" #27 daemon prio=5 os_prio=0 
tid=0x7ff4b1832000 nid=0x21b2 waiting on condition [0x7ff3a8c3e000]
Mar 06 18:28:56java.lang.Thread.State: TIMED_WAITING (sleeping)
Mar 06 18:28:56 at java.lang.Thread.sleep(Native Method)
Mar 06 18:28:56 at 
org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase.waitUntilJobIsRunning(HiveServer2EndpointITCase.java:1004)
Mar 06 18:28:56 at 
org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase.lambda$testExecuteStatementInSyncModeWithRuntimeException2$37(HiveServer2EndpointITCase.java:711)
Mar 06 18:28:56 at 
org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase$$Lambda$2018/2127600974.accept(Unknown
 Source)
Mar 06 18:28:56 at 
org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase.runExecuteStatementInSyncModeWithRuntimeException(HiveServer2EndpointITCase.java:999)
Mar 06 18:28:56 at 
org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase.testExecuteStatementInSyncModeWithRuntimeException2(HiveServer2EndpointITCase.java:709)
Mar 06 18:28:56 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
Mar 06 18:28:56 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Mar 06 18:28:56 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 06 18:28:56 at java.lang.reflect.Method.invoke(Method.java:498)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31350) Support Calcite's 1.30+ UnknownType

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-31350:
---

 Summary: Support Calcite's  1.30+ UnknownType
 Key: FLINK-31350
 URL: https://issues.apache.org/jira/browse/FLINK-31350
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Affects Versions: 1.18.0
Reporter: Sergey Nuyanzin


In 1.30 in Calcite https://issues.apache.org/jira/browse/CALCITE-4872 there has 
been introduced a new type UNKNOWN which is similar to NULL however is not 
nullable by default and as a result breaks some tests.

 

I guess a similar type should be introduced in Flink to support it



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31349) Adapt to Calcite's 1.30+ new ROW null semantics

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-31349:
---

 Summary: Adapt to Calcite's 1.30+ new ROW null semantics
 Key: FLINK-31349
 URL: https://issues.apache.org/jira/browse/FLINK-31349
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.18.0
Reporter: Sergey Nuyanzin


The change has been introduced at 
https://issues.apache.org/jira/browse/CALCITE-3627
It leads to unsync behavior of sql and table api in context of nullability for 
rows.

As a workaround there is a {{SqlRowConstructor}} mimicking Calcite 1.29 
behavior.

After resolving this issue {{SqlRowConstructor}} should be removed



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31348) Documentation fails to build due to unclosed shortcodes

[Jira]

David Morávek created FLINK-31348:
-

 Summary: Documentation fails to build due to unclosed shortcodes
 Key: FLINK-31348
 URL: https://issues.apache.org/jira/browse/FLINK-31348
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: David Morávek
Assignee: David Morávek


After migration to HUGO, there are a bunch of unclosed shortcodes which prevent 
the documentation from being served locally.

 

Example:
{code:java}
docker run -v $(pwd):/src -p 1313:1313 jakejarvis/hugo-extended:latest server 
--buildDrafts --buildFuture --bind 0.0.0.0
 
...

Error: Error building site: 
"/src/content.zh/docs/connectors/datastream/formats/parquet.md:111:1": failed 
to extract shortcode: unclosed shortcode "tabs" {code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31347) AdaptiveSchedulerClusterITCase.testAutomaticScaleUp times out

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31347:
-

 Summary: AdaptiveSchedulerClusterITCase.testAutomaticScaleUp times 
out
 Key: FLINK-31347
 URL: https://issues.apache.org/jira/browse/FLINK-31347
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.17.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46850=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=10451

{code}
Mar 06 14:11:24 "main" #1 prio=5 os_prio=0 tid=0x7f482800b800 nid=0x6eee 
waiting on condition [0x7f48325cd000]
Mar 06 14:11:24java.lang.Thread.State: TIMED_WAITING (sleeping)
Mar 06 14:11:24 at java.lang.Thread.sleep(Native Method)
Mar 06 14:11:24 at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:151)
Mar 06 14:11:24 at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:144)
Mar 06 14:11:24 at 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerClusterITCase.waitUntilParallelismForVertexReached(AdaptiveSchedulerClusterITCase.java:265)
Mar 06 14:11:24 at 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerClusterITCase.testAutomaticScaleUp(AdaptiveSchedulerClusterITCase.java:153)
[...]
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Large schemas lead to long DataStream-to-table transformation names

[Xingcan Cui]

Oh, I just realized that FLIP-195 has already solved this. We'll upgrade
our Flink version to 1.15+. Thanks!

On Mon, Mar 6, 2023 at 10:08 AM Xingcan Cui  wrote:

> Hi Jark,
>
> Yes. I believe field names of the table would be enough to describe the
> conversion operator. I'll try to improve this.
>
> Best,
> Xingcan
>
> On Sun, Mar 5, 2023 at 9:18 PM Jark Wu  wrote:
>
>> Hi Xingcan,
>>
>> I think `physicalDataType.toString()` is indeed verbose in this case.
>> Normal table scan generates descriptions using field names instead of the
>> full schema.
>> Will that help in your case?
>>
>> Best,
>> Jark
>>
>> On Sat, 4 Mar 2023 at 06:57, Xingcan Cui  wrote:
>>
>> > Hi all,
>> >
>> > We are dealing with some streams with large (nested) schemas. When
>> using `t
>> > ableEnv.createTemporaryView()` to register a DataStream to a table, the
>> > transformation always gets a large name. It's not a big problem, but
>> quite
>> > annoying since the UI and logs are hard to read.
>> >
>> > Internally, `ExternalDynamicSource` (and `ExternalDynamicSink`) invokes
>> > `physicalDataType.toString()` to generate an operator name (which will
>> also
>> > be used as the transformation name). I'm thinking to introduce a new
>> table
>> > config to either truncate the name or use a limited level of
>> logicalType to
>> > generate the name (works for nested schemas).
>> >
>> > What do you think?
>> >
>> > Best,
>> > Xingcan
>> >
>>
>

Re: Large schemas lead to long DataStream-to-table transformation names

[Xingcan Cui]

Hi Jark,

Yes. I believe field names of the table would be enough to describe the
conversion operator. I'll try to improve this.

Best,
Xingcan

On Sun, Mar 5, 2023 at 9:18 PM Jark Wu  wrote:

> Hi Xingcan,
>
> I think `physicalDataType.toString()` is indeed verbose in this case.
> Normal table scan generates descriptions using field names instead of the
> full schema.
> Will that help in your case?
>
> Best,
> Jark
>
> On Sat, 4 Mar 2023 at 06:57, Xingcan Cui  wrote:
>
> > Hi all,
> >
> > We are dealing with some streams with large (nested) schemas. When using
> `t
> > ableEnv.createTemporaryView()` to register a DataStream to a table, the
> > transformation always gets a large name. It's not a big problem, but
> quite
> > annoying since the UI and logs are hard to read.
> >
> > Internally, `ExternalDynamicSource` (and `ExternalDynamicSink`) invokes
> > `physicalDataType.toString()` to generate an operator name (which will
> also
> > be used as the transformation name). I'm thinking to introduce a new
> table
> > config to either truncate the name or use a limited level of logicalType
> to
> > generate the name (works for nested schemas).
> >
> > What do you think?
> >
> > Best,
> > Xingcan
> >
>

[jira] [Created] (FLINK-31346) IO scheduler does not throw TimeoutException if numRequestedBuffers is greater than 0

[Weijie Guo (Jira)]

Weijie Guo created FLINK-31346:
--

 Summary: IO scheduler does not throw TimeoutException if 
numRequestedBuffers is greater than 0
 Key: FLINK-31346
 URL: https://issues.apache.org/jira/browse/FLINK-31346
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.16.1
Reporter: Weijie Guo
Assignee: Weijie Guo


We currently rely on throw exception to trigger downstream task failover to 
avoid read buffer request deadlock. But if {{numRequestedBuffers}} is greater 
than 0, IO scheduler does not throw {{TimeoutException}}. This will cause a 
deadlock.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[Lincoln Lee]

hi yuxia,

Thanks for your feedback and tracking the issue of update statement! I've
updated the FLIP[1] and also the poc[2].
Since the bug and flip are orthogonal, we can focus on finalizing the api
changes first, and then work on the flip implementation and bugfix
separately, WDYT?

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
[2] https://github.com/apache/flink/pull/22041

Best,
Lincoln Lee


yuxia  于2023年3月6日周一 21:21写道：

> Hi, Lincoln.
> Thanks for bringing this up. +1 for this FLIP, it's helpful for external
> storage system to implement partial update.
> The FLIP looks good to me. I only want to add one comment, update
> statement also doesn't support updating nested column, I have created
> FLINK-31344[1] to track it.
> Maybe we also need to explain it in this FLIP.
>
> [1] https://issues.apache.org/jira/browse/FLINK-31344
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "Lincoln Lee" 
> 收件人: "dev" 
> 发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
> 主题: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to
> solve the null overwrite problem of partial-insert
>
> Hi everyone,
>
> This FLIP[1] aims to support connectors in avoiding overwriting non-target
> columns with null values when processing partial column updates, we propose
> adding information on the target column list to DynamicTableSink#Context.
>
> FLINK-18726[2] supports inserting statements with specified column list, it
> fills null values (or potentially declared default values in the future)
> for columns not appearing in the column list of insert statement to the
> target table.
> But this behavior does not satisfy some partial column update requirements
> of some storage systems which allow storing null values. The problem is
> that connectors cannot distinguish whether the null value of a column is
> really from the user's data or whether it is a null value populated because
> of partial insert behavior.
>
> Looking forward to your comments or feedback.
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
> [2] https://issues.apache.org/jira/browse/FLINK-18726
>
> Best,
> Lincoln Lee
>

Re: [DISCUSS] String literal behavior in Flink

[Aitozi]

Hi, Jark

FYI, this feature has already been supported in Calcite 1.33.0 [1].
Therefore, I believe we can use it directly after upgrading the Calcite
version.

Best,
Aitozi.
[1]: https://issues.apache.org/jira/browse/CALCITE-5305

Aitozi  于2023年3月6日周一 16:48写道：

> Thanks, will give it a try
>
> Best,
> Aitozi.
>
> Jark Wu  于2023年3月6日周一 15:11写道：
>
>> Hi Aitozi,
>>
>> I would suggest trying to contribute it to the upstream project Calcite
>> first.
>>
>> Best,
>> Jark
>>
>> > 2023年3月6日 11:51，Aitozi  写道：
>> >
>> > Hi Jark,
>> >
>> > Thank you for your helpful suggestion. It appears that 'E'foo\n'' is a
>> more
>> > versatile and widely accepted option. To assess its feasibility, I have
>> > reviewed the relevant Unicode supports and concluded that it may
>> > necessitate modifications to the Parser.jj file to accommodate this new
>> > syntax.
>> >
>> >
>> > I am unsure whether we should initially incorporate this alteration in
>> > Calcite or if we can directly supersede the StringLiteral behavior
>> within
>> > the Flink project. Nevertheless, I believe supporting this change is
>> > achievable.
>> >
>> >
>> >
>> > Thanks,
>> > Aitozi.
>> >
>> > Jark Wu  于2023年3月6日周一 10:16写道：
>> >
>> >> Hi Aitozi,
>> >>
>> >> I think this is a good idea to improve the backslash escape strings.
>> >> However, I lean a bit more toward the Postgres approach[1],
>> >> which is more standard-compliant. PG allows backslash escape
>> >> string by writing the letter E (upper or lower case) just before the
>> >> opening single quote, e.g., E'foo\n'.
>> >>
>> >> Recognizing backslash escapes in both regular and escape string
>> constants
>> >> is not backward compatible in Flink, and is also deprecated in PG.
>> >>
>> >> In addition, Flink also supports Unicode escape string constants by
>> >> writing the U& before the quote[1] which works in the same way with
>> >> backslash escape string.
>> >>
>> >> Best,
>> >> Jark
>> >>
>> >> [1]:
>> >>
>> >>
>> https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-CONSTANTS
>> >> [2]:
>> >>
>> >>
>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/overview/
>> >>
>> >> On Sat, 4 Mar 2023 at 23:31, Aitozi  wrote:
>> >>
>> >>> Hi,
>> >>>  I encountered a problem when using string literal in Flink.
>> Currently,
>> >>> Flink will escape the string literal during codegen, so for the query
>> >>> below:
>> >>>
>> >>> SELECT 'a\nb'; it will print => a\nb
>> >>>
>> >>> then for the query
>> >>>
>> >>> SELECT SPLIT_INDEX(col, '\n', 0);
>> >>>
>> >>> The col can not split by the newline. If we want to split by the
>> newline,
>> >>> we should use
>> >>>
>> >>> SELECT SPLIT_INDEX(col, '
>> >>> ', 0)
>> >>>
>> >>> or
>> >>>
>> >>> SELECT SPLIT_INDEX(col, CHR(10), 0)
>> >>>
>> >>> The above way could be more intuitive. Some other databases support
>> these
>> >>> "Special Character Escape Sequences"[1].
>> >>>
>> >>> In this way, we can directly use
>> >>> SELECT SPLIT_INDEX(col, '\n', 0); for the query.
>> >>>
>> >>> I know this is not standard behavior in ANSI SQL. I'm opening this
>> thread
>> >>> for some opinions from the community guys.
>> >>>
>> >>> [1]:
>> >>>
>> >>>
>> >>
>> https://dev.mysql.com/doc/refman/8.0/en/string-literals.html#character-escape-sequences
>> >>>
>> >>> Thanks,
>> >>> Aitozi
>> >>>
>> >>
>>
>>

[jira] [Created] (FLINK-31345) Trim autoscaler configMap to not exceed 1mb size limit

[Maximilian Michels (Jira)]

Maximilian Michels created FLINK-31345:
--

 Summary: Trim autoscaler configMap to not exceed 1mb size limit
 Key: FLINK-31345
 URL: https://issues.apache.org/jira/browse/FLINK-31345
 Project: Flink
  Issue Type: Bug
  Components: Autoscaler, Kubernetes Operator
Affects Versions: kubernetes-operator-1.4.0
Reporter: Maximilian Michels
 Fix For: kubernetes-operator-1.5.0


When the {{autoscaler-}} ConfigMap which is used to persist 
scaling decisions and metrics becomes too large, the following error is thrown 
consistently:

{noformat}
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [replace]  
for kind: [ConfigMap]  with name: [deployment]  in namespace: [namespace]  
failed.
    at 
io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
    at 
io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$replace$0(HasMetadataOperation.java:169)
    at 
io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:172)
    at 
io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:113)
    at 
io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.replace(HasMetadataOperation.java:41)
    at 
io.fabric8.kubernetes.client.extension.ResourceAdapter.replace(ResourceAdapter.java:252)
    at 
org.apache.flink.kubernetes.operator.autoscaler.AutoScalerInfo.replaceInKubernetes(AutoScalerInfo.java:167)
    at 
org.apache.flink.kubernetes.operator.autoscaler.JobAutoScalerImpl.scale(JobAutoScalerImpl.java:113)
    at 
org.apache.flink.kubernetes.operator.reconciler.deployment.AbstractFlinkResourceReconciler.reconcile(AbstractFlinkResourceReconciler.java:178)
    at 
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:130)
    at 
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)
    at 
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:145)
    at 
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:103)
    at 
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
    at 
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:102)
    at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)
    at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)
    at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)
    at 
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)
    at 
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:406)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown 
Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown 
Source)
    at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: stream was reset: NO_ERROR
    at 
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:514)
    at 
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:551)
    at 
io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleUpdate(OperationSupport.java:347)
    at 
io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleUpdate(BaseOperation.java:680)
    at 
io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$replace$0(HasMetadataOperation.java:167)
    ... 21 more
Caused by: okhttp3.internal.http2.StreamResetException: stream was reset: 
NO_ERROR
    at 
okhttp3.internal.http2.Http2Stream.checkOutNotClosed$okhttp(Http2Stream.kt:646)
    at 
okhttp3.internal.http2.Http2Stream$FramingSink.emitFrame(Http2Stream.kt:557)
    at okhttp3.internal.http2.Http2Stream$FramingSink.write(Http2Stream.kt:532)
    at okio.ForwardingSink.write(ForwardingSink.kt:29)
    at 
okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.kt:218)
    at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255)
    at okio.RealBufferedSink.write(RealBufferedSink.kt:185)
    at okhttp3.RequestBody$Companion$toRequestBody$2.writeTo(RequestBody.kt:152)
    at 
okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:59)
    at 
okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
    at 
okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
    at 

[VOTE] FLIP-299 Pub/Sub Lite Connector

[Daniel Collins]

Hello all,

This is the vote thread for FLIP-299
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885055
to add a Pub/Sub Lite connector. I would like to add this to the same repo
and maven artifacts as the flink pubsub connector, per a suggestion on the
DISCUSS thread, but there were otherwise no suggested changes.

This thread will remain open at least 3 days or until there are at least
3 +1s from committers, whichever is later.

-Daniel

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[yuxia]

Hi, Lincoln.
Thanks for bringing this up. +1 for this FLIP, it's helpful for external 
storage system to implement partial update.
The FLIP looks good to me. I only want to add one comment, update statement 
also doesn't support updating nested column, I have created FLINK-31344[1] to 
track it.
Maybe we also need to explain it in this FLIP.

[1] https://issues.apache.org/jira/browse/FLINK-31344

Best regards,
Yuxia

- 原始邮件 -
发件人: "Lincoln Lee" 
收件人: "dev" 
发送时间: 星期五, 2023年 3 月 03日 下午 12:22:19
主题: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve 
the null overwrite problem of partial-insert

Hi everyone,

This FLIP[1] aims to support connectors in avoiding overwriting non-target
columns with null values when processing partial column updates, we propose
adding information on the target column list to DynamicTableSink#Context.

FLINK-18726[2] supports inserting statements with specified column list, it
fills null values (or potentially declared default values in the future)
for columns not appearing in the column list of insert statement to the
target table.
But this behavior does not satisfy some partial column update requirements
of some storage systems which allow storing null values. The problem is
that connectors cannot distinguish whether the null value of a column is
really from the user's data or whether it is a null value populated because
of partial insert behavior.

Looking forward to your comments or feedback.

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
[2] https://issues.apache.org/jira/browse/FLINK-18726

Best,
Lincoln Lee

[jira] [Created] (FLINK-31344) Support to update nested columns in update statement

[luoyuxia (Jira)]

luoyuxia created FLINK-31344:


 Summary: Support to update nested columns in update statement
 Key: FLINK-31344
 URL: https://issues.apache.org/jira/browse/FLINK-31344
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: luoyuxia


Currently, it'll throw exception while using update statement to update nested 
column;

For the following sql:

 
{code:java}
create table (t ROW<`a` INT>) with (xxx);
update t set s.a = 1;{code}
It'll throw the exception:
{code:java}
Caused by: org.apache.flink.sql.parser.impl.ParseException: Encountered "." at 
line 1, column 15.
Was expecting:
    "=" ...
    
    at 
org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.generateParseException(FlinkSqlParserImpl.java:46382)
    at 
org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.jj_consume_token(FlinkSqlParserImpl.java:46190)
    at 
org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlUpdate(FlinkSqlParserImpl.java:14389)
    at 
org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmt(FlinkSqlParserImpl.java:4121)
    at 
org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.SqlStmtList(FlinkSqlParserImpl.java:2998)
    at 
org.apache.flink.sql.parser.impl.FlinkSqlParserImpl.parseSqlStmtList(FlinkSqlParserImpl.java:306)
    at org.apache.calcite.sql.parser.SqlParser.parseStmtList(SqlParser.java:198)
    ... 33 more {code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31343) Remove JMH dependency in flink-table-store-micro-benchmark

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-31343:


 Summary: Remove JMH dependency in flink-table-store-micro-benchmark
 Key: FLINK-31343
 URL: https://issues.apache.org/jira/browse/FLINK-31343
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: table-store-0.4.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

[Weihua Hu]

Thanks Shammon,

I've updated FLIP to add this redundant Task Manager limitation.


Best,
Weihua


On Mon, Mar 6, 2023 at 5:07 PM Shammon FY  wrote:

> Hi weihua
>
> Can you add content related to `heterogeneous resources` to this FLIP? We
> can record it and consider it in the future. It may be useful for some
> scenarios, such as the combination of streaming and ML.
>
> Best,
> Shammon
>
>
> On Mon, Mar 6, 2023 at 1:39 PM weijie guo 
> wrote:
>
> > Hi Weihua,
> >
> > Thanks for your clarification, SGTM.
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Weihua Hu  于2023年3月6日周一 11:43写道：
> >
> > > Thanks Weijie.
> > >
> > > Heterogeneous task managers will not be considered in this FLIP since
> > > it does not request heterogeneous resources as you said.
> > >
> > > My first thought is we can adjust the meaning of redundant
> configuration
> > > to redundant number of per resource type. These can be considered in
> > > detail when we decide to support heterogeneous task managers.
> > >
> > > Best,
> > > Weihua
> > >
> > >
> > > On Sat, Mar 4, 2023 at 1:13 AM weijie guo 
> > > wrote:
> > >
> > > > Thanks Weihua for preparing this FLIP.
> > > >
> > > > This FLIP overall looks reasonable to me after updating as suggested
> by
> > > > Matthias.
> > > >
> > > > I only have one small question about keeping some redundant task
> > > managers:
> > > > In the fine-grained resource management, theoretically, it can
> support
> > > > heterogeneous taskmanagers. When we complete the missing features for
> > > FGSM,
> > > > do we plan to take this into account?
> > > > Of course, if I remember correctly, FGSM will not request
> heterogeneous
> > > > resources at present, so it is also acceptable to me if there is no
> > > special
> > > > treatment now.
> > > >
> > > > +1 for this changes if we can ensure the test coverage.
> > > >
> > > > Best regards,
> > > >
> > > > Weijie
> > > >
> > > >
> > > > John Roesler  于2023年3月2日周四 12:53写道：
> > > >
> > > > > Thanks for the test plan, Weihua!
> > > > >
> > > > > Yes, it addresses my concerns.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Wed, Mar 1, 2023, at 22:38, Weihua Hu wrote:
> > > > > > Hi, everyone,
> > > > > > Thanks for your suggestions and ideas.
> > > > > > Thanks Xintong for sharing the detailed backgrounds of
> SlotManager.
> > > > > >
> > > > > > *@Matthias
> > > > > >
> > > > > > 1. Did you do a proper test coverage analysis?
> > > > > >
> > > > > >
> > > > > > Just as Xintong said, we already have a CI stage for fine grained
> > > > > resource
> > > > > > managers.
> > > > > > And I will make sure FineGrainedSlotManager as the default
> > > SlotManager
> > > > > can
> > > > > > pass all the tests of CI.
> > > > > > In addition, I will review all unit tests of
> > > > DeclarativeSlotManager(DSM)
> > > > > to
> > > > > > ensure that there are no gaps in the
> > > > > > coverage provided by the FineGrainedSlotManager.
> > > > > > I also added the 'Test Plan' part to the FLIP.
> > > > > > @Matthias @John @Shammon Does this test plan address your
> concerns?
> > > > > >
> > > > > > 2.  DeclarativeSlotManager and FineGrainedSlotManager feel quite
> > big
> > > in
> > > > > >
> > > > > > terms of lines of code
> > > > > >
> > > > > >
> > > > > > IMO, the refactoring of SlotManager does not belong to this FLIP
> > > since
> > > > it
> > > > > > may lead to some unstable risks. For
> > > > > > FineGrainedSlotManager(FGSM), we already split some reasonable
> > > > > components.
> > > > > > They are:
> > > > > > * TaskManagerTracker: Track task managers and their resources.
> > > > > > * ResourceTracker: track requirements of jobs
> > > > > > * ResourceAllocationStrategy: Try to fulfill the resource
> > > requirements
> > > > > with
> > > > > > available/pending resources.
> > > > > > * SlotStatusSyncer: communicate with TaskManager, for
> > > > allocating/freeing
> > > > > > slot and reconciling the slot status
> > > > > > Maybe we can start a discussion about refactoring SlotManager in
> > > > another
> > > > > > FLIP if there are some good suggestions.
> > > > > > WDYT
> > > > > >
> > > > > > 3. For me personally, having a more detailed summary comparing
> the
> > > > > >> subcomponents of both SlotManager implementations with where
> > > > > >> their functionality matches and where they differ might help
> > > > understand
> > > > > the
> > > > > >> consequences of the changes proposed in FLIP-298
> > > > > >
> > > > > > Good suggestion, I have updated the comparison in this FLIP.
> > Looking
> > > > > > forward to any suggestions/thoughts
> > > > > > if they are not described clearly.
> > > > > >
> > > > > > *@John
> > > > > >
> > > > > > 4. In addition to changing the default, would it make sense to
> log
> > a
> > > > > >> deprecation warning on initialization
> > > > > >
> > > > > > if the DeclarativeSlotManager is used?
> > > > > >>
> > > > > > SGTM, We should add Deprecated annotations to DSM for devs. And
> > log a
> > > > > > 

[VOTE] FLIP-297: Improve Auxiliary Sql Statements

[Ran Tao]

Hi Everyone,


I want to start the vote on FLIP-297: Improve Auxiliary Sql Statements [1].
The FLIP was discussed in this thread [2].


The goal of the FLIP is to improve flink auxiliary sql statements(compared
with sql standard or other mature engines).

The vote will last for at least 72 hours (03/09, 19:30 UTC+8)
unless there is an objection or insufficient votes. Thank you all.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-297%3A+Improve+Auxiliary+Sql+Statements
[2] https://lists.apache.org/thread/54fyd27m8on1cf3hn6dz564zqmkobjyd

Best Regards,
Ran Tao
https://github.com/chucheng92

Re: Re: obtain the broadcast stream information in sink

[Weihua Hu]

AFAIK, we can not get the broadcast state in sink.

Maybe you can enrich records with broadcast information, and then get the
information from each record in the Sink function.

Best,
Weihua


On Mon, Mar 6, 2023 at 6:20 PM zhan...@eastcom-sw.com <
zhan...@eastcom-sw.com> wrote:

>
> The sinks needs to get some configuration information while writing. I
> want to get it from the broadcast stream ~
>
>
> From: Weihua Hu
> Date: 2023-03-06 18:06
> To: dev; user
> Subject: Re: obtain the broadcast stream information in sink
> Hi,
>
> Could you describe your usage scenario in detail?
> Why do you need to get the broadcast stream in sink?
> And could you split an operator from the sink to deal with broadcast
> stream?
>
> Best,
> Weihua
>
>
> On Mon, Mar 6, 2023 at 10:57 AM zhan...@eastcom-sw.com <
> zhan...@eastcom-sw.com> wrote:
>
> >
> > hi, all ~
> >
> > how can i obtain the broadcast stream information in sink ?
> >
>

[jira] [Created] (FLINK-31342) SQLClientSchemaRegistryITCase timed out when starting the test container

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31342:
-

 Summary: SQLClientSchemaRegistryITCase timed out when starting the 
test container
 Key: FLINK-31342
 URL: https://issues.apache.org/jira/browse/FLINK-31342
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.17.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46820=logs=fb37c667-81b7-5c22-dd91-846535e99a97=39a035c3-c65e-573c-fb66-104c66c28912=11767

{code}
Mar 06 06:53:47 [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 1, Time 
elapsed: 1,037.927 s <<< FAILURE! - in 
org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase
Mar 06 06:53:47 [ERROR] 
org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase  Time elapsed: 
1,037.927 s  <<< ERROR!
Mar 06 06:53:47 org.junit.runners.model.TestTimedOutException: test timed out 
after 10 minutes
Mar 06 06:53:47 at sun.misc.Unsafe.park(Native Method)
Mar 06 06:53:47 at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
Mar 06 06:53:47 at 
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
Mar 06 06:53:47 at 
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
Mar 06 06:53:47 at 
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
Mar 06 06:53:47 at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
Mar 06 06:53:47 at 
org.testcontainers.containers.GenericContainer.start(GenericContainer.java:323)
Mar 06 06:53:47 at 
org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1063)
Mar 06 06:53:47 at 
org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:29)
Mar 06 06:53:47 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
Mar 06 06:53:47 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
Mar 06 06:53:47 at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
Mar 06 06:53:47 at java.lang.Thread.run(Thread.java:750)
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31341) OutOfMemoryError in Kafka e2e tests

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31341:
-

 Summary: OutOfMemoryError in Kafka e2e tests
 Key: FLINK-31341
 URL: https://issues.apache.org/jira/browse/FLINK-31341
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.17.0
Reporter: Matthias Pohl


We experience a OOM in Kafka e2e tests:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46820=logs=fb37c667-81b7-5c22-dd91-846535e99a97=39a035c3-c65e-573c-fb66-104c66c28912=11726

{code}
ar 06 06:22:30 [ERROR] Exception: java.lang.OutOfMemoryError thrown from the 
UncaughtExceptionHandler in thread "ForkJoinPool-1-worker-0"
Exception in thread "ForkJoinPool-1-worker-0" Mar 06 06:27:30 [ERROR] Tests 
run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1,094.139 s <<< 
FAILURE! - in JUnit Jupiter
Mar 06 06:27:30 [ERROR] JUnit Jupiter.JUnit Jupiter  Time elapsed: 947.463 s  
<<< ERROR!
Mar 06 06:27:30 org.junit.platform.commons.JUnitException: TestEngine with ID 
'junit-jupiter' failed to execute tests
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:153)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
Mar 06 06:27:30 at 
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:188)
Mar 06 06:27:30 at 
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:154)
Mar 06 06:27:30 at 
org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:128)
Mar 06 06:27:30 at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
Mar 06 06:27:30 at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
Mar 06 06:27:30 at 
org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
Mar 06 06:27:30 at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
Mar 06 06:27:30 Caused by: org.junit.platform.commons.JUnitException: Error 
executing tests for engine junit-jupiter
Mar 06 06:27:30 at 
org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:57)
Mar 06 06:27:30 at 
org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
Mar 06 06:27:30 ... 16 more
Mar 06 06:27:30 Caused by: java.util.concurrent.ExecutionException: 
java.lang.OutOfMemoryError
Mar 06 06:27:30 at 
java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
Mar 06 06:27:30 at 
org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
Mar 06 06:27:30 ... 17 more
Mar 06 06:27:30 Caused by: java.lang.OutOfMemoryError
Mar 06 06:27:30 at 
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Mar 06 06:27:30 at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Mar 06 06:27:30 at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Mar 06 06:27:30 at 
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
Mar 06 06:27:30 at 
java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
Mar 06 06:27:30 at 
java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
Mar 06 06:27:30 ... 18 more
Mar 06 06:27:30 Caused by: java.lang.OutOfMemoryError: Java heap space
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31340) HiveTableSourceStatisticsReportTest failed with multiple test failures

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31340:
-

 Summary: HiveTableSourceStatisticsReportTest failed with multiple 
test failures
 Key: FLINK-31340
 URL: https://issues.apache.org/jira/browse/FLINK-31340
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Affects Versions: 1.17.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42680=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=d04c9862-880c-52f5-574b-a7a79fef8e0f=23985



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: obtain the broadcast stream information in sink

[Weihua Hu]

Hi,

Could you describe your usage scenario in detail?
Why do you need to get the broadcast stream in sink?
And could you split an operator from the sink to deal with broadcast stream?

Best,
Weihua


On Mon, Mar 6, 2023 at 10:57 AM zhan...@eastcom-sw.com <
zhan...@eastcom-sw.com> wrote:

>
> hi, all ~
>
> how can i obtain the broadcast stream information in sink ?
>

[jira] [Created] (FLINK-31339) PlannerScalaFreeITCase.testImperativeUdaf

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31339:
-

 Summary: PlannerScalaFreeITCase.testImperativeUdaf
 Key: FLINK-31339
 URL: https://issues.apache.org/jira/browse/FLINK-31339
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Matthias Pohl


{{PlannerScalaFreeITCase.testImperativeUdaf}} failed:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46812=logs=87489130-75dc-54e4-1f45-80c30aa367a3=73da6d75-f30d-5d5a-acbe-487a9dcff678=15012

{code}
Mar 05 05:55:50 [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time 
elapsed: 62.028 s <<< FAILURE! - in 
org.apache.flink.table.sql.codegen.PlannerScalaFreeITCase
Mar 05 05:55:50 [ERROR] PlannerScalaFreeITCase.testImperativeUdaf  Time 
elapsed: 40.924 s  <<< FAILURE!
Mar 05 05:55:50 org.opentest4j.AssertionFailedError: Did not get expected 
results before timeout, actual result: 
[{"before":null,"after":{"user_name":"Bob","order_cnt":1},"op":"c"}, 
{"before":null,"after":{"user_name":"Alice","order_cnt":1},"op":"c"}, 
{"before":{"user_name":"Bob","order_cnt":1},"after":null,"op":"d"}, 
{"before":null,"after":{"user_name":"Bob","order_cnt":2},"op":"c"}]. ==> 
expected:  but was: 
Mar 05 05:55:50 at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
Mar 05 05:55:50 at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
Mar 05 05:55:50 at 
org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
Mar 05 05:55:50 at 
org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
Mar 05 05:55:50 at 
org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:211)
Mar 05 05:55:50 at 
org.apache.flink.table.sql.codegen.SqlITCaseBase.checkJsonResultFile(SqlITCaseBase.java:168)
Mar 05 05:55:50 at 
org.apache.flink.table.sql.codegen.SqlITCaseBase.runAndCheckSQL(SqlITCaseBase.java:111)
Mar 05 05:55:50 at 
org.apache.flink.table.sql.codegen.PlannerScalaFreeITCase.testImperativeUdaf(PlannerScalaFreeITCase.java:43)
[...]
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31338) support infer parallelism for flink table store

[Jun Zhang (Jira)]

Jun Zhang created FLINK-31338:
-

 Summary: support  infer parallelism for flink table store
 Key: FLINK-31338
 URL: https://issues.apache.org/jira/browse/FLINK-31338
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Affects Versions: table-store-0.3.0
Reporter: Jun Zhang
 Fix For: table-store-0.4.0


When using flink  to query the fts table, we can config the scan parallelism by 
set the scan.parallelism, but the user may do not know how much parallelism 
should be used,  setting a too large parallelism will cause resource waste, 
setting the parallelism too small will cause the query to be slow, so we can 
add parallelism infer.

The function is enabled by default. the parallelism is equal to the number of 
read splits. Of course, the user can manually turn off the infer function. In 
order to prevent too many datafiles from causing excessive parallelism, we also 
set a max infer parallelism. When the infer parallelism exceeds the setting, 
use the max parallelism.

In addition, we also need to compare with the limit in the select query 
statement to get a more appropriate parallelism in the case of limit pushdown, 
for example we have a sql select * from table limit 1, and finally we infer the 
parallelism is 10, but we only one parallel is needed , besause we only need 
one data .



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: help: [FLINK-31321] @flinkbot run azure does not work?

[Martijn Visser]

Hi,

Please make sure that you have rebased on the latest changes. This was
already tracked and fixed with
https://issues.apache.org/jira/browse/FLINK-30972

Best regards,

Martijn

On Mon, Mar 6, 2023 at 10:12 AM yuxia  wrote:

> Hi, I have created FLINK-31334[1] to track it.
> [1] https://issues.apache.org/jira/browse/FLINK-31334
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "felixzh" 
> 收件人: "dev" 
> 发送时间: 星期一, 2023年 3 月 06日 上午 10:18:12
> 主题: Re:Re: help: [FLINK-31321] @flinkbot run azure does not work?
>
>
> http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.11_amd64.deb
> The above url exists.
>
> http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb
> The above url doex not exists.
> ubuntu5.10 -> ubuntu5.11 ?
>
>
>
>
>
>
>
> 在 2023-03-06 09:45:10，"yuxia"  写道：
> >I also encouter the same problem. I have no idea why it happens, but hope
> it can ne fixed assp.
> >
> >Best regards,
> >Yuxia
> >
> >- 原始邮件 -
> >发件人: "felixzh" 
> >收件人: "dev" 
> >发送时间: 星期一, 2023年 3 月 06日 上午 8:46:49
> >主题: help: [FLINK-31321] @flinkbot run azure does not work?
> >
> >2023-03-05T05:38:27.1951220Z libapr1 is already the newest version
> (1.6.5-1ubuntu1).
> >2023-03-05T05:38:27.1951745Z libapr1 set to manually installed.
> >2023-03-05T05:38:27.1952264Z 0 upgraded, 0 newly installed, 0 to remove
> and 13 not upgraded.
> >2023-03-05T05:38:27.1984256Z --2023-03-05 05:38:27--
> http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb
> >2023-03-05T05:38:27.2104330Z Resolving security.ubuntu.com (
> security.ubuntu.com)... 91.189.91.39, 91.189.91.38, 185.125.190.39, ...
> >2023-03-05T05:38:27.2904245Z Connecting to security.ubuntu.com (
> security.ubuntu.com)|91.189.91.39|:80... connected.
> >2023-03-05T05:38:27.3707348Z HTTP request sent, awaiting response... 404
> Not Found
> >2023-03-05T05:38:27.3708310Z 2023-03-05 05:38:27 ERROR 404: Not Found.
> >2023-03-05T05:38:27.3708467Z
> >2023-03-05T05:38:27.4023505Z
> >2023-03-05T05:38:27.4024204Z WARNING: apt does not have a stable CLI
> interface. Use with caution in scripts.
> >2023-03-05T05:38:27.4024423Z
> >2023-03-05T05:38:27.4566409Z Reading package lists...
> >2023-03-05T05:38:27.4595509Z E: Unsupported file
> ./libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb given on commandline
> >2023-03-05T05:38:27.4659700Z ##[error]Bash exited with code '100'.
> >2023-03-05T05:38:27.4677676Z ##[section]Finishing: Prepare E2E run
>

[jira] [Created] (FLINK-31337) EmbeddedDataStreamBatchTests.test_keyed_co_broadcast_side_output

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31337:
-

 Summary: 
EmbeddedDataStreamBatchTests.test_keyed_co_broadcast_side_output
 Key: FLINK-31337
 URL: https://issues.apache.org/jira/browse/FLINK-31337
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.17.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46799=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=24566

{code}
Mar 04 01:21:35 pyflink/datastream/tests/test_data_stream.py:743: 
Mar 04 01:21:35 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
_ _ _ _ _ _ _ _ 
Mar 04 01:21:35 pyflink/datastream/tests/test_data_stream.py:63: in 
assert_equals_sorted
Mar 04 01:21:35 self.assertEqual(expected, actual)
Mar 04 01:21:35 E   AssertionError: Lists differ: ['0', '1', '2', '4', '5', 
'5', '6', '6'] != ['0', '1', '2', '3', '5', '5', '6', '6']
Mar 04 01:21:35 E   
Mar 04 01:21:35 E   First differing element 3:
Mar 04 01:21:35 E   '4'
Mar 04 01:21:35 E   '3'
Mar 04 01:21:35 E   
Mar 04 01:21:35 E   - ['0', '1', '2', '4', '5', '5', '6', '6']
Mar 04 01:21:35 E   ?  ^
Mar 04 01:21:35 E   
Mar 04 01:21:35 E   + ['0', '1', '2', '3', '5', '5', '6', '6']
Mar 04 01:21:35 E   ?   
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31336) interval type process has problem in table api and sql

[jackylau (Jira)]

jackylau created FLINK-31336:


 Summary: interval type process has problem in table api and sql
 Key: FLINK-31336
 URL: https://issues.apache.org/jira/browse/FLINK-31336
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.18.0
Reporter: jackylau
 Fix For: 1.18.0


{code:java}
// code placeholder
select typeof(interval '1' day);
 - INTERVAL SECOND(3) NOT NULL {code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

[Kui Yuan]

Hi Jark,

Thanks for your kind reply. I think your questioning is right, and my
initial meaning is that we don't need to emit every watermark on each
event. Sorry for my previous not-rigorous descrptions.

The reason we add the 'on-event.gap' option is that we need to verify the
watermark alignment performance under different conditions. Compared to
'on-periodic' watermark strategy, we need to configure the 'on-event'
strategy and the gap to emit the watermark is a good choice. The
experimental results show that configure the gap as 1 (which is also the
actual default value) could be better In some watermark-aligned cases (The
large gap will cause the misaligned watermark, just like the large periodic
time). Thus, considering that current Datastream API does not support such
capability, and regarding to people's review comments, I agree we don't
need to introduce such SQL option in this FLIP.

Last but not least, with concerns about the changes to complied plan from
@Timo and @Jark, I've also added a description of the public interface
changes in the FLIP.

Best,
Kui Yuan

Jark Wu  于2023年3月3日周五 16:39写道：

> Hi Kui,
>
> I left my comments below.
>
> > we don't need to calculate watermark for each event.
>
> This is not true. We have to calculate the watermark for every event.
> Otherwise, we may miss advancing the watermark.
> We just don't need to emit every watermark.
>
> > There is no default features like 'on-event-gap' in
> > DataStream API, but the users can achieve the 'on-event-gap' feature by
> > using `WatermarkGenerator` interface, just like the implemention in my
> > POC[1]. However, If we don't provide it  in SQL layer, there is no way
> for
> > users to use similar features.
>
> My core point is that if DataStream API doesn't provide an 'on-event-gap'
> built-in strategy, which may mean it is not a true demand. We should be
> careful when adding this to SQL. However, I'm not against supporting
> user-defined WatermarkGenerator in SQL.
>
> The reason why I have concerns about adding 'on-event-gap' strategy is
> that Punctuated WatermarkGenerator is not designed for emitting watermark
> on event gap. It is designed to emit watermarks when it observes special
> events in the stream that carries watermark information. That's why the
> `WatermarkGenerator#onEvent(event, eventTimestamp, output)` carries an
> input event. See the official example of PunctuatedAssigner [1].
>
> That means, even if we introduce the 'on-event-gap' strategy in SQL, users
> still
> can't achieve the true ability of WatermarkGenerator.
>
> > I'm sorry, perhaps I don't understand what are your concerns
> > about CompiledPlan.
>
> You can consider CompiledPlan and the JSON representation is also a public
> API
> that should be backward compatible, e.g., COMPILE PLAN + EXECUTE PLAN
> should work, and execute a previous plan should also be compatible.
> Therefore,
>  the changes to them may need to be included in the FLIP.
>
> Best,
> Jark
>
> [1]:
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#writing-a-punctuated-watermarkgenerator
>
>
> On Fri, 3 Mar 2023 at 15:46, Kui Yuan  wrote:
>
> > Hi all,
> >
> > Thanks for all. There are more questions and I will answer one by one.
> >
> > @Jark Thanks for your tips. For the first question, I will add more
> details
> > in the flip, and give a POC[1] so that pepole can know how I'm currently
> > implementing these features.
> >
> > > IIRC, this is the first time we introduce the framework-level connector
> > > options that the option is not recognized and handled by connectors.
> > > The FLIP should cover how framework filters the watermark related
> options
> > > to avoid discover connector factory failed, and what happens if the
> > > connector already supported the conflict options
> >
> > For the second question, We know that the default strategy is
> 'on-periodic'
> > in SQL layer, and the default interval is 200ms. The reason for emiting
> > watermark periodically is that the time advancement of consecutive events
> > may be very small, we don't need to calculate watermark for each event.
> > Same for 'on-event' strategy, so my idea is that we can set a fixed gap
> for
> > 'on-event' strategy.
> >
> > > I'm not sure about the usage scenarios of event gap emit strategy. Do
> > > you have any specific use case of this strategy? I'm confused why no
> one
> > > requested this strategy before no matter in DataStream or SQL, but
> maybe
> > > I missed something. I'm not against to add this option, but just want
> to
> > be
> > > careful when adding new API because it's hard to remove in the future.
> >
> > As @Timo said, There is no default features like 'on-event-gap' in
> > DataStream API, but the users can achieve the 'on-event-gap' feature by
> > using `WatermarkGenerator` interface, just like the implemention in my
> > POC[1]. However, If we don't provide it  in SQL layer, there is no way
> for
> > users to use 

[jira] [Created] (FLINK-31335) using sql-gateway to submit job to yarn cluster, sql-gateway don't support kerberos

[felixzh (Jira)]

felixzh created FLINK-31335:
---

 Summary: using sql-gateway to submit job to yarn cluster, 
sql-gateway don't support kerberos
 Key: FLINK-31335
 URL: https://issues.apache.org/jira/browse/FLINK-31335
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Gateway
Affects Versions: 1.16.1
Reporter: felixzh


when submit job to yarn cluster, sql-gateway don't support kerberos.

1. yarn-per-job mode

-Dexecution.target=yarn-per-job

2. yarn-session mode

-Dexecution.target=yarn-session -Dyarn.application.id=yarnSessionAppID(eg: 
application_1677479737242_0052)

sql-gateway need to use SecurityUtils Modules



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: help: [FLINK-31321] @flinkbot run azure does not work?

[yuxia]

Hi, I have created FLINK-31334[1] to track it.
[1] https://issues.apache.org/jira/browse/FLINK-31334

Best regards,
Yuxia

- 原始邮件 -
发件人: "felixzh" 
收件人: "dev" 
发送时间: 星期一, 2023年 3 月 06日 上午 10:18:12
主题: Re:Re: help: [FLINK-31321] @flinkbot run azure does not work?

http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.11_amd64.deb
The above url exists. 
http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb
The above url doex not exists. 
ubuntu5.10 -> ubuntu5.11 ?







在 2023-03-06 09:45:10，"yuxia"  写道：
>I also encouter the same problem. I have no idea why it happens, but hope it 
>can ne fixed assp.
>
>Best regards,
>Yuxia
>
>- 原始邮件 -
>发件人: "felixzh" 
>收件人: "dev" 
>发送时间: 星期一, 2023年 3 月 06日 上午 8:46:49
>主题: help: [FLINK-31321] @flinkbot run azure does not work?
>
>2023-03-05T05:38:27.1951220Z libapr1 is already the newest version 
>(1.6.5-1ubuntu1).
>2023-03-05T05:38:27.1951745Z libapr1 set to manually installed.
>2023-03-05T05:38:27.1952264Z 0 upgraded, 0 newly installed, 0 to remove and 13 
>not upgraded.
>2023-03-05T05:38:27.1984256Z --2023-03-05 05:38:27--  
>http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb
>2023-03-05T05:38:27.2104330Z Resolving security.ubuntu.com 
>(security.ubuntu.com)... 91.189.91.39, 91.189.91.38, 185.125.190.39, ...
>2023-03-05T05:38:27.2904245Z Connecting to security.ubuntu.com 
>(security.ubuntu.com)|91.189.91.39|:80... connected.
>2023-03-05T05:38:27.3707348Z HTTP request sent, awaiting response... 404 Not 
>Found
>2023-03-05T05:38:27.3708310Z 2023-03-05 05:38:27 ERROR 404: Not Found.
>2023-03-05T05:38:27.3708467Z 
>2023-03-05T05:38:27.4023505Z 
>2023-03-05T05:38:27.4024204Z WARNING: apt does not have a stable CLI 
>interface. Use with caution in scripts.
>2023-03-05T05:38:27.4024423Z 
>2023-03-05T05:38:27.4566409Z Reading package lists...
>2023-03-05T05:38:27.4595509Z E: Unsupported file 
>./libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb given on commandline
>2023-03-05T05:38:27.4659700Z ##[error]Bash exited with code '100'.
>2023-03-05T05:38:27.4677676Z ##[section]Finishing: Prepare E2E run

[jira] [Created] (FLINK-31334) E2e ci fail with unsupported file exception

[luoyuxia (Jira)]

luoyuxia created FLINK-31334:


 Summary: E2e ci fail with unsupported file exception 
 Key: FLINK-31334
 URL: https://issues.apache.org/jira/browse/FLINK-31334
 Project: Flink
  Issue Type: Bug
  Components: Build System / CI
Reporter: luoyuxia


The e2e ci throw

"E: Unsupported file ./libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb given on 
commandline".

The full exception message is 
{code:java}
Installing required software
Reading package lists...
Building dependency tree...
Reading state information...
bc is already the newest version (1.07.1-2build1).
bc set to manually installed.
libapr1 is already the newest version (1.6.5-1ubuntu1).
libapr1 set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
--2023-03-06 07:22:17--  
http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb
Resolving security.ubuntu.com (security.ubuntu.com)... 185.125.190.39, 
185.125.190.36, 91.189.91.39, ...
Connecting to security.ubuntu.com (security.ubuntu.com)|185.125.190.39|:80... 
connected.
HTTP request sent, awaiting response... 404 Not Found
2023-03-06 07:22:17 ERROR 404: Not Found.


WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
E: Unsupported file ./libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb given on 
commandline
##[error]Bash exited with code '100'.
Finishing: Prepare E2E run
 {code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46628=logs=81be5d54-0dc6-5130-d390-233dd2956037=d72874ca-f446-5272-2efd-0705f108dbf6



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

[Shammon FY]

Hi weihua

Can you add content related to `heterogeneous resources` to this FLIP? We
can record it and consider it in the future. It may be useful for some
scenarios, such as the combination of streaming and ML.

Best,
Shammon


On Mon, Mar 6, 2023 at 1:39 PM weijie guo  wrote:

> Hi Weihua,
>
> Thanks for your clarification, SGTM.
>
> Best regards,
>
> Weijie
>
>
> Weihua Hu  于2023年3月6日周一 11:43写道：
>
> > Thanks Weijie.
> >
> > Heterogeneous task managers will not be considered in this FLIP since
> > it does not request heterogeneous resources as you said.
> >
> > My first thought is we can adjust the meaning of redundant configuration
> > to redundant number of per resource type. These can be considered in
> > detail when we decide to support heterogeneous task managers.
> >
> > Best,
> > Weihua
> >
> >
> > On Sat, Mar 4, 2023 at 1:13 AM weijie guo 
> > wrote:
> >
> > > Thanks Weihua for preparing this FLIP.
> > >
> > > This FLIP overall looks reasonable to me after updating as suggested by
> > > Matthias.
> > >
> > > I only have one small question about keeping some redundant task
> > managers:
> > > In the fine-grained resource management, theoretically, it can support
> > > heterogeneous taskmanagers. When we complete the missing features for
> > FGSM,
> > > do we plan to take this into account?
> > > Of course, if I remember correctly, FGSM will not request heterogeneous
> > > resources at present, so it is also acceptable to me if there is no
> > special
> > > treatment now.
> > >
> > > +1 for this changes if we can ensure the test coverage.
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > John Roesler  于2023年3月2日周四 12:53写道：
> > >
> > > > Thanks for the test plan, Weihua!
> > > >
> > > > Yes, it addresses my concerns.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Mar 1, 2023, at 22:38, Weihua Hu wrote:
> > > > > Hi, everyone,
> > > > > Thanks for your suggestions and ideas.
> > > > > Thanks Xintong for sharing the detailed backgrounds of SlotManager.
> > > > >
> > > > > *@Matthias
> > > > >
> > > > > 1. Did you do a proper test coverage analysis?
> > > > >
> > > > >
> > > > > Just as Xintong said, we already have a CI stage for fine grained
> > > > resource
> > > > > managers.
> > > > > And I will make sure FineGrainedSlotManager as the default
> > SlotManager
> > > > can
> > > > > pass all the tests of CI.
> > > > > In addition, I will review all unit tests of
> > > DeclarativeSlotManager(DSM)
> > > > to
> > > > > ensure that there are no gaps in the
> > > > > coverage provided by the FineGrainedSlotManager.
> > > > > I also added the 'Test Plan' part to the FLIP.
> > > > > @Matthias @John @Shammon Does this test plan address your concerns?
> > > > >
> > > > > 2.  DeclarativeSlotManager and FineGrainedSlotManager feel quite
> big
> > in
> > > > >
> > > > > terms of lines of code
> > > > >
> > > > >
> > > > > IMO, the refactoring of SlotManager does not belong to this FLIP
> > since
> > > it
> > > > > may lead to some unstable risks. For
> > > > > FineGrainedSlotManager(FGSM), we already split some reasonable
> > > > components.
> > > > > They are:
> > > > > * TaskManagerTracker: Track task managers and their resources.
> > > > > * ResourceTracker: track requirements of jobs
> > > > > * ResourceAllocationStrategy: Try to fulfill the resource
> > requirements
> > > > with
> > > > > available/pending resources.
> > > > > * SlotStatusSyncer: communicate with TaskManager, for
> > > allocating/freeing
> > > > > slot and reconciling the slot status
> > > > > Maybe we can start a discussion about refactoring SlotManager in
> > > another
> > > > > FLIP if there are some good suggestions.
> > > > > WDYT
> > > > >
> > > > > 3. For me personally, having a more detailed summary comparing the
> > > > >> subcomponents of both SlotManager implementations with where
> > > > >> their functionality matches and where they differ might help
> > > understand
> > > > the
> > > > >> consequences of the changes proposed in FLIP-298
> > > > >
> > > > > Good suggestion, I have updated the comparison in this FLIP.
> Looking
> > > > > forward to any suggestions/thoughts
> > > > > if they are not described clearly.
> > > > >
> > > > > *@John
> > > > >
> > > > > 4. In addition to changing the default, would it make sense to log
> a
> > > > >> deprecation warning on initialization
> > > > >
> > > > > if the DeclarativeSlotManager is used?
> > > > >>
> > > > > SGTM, We should add Deprecated annotations to DSM for devs. And
> log a
> > > > > deprecation warning for users.
> > > > >
> > > > > *@Shammon
> > > > >
> > > > > 1. For their functional differences, can you give some detailed
> tests
> > > to
> > > > >> verify that the new FineGrainedSlotManager has these capabilities?
> > > This
> > > > can
> > > > >> effectively verify the new functions
> > > > >>
> > > > > As just maintained, there is already a CI stage of FGSM, and I will
> > do
> > > > more
> > > > > review of unit tests for DSM.
> > > > 

[jira] [Created] (FLINK-31333) KafkaPartitionSplitReaderTest.testPendingRecordsGauge failed to create topic

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31333:
-

 Summary: KafkaPartitionSplitReaderTest.testPendingRecordsGauge 
failed to create topic
 Key: FLINK-31333
 URL: https://issues.apache.org/jira/browse/FLINK-31333
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Kafka
Affects Versions: 1.15.3
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46765=logs=ce8f3cc3-c1ea-5281-f5eb-df9ebd24947f=918e890f-5ed9-5212-a25e-962628fb4bc5
{code}
Mar 03 03:17:58 [INFO] Running 
org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReaderTest
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: The request timed out.
at 
org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
at 
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:175)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:97)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:216)
at 
org.apache.flink.connector.kafka.testutils.KafkaSourceTestEnv.createTestTopic(KafkaSourceTestEnv.java:217)
at 
org.apache.flink.connector.kafka.testutils.KafkaSourceTestEnv.setupTopic(KafkaSourceTestEnv.java:261)
at 
org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReaderTest.testPendingRecordsGauge(KafkaPartitionSplitReaderTest.java:198)
[...]
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] String literal behavior in Flink

[Aitozi]

Thanks, will give it a try

Best,
Aitozi.

Jark Wu  于2023年3月6日周一 15:11写道：

> Hi Aitozi,
>
> I would suggest trying to contribute it to the upstream project Calcite
> first.
>
> Best,
> Jark
>
> > 2023年3月6日 11:51，Aitozi  写道：
> >
> > Hi Jark,
> >
> > Thank you for your helpful suggestion. It appears that 'E'foo\n'' is a
> more
> > versatile and widely accepted option. To assess its feasibility, I have
> > reviewed the relevant Unicode supports and concluded that it may
> > necessitate modifications to the Parser.jj file to accommodate this new
> > syntax.
> >
> >
> > I am unsure whether we should initially incorporate this alteration in
> > Calcite or if we can directly supersede the StringLiteral behavior within
> > the Flink project. Nevertheless, I believe supporting this change is
> > achievable.
> >
> >
> >
> > Thanks,
> > Aitozi.
> >
> > Jark Wu  于2023年3月6日周一 10:16写道：
> >
> >> Hi Aitozi,
> >>
> >> I think this is a good idea to improve the backslash escape strings.
> >> However, I lean a bit more toward the Postgres approach[1],
> >> which is more standard-compliant. PG allows backslash escape
> >> string by writing the letter E (upper or lower case) just before the
> >> opening single quote, e.g., E'foo\n'.
> >>
> >> Recognizing backslash escapes in both regular and escape string
> constants
> >> is not backward compatible in Flink, and is also deprecated in PG.
> >>
> >> In addition, Flink also supports Unicode escape string constants by
> >> writing the U& before the quote[1] which works in the same way with
> >> backslash escape string.
> >>
> >> Best,
> >> Jark
> >>
> >> [1]:
> >>
> >>
> https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-CONSTANTS
> >> [2]:
> >>
> >>
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/overview/
> >>
> >> On Sat, 4 Mar 2023 at 23:31, Aitozi  wrote:
> >>
> >>> Hi,
> >>>  I encountered a problem when using string literal in Flink. Currently,
> >>> Flink will escape the string literal during codegen, so for the query
> >>> below:
> >>>
> >>> SELECT 'a\nb'; it will print => a\nb
> >>>
> >>> then for the query
> >>>
> >>> SELECT SPLIT_INDEX(col, '\n', 0);
> >>>
> >>> The col can not split by the newline. If we want to split by the
> newline,
> >>> we should use
> >>>
> >>> SELECT SPLIT_INDEX(col, '
> >>> ', 0)
> >>>
> >>> or
> >>>
> >>> SELECT SPLIT_INDEX(col, CHR(10), 0)
> >>>
> >>> The above way could be more intuitive. Some other databases support
> these
> >>> "Special Character Escape Sequences"[1].
> >>>
> >>> In this way, we can directly use
> >>> SELECT SPLIT_INDEX(col, '\n', 0); for the query.
> >>>
> >>> I know this is not standard behavior in ANSI SQL. I'm opening this
> thread
> >>> for some opinions from the community guys.
> >>>
> >>> [1]:
> >>>
> >>>
> >>
> https://dev.mysql.com/doc/refman/8.0/en/string-literals.html#character-escape-sequences
> >>>
> >>> Thanks,
> >>> Aitozi
> >>>
> >>
>
>

[jira] [Created] (FLINK-31332) Limit the use of ExecutionConfig on JdbcOutputFormat

[Jira]

João Boto created FLINK-31332:
-

 Summary: Limit the use of ExecutionConfig on JdbcOutputFormat
 Key: FLINK-31332
 URL: https://issues.apache.org/jira/browse/FLINK-31332
 Project: Flink
  Issue Type: Improvement
Reporter: João Boto


This is for limiting the use of ExecutionConfig on JdbcOutputFormat and 
centralize the logic of serialisation of records in one place.

Also current the record will be copied at least 2/3 times if isObjectReuse is 
enabled



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31331) Flink 1.16 should implement new LookupFunction

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-31331:


 Summary: Flink 1.16 should implement new LookupFunction
 Key: FLINK-31331
 URL: https://issues.apache.org/jira/browse/FLINK-31331
 Project: Flink
  Issue Type: Bug
  Components: Table Store
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: table-store-0.4.0


Only implements new LookupFunction, retry lookup join can work.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31330) Batch shuffle may deadlock for operator with priority input

[Weijie Guo (Jira)]

Weijie Guo created FLINK-31330:
--

 Summary: Batch shuffle may deadlock for operator with priority 
input
 Key: FLINK-31330
 URL: https://issues.apache.org/jira/browse/FLINK-31330
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Network
Affects Versions: 1.16.1
Reporter: Weijie Guo
Assignee: Weijie Guo


For batch job, some operator's input have priority. For example, hash join 
operator has two inputs called {{build}} and {{probe}} respectively. Only after 
the build input is finished can the probe input start consuming. Unfortunately, 
the priority of input will not affect multiple inputs to request upstream 
data(i.e. request partition). In current implementation, when all states are 
restored, inputGate will start to request partition. This will enable the 
upstream {{IO scheduler}} to register readers for all downstream channels, so 
there is the possibility of deadlock.
Assume that the build and probe input's upstream tasks of hash join are 
deployed in the same TM. Then the corresponding readers will be registered to 
an single {{IO scheduler}}, and they share the same 
{{BatchShuffleReadBufferPool}}.  If the IO thread happens to load too many 
buffers for the probe reader, but the downstream will not consume the data, 
which will cause the build reader to be unable to request enough buffers. 
Therefore, deadlock occurs.
In fact, we realized this problem at the beginning of the design of 
{{SortMergeShuffle}}, so we introduced a timeout mechanism when requesting read 
buffers. If this happens, the downstream task will trigger failover to avoid 
permanent blocking. However, under the default configuration, TPC-DS test with 
10T data can easily cause the job to fail because of this reason. It seems that 
this problem needs to be solved more better.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)