[jira] [Created] (FLINK-31323) Fix unstable merge-into E2E test

[yuzelin (Jira)]

yuzelin created FLINK-31323:
---

 Summary: Fix unstable merge-into E2E test
 Key: FLINK-31323
 URL: https://issues.apache.org/jira/browse/FLINK-31323
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Affects Versions: table-store-0.4.0
Reporter: yuzelin


A complex test of merge-into action in docker environment may fail. So the test 
need to be simplified.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31322) Improve merge-into action

[yuzelin (Jira)]

yuzelin created FLINK-31322:
---

 Summary: Improve merge-into action
 Key: FLINK-31322
 URL: https://issues.apache.org/jira/browse/FLINK-31322
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Affects Versions: table-store-0.4.0
Reporter: yuzelin


Umbrella issue for bug fixing and code improvement.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Flink k8s native support - pod deployments and upgrades

[Gyula Fóra]

In general there are no in place upgrades in Flink for either jobs or
processes . This is not specific to Kubernetes, Yarn or any deployment mode.

Gyula

On Sat, 4 Mar 2023 at 06:51, ramkrishna vasudevan 
wrote:

> Thanks for the reply. Yes I get that the operator helps out in upgrades
> also . But when the operator works with flink k8s native clusters, it can
> help with savepoints and helps in creating a new upgraded cluster, but the
> existing cluster is not generally upgraded due to the design as how
> clusters are created.
>
> Is my understanding correct?
>
> Regards
> Ram
>
> On Fri, Mar 3, 2023 at 9:49 PM Thomas Weise  wrote:
>
> > Hi,
> >
> > The Flink k8s native integration does not handle upgrades. That's what
> > flink-kubernates-operator was built for. Please check out:
> >
> > https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/
> >
> > Thanks,
> > Thomas
> >
> >
> > On Fri, Mar 3, 2023 at 2:02 AM ramkrishna vasudevan <
> > ramvasu.fl...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > The native implementation of the App mode and session mode - does not
> > have
> > > any replica set .
> > > Instead it just allows the JM to create TM pods on demand.
> > >
> > > This is simple and easy in terms of creation of resources, but for an
> > > upgrade story, how is this managed? Leaving K8s to manage a replica set
> > > based upgrade might be easier right?
> > >
> > > Just wanted to understand how upgrades are handled in native K8s mode.
> > >
> > > Regards
> > > Ram
> > >
> >
>

Re: Flink k8s native support - pod deployments and upgrades

[ramkrishna vasudevan]

Thanks for the reply. Yes I get that the operator helps out in upgrades
also . But when the operator works with flink k8s native clusters, it can
help with savepoints and helps in creating a new upgraded cluster, but the
existing cluster is not generally upgraded due to the design as how
clusters are created.

Is my understanding correct?

Regards
Ram

On Fri, Mar 3, 2023 at 9:49 PM Thomas Weise  wrote:

> Hi,
>
> The Flink k8s native integration does not handle upgrades. That's what
> flink-kubernates-operator was built for. Please check out:
>
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/
>
> Thanks,
> Thomas
>
>
> On Fri, Mar 3, 2023 at 2:02 AM ramkrishna vasudevan <
> ramvasu.fl...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > The native implementation of the App mode and session mode - does not
> have
> > any replica set .
> > Instead it just allows the JM to create TM pods on demand.
> >
> > This is simple and easy in terms of creation of resources, but for an
> > upgrade story, how is this managed? Leaving K8s to manage a replica set
> > based upgrade might be easier right?
> >
> > Just wanted to understand how upgrades are handled in native K8s mode.
> >
> > Regards
> > Ram
> >
>

[jira] [Created] (FLINK-31321) Yarn-session mode, securityConfiguration supports dynamic configuration

[felixzh (Jira)]

felixzh created FLINK-31321:
---

 Summary: Yarn-session mode, securityConfiguration supports dynamic 
configuration
 Key: FLINK-31321
 URL: https://issues.apache.org/jira/browse/FLINK-31321
 Project: Flink
  Issue Type: Improvement
Reporter: felixzh






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Large schemas lead to long DataStream-to-table transformation names

[Xingcan Cui]

Hi all,

We are dealing with some streams with large (nested) schemas. When using `t
ableEnv.createTemporaryView()` to register a DataStream to a table, the
transformation always gets a large name. It's not a big problem, but quite
annoying since the UI and logs are hard to read.

Internally, `ExternalDynamicSource` (and `ExternalDynamicSink`) invokes
`physicalDataType.toString()` to generate an operator name (which will also
be used as the transformation name). I'm thinking to introduce a new table
config to either truncate the name or use a limited level of logicalType to
generate the name (works for nested schemas).

What do you think?

Best,
Xingcan

[jira] [Created] (FLINK-31320) Modify DATE_FORMAT system (built-in) function to accepts DATEs

[James Mcguire (Jira)]

James Mcguire created FLINK-31320:
-

 Summary: Modify DATE_FORMAT system (built-in) function to accepts 
DATEs
 Key: FLINK-31320
 URL: https://issues.apache.org/jira/browse/FLINK-31320
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: James Mcguire


The current {{DATE_FORMAT}} function only supports {{{}TIMESTAMP{}}}s. 

(See 
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/systemfunctions/#temporal-functions)

 

Ideally, it should be able to format {{{}DATE{}}}'s as well as {{TIMESTAMPs}}

 

Example usage:
{noformat}
Flink SQL> CREATE TABLE test_table (
>   some_date DATE,
>   object AS JSON_OBJECT(
> KEY 'some_date' VALUE DATE_FORMAT(some_date, '-MM-dd')
>   )
> )
> COMMENT ''
> WITH (
>   'connector'='datagen'
> )
> ;
> 
[ERROR] Could not execute SQL statement. Reason:
org.apache.calcite.sql.validate.SqlValidatorException: Cannot apply 
'DATE_FORMAT' to arguments of type 'DATE_FORMAT(, )'. Supported 
form(s): 'DATE_FORMAT(, )'
'DATE_FORMAT(, )'{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Regarding new command to download jars in flink cluster

[Surendra Singh Lilhore]

Hi Team,



According to the Flink documentation, in the APP mode, the application jar
should be bundled with the Flink image. However, building an image for each
new application can be difficult. Can we introduce new commands that will
help to download the required jar locally before starting Flink JM or TM
containers? This should be a simple command that depends on the supported
file system (S3, HDFS, ABFS) in Flink, and the command format should be
something like this:

*./flink fs-download  *

Example:

*./flink fs-download
abfs://mycontai...@storageaccount.dfs.core.windows.net/jars /tmp/appjars
*

I have already tested this in my cluster, and it is working fine. Before
raising a JIRA ticket, I would like to get suggestions from the community.


Thanks and Regards
Surendra

[jira] [Created] (FLINK-31319) Inconsistent condition judgement about kafka partitionDiscoveryIntervalMs cause potential bug

[Ran Tao (Jira)]

Ran Tao created FLINK-31319:
---

 Summary: Inconsistent condition judgement about kafka 
partitionDiscoveryIntervalMs cause potential bug
 Key: FLINK-31319
 URL: https://issues.apache.org/jira/browse/FLINK-31319
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.16.1
Reporter: Ran Tao
 Attachments: image-2023-03-04-01-37-29-360.png, 
image-2023-03-04-01-39-20-352.png, image-2023-03-04-01-40-44-124.png, 
image-2023-03-04-01-41-55-664.png

As kafka option description, partitionDiscoveryIntervalMs <=0 means disabled.  
!image-2023-03-04-01-37-29-360.png|width=781,height=147!

just like start kafka enum:

!image-2023-03-04-01-39-20-352.png|width=465,height=311!

but inner 
handlePartitionSplitChanges use error if condition( < 0):


!image-2023-03-04-01-40-44-124.png|width=576,height=237!
 
it will cause noMoreNewPartitionSplits can not be set to true. Finally cause
!image-2023-03-04-01-41-55-664.png|width=522,height=610!

bounded source can not signalNoMoreSplits.

Anyway，Both ends of the if condition should be mutually exclusive. WDYT?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31318) Do not scale down operators while processing backlog

[Gyula Fora (Jira)]

Gyula Fora created FLINK-31318:
--

 Summary: Do not scale down operators while processing backlog
 Key: FLINK-31318
 URL: https://issues.apache.org/jira/browse/FLINK-31318
 Project: Flink
  Issue Type: Improvement
  Components: Autoscaler, Kubernetes Operator
Reporter: Gyula Fora
Assignee: Gyula Fora


Currently the autoscaler may try to scale down some operators even when the job 
is struggling to catch up. 

This can lead to a vicious cycle where the backlog keeps increasing. It makes 
sense to hold off scale down decisions until the job has caught up.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

[weijie guo]

Thanks Weihua for preparing this FLIP.

This FLIP overall looks reasonable to me after updating as suggested by
Matthias.

I only have one small question about keeping some redundant task managers:
In the fine-grained resource management, theoretically, it can support
heterogeneous taskmanagers. When we complete the missing features for FGSM,
do we plan to take this into account?
Of course, if I remember correctly, FGSM will not request heterogeneous
resources at present, so it is also acceptable to me if there is no special
treatment now.

+1 for this changes if we can ensure the test coverage.

Best regards,

Weijie


John Roesler  于2023年3月2日周四 12:53写道：

> Thanks for the test plan, Weihua!
>
> Yes, it addresses my concerns.
>
> Thanks,
> John
>
> On Wed, Mar 1, 2023, at 22:38, Weihua Hu wrote:
> > Hi, everyone,
> > Thanks for your suggestions and ideas.
> > Thanks Xintong for sharing the detailed backgrounds of SlotManager.
> >
> > *@Matthias
> >
> > 1. Did you do a proper test coverage analysis?
> >
> >
> > Just as Xintong said, we already have a CI stage for fine grained
> resource
> > managers.
> > And I will make sure FineGrainedSlotManager as the default SlotManager
> can
> > pass all the tests of CI.
> > In addition, I will review all unit tests of DeclarativeSlotManager(DSM)
> to
> > ensure that there are no gaps in the
> > coverage provided by the FineGrainedSlotManager.
> > I also added the 'Test Plan' part to the FLIP.
> > @Matthias @John @Shammon Does this test plan address your concerns?
> >
> > 2.  DeclarativeSlotManager and FineGrainedSlotManager feel quite big in
> >
> > terms of lines of code
> >
> >
> > IMO, the refactoring of SlotManager does not belong to this FLIP since it
> > may lead to some unstable risks. For
> > FineGrainedSlotManager(FGSM), we already split some reasonable
> components.
> > They are:
> > * TaskManagerTracker: Track task managers and their resources.
> > * ResourceTracker: track requirements of jobs
> > * ResourceAllocationStrategy: Try to fulfill the resource requirements
> with
> > available/pending resources.
> > * SlotStatusSyncer: communicate with TaskManager, for allocating/freeing
> > slot and reconciling the slot status
> > Maybe we can start a discussion about refactoring SlotManager in another
> > FLIP if there are some good suggestions.
> > WDYT
> >
> > 3. For me personally, having a more detailed summary comparing the
> >> subcomponents of both SlotManager implementations with where
> >> their functionality matches and where they differ might help understand
> the
> >> consequences of the changes proposed in FLIP-298
> >
> > Good suggestion, I have updated the comparison in this FLIP. Looking
> > forward to any suggestions/thoughts
> > if they are not described clearly.
> >
> > *@John
> >
> > 4. In addition to changing the default, would it make sense to log a
> >> deprecation warning on initialization
> >
> > if the DeclarativeSlotManager is used?
> >>
> > SGTM, We should add Deprecated annotations to DSM for devs. And log a
> > deprecation warning for users.
> >
> > *@Shammon
> >
> > 1. For their functional differences, can you give some detailed tests to
> >> verify that the new FineGrainedSlotManager has these capabilities? This
> can
> >> effectively verify the new functions
> >>
> > As just maintained, there is already a CI stage of FGSM, and I will do
> more
> > review of unit tests for DSM.
> >
> >  2. I'm worried that many functions are not independent and it is
> difficult
> >> to migrate step-by-step. You can list the relationship between them in
> >> detail.
> >
> >  As Xintong saied the DSM is a subset of FGSM by design. But as time goes
> > on, FGSM has some lacking
> > functions as I listed in this FLIP. And I have added the comparison
> between
> > DSM and FGSM in this FLIP.
> >
> >
> > Thanks again for all your thoughts. Any feedback is appreciated!
> >
> > Best,
> > Weihua
> >
> >
> > On Wed, Mar 1, 2023 at 2:17 PM Xintong Song 
> wrote:
> >
> >> Thanks Weihua for preparing this FLIP. +1 for the proposal.
> >>
> >>
> >> As one of the contributors of the fine-grained slot manager, I'd like to
> >> share some backgrounds here.
> >>
> >> - There used to be a defaut slot manager implementation, which is
> >> non-declarative and has been removed now. The two features, declarative
> /
> >> reactive resource management and fine-grained resource management, were
> >> proposed at about the same time. We were aware that by design the
> >> declarative slot manager is a subset of fine-grained slot manager at
> that
> >> time, but still decided to implement two slot managers for the purpose
> of
> >> decoupling efforts and reducing cross-team synchronization overhead.
> >> Merging the two slot managers once they are proved stable is IMO a
> >> technical debt that was planned at the very beginning.
> >>
> >> - The FineGrainedSlotManager has been verified in Alibaba's internal
> >> production as well as Alibaba Cloud services as the default 

Re: Flink k8s native support - pod deployments and upgrades

[Thomas Weise]

Hi,

The Flink k8s native integration does not handle upgrades. That's what
flink-kubernates-operator was built for. Please check out:

https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/

Thanks,
Thomas


On Fri, Mar 3, 2023 at 2:02 AM ramkrishna vasudevan 
wrote:

> Hi All,
>
> The native implementation of the App mode and session mode - does not have
> any replica set .
> Instead it just allows the JM to create TM pods on demand.
>
> This is simple and easy in terms of creation of resources, but for an
> upgrade story, how is this managed? Leaving K8s to manage a replica set
> based upgrade might be easier right?
>
> Just wanted to understand how upgrades are handled in native K8s mode.
>
> Regards
> Ram
>

[jira] [Created] (FLINK-31317) Introduce data structure for managing resource requirements of a job

[Jira]

David Morávek created FLINK-31317:
-

 Summary: Introduce data structure for managing resource 
requirements of a job
 Key: FLINK-31317
 URL: https://issues.apache.org/jira/browse/FLINK-31317
 Project: Flink
  Issue Type: Sub-task
Reporter: David Morávek






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31316) FLIP-291: Externalized Declarative Resource Management

[Jira]

David Morávek created FLINK-31316:
-

 Summary: FLIP-291: Externalized Declarative Resource Management
 Key: FLINK-31316
 URL: https://issues.apache.org/jira/browse/FLINK-31316
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Coordination, Runtime / REST
Reporter: David Morávek
Assignee: David Morávek


This is an umbrella ticket for 
[FLIP-291|https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 1.15.4, release candidate #1

[Danny Cranmer]

Hello Dian,

As per the new version support strategy [1], we will perform a Flink 1.15.5
patch if there are any resolved critical/blocker issues at the point in
which 1.17.0 is released. The issue referenced is a Major. I am not
inclined to cancel the release candidate at this point unless I get a -1
veto. Is this issue impacting all Python Datastream apps with a Tumbling
window or a subset? I am wondering why it is only a Major?

Thanks,
Danny

[1] https://lists.apache.org/thread/9w99mgx3nw5tc0v26wcvlyqxrcrkpzdz

On Fri, Mar 3, 2023 at 2:09 AM Dian Fu  wrote:

> Hi Danny,
>
> I'm sorry that I'm coming to this thread a little late. It seems that this
> will be the last bugfix release of Flink 1.15? If so, I'd like to also
> include https://issues.apache.org/jira/browse/FLINK-31272 into this
> release
> which fixes a serious issue of PyFlink.
>
> Regards,
> Dian
>
>
>
> On Thu, Mar 2, 2023 at 5:51 PM Yu Li  wrote:
>
> > +1 (binding)
> >
> >
> > - Checked the diff between 1.15.3 and 1.15.4-rc1: *OK* (
> >
> https://github.com/apache/flink/compare/release-1.15.3...release-1.15.4-rc1
> > )
> >
> >   - AWS SDKv2 version has been bumped to 2.19.14 through FLINK-30633 and
> > all NOTICE files updated correctly
> >
> > - Checked release notes: *OK*
> >
> > - Checked sums and signatures: *OK*
> >
> > - Maven clean install from source: *OK* (8u181)
> >
> > - Checked the jars in the staging repo: *OK*
> >
> > - Checked the website updates: *OK*
> >
> >
> > Thanks for driving this release, Danny!
> >
> >
> > Best Regards,
> > Yu
> >
> >
> > On Wed, 1 Mar 2023 at 02:01, Ahmed Hamdy  wrote:
> >
> > > Thanks Danny,
> > >
> > > +1 (non-binding)
> > >
> > > - Verified hashes and signatures
> > > - Built Source archive using maven
> > > - Web PR looks good.
> > > - Started WordCount Example
> > >
> > > On Tue, 28 Feb 2023 at 16:37, Jing Ge 
> > wrote:
> > >
> > > > Thanks Danny,
> > > >
> > > > +1 (non-binding)
> > > >
> > > >  - GPG signatures looks good
> > > >  - checked dist and maven repo
> > > >  - maven clean install from source
> > > >  - checked version consistency in pom files
> > > >  - went through the web release notes and found one task is still
> open:
> > > > FLINK-31133 [1]
> > > >  - download artifacts
> > > >  - started/stopped local cluster and ran WordCount job in streaming
> and
> > > > batch
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-31133
> > > >
> > > > On Tue, Feb 28, 2023 at 3:12 PM Matthias Pohl
> > > >  wrote:
> > > >
> > > > > Thanks Danny.
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > * Downloaded artifacts
> > > > > * Built Flink from sources
> > > > > * Verified SHA512 checksums GPG signatures
> > > > > * Compared checkout with provided sources
> > > > > * Verified pom file versions
> > > > > * Went over NOTICE file/pom files changes without finding anything
> > > > > suspicious
> > > > > * Deployed standalone session cluster and ran WordCount example in
> > > batch
> > > > > and streaming: Nothing suspicious in log files found
> > > > >
> > > > > On Tue, Feb 28, 2023 at 9:50 AM Teoh, Hong
> > >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Danny for driving this
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > * Hashes and Signatures look good
> > > > > > * All required files on dist.apache.org
> > > > > > * Source archive builds using maven
> > > > > > * Started packaged example WordCountSQLExample job
> > > > > > * Web PR looks good.
> > > > > >
> > > > > > Cheers,
> > > > > > Hong
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On 24 Feb 2023, at 05:36, Weihua Hu 
> > > wrote:
> > > > > > >
> > > > > > > CAUTION: This email originated from outside of the
> organization.
> > Do
> > > > not
> > > > > > click links or open attachments unless you can confirm the sender
> > and
> > > > > know
> > > > > > the content is safe.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks Danny.
> > > > > > >
> > > > > > > +1(non-binding)
> > > > > > >
> > > > > > > Tested the following:
> > > > > > > - Download the artifacts and build image
> > > > > > > - Ran WordCount on Kubernetes(session mode and application
> mode)
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Weihua
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 24, 2023 at 12:29 PM Yanfei Lei <
> fredia...@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > >> Thanks Danny.
> > > > > > >> +1 (non-binding)
> > > > > > >>
> > > > > > >> - Downloaded artifacts & built Flink from sources
> > > > > > >> - Verified GPG signatures of bin and source.
> > > > > > >> - Verified version in poms
> > > > > > >> - Ran WordCount example in streaming and batch mode(standalone
> > > > > cluster)
> > > > > > >> - Went over flink-web PR, looks good except for Sergey's
> remark.
> > > > > > >>
> > > > > > >> Danny Cranmer  于2023年2月24日周五
> 02:08写道：
> > > > > > >>>
> > > > > > >>> Hi everyone,
> > > > > > >>> Please review and 

[jira] [Created] (FLINK-31315) FlinkActionsE2eTest.testMergeInto is unstable

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-31315:


 Summary: FlinkActionsE2eTest.testMergeInto is unstable
 Key: FLINK-31315
 URL: https://issues.apache.org/jira/browse/FLINK-31315
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.4.0


{code:java}
Error:  Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 320.272 
s <<< FAILURE! - in org.apache.flink.table.store.tests.FlinkActionsE2eTest
82Error:  testMergeInto  Time elapsed: 111.826 s  <<< FAILURE!
83org.opentest4j.AssertionFailedError: 
84Result is still unexpected after 60 retries.
85Expected: {3, v_3, creation, 02-27=1, 2, v_2, creation, 02-27=1, 6, v_6, 
creation, 02-28=1, 1, v_1, creation, 02-27=1, 8, v_8, insert, 02-29=1, 11, 
v_11, insert, 02-29=1, 7, Seven, matched_upsert, 02-28=1, 5, v_5, creation, 
02-28=1, 10, v_10, creation, 02-28=1, 9, v_9, creation, 02-28=1}
86Actual: {4, v_4, creation, 02-27=1, 8, v_8, creation, 02-28=1, 3, v_3, 
creation, 02-27=1, 7, v_7, creation, 02-28=1, 2, v_2, creation, 02-27=1, 6, 
v_6, creation, 02-28=1, 1, v_1, creation, 02-27=1, 5, v_5, creation, 02-28=1, 
10, v_10, creation, 02-28=1, 9, v_9, creation, 02-28=1}
87  at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:39)
88  at org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
89  at 
org.apache.flink.table.store.tests.E2eTestBase.checkResult(E2eTestBase.java:261)
90  at 
org.apache.flink.table.store.tests.FlinkActionsE2eTest.testMergeInto(FlinkActionsE2eTest.java:355)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-291: Externalized Declarative Resource Management

[David Morávek]

I'm happy to announce that we have unanimously approved this FLIP.

There are 8 approving votes, 3 of which are binding:

* John Roesler
* Konstantin Knauf (binding)
* Zhanghao Chen
* ConradJam
* Feng Xiangyu
* Gyula Fóra (binding)
* Roman Khachatryan (binding)
* Shammon FY

There are no disapproving votes.

Thanks everyone for participating!

Best,
D.

Re: [VOTE] FLIP-291: Externalized Declarative Resource Management

[David Morávek]

Thanks, everyone, I'm closing this vote now. I'll follow up with the result
in a separate email.

On Wed, Mar 1, 2023 at 10:01 AM Shammon FY  wrote:

> +1 (non-binding)
>
> Best,
> Shammon
>
>
> On Wed, Mar 1, 2023 at 4:51 PM Roman Khachatryan  wrote:
>
> > +1 (binding)
> >
> > Thanks David, and everyone involved :)
> >
> > Regards,
> > Roman
> >
> >
> > On Wed, Mar 1, 2023 at 8:01 AM Gyula Fóra  wrote:
> >
> > > +1 (binding)
> > >
> > > Looking forward to this :)
> > >
> > > Gyula
> > >
> > > On Wed, 1 Mar 2023 at 04:02, feng xiangyu 
> wrote:
> > >
> > > > +1  (non-binding)
> > > >
> > > > ConradJam  于2023年3月1日周三 10:37写道：
> > > >
> > > > > +1  (non-binding)
> > > > >
> > > > > Zhanghao Chen  于2023年3月1日周三 10:18写道：
> > > > >
> > > > > > Thanks for driving this. +1 (non-binding)
> > > > > >
> > > > > > Best,
> > > > > > Zhanghao Chen
> > > > > > 
> > > > > > From: David Mor?vek 
> > > > > > Sent: Tuesday, February 28, 2023 21:46
> > > > > > To: dev 
> > > > > > Subject: [VOTE] FLIP-291: Externalized Declarative Resource
> > > Management
> > > > > >
> > > > > > Hi Everyone,
> > > > > >
> > > > > > I want to start the vote on FLIP-291: Externalized Declarative
> > > Resource
> > > > > > Management [1]. The FLIP was discussed in this thread [2].
> > > > > >
> > > > > > The goal of the FLIP is to enable external declaration of the
> > > resource
> > > > > > requirements of a running job.
> > > > > >
> > > > > > The vote will last for at least 72 hours (Friday, 3rd of March,
> > 15:00
> > > > > CET)
> > > > > > unless
> > > > > > there is an objection or insufficient votes.
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/b8fnj127jsl5ljg6p4w3c4wvq30cnybh
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management
> > > > > >
> > > > > > Best,
> > > > > > D.
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-31314) Hybrid

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-31314:
-

 Summary: Hybrid 
 Key: FLINK-31314
 URL: https://issues.apache.org/jira/browse/FLINK-31314
 Project: Flink
  Issue Type: Improvement
Reporter: Yuxin Tan






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31313) Unsupported meta columns in column list of insert statement

[lincoln lee (Jira)]

lincoln lee created FLINK-31313:
---

 Summary: Unsupported meta columns in column list of insert 
statement
 Key: FLINK-31313
 URL: https://issues.apache.org/jira/browse/FLINK-31313
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.16.1, 1.17.0
Reporter: lincoln lee


Currently an error will be raised when ref meta columns in column list of 
insert statement, e.g.,
{code}
INSERT INTO sink (a,b,f) -- here `f` is a metadata column of sink table
SELECT ...{code}

{code}
Caused by: org.apache.calcite.runtime.CalciteContextException: At line 1, 
column 44: Unknown target column 'f'
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.calcite.runtime.Resources$ExInstWithCause.ex(Resources.java:505)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:932)
at org.apache.calcite.sql.SqlUtil.newContextException(SqlUtil.java:917)
at 
org.apache.flink.table.planner.calcite.PreValidateReWriter$.newValidationError(PreValidateReWriter.scala:440)
at 
org.apache.flink.table.planner.calcite.PreValidateReWriter$.validateField(PreValidateReWriter.scala:428)
at 
org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$appendPartitionAndNullsProjects$3(PreValidateReWriter.scala:169)
at 
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike.map(TraversableLike.scala:233)
at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at 
org.apache.flink.table.planner.calcite.PreValidateReWriter$.appendPartitionAndNullsProjects(PreValidateReWriter.scala:161)
at 
org.apache.flink.table.planner.calcite.PreValidateReWriter.rewriteInsert(PreValidateReWriter.scala:72)
{code}

The cause is current PreValidateReWriter in validation phase uses the physical 
types of sink table which does not include metadata columns



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

[Jark Wu]

Hi Kui,

I left my comments below.

> we don't need to calculate watermark for each event.

This is not true. We have to calculate the watermark for every event.
Otherwise, we may miss advancing the watermark.
We just don't need to emit every watermark.

> There is no default features like 'on-event-gap' in
> DataStream API, but the users can achieve the 'on-event-gap' feature by
> using `WatermarkGenerator` interface, just like the implemention in my
> POC[1]. However, If we don't provide it  in SQL layer, there is no way for
> users to use similar features.

My core point is that if DataStream API doesn't provide an 'on-event-gap'
built-in strategy, which may mean it is not a true demand. We should be
careful when adding this to SQL. However, I'm not against supporting
user-defined WatermarkGenerator in SQL.

The reason why I have concerns about adding 'on-event-gap' strategy is
that Punctuated WatermarkGenerator is not designed for emitting watermark
on event gap. It is designed to emit watermarks when it observes special
events in the stream that carries watermark information. That's why the
`WatermarkGenerator#onEvent(event, eventTimestamp, output)` carries an
input event. See the official example of PunctuatedAssigner [1].

That means, even if we introduce the 'on-event-gap' strategy in SQL, users
still
can't achieve the true ability of WatermarkGenerator.

> I'm sorry, perhaps I don't understand what are your concerns
> about CompiledPlan.

You can consider CompiledPlan and the JSON representation is also a public
API
that should be backward compatible, e.g., COMPILE PLAN + EXECUTE PLAN
should work, and execute a previous plan should also be compatible.
Therefore,
 the changes to them may need to be included in the FLIP.

Best,
Jark

[1]:
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/event-time/generating_watermarks/#writing-a-punctuated-watermarkgenerator


On Fri, 3 Mar 2023 at 15:46, Kui Yuan  wrote:

> Hi all,
>
> Thanks for all. There are more questions and I will answer one by one.
>
> @Jark Thanks for your tips. For the first question, I will add more details
> in the flip, and give a POC[1] so that pepole can know how I'm currently
> implementing these features.
>
> > IIRC, this is the first time we introduce the framework-level connector
> > options that the option is not recognized and handled by connectors.
> > The FLIP should cover how framework filters the watermark related options
> > to avoid discover connector factory failed, and what happens if the
> > connector already supported the conflict options
>
> For the second question, We know that the default strategy is 'on-periodic'
> in SQL layer, and the default interval is 200ms. The reason for emiting
> watermark periodically is that the time advancement of consecutive events
> may be very small, we don't need to calculate watermark for each event.
> Same for 'on-event' strategy, so my idea is that we can set a fixed gap for
> 'on-event' strategy.
>
> > I'm not sure about the usage scenarios of event gap emit strategy. Do
> > you have any specific use case of this strategy? I'm confused why no one
> > requested this strategy before no matter in DataStream or SQL, but maybe
> > I missed something. I'm not against to add this option, but just want to
> be
> > careful when adding new API because it's hard to remove in the future.
>
> As @Timo said, There is no default features like 'on-event-gap' in
> DataStream API, but the users can achieve the 'on-event-gap' feature by
> using `WatermarkGenerator` interface, just like the implemention in my
> POC[1]. However, If we don't provide it  in SQL layer, there is no way for
> users to use similar features.
>
> > Jark raised a very good point. I thought we only expose what is
> > contained in DataStream API already. If this strategy is not part of
> > DataStream API, would like to exclude it from the FLIP. We need to be
> > careful which strategies we offer by default.
>
> @Jark @Timo I'm sorry, perhaps I don't understand what are your concerns
> about CompiledPlan, maybe I missed something else, maybe you can look at my
> POC first to see if there is somewhere to worry about.
>
> > Sorry, I forgot to remind you that Timo's concern about the changes to
> the
> > CompiledPlan looks like is still not covered in the FLIP.
>
> @Jing We could have more discussion about naming, but I prefer that the
> naming should be consistent with the DataStream API.
> About aligning splits/partitions/shards, maybe you missed FLIP-217[2] which
> aims to support watermark alignment of source splits.
>
> > After reading the most up-to-date Flip, I didn't find any information if
> > this solution will support aligning splits/partitions/shards [1]. Did I
> > miss anything?
>
> Best
> Kui Yuan
>
> [1] the POC:
> https://github.com/yuchengxin/flink/tree/yuankui/watermark_params
> [2] FLIP-217:
>
>