[jira] [Created] (FLINK-26575) Improve the info message when restoring keyed state backend

[Yun Tang (Jira)]

Yun Tang created FLINK-26575:


 Summary: Improve the info message when restoring keyed state 
backend
 Key: FLINK-26575
 URL: https://issues.apache.org/jira/browse/FLINK-26575
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Yun Tang
Assignee: Yun Tang


Currently, Flink would print the message when starts and finishes to restore 
keyed state backend. For {{KeyGroupStateHandle}}, it would print all the 
offsets, which could be extremely annoying if the state handle contains many 
key groups.


{code:java}
17183 [OverAggregate[4] -> Calc[5] -> SinkConversion[6] -> Sink: Unnamed 
(2/4)#1] INFO  org.apache.flink.runtime.state.heap.HeapRestoreOperation [] - 
Starting to restore from state handle: 
KeyGroupsStateHandle{groupRangeOffsets=KeyGroupRangeOffsets{keyGroupRange=KeyGroupRange{startKeyGroup=8192,
 endKeyGroup=16383}, offsets=[6837, 6877, 6917, 6957, 6997, 7037, 7077, 7117, 
7157, 7197, 7237, 7277, 7317, 7357, 7397, 7437, 7477, 7517, 7557, 7597, 7637, 
7677, 7717, 7757, 7797, 7837, 7877, 7917, 7957, 7997, 8037, 8077, 8117, 8157, 
8197, 8237, 8277, 8317, 8357, 8397, 8437, 8477, 8517, 8557, 8597, 8637, 8677, 
8717, 8757, 8797, 8837, 8877, 8917, 8957, 8997, 9037, 9077, 9117, 9157, 9197, 
9237, 9277, 9317, 9357, 9397, 9437, 9477, 9517, 9557, 9597, 9637, 9677, 9717, 
9757, 9797, 9837, 9877, 9917, 9957, 9997, 10037, 10077, 10117, 10157, 10197, 
10237, 10277, 10317, 10357, 10397, 10437, 10477, 10517, 10557, 10597, 10637, 
10677, 10717, 10757, 10797, 10837, 10877, 10917, 10957, 10997, 11037, 11077, 
7, 11157, 11197, 11237, 11277, 11317, 11357, 11397, 11437, 11477, 11517, 
11557, 11597, 11637, 11677, 11717, 11757, 11797, 11837, 11877, 11917, 11957, 
11997, 12037, 12077, 12117, 12157, 12197, 12237, 12277, 12317, 12357, 12397, 
12437, 12477, 12517, 12557, 12597, 12637, 12677, 12717, 12757, 12797, 12837, 
12877, 12917, 12957, 12997, 13037, 13077, 13117, 13157, 13197, 13237, 13277, 
13317, 13357, 13397, 13437, 13477, 13517, 13557, 13597, 13637, 13677, 13717, 
13757, 13797, 13837, 13877, 13917, 13957, 13997, 14037, 14077, 14117, 14157, 
14197, 14237, 14277, 14317, 14357, 14397, 14437, 14477, 14517, 14557, 14597, 
14637, 14677, 14717, 14757, 14797, 14837, 14877, 14917, 14957, 14997, 15037, 
15077, 15117, 15157, 15197, 15237, 15277, 15317, 15357, 15397, 15437, 15477, 
15517, 15557, 15597, 15637, 15677, 15717, 15757, 15797, 15837, 15877, 15917, 
15957, 15997, 16037, 16077, 16117, 16157, 16197, 16237, 16277, 16317, 16357, 
16397, 16437, 16477, 16517, 16557, 16597, 16637, 16677, 16717, 16757, 16797, 
16837, 16877, 16917, 16957, 16997, 17037, 17077, 17117, 17157, 17197, 17237, 
17277, 17317, 17357, 17397, 17437, 17477, 17517, 17557, 17597, 17637, 17677, 
17717, 17757, 17797, 17837, 17877, 17917, 17957, 17997, 18037, 18077, 18117, 
18157, 18197, 18237, 18277, 18317, 18357, 18397, 18437, 18477, 18517, 18557, 
18597, 18637, 18677, 18717, 18757, 18797, 18837, 18877, 18917, 18957, 18997, 
19037, 19077, 19117, 19157, 19197, 19237, 19277, 19317, 19357, 19397, 19437, 
19477, 19517, 19557, 19597, 19637, 19677, 19717, 19757, 19797, 19837, 19877, 
19917, 19957, 19997, 20037, 20077, 20117, 20157, 20197, 20237, 20277, 20317, 
20357, 20397, 20437, 20477, 20517, 20557, 20597, 20637, 20677, 20717, 20757, 
20797, 20837, 20877, 20917, 20957, 20997, 21037, 21077, 21117, 21157, 21197, 
21237, 21277, 21317, 21357, 21397, 21437, 21477, 21517, 21557, 21597, 21637, 
21677, 21717, 21757, 21797, 21837, 21877, 21917, 21957, 21997, 22037, 22077, 
22117, 22157, 22197, 22237, 22277, 22317, 22357, 22397, 22437, 22477, 22517, 
22557, 22597, 22637, 22677, 22717, 22757, 22797, 22837, 22877, 22917, 22957, 
22997, 23037, 23077, 23117, 23157, 23197, 23237, 23277, 23317, 23357, 23397, 
23437, 23477, 23517, 23557, 23597, 23637, 23677, 23717, 23757, 23797, 23837, 
23877, 23917, 23957, 23997, 24037, 24077, 24117, 24157, 24197, 24237, 24277, 
24317, 24357, 24397, 24437, 24477, 24517, 24557, 24597, 24637, 24677, 24717, 
24757, 24797, 24837, 24877, 24917, 24957, 24997, 25037, 25077, 25117, 25157, 
25197, 25237, 25277, 25317, 25357, 25397, 25437, 25477, 25517, 25557, 25597, 
25637, 25677, 25717, 25757, 25797, 25837, 25877, 25917, 25957, 25997, 26037, 
26077, 26117, 26157, 26197, 26237, 26277, 26317, 26357, 26397, 26437, 26477, 
26517, 26557, 26597, 26637, 26677, 26717, 26757, 26797, 26837, 26877, 26917, 
26957, 26997, 27037, 27077, 27117, 27157, 27197, 27237, 27277, 27317, 27357, 
27397, 27437, 27477, 27517, 27557, 27597, 27637, 27677, 27717, 27757, 27797, 
27837, 27877, 27917, 27957, 27997, 28037, 28077, 28117, 28157, 28197, 28237, 
28277, 28317, 28357, 28397, 28437, 28477, 28517, 28557, 28597, 28637, 28677, 
28717, 28757, 28797, 28837, 28877, 28917, 28957, 28997, 29037, 29077, 29117, 
29157, 29197, 29237, 29277, 29317, 29357, 29397, 29437, 29477, 29517, 29557, 
29597, 29637, 29677, 29717, 

[jira] [Created] (FLINK-26574) Allow definining Operator configuration in Helm chart Values

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26574:
--

 Summary: Allow definining Operator configuration in Helm chart 
Values
 Key: FLINK-26574
 URL: https://issues.apache.org/jira/browse/FLINK-26574
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Reporter: Gyula Fora


Currently the Helm chart hardodes the flink-operator-config configmap that 
contains the operator configurations (logging and yaml).

We should allow the users to define the contents of these config files directly 
in the helm Values yaml.

Also we should probably rename the flink-conf.yaml config key to 
operator-conf.yaml to not confuse it with flink accidentally.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26573) ChangelogPeriodicMaterializationRescaleITCase.testRescaleIn failed on azure

[Yun Gao (Jira)]

Yun Gao created FLINK-26573:
---

 Summary: 
ChangelogPeriodicMaterializationRescaleITCase.testRescaleIn failed on azure
 Key: FLINK-26573
 URL: https://issues.apache.org/jira/browse/FLINK-26573
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Affects Versions: 1.15.0
Reporter: Yun Gao



{code:java}
Mar 09 18:27:45 [INFO] Running 
org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationRescaleITCase
Mar 09 18:27:58 [ERROR] Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 12.81 s <<< FAILURE! - in 
org.apache.flink.test.checkpointing.ChangelogPeriodicMaterializationRescaleITCase
Mar 09 18:27:58 [ERROR] 
ChangelogPeriodicMaterializationRescaleITCase.testRescaleIn  Time elapsed: 
5.292 s  <<< ERROR!
Mar 09 18:27:58 org.apache.flink.runtime.JobException: Recovery is suppressed 
by FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=0, 
backoffTimeMS=0)
Mar 09 18:27:58 at 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:138)
Mar 09 18:27:58 at 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:82)
Mar 09 18:27:58 at 
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:301)
Mar 09 18:27:58 at 
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:291)
Mar 09 18:27:58 at 
org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:282)
Mar 09 18:27:58 at 
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:739)
Mar 09 18:27:58 at 
org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:78)
Mar 09 18:27:58 at 
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:443)
Mar 09 18:27:58 at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown 
Source)
Mar 09 18:27:58 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 09 18:27:58 at java.lang.reflect.Method.invoke(Method.java:498)
Mar 09 18:27:58 at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$handleRpcInvocation$1(AkkaRpcActor.java:304)
Mar 09 18:27:58 at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
Mar 09 18:27:58 at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:302)
Mar 09 18:27:58 at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:217)
Mar 09 18:27:58 at 
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:78)
Mar 09 18:27:58 at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:163)
Mar 09 18:27:58 at 
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24)
Mar 09 18:27:58 at 
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20)
Mar 09 18:27:58 at 
scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
Mar 09 18:27:58 at 
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
Mar 09 18:27:58 at 
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20)
Mar 09 18:27:58 at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
Mar 09 18:27:58 at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
Mar 09 18:27:58 at 
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
Mar 09 18:27:58 at akka.actor.Actor.aroundReceive(Actor.scala:537)
Mar 09 18:27:58 at akka.actor.Actor.aroundReceive$(Actor.scala:535)
Mar 09 18:27:58 at 
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220)
Mar 09 18:27:58 at 
akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
Mar 09 18:27:58 at akka.actor.ActorCell.invoke(ActorCell.scala:548)
Mar 09 18:27:58 at 
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
Mar 09 18:27:58 at akka.dispatch.Mailbox.run(Mailbox.scala:231)
Mar 09 18:27:58 at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
Mar 09 18:27:58 at 
java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
Mar 09 18:27:58 at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)

{code}

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32774=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=5593



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26572) Re-schedule reconcile more often until job is in ready state

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26572:
--

 Summary: Re-schedule reconcile more often until job is in ready 
state
 Key: FLINK-26572
 URL: https://issues.apache.org/jira/browse/FLINK-26572
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Reporter: Gyula Fora


Currently we only use 2 reconcile reschedule configs.  One that is specific to 
when the port is ready but we need to reschedule one more time (this is 10 
seconds by default) and another general reconcile reschedule delay (60 seconds)

We should introduce another setting to use when the job is in a deploying or 
savepointing state to allow for reaching the READY status faster.

We could call it: 
operator.observer.progress-check.interval.sec
or
operator.observer.operation-check.interval.sec

I suggest to use 10 seconds here by default also.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26571) Savepoint trigger/tracking improvements

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26571:
--

 Summary: Savepoint trigger/tracking improvements
 Key: FLINK-26571
 URL: https://issues.apache.org/jira/browse/FLINK-26571
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Reporter: Gyula Fora


This Jira covers a few small fixes / improvements we should make to the manual 
savepoint trigger/tracking logic:

 - JobSpec.savepointTriggerNonce should be Long instead long with null default 
value. 

 - SavepointInfo.triggerTimestamp should be Long type and nulled out together 
with triggerId when savepoint is complete



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26570) Remote module configuration interpolation

[Fil Karnicki (Jira)]

Fil Karnicki created FLINK-26570:


 Summary: Remote module configuration interpolation
 Key: FLINK-26570
 URL: https://issues.apache.org/jira/browse/FLINK-26570
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Fil Karnicki


Add the ability for users to provide placeholders in module.yaml, e.g.
{code:java}
kind: com.foo.bar/test
spec:
  something: ${REPLACE_ME}
  transport:
password: ${REPLACE_ME_WITH_A_SECRET}
array:
  - ${REPLACE_ME}
  - sthElse {code}
These placeholders would be resolved in 

org.apache.flink.statefun.flink.core.jsonmodule.RemoteModule#bindComponent

using 
{code:java}
ParameterTool.fromSystemProperties().mergeWith(ParameterTool.fromMap(globalConfiguration)
 {code}
by traversing the ComponentJsonObject.specJsonNode() and replacing values that 
contain placeholders with values from the combined system+globalConfig map



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26569) testResumeWithWrongOffset(org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriterTest) failing on master

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-26569:
-

 Summary: 
testResumeWithWrongOffset(org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriterTest)
 failing on master
 Key: FLINK-26569
 URL: https://issues.apache.org/jira/browse/FLINK-26569
 Project: Flink
  Issue Type: Bug
  Components: Connectors / FileSystem, Connectors / Hadoop Compatibility
Affects Versions: 1.15.0
Reporter: Matthias Pohl
 Fix For: 1.15.0


There's a test failure in [this 
build|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32792=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=bfbc6239-57a0-5db0-63f3-41551b4f7d51=10107]
 due to {{HadoopRecoverableWriterTest.testResumeWithWrongOffset}}:
{code}
Mar 10 00:46:04 [ERROR] 
testResumeWithWrongOffset(org.apache.flink.runtime.fs.hdfs.HadoopRecoverableWriterTest)
  Time elapsed: 100.39 s  <<< FAILURE!
Mar 10 00:46:04 java.lang.AssertionError
Mar 10 00:46:04 at org.junit.Assert.fail(Assert.java:86)
Mar 10 00:46:04 at org.junit.Assert.fail(Assert.java:95)
Mar 10 00:46:04 at 
org.apache.flink.core.fs.AbstractRecoverableWriterTest.testResumeWithWrongOffset(AbstractRecoverableWriterTest.java:381)
Mar 10 00:46:04 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
Mar 10 00:46:04 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Mar 10 00:46:04 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 10 00:46:04 at java.lang.reflect.Method.invoke(Method.java:498)
Mar 10 00:46:04 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Mar 10 00:46:04 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Mar 10 00:46:04 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
Mar 10 00:46:04 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Mar 10 00:46:04 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Mar 10 00:46:04 at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
Mar 10 00:46:04 at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Mar 10 00:46:04 at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
Mar 10 00:46:04 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Mar 10 00:46:04 at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
Mar 10 00:46:04 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
Mar 10 00:46:04 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
Mar 10 00:46:04 at 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
Mar 10 00:46:04 at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
Mar 10 00:46:04 at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
Mar 10 00:46:04 at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
Mar 10 00:46:04 at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
Mar 10 00:46:04 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Mar 10 00:46:04 at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
Mar 10 00:46:04 at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
Mar 10 00:46:04 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
Mar 10 00:46:04 at 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
Mar 10 00:46:04 at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
Mar 10 00:46:04 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
Mar 10 00:46:04 at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
Mar 10 00:46:04 at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
Mar 10 00:46:04 at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
Mar 10 00:46:04 at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
Mar 10 00:46:04 at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
Mar 10 00:46:04 at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Enable scala formatting check

[Jark Wu]

I also have some concerns because it's a huge change and the 1.15 will be
released soon.
I remember the last time when we merged Java formatting, and all the
pending bugfix PRs
needed to be refactored. I'm afraid this may delay the 1.15 release.  I
guess the scala formatting
is a nice-to-have feature, so we don't need to rush into 1.15.

Best,
Jark

On Thu, 10 Mar 2022 at 14:24, Yun Tang  wrote:

> The feature freeze is already passed, and we just not cut the release
> branch yet.
> I am a bit against the idea to include this in Flink-1.15 as this is
> really a feature change to build system not to mentition it includes 118k
> lines change!
>
> Thanks
> Yun Tang
>
> On 2022/03/09 16:48:10 Francesco Guardiani wrote:
> > It would be nice to merge it before the release branch cut, but I'm not
> > sure we're on time for that...
> >
> > On Wed, Mar 9, 2022 at 4:58 PM Martijn Visser 
> > wrote:
> >
> > > I think it would actually be better to merge it before the release
> branch
> > > is cut to avoid potential issues when needing to backport bugfixes?
> > >
> > > Thanks, Martijn
> > >
> > > On Wed, 9 Mar 2022 at 16:55, Seth Wiesman  wrote:
> > >
> > > > Happy to help get this merged.
> > > >
> > > > Do we want to wait until the 1.15 branch is cut? The change is mostly
> > > > trivial (reformatting) but does make changes to the build system.
> > > >
> > > > Seth
> > > >
> > > > On Wed, Mar 9, 2022 at 9:45 AM Francesco Guardiani <
> > > > france...@ververica.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > > I've been spending some time prototyping a scalafmt conf, which
> doesn't
> > > > > look too different from our java style and tries to keep the same
> > > > > properties from our scalastyle conf. Here is the PR:
> > > > > https://github.com/apache/flink/pull/19025
> > > > >
> > > > > In particular, this is the scalafmt config commit:
> > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/flink/pull/19025/commits/cb32893df4b554e4526324c43c86681cc9fe8169
> > > > > And this is the commit removing scalastyle:
> > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/flink/pull/19025/commits/9ffe7d52e3368c5c40f15e3dc48f6d81691a8dd0
> > > > >
> > > > > I need some committer to pair with to merge the big PR, any
> volunteers?
> > > > :)
> > > > >
> > > > > After we merge it I will also update the contributor guide doc to
> > > remove
> > > > > scalastyle.
> > > > >
> > > > > FG
> > > > >
> > > > > On Tue, Mar 8, 2022 at 10:07 AM David Anderson <
> dander...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > For flink-training we initially tried cloning the scalastyle
> setup
> > > from
> > > > > > flink, but we decided to use spotless + scalafmt instead.
> > > > > >
> > > > > > David
> > > > > >
> > > > > > On Mon, Mar 7, 2022 at 1:12 PM Timo Walther 
> > > > wrote:
> > > > > >
> > > > > > > Big +1
> > > > > > >
> > > > > > > This will improve the contribution experience. Even though we
> > > stopped
> > > > > > > adding more Scala code, it is still necessary from time to
> time.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Timo
> > > > > > >
> > > > > > > Am 02.03.22 um 09:29 schrieb 刘首维:
> > > > > > > > +1
> > > > > > > >
> > > > > > > >
> > > > > > > > I still remember my first pr. Lack of experience, I had to
> pay
> > > > > > attention
> > > > > > > to Scala code format and corrected the format manually, which
> made
> > > > me a
> > > > > > > littleembarrassed(though I'm a big fan of Scala). I think
> > > this
> > > > > > > proposal will lighten the burden of writing Scala code.
> > > > > > > >
> > > > > > > >
> > > > > > > > Shouwei Liu
> > > > > > > >
> > > > > > > >
> > > > > > > > --原始邮件--
> > > > > > > > 发件人:
> > > > > > > "dev"
> > > > > > >
><
> > > > > > > kna...@apache.org;
> > > > > > > > 发送时间:2022年3月2日(星期三) 下午3:01
> > > > > > > > 收件人:"dev" > > > > > > >
> > > > > > > > 主题:Re: [DISCUSS] Enable scala formatting check
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > +1 I've never written any Scala in Flink, but this makes a
> lot of
> > > > > sense
> > > > > > > to
> > > > > > > > me. Converging on a smaller set of tools and simplifying the
> > > build
> > > > is
> > > > > > > > always a good idea and the Community already concluded before
> > > that
> > > > > > > spotless
> > > > > > > > is generally a good approach.
> > > > > > > >
> > > > > > > > On Tue, Mar 1, 2022 at 5:52 PM Francesco Guardiani <
> > > > > > > france...@ververica.com
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >  Hi all,
> > > > > > > > 
> > > > > > > >  I want to propose to enable the spotless scalafmt
> > > integration
> > > > > and
> > > > > > > remove
> > > > > > > >  the scalastyle plugin.
> > > > > > > > 
> > > > > > > >  From an initial analysis, scalafmt can do everything
> > > > scalastyle
> > > > > > can
> > > > > > > do, and
> > > > > > > >  the integration 

[jira] [Created] (FLINK-26568) BlockingShuffleITCase.testDeletePartitionFileOfBoundedBlockingShuffle timing out on Azure

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-26568:
-

 Summary: 
BlockingShuffleITCase.testDeletePartitionFileOfBoundedBlockingShuffle timing 
out on Azure
 Key: FLINK-26568
 URL: https://issues.apache.org/jira/browse/FLINK-26568
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task, Tests
Affects Versions: 1.15.0
Reporter: Matthias Pohl
 Fix For: 1.15.0


[This 
build|https://dev.azure.com/mapohl/flink/_build/results?buildId=845=logs=0a15d512-44ac-5ba5-97ab-13a5d066c22c=9a028d19-6c4b-5a4e-d378-03fca149d0b1=12865]
 timed out due the test 
{{BlockingShuffleITCase.testDeletePartitionFileOfBoundedBlockingShuffle}} not 
finishing.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Enable scala formatting check

[Yun Tang]

The feature freeze is already passed, and we just not cut the release branch 
yet.
I am a bit against the idea to include this in Flink-1.15 as this is really a 
feature change to build system not to mentition it includes 118k lines change!

Thanks
Yun Tang

On 2022/03/09 16:48:10 Francesco Guardiani wrote:
> It would be nice to merge it before the release branch cut, but I'm not
> sure we're on time for that...
> 
> On Wed, Mar 9, 2022 at 4:58 PM Martijn Visser 
> wrote:
> 
> > I think it would actually be better to merge it before the release branch
> > is cut to avoid potential issues when needing to backport bugfixes?
> >
> > Thanks, Martijn
> >
> > On Wed, 9 Mar 2022 at 16:55, Seth Wiesman  wrote:
> >
> > > Happy to help get this merged.
> > >
> > > Do we want to wait until the 1.15 branch is cut? The change is mostly
> > > trivial (reformatting) but does make changes to the build system.
> > >
> > > Seth
> > >
> > > On Wed, Mar 9, 2022 at 9:45 AM Francesco Guardiani <
> > > france...@ververica.com>
> > > wrote:
> > >
> > > > Hi all,
> > > > I've been spending some time prototyping a scalafmt conf, which doesn't
> > > > look too different from our java style and tries to keep the same
> > > > properties from our scalastyle conf. Here is the PR:
> > > > https://github.com/apache/flink/pull/19025
> > > >
> > > > In particular, this is the scalafmt config commit:
> > > >
> > > >
> > >
> > https://github.com/apache/flink/pull/19025/commits/cb32893df4b554e4526324c43c86681cc9fe8169
> > > > And this is the commit removing scalastyle:
> > > >
> > > >
> > >
> > https://github.com/apache/flink/pull/19025/commits/9ffe7d52e3368c5c40f15e3dc48f6d81691a8dd0
> > > >
> > > > I need some committer to pair with to merge the big PR, any volunteers?
> > > :)
> > > >
> > > > After we merge it I will also update the contributor guide doc to
> > remove
> > > > scalastyle.
> > > >
> > > > FG
> > > >
> > > > On Tue, Mar 8, 2022 at 10:07 AM David Anderson 
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > For flink-training we initially tried cloning the scalastyle setup
> > from
> > > > > flink, but we decided to use spotless + scalafmt instead.
> > > > >
> > > > > David
> > > > >
> > > > > On Mon, Mar 7, 2022 at 1:12 PM Timo Walther 
> > > wrote:
> > > > >
> > > > > > Big +1
> > > > > >
> > > > > > This will improve the contribution experience. Even though we
> > stopped
> > > > > > adding more Scala code, it is still necessary from time to time.
> > > > > >
> > > > > > Regards,
> > > > > > Timo
> > > > > >
> > > > > > Am 02.03.22 um 09:29 schrieb 刘首维:
> > > > > > > +1
> > > > > > >
> > > > > > >
> > > > > > > I still remember my first pr. Lack of experience, I had to pay
> > > > > attention
> > > > > > to Scala code format and corrected the format manually, which made
> > > me a
> > > > > > littleembarrassed(though I'm a big fan of Scala). I think
> > this
> > > > > > proposal will lighten the burden of writing Scala code.
> > > > > > >
> > > > > > >
> > > > > > > Shouwei Liu
> > > > > > >
> > > > > > >
> > > > > > > --原始邮件--
> > > > > > > 发件人:
> > > > > > "dev"
> > > > > >   <
> > > > > > kna...@apache.org;
> > > > > > > 发送时间:2022年3月2日(星期三) 下午3:01
> > > > > > > 收件人:"dev" > > > > > >
> > > > > > > 主题:Re: [DISCUSS] Enable scala formatting check
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > +1 I've never written any Scala in Flink, but this makes a lot of
> > > > sense
> > > > > > to
> > > > > > > me. Converging on a smaller set of tools and simplifying the
> > build
> > > is
> > > > > > > always a good idea and the Community already concluded before
> > that
> > > > > > spotless
> > > > > > > is generally a good approach.
> > > > > > >
> > > > > > > On Tue, Mar 1, 2022 at 5:52 PM Francesco Guardiani <
> > > > > > france...@ververica.com
> > > > > > > wrote:
> > > > > > >
> > > > > > >  Hi all,
> > > > > > > 
> > > > > > >  I want to propose to enable the spotless scalafmt
> > integration
> > > > and
> > > > > > remove
> > > > > > >  the scalastyle plugin.
> > > > > > > 
> > > > > > >  From an initial analysis, scalafmt can do everything
> > > scalastyle
> > > > > can
> > > > > > do, and
> > > > > > >  the integration with spotless looks easy to enable:
> > > > > > > 
> > > > https://github.com/diffplug/spotless/tree/main/plugin-maven#scala
> > > > > .
> > > > > > The
> > > > > > >  scalafmt conf file gets picked up automatically from every
> > > IDE,
> > > > > and
> > > > > > it can
> > > > > > >  be heavily tuned.
> > > > > > > 
> > > > > > >  This way we can unify the formatting and integrate with our
> > CI
> > > > > > without any
> > > > > > >  additional configurations. And we won't need scalastyle
> > > anymore,
> > > > > as
> > > > > > >  scalafmt will take care of the checks:
> > > > > > > 
> > > > > > >  * mvn spotless:check will check both java and scala

[jira] [Created] (FLINK-26567) FileStoreSourceSplitReader should deal with value count

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-26567:


 Summary: FileStoreSourceSplitReader should deal with value count
 Key: FLINK-26567
 URL: https://issues.apache.org/jira/browse/FLINK-26567
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: table-store-0.1.0


There is a keyAsRecord in FileStoreSourceSplitReader, but this should only be 
keyAsRecord.

When keyAsRecord, it should emit the same number of records as value count.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26566) FlinkKafkaProducerITCase.testFailAndRecoverSameCheckpointTwice failed on azure

[Yun Gao (Jira)]

Yun Gao created FLINK-26566:
---

 Summary: 
FlinkKafkaProducerITCase.testFailAndRecoverSameCheckpointTwice failed on azure
 Key: FLINK-26566
 URL: https://issues.apache.org/jira/browse/FLINK-26566
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.14.3
Reporter: Yun Gao



{code:java}
Mar 09 20:11:28 [ERROR] Tests run: 15, Failures: 1, Errors: 0, Skipped: 0, Time 
elapsed: 274.396 s <<< FAILURE! - in 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
Mar 09 20:11:28 [ERROR] testFailAndRecoverSameCheckpointTwice  Time elapsed: 
74.511 s  <<< FAILURE!
Mar 09 20:11:28 java.lang.AssertionError: Expected elements: <[42, 43]>, but 
was: elements: <[42, 43, 42, 43, 42, 43, 42, 43]>
Mar 09 20:11:28 at org.junit.Assert.fail(Assert.java:89)
Mar 09 20:11:28 at 
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.assertExactlyOnceForTopic(KafkaTestBase.java:331)
Mar 09 20:11:28 at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.testFailAndRecoverSameCheckpointTwice(FlinkKafkaProducerITCase.java:316)
Mar 09 20:11:28 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
Mar 09 20:11:28 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Mar 09 20:11:28 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Mar 09 20:11:28 at java.lang.reflect.Method.invoke(Method.java:498)
Mar 09 20:11:28 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
Mar 09 20:11:28 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Mar 09 20:11:28 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
Mar 09 20:11:28 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Mar 09 20:11:28 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Mar 09 20:11:28 at 
org.apache.flink.testutils.junit.RetryRule$RetryOnFailureStatement.evaluate(RetryRule.java:135)
Mar 09 20:11:28 at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
Mar 09 20:11:28 at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
Mar 09 20:11:28 at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
Mar 09 20:11:28 at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
Mar 09 20:11:28 at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
Mar 09 20:11:28 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
Mar 09 20:11:28 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
Mar 09 20:11:28 at 
org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
Mar 09 20:11:28 at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
Mar 09 20:11:28 at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
Mar 09 20:11:28 at 
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
Mar 09 20:11:28 at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
Mar 09 20:11:28 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Mar 09 20:11:28 at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32778=logs=c5f0071e-1851-543e-9a45-9ac140befc32=15a22db7-8faa-5b34-3920-d33c9f0ca23c=7412



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26565) Use lateTrigger when the window maxtimestap of data is less than currentwatermark and it is not discarded because the allow latency parameter

[hehuiyuan (Jira)]

hehuiyuan created FLINK-26565:
-

 Summary: Use lateTrigger when the window maxtimestap of data is 
less than currentwatermark and  it is not discarded because the allow latency 
parameter 
 Key: FLINK-26565
 URL: https://issues.apache.org/jira/browse/FLINK-26565
 Project: Flink
  Issue Type: Improvement
Reporter: hehuiyuan
 Attachments: image-2022-03-10-11-27-52-891.png

Use lateTrigger when the window maxtimestap of data is less than 
currentwatermark and  it is not discarded because the allow latency parameter.

!image-2022-03-10-11-27-52-891.png|width=543,height=318!



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26564) CompactCoordinatorStateHandler doesn't properly handle the cleanup-in-progress requests.

[Gen Luo (Jira)]

Gen Luo created FLINK-26564:
---

 Summary: CompactCoordinatorStateHandler doesn't properly handle 
the cleanup-in-progress requests.
 Key: FLINK-26564
 URL: https://issues.apache.org/jira/browse/FLINK-26564
 Project: Flink
  Issue Type: Bug
  Components: Connectors / FileSystem
Affects Versions: 1.15.0
Reporter: Gen Luo
 Fix For: 1.15.0


It is found in FLINK-26322 that the CompactCoordinatorStateHandler doesn't 
properly handle the cleanup-in-progress requests but submit them as compacting 
requests. The issue happens when a job with compaction enabled is 
stop-with-savepoint and restarted with compaction disabled.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26563) HadoopS3RecoverableWriterITCase hang on azure

[Yun Gao (Jira)]

Yun Gao created FLINK-26563:
---

 Summary: HadoopS3RecoverableWriterITCase hang on azure
 Key: FLINK-26563
 URL: https://issues.apache.org/jira/browse/FLINK-26563
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Affects Versions: 1.15.0
Reporter: Yun Gao



{code:java}
2022-03-09T09:44:11.6454998Z Mar 09 09:44:11 "main" #1 prio=5 os_prio=0 
tid=0x7f331000b800 nid=0x7601 runnable [0x7f3318203000]
2022-03-09T09:44:11.6455475Z Mar 09 09:44:11java.lang.Thread.State: RUNNABLE
2022-03-09T09:44:11.6455962Z Mar 09 09:44:11at 
java.net.SocketInputStream.socketRead0(Native Method)
2022-03-09T09:44:11.6456563Z Mar 09 09:44:11at 
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
2022-03-09T09:44:11.6457422Z Mar 09 09:44:11at 
java.net.SocketInputStream.read(SocketInputStream.java:171)
2022-03-09T09:44:11.6458036Z Mar 09 09:44:11at 
java.net.SocketInputStream.read(SocketInputStream.java:141)
2022-03-09T09:44:11.6458667Z Mar 09 09:44:11at 
sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
2022-03-09T09:44:11.6459649Z Mar 09 09:44:11at 
sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:237)
2022-03-09T09:44:11.6460672Z Mar 09 09:44:11at 
sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:190)
2022-03-09T09:44:11.6461267Z Mar 09 09:44:11at 
sun.security.ssl.SSLTransport.decode(SSLTransport.java:109)
2022-03-09T09:44:11.6462110Z Mar 09 09:44:11at 
sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1386)
2022-03-09T09:44:11.6463039Z Mar 09 09:44:11at 
sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1354)
2022-03-09T09:44:11.6464168Z Mar 09 09:44:11at 
sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
2022-03-09T09:44:11.6465097Z Mar 09 09:44:11at 
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:948)
2022-03-09T09:44:11.6466100Z Mar 09 09:44:11at 
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
2022-03-09T09:44:11.6467149Z Mar 09 09:44:11at 
org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:197)
2022-03-09T09:44:11.6468145Z Mar 09 09:44:11at 
org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:176)
2022-03-09T09:44:11.6469179Z Mar 09 09:44:11at 
org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:135)
2022-03-09T09:44:11.6470153Z Mar 09 09:44:11at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
2022-03-09T09:44:11.6471929Z Mar 09 09:44:11at 
com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
2022-03-09T09:44:11.6472974Z Mar 09 09:44:11at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
2022-03-09T09:44:11.6474107Z Mar 09 09:44:11at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
2022-03-09T09:44:11.6474724Z Mar 09 09:44:11at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
2022-03-09T09:44:11.6475308Z Mar 09 09:44:11at 
com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
2022-03-09T09:44:11.6475917Z Mar 09 09:44:11at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
2022-03-09T09:44:11.6476713Z Mar 09 09:44:11at 
com.amazonaws.util.LengthCheckInputStream.read(LengthCheckInputStream.java:107)
2022-03-09T09:44:11.6477327Z Mar 09 09:44:11at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
2022-03-09T09:44:11.6478193Z Mar 09 09:44:11at 
com.amazonaws.services.s3.internal.S3AbortableInputStream.read(S3AbortableInputStream.java:125)
2022-03-09T09:44:11.6478853Z Mar 09 09:44:11at 
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:90)
2022-03-09T09:44:11.6479480Z Mar 09 09:44:11at 
org.apache.hadoop.fs.s3a.S3AInputStream.lambda$read$3(S3AInputStream.java:468)
2022-03-09T09:44:11.6480092Z Mar 09 09:44:11at 
org.apache.hadoop.fs.s3a.S3AInputStream$$Lambda$312/577682023.execute(Unknown 
Source)
2022-03-09T09:44:11.6480876Z Mar 09 09:44:11at 
org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
2022-03-09T09:44:11.6481443Z Mar 09 09:44:11at 
org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
2022-03-09T09:44:11.6482012Z Mar 09 09:44:11at 
org.apache.hadoop.fs.s3a.Invoker$$Lambda$289/1921012072.execute(Unknown Source)
2022-03-09T09:44:11.6482581Z Mar 09 09:44:11at 
org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
2022-03-09T09:44:11.6483146Z Mar 09 09:44:11at 
org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
2022-03-09T09:44:11.6483690Z Mar 09 09:44:11at 

[jira] [Created] (FLINK-26562) Introduce table path option

[Jane Chan (Jira)]

Jane Chan created FLINK-26562:
-

 Summary: Introduce table path option
 Key: FLINK-26562
 URL: https://issues.apache.org/jira/browse/FLINK-26562
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Affects Versions: 0.1.0
Reporter: Jane Chan


Currently, the {{FileStoreOptions}} only has the {{FILE_PATH}} option as the 
table store root dir, we should have another {{TABLE_PATH}} for generated 
{{sst/manifest/snapshot}} per table.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26561) Deployment Status cannot be updated

[leinenglian (Jira)]

leinenglian created FLINK-26561:
---

 Summary: Deployment Status cannot be updated
 Key: FLINK-26561
 URL: https://issues.apache.org/jira/browse/FLINK-26561
 Project: Flink
  Issue Type: Bug
 Environment: k8s version: 1.20

flink version: 1.14.3
Reporter: leinenglian


I'd like to know when deployment  status will change to ready, beacuse i see  
deployment  status  is MISSING

flink operator log is always printing this info:

2022-03-10 02:35:46,774 o.a.f.k.o.s.FlinkService       [DEBUG] Creating 
RestClusterClient(http://xx:8081)

2022-03-10 02:35:46,774 o.a.f.r.r.RestClient           [DEBUG] Rest client 
endpoint started.

2022-03-10 02:35:46,774 o.a.f.r.r.RestClient           [DEBUG] Shutting down 
rest endpoint.

2022-03-10 02:35:46,774 o.a.f.r.r.RestClient           [DEBUG] Rest endpoint 
shutdown complete.

2022-03-10 02:35:46,775 o.a.f.k.o.o.Observer           [INFO ] JobManager 
deployment xx in namespace flink-operator port ready, waiting for the REST 
API...

2022-03-10 02:35:46,775 i.j.o.p.e.ReconciliationDispatcher [DEBUG] Updating 
resource: 6b6d6f59-b2db-4490-abe5-9d5843268653 with version: 52817831

2022-03-10 02:35:46,775 i.j.o.p.e.ReconciliationDispatcher [DEBUG] Trying to 
replace resource xx, version: 52817831

2022-03-10 02:35:46,781 i.f.k.c.i.c.Reflector          [DEBUG] Event received 
MODIFIED FlinkDeployment resourceVersion 52817832

2022-03-10 02:35:46,781 i.j.o.p.e.s.c.ControllerResourceEventSource [DEBUG] 
Event received for resource:xx

2022-03-10 02:35:46,781 i.j.o.p.e.EventProcessor       [DEBUG] Received event: 
ResourceEvent\{action=UPDATED, associated resource 
id=CustomResourceID{name='xx, namespace='flink-operator'}}

2022-03-10 02:35:46,781 i.j.o.p.e.EventProcessor       [DEBUG] Skipping 
executing controller for resource id: CustomResourceID\{name='xx', 
namespace='flink-operator'}. Controller in execution: true. Latest Resource 
present: true



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26560) make the threshold of the overlap fraction of incremental restoring configurable

[Yanfei Lei (Jira)]

Yanfei Lei created FLINK-26560:
--

 Summary: make the threshold of the overlap fraction of incremental 
restoring configurable
 Key: FLINK-26560
 URL: https://issues.apache.org/jira/browse/FLINK-26560
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / State Backends
Affects Versions: 1.15.0
Reporter: Yanfei Lei


Currently, the threshold of the overlap fraction of incremental restoring 
`OVERLAP_FRACTION_THRESHOLD` is a hard-coded, fixed value.

 
{code:java}
public class RocksDBIncrementalCheckpointUtils {

/**
 * The threshold of the overlap fraction of the handle's key-group range 
with target key-group
 * range to be an initial handle.
 */
private static final double OVERLAP_FRACTION_THRESHOLD = 0.75;
...
} {code}
 

`OVERLAP_FRACTION_THRESHOLD` is used to control how to restore a state handle, 
different thresholds can affect the performance of restoring. The behavior of 
deletion in restoring has been changed after FLINK-21321, the old threshold no 
longer fits the current situation.

To make it easier to modify the threshold according to different situations, 
changing `OVERLAP_FRACTION_THRESHOLD` to be configurable is suggested.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26559) Support Customized Kubernetes Schedulers for Flink Kubernetes

[bo zhao (Jira)]

bo zhao created FLINK-26559:
---

 Summary: Support Customized Kubernetes Schedulers for Flink 
Kubernetes
 Key: FLINK-26559
 URL: https://issues.apache.org/jira/browse/FLINK-26559
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes
Reporter: bo zhao


This is an umbrella issue for tracking the work for supporting Volcano & 
Yunikorn on FLink Kubernetes. These schedulers provide more YARN like features 
(such as queues and minimum resources before scheduling jobs) that many folks 
want on Kubernetes.

 

Yunikorn is an ASF project & Volcano is a CNCF project (sig-batch).

 

They've taken slightly different approaches to solving the same problem, but 
there must have the same high level logic code in Flink.

 

DISCUSSION: TBD

Design DOC: [DRAFT] 
https://docs.google.com/document/d/1n0ZF4ssM8syLhMy1XAiv9LiU3Ig7udOeWaKi60mXQV8/edit?usp=sharing

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26558) Errors when reporting on Job status

[Jeremy DeGroot (Jira)]

Jeremy DeGroot created FLINK-26558:
--

 Summary: Errors when reporting on Job status
 Key: FLINK-26558
 URL: https://issues.apache.org/jira/browse/FLINK-26558
 Project: Flink
  Issue Type: Bug
  Components: Runtime / REST
Affects Versions: 1.14.3
Reporter: Jeremy DeGroot


This error is showing up very frequently in my JobManager logs since I upgraded 
from 1.14.2 to 1.14.3. The Flink Rest dashboard also fails to load either 
Running or Completed jobs when this happens.

The Job Managers are HA, running on Kubernetes. The Task managers are running 
on Kubernetes as well, and were also upgraded to 1.14.3.

 

Please advise

 
{{2022-03-09 22:12:40,925 ERROR 
org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler [] - Unhandled 
exception.}}
{{org.apache.flink.runtime.rpc.akka.exceptions.AkkaRpcException: Failed to 
serialize the result for RPC call : requestMultipleJobDetails.}}
{{at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.serializeRemoteResultAndVerifySize(AkkaRpcActor.java:417)
 ~[?:?]}}
{{at 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.lambda$sendAsyncResponse$2(AkkaRpcActor.java:373)
 ~[?:?]}}
{{at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) ~[?:?]}}
{{at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) 
~[?:?]}}
{{at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) 
~[?:?]}}
{{at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]}}
{{at 
org.apache.flink.util.concurrent.FutureUtils$ResultConjunctFuture.handleCompletedFuture(FutureUtils.java:858)
 ~[flink-dist_2.11-1.14.3.jar:1.14.3]}}
{{at 
org.apache.flink.util.concurrent.FutureUtils$ResultConjunctFuture.lambda$new$0(FutureUtils.java:876)
 ~[flink-dist_2.11-1.14.3.jar:1.14.3]}}
{{at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) 
~[?:?]}}
{{at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown 
Source) ~[?:?]}}
{{at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) 
~[?:?]}}
{{at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]}}
{{at 
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$1(AkkaInvocationHandler.java:258)
 ~[?:?]}}
{{at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) 
~[?:?]}}
{{at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown 
Source) ~[?:?]}}
{{at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) 
~[?:?]}}
{{at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]}}
{{at 
org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1389) 
~[flink-dist_2.11-1.14.3.jar:1.14.3]}}
{{at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$null$1(ClassLoadingUtils.java:93)
 ~[?:?]}}
{{at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
 ~[?:?]}}
{{at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
 ~[?:?]}}
{{at java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) 
~[?:?]}}
{{at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown 
Source) ~[?:?]}}
{{at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) 
~[?:?]}}
{{at java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]}}
{{at 
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$1.onComplete(AkkaFutureUtils.java:47)
 ~[?:?]}}
{{at akka.dispatch.OnComplete.internal(Future.scala:300) ~[?:?]}}
{{at akka.dispatch.OnComplete.internal(Future.scala:297) ~[?:?]}}
{{at akka.dispatch.japi$CallbackBridge.apply(Future.scala:224) ~[?:?]}}
{{at akka.dispatch.japi$CallbackBridge.apply(Future.scala:221) ~[?:?]}}
{{at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) 
~[flink-dist_2.11-1.14.3.jar:1.14.3]}}
{{at 
org.apache.flink.runtime.concurrent.akka.AkkaFutureUtils$DirectExecutionContext.execute(AkkaFutureUtils.java:65)
 ~[?:?]}}
{{at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68) 
~[flink-dist_2.11-1.14.3.jar:1.14.3]}}
{{at 
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
 ~[flink-dist_2.11-1.14.3.jar:1.14.3]}}
{{at 
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
 ~[flink-dist_2.11-1.14.3.jar:1.14.3]}}
{{at 
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284) 
~[flink-dist_2.11-1.14.3.jar:1.14.3]}}
{{at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:621) ~[?:?]}}
{{at 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:24)
 ~[?:?]}}
{{at 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
 ~[?:?]}}
{{at 

[jira] [Created] (FLINK-26557) Extend CsvFormat documentation based on release-testing feedback

[Alexander Fedulov (Jira)]

Alexander Fedulov created FLINK-26557:
-

 Summary: Extend CsvFormat documentation based on release-testing 
feedback
 Key: FLINK-26557
 URL: https://issues.apache.org/jira/browse/FLINK-26557
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Alexander Fedulov
 Fix For: 1.15.0


Incorporate feedback about the documentation from CsvFormat release testing:

https://issues.apache.org/jira/browse/FLINK-26311 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Flink's supported APIs and Hive query syntax

[Martijn Visser]

Hi everyone,

Thank you all very much for your input. From my perspective, I consider
batch as a special case of streaming. So with Flink SQL, we can support
both batch and streaming use cases and I think we should use Flink SQL as
our target.

To reply on some of the comments:

@Jing on your remark:
> Since Flink has a clear vision of unified batch and stream processing,
supporting batch jobs will be one of the critical core features to help us
reach the vision and let Flink have an even bigger impact in the industry.

I fully agree with that statement. I do think that having Hive syntax
support doesn't help in that unified batch and stream processing. We're
making it easier for batch users to run their Hive batch jobs on Flink, but
that doesn't fit the "unified" part since it's focussed on batch, while
Flink SQL focusses on batch and streaming. I would have rather invested
time in making batch improvements to Flink and Flink SQL vs investing in
Hive syntax support. I do understand from the given replies that Hive
syntax support is valuable for those that are already running batch
processing and would like to run these queries on Flink. I do think that's
limited to mostly Chinese companies at the moment.

@Jark I think you've provided great input and are spot on with:
> Regarding the maintenance concern you raised, I think that's a good point
and they are in the plan. The Hive dialect has already been a plugin and
option now, and the implementation is located in hive-connector module. We
still need some work to make the Hive dialect purely rely on public APIs,
and the Hive connector should be decopule with table planner. At that time,
we can move the whole Hive connector into a separate repository (I guess
this is also in the externalize connectors plan).

I'm looping in Francesco and Timo who can elaborate more in depth on the
current maintenance issues. I think we need to have a proper plan on how
this tech debt/maintenance can be addressed and to get commitment that this
will be resolved in Flink 1.16, since we indeed need to move out all
previously agreed connectors before Flink 1.16 is released.

> From my perspective, Hive is still widely used and there exists many
running Hive SQL jobs, so why not to provide users a better experience to
help them migrate Hive jobs to Flink? Also, it doesn't conflict with Flink
SQL as it's just a dialect option.

I do think there is a conflict with Flink SQL; you can't use both of them
at the same time, so you don't have access to all features in Flink. That
increases feature sparsity and user friction. It also puts a bigger burden
on the Flink community, because having both options available means more
maintenance work. For example, an upgrade of Calcite is more impactful. The
Flink codebase is already rather large and CI build times are already too
long. More code means more risk of bugs. If a user at some point wants to
change his Hive batch job to a streaming Flink SQL job, there's still
migration work for the user, it just needs to happen at a later stage.

@Jingsong I think you have a good argument that migrating SQL for Batch ETL
is indeed an expensive effort.

Last but not least, there was no one who has yet commented on the supported
Hive versions and security issues. I've reached out to the Hive community
and from the info I've received so far is that only Hive 3.1.x and Hive
2.3.x are still supported. The older Hive versions are no longer maintained
and also don't receive security updates. This is important because many
companies scan the Flink project for vulnerabilities and won't allow using
it when these types of vulnerabilities are included.

My summary would be the following:
* Like Jark said, in the short term, Hive syntax compatibility is the
ticket for us to have a seat in the batch processing. Having improved Hive
syntax support for that in Flink can help in this.
* In the long term, we can and should drop it and focus on Flink SQL itself
both for batch and stream processing.
* The Hive maintainers/volunteers should come up with a plan on how the
tech debt/maintenance with regards to Hive query syntax can be addressed
and will be resolved for Flink 1.16. This includes stuff like using public
APIs and decoupling it from the planner. This is also extremely important
since we want to move out connectors with Flink 1.16 (next Flink release).
I'm hoping that those who can help out with this will chip-in.
* We follow the Hive versions that are still supported, which means we drop
support for Hive 1.*, 2.1.x and 2.2.x and upgrade Hive 2.3 and Hive 3.1 to
the latest version.

Thanks again for your input and looking forward to your thoughts on this.

Best regards,

Martijn

On Tue, 8 Mar 2022 at 10:39, 罗宇侠(莫辞) 
wrote:

> Hi Martijn,
> Thanks for driving this discussion.
>
> About your concerns, I would like to share my opinion.
>
> Actually, more exactly, FLIP-152 [1] is not to extend Flink SQL to support 
> Hive query synax, it provides a Hive dialect 

[jira] [Created] (FLINK-26556) Refactoring MiniCluster and TestingMiniCluster

[Niklas Semmler (Jira)]

Niklas Semmler created FLINK-26556:
--

 Summary: Refactoring MiniCluster and TestingMiniCluster
 Key: FLINK-26556
 URL: https://issues.apache.org/jira/browse/FLINK-26556
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.16.0
Reporter: Niklas Semmler


After working with the MiniCluster & TestingMiniCluster on FLINK-25235, I saw 
some shortcomings that I patched up but did not fix. I would recommend that we 
change it for 1.16, because it is becoming more and more difficult to 
understand what is happening.

*Background*
- We recently changed the implementation of the leader election from a separate 
{{LeaderElectionDriver}} per JobManager component (e.g., dispatcher, rest 
server, etc.) to a combined {{LeaderElectionDriver}} for the JobManager as a 
whole. (For Zoo Keeper based leader election we previously used a 
{{ZooKeeperLeaderElectionDriver}}. Now we use 
{{MultipleComponentLeaderElectionDriverAdapter}} which de-multiplexes the 
leader election for all JobManager components via a single 
{{MultipleComponentLeaderElectionService}} to a single 
{{ZooKeeperMultipleComponentLeaderElectionDriver}} underneath.) This changed 
the mapping between {{HighAvailabilityServices}} and {{LeaderElectionDriver}} 
from a one-to-many to a one-to-one relationship.
- {{HighAvailabilityServices}} is the class responsible for high availability 
(e.g., leader fail-over). However, even though JobManager and {{TaskManager}} 
have a dependecy on this class, not all scenarios require high availability. 
The implementation {{AbstractNonHaServices}} and its {{EmeddedHaServices}} 
(single JVM setup) and {{StandaloneHaServices}} (no support for JobManager 
failures) are used for these scenarios.
- {{MiniCluster}} is the class responsible for managing a single-node Flink 
cluster. It contains a single {{HighAvailabilityServices}} object that is 
shared by the JobManager and multiple {{TaskManager}}.
- {{TestingMiniCluster}} extends the {{MiniCluster}} for test purposes. Among 
others, it is used for tests on leader election between multiple JobManager.

*Issues with MiniCluster*
- The {{MiniCluster}} is meant to "to execute Flink jobs locally". To me this 
means that the {{MiniCluster}} _should_ only use {{EmbeddedHaServices}} as it 
does not need high availability on a single JVM. However, it does not put any 
constraints on the type of high availability services used.
- Although the {{MiniCluster}} is production code. it contains code that is 
meant exclusively for testing. {{MiniCluster#getHaLeadershipControl}} is used 
to give a test explicit control over the leader election of an 
{{EmbeddedHaService}}. Btw, this method depends on {{HighAvailabilityServices}} 
being an {{EmbeddedHaServicesWithLeadershipControl}} object.
- The code to create the {{HighAvailabilityServices}} object seems needlessly 
complex. In {{MiniCluster#createHighAvailabilityServices}} we differentiate 
between {{EmbeddedWithControlHighAvailabilityServices}} and everything else. In 
{{HighAvailabilityServicesUtils}} we distinguish between 
{{EmbeddedHaServices}}, {{ZooKeeperHaServices}} 
&{{ZooKeeperMultipleComponentLeaderElectionHaServices}}, and ones that are 
produced by {{HighAvailabilityServicesFactory}}. It would be nice, if we could 
flatten this into one method. Especially with the additional option to 
configure it via the {{TestingMiniCluster}} (see below). (I also don't 
understand why there is no Kubernetes option. Even though there is a 
{{KubernetesHaServicesFactory}}.) 

*Issues with TestingMiniCluster*
- The name TestingSomething is usually used for mock objects. In contrast, the 
{{TestingMiniCluster}} does not mock anything.
- Its doc string says its used "to set a custom {@link 
HighAvailabilityServices}", but the {{MiniCluster}} already allows it.
- The use of the {{TestingMiniCluster}} seems to be to configure the number of 
{{JobManagers}}, configure the {{HighAvailabilityServices}} (which looks 
redundant see below), and to configure the {{TaskManager}} to use only local 
communication even when more than one {{TaskManager}} exists. Why can this not 
be optional settings on the MiniCluster itself?
- The {{TestingMiniCluster}} is used by tests that need multiple JobManager 
with separate leader election (e.g., ZooKeeperLeaderElectionITCase) and some 
that require that all parts including the {{TaskExecutor}} share the same 
{{HighAvailabilityServices}} (e.g., JobExecutionITCase).
- {{TestingMiniCluster}} allows overriding the method 
{{MiniCluster#createHighAvailabilityServices}} for creating a 
{{HighAvailabilityServices}}. However, that method already has a two step 
process of creating the {{HighAvailabilityServices}}. The existing process even 
includes the option of using a custom factory. Again, this redundancy makes the 
code hard to understand.

Re: [DISCUSS] Enable scala formatting check

[Francesco Guardiani]

It would be nice to merge it before the release branch cut, but I'm not
sure we're on time for that...

On Wed, Mar 9, 2022 at 4:58 PM Martijn Visser 
wrote:

> I think it would actually be better to merge it before the release branch
> is cut to avoid potential issues when needing to backport bugfixes?
>
> Thanks, Martijn
>
> On Wed, 9 Mar 2022 at 16:55, Seth Wiesman  wrote:
>
> > Happy to help get this merged.
> >
> > Do we want to wait until the 1.15 branch is cut? The change is mostly
> > trivial (reformatting) but does make changes to the build system.
> >
> > Seth
> >
> > On Wed, Mar 9, 2022 at 9:45 AM Francesco Guardiani <
> > france...@ververica.com>
> > wrote:
> >
> > > Hi all,
> > > I've been spending some time prototyping a scalafmt conf, which doesn't
> > > look too different from our java style and tries to keep the same
> > > properties from our scalastyle conf. Here is the PR:
> > > https://github.com/apache/flink/pull/19025
> > >
> > > In particular, this is the scalafmt config commit:
> > >
> > >
> >
> https://github.com/apache/flink/pull/19025/commits/cb32893df4b554e4526324c43c86681cc9fe8169
> > > And this is the commit removing scalastyle:
> > >
> > >
> >
> https://github.com/apache/flink/pull/19025/commits/9ffe7d52e3368c5c40f15e3dc48f6d81691a8dd0
> > >
> > > I need some committer to pair with to merge the big PR, any volunteers?
> > :)
> > >
> > > After we merge it I will also update the contributor guide doc to
> remove
> > > scalastyle.
> > >
> > > FG
> > >
> > > On Tue, Mar 8, 2022 at 10:07 AM David Anderson 
> > > wrote:
> > >
> > > > +1
> > > >
> > > > For flink-training we initially tried cloning the scalastyle setup
> from
> > > > flink, but we decided to use spotless + scalafmt instead.
> > > >
> > > > David
> > > >
> > > > On Mon, Mar 7, 2022 at 1:12 PM Timo Walther 
> > wrote:
> > > >
> > > > > Big +1
> > > > >
> > > > > This will improve the contribution experience. Even though we
> stopped
> > > > > adding more Scala code, it is still necessary from time to time.
> > > > >
> > > > > Regards,
> > > > > Timo
> > > > >
> > > > > Am 02.03.22 um 09:29 schrieb 刘首维:
> > > > > > +1
> > > > > >
> > > > > >
> > > > > > I still remember my first pr. Lack of experience, I had to pay
> > > > attention
> > > > > to Scala code format and corrected the format manually, which made
> > me a
> > > > > littleembarrassed(though I'm a big fan of Scala). I think
> this
> > > > > proposal will lighten the burden of writing Scala code.
> > > > > >
> > > > > >
> > > > > > Shouwei Liu
> > > > > >
> > > > > >
> > > > > > --原始邮件--
> > > > > > 发件人:
> > > > > "dev"
> > > > >   <
> > > > > kna...@apache.org;
> > > > > > 发送时间:2022年3月2日(星期三) 下午3:01
> > > > > > 收件人:"dev" > > > > >
> > > > > > 主题:Re: [DISCUSS] Enable scala formatting check
> > > > > >
> > > > > >
> > > > > >
> > > > > > +1 I've never written any Scala in Flink, but this makes a lot of
> > > sense
> > > > > to
> > > > > > me. Converging on a smaller set of tools and simplifying the
> build
> > is
> > > > > > always a good idea and the Community already concluded before
> that
> > > > > spotless
> > > > > > is generally a good approach.
> > > > > >
> > > > > > On Tue, Mar 1, 2022 at 5:52 PM Francesco Guardiani <
> > > > > france...@ververica.com
> > > > > > wrote:
> > > > > >
> > > > > >  Hi all,
> > > > > > 
> > > > > >  I want to propose to enable the spotless scalafmt
> integration
> > > and
> > > > > remove
> > > > > >  the scalastyle plugin.
> > > > > > 
> > > > > >  From an initial analysis, scalafmt can do everything
> > scalastyle
> > > > can
> > > > > do, and
> > > > > >  the integration with spotless looks easy to enable:
> > > > > > 
> > > https://github.com/diffplug/spotless/tree/main/plugin-maven#scala
> > > > .
> > > > > The
> > > > > >  scalafmt conf file gets picked up automatically from every
> > IDE,
> > > > and
> > > > > it can
> > > > > >  be heavily tuned.
> > > > > > 
> > > > > >  This way we can unify the formatting and integrate with our
> CI
> > > > > without any
> > > > > >  additional configurations. And we won't need scalastyle
> > anymore,
> > > > as
> > > > > >  scalafmt will take care of the checks:
> > > > > > 
> > > > > >  * mvn spotless:check will check both java and scala
> > > > > >  * mvn spotless:apply will format both java and scala
> > > > > > 
> > > > > >  WDYT?
> > > > > > 
> > > > > >  FG
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > >  --
> > > > > > 
> > > > > >  Francesco Guardiani | Software Engineer
> > > > > > 
> > > > > >  france...@ververica.com
> > > > > > 
> > > > > > 
> > > > > >   > > > > > 
> > > > > >  Follow us @VervericaData
> > > > > > 
> > > > > >  --
> > > > > > 
> > > > > >  Join Flink Forward  > Apache
> > > > > Flink
> > > > > >  Conference
> > > > > > 
> > > > > >  

[jira] [Created] (FLINK-26555) Missing close in FileSystemJobResultStore

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-26555:
-

 Summary: Missing close in FileSystemJobResultStore
 Key: FLINK-26555
 URL: https://issues.apache.org/jira/browse/FLINK-26555
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Matthias Pohl


{{FileSystemJobResultStore.createDirtyResultInternal}} does not close the 
opened {{OutputStream}}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26554) Clean termination of FlinkDeployment

[Thomas Weise (Jira)]

Thomas Weise created FLINK-26554:


 Summary: Clean termination of FlinkDeployment
 Key: FLINK-26554
 URL: https://issues.apache.org/jira/browse/FLINK-26554
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Reporter: Thomas Weise


After stopping the deployment, operator attempts to list jobs:


 2022-03-09 07:55:37,114 o.a.f.k.o.u.FlinkUtils         [INFO ] 
[default.basic-example] Waiting for cluster shutdown... (16)
2022-03-09 07:55:38,123 o.a.f.k.o.u.FlinkUtils         [INFO ] 
[default.basic-example] Cluster shutdown completed.
2022-03-09 07:55:38,160 o.a.f.k.o.c.FlinkDeploymentController [INFO ] 
[default.basic-example] Stopping cluster basic-example
2022-03-09 07:55:38,160 o.a.f.k.o.o.Observer           [INFO ] 
[default.basic-example] Getting job statuses for basic-example

 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Enable scala formatting check

[Martijn Visser]

I think it would actually be better to merge it before the release branch
is cut to avoid potential issues when needing to backport bugfixes?

Thanks, Martijn

On Wed, 9 Mar 2022 at 16:55, Seth Wiesman  wrote:

> Happy to help get this merged.
>
> Do we want to wait until the 1.15 branch is cut? The change is mostly
> trivial (reformatting) but does make changes to the build system.
>
> Seth
>
> On Wed, Mar 9, 2022 at 9:45 AM Francesco Guardiani <
> france...@ververica.com>
> wrote:
>
> > Hi all,
> > I've been spending some time prototyping a scalafmt conf, which doesn't
> > look too different from our java style and tries to keep the same
> > properties from our scalastyle conf. Here is the PR:
> > https://github.com/apache/flink/pull/19025
> >
> > In particular, this is the scalafmt config commit:
> >
> >
> https://github.com/apache/flink/pull/19025/commits/cb32893df4b554e4526324c43c86681cc9fe8169
> > And this is the commit removing scalastyle:
> >
> >
> https://github.com/apache/flink/pull/19025/commits/9ffe7d52e3368c5c40f15e3dc48f6d81691a8dd0
> >
> > I need some committer to pair with to merge the big PR, any volunteers?
> :)
> >
> > After we merge it I will also update the contributor guide doc to remove
> > scalastyle.
> >
> > FG
> >
> > On Tue, Mar 8, 2022 at 10:07 AM David Anderson 
> > wrote:
> >
> > > +1
> > >
> > > For flink-training we initially tried cloning the scalastyle setup from
> > > flink, but we decided to use spotless + scalafmt instead.
> > >
> > > David
> > >
> > > On Mon, Mar 7, 2022 at 1:12 PM Timo Walther 
> wrote:
> > >
> > > > Big +1
> > > >
> > > > This will improve the contribution experience. Even though we stopped
> > > > adding more Scala code, it is still necessary from time to time.
> > > >
> > > > Regards,
> > > > Timo
> > > >
> > > > Am 02.03.22 um 09:29 schrieb 刘首维:
> > > > > +1
> > > > >
> > > > >
> > > > > I still remember my first pr. Lack of experience, I had to pay
> > > attention
> > > > to Scala code format and corrected the format manually, which made
> me a
> > > > littleembarrassed(though I'm a big fan of Scala). I think this
> > > > proposal will lighten the burden of writing Scala code.
> > > > >
> > > > >
> > > > > Shouwei Liu
> > > > >
> > > > >
> > > > > --原始邮件--
> > > > > 发件人:
> > > > "dev"
> > > >   <
> > > > kna...@apache.org;
> > > > > 发送时间:2022年3月2日(星期三) 下午3:01
> > > > > 收件人:"dev" > > > >
> > > > > 主题:Re: [DISCUSS] Enable scala formatting check
> > > > >
> > > > >
> > > > >
> > > > > +1 I've never written any Scala in Flink, but this makes a lot of
> > sense
> > > > to
> > > > > me. Converging on a smaller set of tools and simplifying the build
> is
> > > > > always a good idea and the Community already concluded before that
> > > > spotless
> > > > > is generally a good approach.
> > > > >
> > > > > On Tue, Mar 1, 2022 at 5:52 PM Francesco Guardiani <
> > > > france...@ververica.com
> > > > > wrote:
> > > > >
> > > > >  Hi all,
> > > > > 
> > > > >  I want to propose to enable the spotless scalafmt integration
> > and
> > > > remove
> > > > >  the scalastyle plugin.
> > > > > 
> > > > >  From an initial analysis, scalafmt can do everything
> scalastyle
> > > can
> > > > do, and
> > > > >  the integration with spotless looks easy to enable:
> > > > > 
> > https://github.com/diffplug/spotless/tree/main/plugin-maven#scala
> > > .
> > > > The
> > > > >  scalafmt conf file gets picked up automatically from every
> IDE,
> > > and
> > > > it can
> > > > >  be heavily tuned.
> > > > > 
> > > > >  This way we can unify the formatting and integrate with our CI
> > > > without any
> > > > >  additional configurations. And we won't need scalastyle
> anymore,
> > > as
> > > > >  scalafmt will take care of the checks:
> > > > > 
> > > > >  * mvn spotless:check will check both java and scala
> > > > >  * mvn spotless:apply will format both java and scala
> > > > > 
> > > > >  WDYT?
> > > > > 
> > > > >  FG
> > > > > 
> > > > > 
> > > > > 
> > > > >  --
> > > > > 
> > > > >  Francesco Guardiani | Software Engineer
> > > > > 
> > > > >  france...@ververica.com
> > > > > 
> > > > > 
> > > > >   > > > > 
> > > > >  Follow us @VervericaData
> > > > > 
> > > > >  --
> > > > > 
> > > > >  Join Flink Forward  Apache
> > > > Flink
> > > > >  Conference
> > > > > 
> > > > >  Stream Processing | Event Driven | Real Time
> > > > > 
> > > > >  --
> > > > > 
> > > > >  Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> > > > > 
> > > > >  --
> > > > > 
> > > > >  Ververica GmbH
> > > > > 
> > > > >  Registered at Amtsgericht Charlottenburg: HRB 158244 B
> > > > > 
> > > > >  Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park
> > Tung
> > > > Jason,
> > > > >  Jinwei (Kevin) Zhang
> > > > > 
> > > > >
> > > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Enable scala formatting check

[Seth Wiesman]

Happy to help get this merged.

Do we want to wait until the 1.15 branch is cut? The change is mostly
trivial (reformatting) but does make changes to the build system.

Seth

On Wed, Mar 9, 2022 at 9:45 AM Francesco Guardiani 
wrote:

> Hi all,
> I've been spending some time prototyping a scalafmt conf, which doesn't
> look too different from our java style and tries to keep the same
> properties from our scalastyle conf. Here is the PR:
> https://github.com/apache/flink/pull/19025
>
> In particular, this is the scalafmt config commit:
>
> https://github.com/apache/flink/pull/19025/commits/cb32893df4b554e4526324c43c86681cc9fe8169
> And this is the commit removing scalastyle:
>
> https://github.com/apache/flink/pull/19025/commits/9ffe7d52e3368c5c40f15e3dc48f6d81691a8dd0
>
> I need some committer to pair with to merge the big PR, any volunteers? :)
>
> After we merge it I will also update the contributor guide doc to remove
> scalastyle.
>
> FG
>
> On Tue, Mar 8, 2022 at 10:07 AM David Anderson 
> wrote:
>
> > +1
> >
> > For flink-training we initially tried cloning the scalastyle setup from
> > flink, but we decided to use spotless + scalafmt instead.
> >
> > David
> >
> > On Mon, Mar 7, 2022 at 1:12 PM Timo Walther  wrote:
> >
> > > Big +1
> > >
> > > This will improve the contribution experience. Even though we stopped
> > > adding more Scala code, it is still necessary from time to time.
> > >
> > > Regards,
> > > Timo
> > >
> > > Am 02.03.22 um 09:29 schrieb 刘首维:
> > > > +1
> > > >
> > > >
> > > > I still remember my first pr. Lack of experience, I had to pay
> > attention
> > > to Scala code format and corrected the format manually, which made me a
> > > littleembarrassed(though I'm a big fan of Scala). I think this
> > > proposal will lighten the burden of writing Scala code.
> > > >
> > > >
> > > > Shouwei Liu
> > > >
> > > >
> > > > --原始邮件--
> > > > 发件人:
> > > "dev"
> > >   <
> > > kna...@apache.org;
> > > > 发送时间:2022年3月2日(星期三) 下午3:01
> > > > 收件人:"dev" > > >
> > > > 主题:Re: [DISCUSS] Enable scala formatting check
> > > >
> > > >
> > > >
> > > > +1 I've never written any Scala in Flink, but this makes a lot of
> sense
> > > to
> > > > me. Converging on a smaller set of tools and simplifying the build is
> > > > always a good idea and the Community already concluded before that
> > > spotless
> > > > is generally a good approach.
> > > >
> > > > On Tue, Mar 1, 2022 at 5:52 PM Francesco Guardiani <
> > > france...@ververica.com
> > > > wrote:
> > > >
> > > >  Hi all,
> > > > 
> > > >  I want to propose to enable the spotless scalafmt integration
> and
> > > remove
> > > >  the scalastyle plugin.
> > > > 
> > > >  From an initial analysis, scalafmt can do everything scalastyle
> > can
> > > do, and
> > > >  the integration with spotless looks easy to enable:
> > > > 
> https://github.com/diffplug/spotless/tree/main/plugin-maven#scala
> > .
> > > The
> > > >  scalafmt conf file gets picked up automatically from every IDE,
> > and
> > > it can
> > > >  be heavily tuned.
> > > > 
> > > >  This way we can unify the formatting and integrate with our CI
> > > without any
> > > >  additional configurations. And we won't need scalastyle anymore,
> > as
> > > >  scalafmt will take care of the checks:
> > > > 
> > > >  * mvn spotless:check will check both java and scala
> > > >  * mvn spotless:apply will format both java and scala
> > > > 
> > > >  WDYT?
> > > > 
> > > >  FG
> > > > 
> > > > 
> > > > 
> > > >  --
> > > > 
> > > >  Francesco Guardiani | Software Engineer
> > > > 
> > > >  france...@ververica.com
> > > > 
> > > > 
> > > >   > > > 
> > > >  Follow us @VervericaData
> > > > 
> > > >  --
> > > > 
> > > >  Join Flink Forward  > > Flink
> > > >  Conference
> > > > 
> > > >  Stream Processing | Event Driven | Real Time
> > > > 
> > > >  --
> > > > 
> > > >  Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> > > > 
> > > >  --
> > > > 
> > > >  Ververica GmbH
> > > > 
> > > >  Registered at Amtsgericht Charlottenburg: HRB 158244 B
> > > > 
> > > >  Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park
> Tung
> > > Jason,
> > > >  Jinwei (Kevin) Zhang
> > > > 
> > > >
> > > >
> > >
> > >
> >
>

Re: [DISCUSS] Enable scala formatting check

[Francesco Guardiani]

Hi all,
I've been spending some time prototyping a scalafmt conf, which doesn't
look too different from our java style and tries to keep the same
properties from our scalastyle conf. Here is the PR:
https://github.com/apache/flink/pull/19025

In particular, this is the scalafmt config commit:
https://github.com/apache/flink/pull/19025/commits/cb32893df4b554e4526324c43c86681cc9fe8169
And this is the commit removing scalastyle:
https://github.com/apache/flink/pull/19025/commits/9ffe7d52e3368c5c40f15e3dc48f6d81691a8dd0

I need some committer to pair with to merge the big PR, any volunteers? :)

After we merge it I will also update the contributor guide doc to remove
scalastyle.

FG

On Tue, Mar 8, 2022 at 10:07 AM David Anderson  wrote:

> +1
>
> For flink-training we initially tried cloning the scalastyle setup from
> flink, but we decided to use spotless + scalafmt instead.
>
> David
>
> On Mon, Mar 7, 2022 at 1:12 PM Timo Walther  wrote:
>
> > Big +1
> >
> > This will improve the contribution experience. Even though we stopped
> > adding more Scala code, it is still necessary from time to time.
> >
> > Regards,
> > Timo
> >
> > Am 02.03.22 um 09:29 schrieb 刘首维:
> > > +1
> > >
> > >
> > > I still remember my first pr. Lack of experience, I had to pay
> attention
> > to Scala code format and corrected the format manually, which made me a
> > littleembarrassed(though I'm a big fan of Scala). I think this
> > proposal will lighten the burden of writing Scala code.
> > >
> > >
> > > Shouwei Liu
> > >
> > >
> > > --原始邮件--
> > > 发件人:
> > "dev"
> >   <
> > kna...@apache.org;
> > > 发送时间:2022年3月2日(星期三) 下午3:01
> > > 收件人:"dev" > >
> > > 主题:Re: [DISCUSS] Enable scala formatting check
> > >
> > >
> > >
> > > +1 I've never written any Scala in Flink, but this makes a lot of sense
> > to
> > > me. Converging on a smaller set of tools and simplifying the build is
> > > always a good idea and the Community already concluded before that
> > spotless
> > > is generally a good approach.
> > >
> > > On Tue, Mar 1, 2022 at 5:52 PM Francesco Guardiani <
> > france...@ververica.com
> > > wrote:
> > >
> > >  Hi all,
> > > 
> > >  I want to propose to enable the spotless scalafmt integration and
> > remove
> > >  the scalastyle plugin.
> > > 
> > >  From an initial analysis, scalafmt can do everything scalastyle
> can
> > do, and
> > >  the integration with spotless looks easy to enable:
> > >  https://github.com/diffplug/spotless/tree/main/plugin-maven#scala
> .
> > The
> > >  scalafmt conf file gets picked up automatically from every IDE,
> and
> > it can
> > >  be heavily tuned.
> > > 
> > >  This way we can unify the formatting and integrate with our CI
> > without any
> > >  additional configurations. And we won't need scalastyle anymore,
> as
> > >  scalafmt will take care of the checks:
> > > 
> > >  * mvn spotless:check will check both java and scala
> > >  * mvn spotless:apply will format both java and scala
> > > 
> > >  WDYT?
> > > 
> > >  FG
> > > 
> > > 
> > > 
> > >  --
> > > 
> > >  Francesco Guardiani | Software Engineer
> > > 
> > >  france...@ververica.com
> > > 
> > > 
> > >   > > 
> > >  Follow us @VervericaData
> > > 
> > >  --
> > > 
> > >  Join Flink Forward  > Flink
> > >  Conference
> > > 
> > >  Stream Processing | Event Driven | Real Time
> > > 
> > >  --
> > > 
> > >  Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> > > 
> > >  --
> > > 
> > >  Ververica GmbH
> > > 
> > >  Registered at Amtsgericht Charlottenburg: HRB 158244 B
> > > 
> > >  Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park Tung
> > Jason,
> > >  Jinwei (Kevin) Zhang
> > > 
> > >
> > >
> >
> >
>

[jira] [Created] (FLINK-26553) Enable scalafmt for scala codebase

[Francesco Guardiani (Jira)]

Francesco Guardiani created FLINK-26553:
---

 Summary: Enable scalafmt for scala codebase
 Key: FLINK-26553
 URL: https://issues.apache.org/jira/browse/FLINK-26553
 Project: Flink
  Issue Type: Bug
  Components: Build System
Reporter: Francesco Guardiani
Assignee: Francesco Guardiani


As discussed in 
https://lists.apache.org/thread/97398pc9cb8y922xlb6mzlsbjtjf5jnv, we should 
enable scalafmt in our codebase



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] [statefun] resolve placeholders in module.yaml

[Igal Shilman]

Hello Fil,
I think that adding a very simple interpolation mechanism for remote
functions can be useful.
And also your suggested place should be good.
Can you create a JIRA issue with this description, and there we can
continue the conversation and scope this?

Thanks!
Igal.


On Mon, Mar 7, 2022 at 5:50 PM Filip Karnicki 
wrote:

> Hi, as far as I can tell, the way to provide a keystore/truststore password
> to the kafka ingress/egress config is to put it in plaintext in
> module.yaml, like so:
>
> kind: io.statefun.kafka.v1/ingressspec:  #(...)  properties:-
> ssl.truststore.password: changeme
>
> This isn't ideal and I think it would be neater to be able to replace a
> placeholder with something from the parameter tool / global config
>
> kind: io.statefun.kafka.v1/ingressspec:  #(...)  properties:-
> ssl.truststore.password: ${SSL_TRUSTSTORE_PASS}
>
> Similarly, we need to get our hands on a kerberos keytab location inside
> module.yaml. This is not a problem when the location is static and
> available to all cluster nodes, but when yarn gets involved, it's only the
> yarn client (?) that has the keytab file in a static location. As far as I
> can tell, task manager nodes get a 'resolved' and node/container-specific
> location, something along the lines of
> "/JBOD_D01/yarn/application_12345667_0001", which is different for every
> node. I think I could get my hands on that location from the global config,
> seeing as YarnTaskExecutorRunner sets
> '-Dsecurity.kerberos.login.keytab=/container/specific/path/here'
>
> To achieve all of this, we could alter RemoteModule#bindComponent to
> replace instances of ${PLACEHOLDERs} with values from the global config
> using regex.
>
> Please let me know what you think
> Fil
>

Re: [VOTE] Release 1.14.4, release candidate #1

[Timo Walther]

+1 (binding)

- I scanned the commit diff and affected files.
- I could not find major API changes or otherwise problematic changes.
- The biggest changes I could spot were around Kubernetes (see FLINK-20830).

Thanks for taking care of this Konstantin.

Timo


Am 02.03.22 um 10:04 schrieb Yun Tang:

+1 non-binding

- Checked the signatures for pending release artifacts.
- Download the pre-built flink-dist of both scala_2.11 and scala_2.12 versions 
and run them locally with the StateMachine example.
- Reviewed the flink-web PR

Best
Yun Tang

From: Seth Wiesman 
Sent: Monday, February 28, 2022 23:02
To: dev 
Subject: Re: [VOTE] Release 1.14.4, release candidate #1

+1 non-binding

- built from source
- checked hashes and signatures
- started locally and deployed wordcount / stopped with savepoint /
restarted
- reviewed announcement post

Thanks for managing the release!

Seth

On Fri, Feb 25, 2022 at 7:30 AM Konstantin Knauf  wrote:


Hi everyone,

Please review and vote on the release candidate #1 for the version 1.14.4,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

  The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be
deployed to dist.apache.org [2], which are signed with the key with
fingerprint 8C3FB007FE60 DEFA [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.14.4-rc1" [5],
* website pull request listing the new release and adding announcement blog
post [6].

The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Konstantin

[1]

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351231
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.14.4-rc1/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
https://repository.apache.org/content/repositories/orgapacheflink-1487/
[5] https://github.com/apache/flink/tree/release-1.14.4-rc1
[6] https://github.com/apache/flink-web/pull/510

--

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

[jira] [Created] (FLINK-26552) Try to use @EnableKubernetesMockClient(crud = true) in controller test

[Gyula Fora (Jira)]

Gyula Fora created FLINK-26552:
--

 Summary: Try to use @EnableKubernetesMockClient(crud = true) in 
controller test
 Key: FLINK-26552
 URL: https://issues.apache.org/jira/browse/FLINK-26552
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Reporter: Gyula Fora


The controller test currently uses the KubernetesMockserver directly.

As [~wangyang0918] pointed out we could try using 
@EnableKubernetesMockClient(crud = true) like in the FlinkService test.

At my initial attempt I ran into some problems with event creation and listing 
them back.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] CAST legacy behaviour

[Francesco Guardiani]

Hi all,
As I see this thread has consensus, here is the issue and PR to disable the
legacy behavior by default:
https://issues.apache.org/jira/browse/FLINK-26551
https://github.com/apache/flink/pull/19020

We target to merge it before the release of 1.15, unless there are any
objections.

FG

On Tue, Mar 1, 2022 at 2:18 PM Marios Trivyzas  wrote:

> Indeed, if we manage to use the configuration from *flink-conf.yaml* down
> the stack,
> it would be easy for everyone to configure a "system-wide" legacy cast
> behaviour.
>
> Best regards,
> Marios
>
> On Tue, Mar 1, 2022 at 2:52 PM Timo Walther  wrote:
>
> > +1
> >
> > Thanks for bringing up this discussion one more time Marios.
> >
> > I strongly support enabling the new behavior in 1.15. It definitely has
> > implications on existing users, but as Seth said, thinking about the
> > upcoming upgrade story we need to make sure that at least the core/basic
> > operations are correct. Otherwise we will have to maintain multiple
> > versions of functions with broken semantics.
> >
> > I since we also try to fix various issues around configuration, maybe it
> > might still be possible to configure the legacy cast behavior globally
> > via flink-conf.yaml. This should make the transitioning period easier in
> > production.
> >
> > Regards,
> > Timo
> >
> > Am 28.02.22 um 19:04 schrieb Seth Wiesman:
> > > +1
> > >
> > > Especially as SQL upgrades are right around the corner, it makes sense
> to
> > > get our defaults right.
> > >
> > > Seth
> > >
> > > On Mon, Feb 28, 2022 at 7:14 AM Martijn Visser 
> > > wrote:
> > >
> > >> +1 for setting this option to disabled by default. I believe failures
> > >> should be brought forward as soon as possible, so they can be fixed as
> > fast
> > >> as possible. It will also be less confusing for new users. Last but
> not
> > >> least, I believe the impact on existing users will be minimal (since
> it
> > can
> > >> be changed by changing one flag).
> > >>
> > >> Best regards,
> > >>
> > >> Martijn
> > >>
> > >> On Tue, 22 Feb 2022 at 17:55, Marios Trivyzas 
> wrote:
> > >>
> > >>> Thanks Francesco,
> > >>>
> > >>> The two arguments you posted, further strengthen the need to make it
> > >>> DISABLED by default.
> > >>>
> > >>> On Tue, Feb 22, 2022 at 12:10 PM Francesco Guardiani <
> > >>> france...@ververica.com> wrote:
> > >>>
> >  Hi all,
> >  I'm +1 with what everything you said Marios.
> > 
> >  I'm gonna add another argument on top of that: the
> > >> "legacy-cast-behavior"
> >  has also a broken type inference, leading to incorrect results or
> > >> further
> >  errors down in the pipeline[1]. For example, take this:
> > 
> >  SELECT COALESCE(CAST('a' AS INT), 0) ...
> > 
> >  With the legacy cast behavior ENABLED, this is going to lead to the
> > >> wrong
> >  result, as 'a' is inferred as VARCHAR NOT NULL, the CAST return
> value
> > >> is
> >  inferred as INT NOT NULL, so the planner will drop COALESCE, and
> will
> > >>> never
> >  return 0. Essentially, CAST will infer the wrong nullability leading
> > to
> >  assigning its result to a NOT NULL type, when its value can
> > effectively
> > >>> be
> >  NULL.
> > 
> > > You introduce a deprecated flag to help users
> >  using old versions of the software to smoothly transition to the new
> >  version, while the new users experience the new features/behavior,
> >  without the need to set a flag.
> > 
> >  This is IMO the major point on why we should disable the legacy cast
> >  behavior by default. This is even more relevant with 1.15 and the
> > >> upgrade
> >  story, as per the problem described above, because users will now
> end
> > >> up
> >  generating plans with wrong type inference, which will be hard to
> > >> migrate
> >  in the next flink versions.
> > 
> >  FG
> > 
> >  [1] In case you're wondering why it wasn't fixed, the reason is that
> > >>> fixing
> >  it means creating a breaking change, for details
> >  https://github.com/apache/flink/pull/18611#issuecomment-1028174877
> > 
> > 
> >  On Tue, Feb 22, 2022 at 10:07 AM Marios Trivyzas 
> > >>> wrote:
> > > Hello all!
> > >
> > > I would like to bring back the discussion regarding the
> > > *table.exec.legacy-cast-behaviour*
> > > configuration option which we are introducing with Flink *1.15*.
> This
> > > option provides the users
> > > with the flexibility to continue using the old (incorrect,
> according
> > >> to
> >  SQL
> > > standards) behaviour
> > > of *CAST.*
> > >
> > > With Flink *1.15* we have introduced a bunch of fixes, improvements
> > >> and
> >  new
> > > casting functionality
> > > between types, see
> https://issues.apache.org/jira/browse/FLINK-24403
> > >> ,
> >  and
> > > some of them are
> > > guarded behind the legacy behaviour option:
> > >
> > > - Trimming and 

[jira] [Created] (FLINK-26551) Make the legacy behavior disabled by default

[Francesco Guardiani (Jira)]

Francesco Guardiani created FLINK-26551:
---

 Summary: Make the legacy behavior disabled by default
 Key: FLINK-26551
 URL: https://issues.apache.org/jira/browse/FLINK-26551
 Project: Flink
  Issue Type: Sub-task
Reporter: Francesco Guardiani


Followup of https://issues.apache.org/jira/browse/FLINK-25111

For the discussion, see 
https://lists.apache.org/thread/r13y3plwwyg3sngh8cz47flogq621txv



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: status of Apple Silicon (M1) as Flink dev platform?

[Dian Fu]

Hi David,

Thanks for bringing up this question.

Regarding PyFlink, the dependent Python libraries have already supported M1
recently and so I guess we could also make it in the next release, e.g.
Flink 1.16. I will follow up with it.

Regards,
Dian

On Wed, Mar 9, 2022 at 3:27 PM Yun Tang  wrote:

> Hi David,
>
> For the problem of RocksDB running on M1 machines, RocksDB community has
> recently resolved this in RocksDB-6.29 [1], maybe we could involve this
> change in next Flink-1.16.
>
>
> [1] https://github.com/facebook/rocksdb/pull/9662
>
> Best
> Yun Tang
> 
> From: David Anderson 
> Sent: Tuesday, March 8, 2022 16:25
> To: dev 
> Subject: status of Apple Silicon (M1) as Flink dev platform?
>
> What's the current status of using the Apple Silicon (M1) platform for
> Flink development? Have we reached the point where everything "just works",
> or do there remain lingering annoyances (or worse)?
>
> In the past, I've seen reports of issues involving, e.g., RocksDB, nodejs,
> protobuf, and pyflink. Looking in Jira, I see these issues haven't been
> resolved yet, but I'm not sure what to read into that:
>
> https://issues.apache.org/jira/browse/FLINK-24932 (Frocksdb cannot run on
> Apple M1)
> https://issues.apache.org/jira/browse/FLINK-25188 (Cannot install PyFlink
> on MacOS with M1 chip)
>
> Best,
> David
>

[jira] [Created] (FLINK-26550) Correct the information of checkpoint failure

[Yun Tang (Jira)]

Yun Tang created FLINK-26550:


 Summary: Correct the information of checkpoint failure 
 Key: FLINK-26550
 URL: https://issues.apache.org/jira/browse/FLINK-26550
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Reporter: Yun Tang
Assignee: Yun Tang
 Fix For: 1.15.0, 1.14.5


After FLINK-26049, all failed checkpoint would print message with {{ Failed to 
trigger checkpoint }}:


{code:java}
5812 [pool-5-thread-1] INFO  
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering 
checkpoint 1 (type=CheckpointType{name='Checkpoint', 
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1646825286424 for job 
d2fd07b3b33af453a4e115f3197f81bb.
5913 [pool-5-thread-1] INFO  
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Checkpoint 1 of 
job d2fd07b3b33af453a4e115f3197f81bb expired before completing.
451518 [pool-5-thread-1] WARN  
org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to 
trigger checkpoint 1 for job d2fd07b3b33af453a4e115f3197f81bb. (0 consecutive 
failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired 
before completing.
at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2172)
 [classes/:?]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_292]
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) 
[?:1.8.0_292]
at java.util.concurrent.FutureTask.run(FutureTask.java) [?:1.8.0_292]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 [?:1.8.0_292]
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [?:1.8.0_292]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_292]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
{code}

This is extremely strange as the failure does not happen during the trigger 
phase.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26549) INSERT INTO with VALUES leads to wrong type inference with nested types

[Francesco Guardiani (Jira)]

Francesco Guardiani created FLINK-26549:
---

 Summary: INSERT INTO with VALUES leads to wrong type inference 
with nested types
 Key: FLINK-26549
 URL: https://issues.apache.org/jira/browse/FLINK-26549
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Francesco Guardiani


While working on casting, I've found out we have an interesting bug in the 
insert values type inference. This comes from the 
{{KafkaTableITCase#testKafkaSourceSinkWithMetadata}} (look at this version in 
particular 
https://github.com/apache/flink/blob/567440115bcacb5aceaf3304e486281c7da8c14f/flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/KafkaTableITCase.java).

The test scenario is an INSERT INTO VALUES query which is also pushing some 
metadata to a Kafka table, in particular is writing the headers metadata.

The table is declared like that:

{code:sql}
 CREATE TABLE kafka (
  `physical_1` STRING,
  `physical_2` INT,
  `timestamp-type` STRING METADATA VIRTUAL,
  `timestamp` TIMESTAMP(3) METADATA,
  `leader-epoch` INT METADATA VIRTUAL,
  `headers` MAP METADATA,
  `partition` INT METADATA VIRTUAL,
  `topic` STRING METADATA VIRTUAL,
  `physical_3` BOOLEAN
) WITH (
   'connector' = 'kafka',
   [...]
)
{code}

The insert into query looks like:

{code:sql}
INSERT INTO kafka VALUES
('data 1', 1, TIMESTAMP '2020-03-08 13:12:11.123', MAP['k1', x'C0FFEE', 'k2', 
x'BABE'], TRUE),
('data 2', 2, TIMESTAMP '2020-03-09 13:12:11.123', CAST(NULL AS MAP), FALSE),
('data 3', 3, TIMESTAMP '2020-03-10 13:12:11.123', MAP['k1', X'10', 'k2', 
X'20'], TRUE)
{code}

Note that in the first row, the byte literal is of length 3, while in the last 
row the byte literal is of length 1.

The generated plan of this INSERT INTO is:

{code}
== Abstract Syntax Tree ==
LogicalSink(table=[default_catalog.default_database.kafka], fields=[physical_1, 
physical_2, physical_3, headers, timestamp])
+- LogicalProject(physical_1=[$0], physical_2=[$1], physical_3=[$4], 
headers=[CAST($3):(VARCHAR(2147483647) CHARACTER SET "UTF-16LE", 
VARBINARY(2147483647)) MAP], 
timestamp=[CAST($2):TIMESTAMP_WITH_LOCAL_TIME_ZONE(3)])
   +- LogicalUnion(all=[true])
  :- LogicalProject(EXPR$0=[_UTF-16LE'data 1'], EXPR$1=[1], 
EXPR$2=[2020-03-08 13:12:11.123:TIMESTAMP(3)], EXPR$3=[MAP(_UTF-16LE'k1', 
X'c0ffee':VARBINARY(3), _UTF-16LE'k2', X'babe':VARBINARY(3))], EXPR$4=[true])
  :  +- LogicalValues(tuples=[[{ 0 }]])
  :- LogicalProject(EXPR$0=[_UTF-16LE'data 2'], EXPR$1=[2], 
EXPR$2=[2020-03-09 13:12:11.123:TIMESTAMP(3)], 
EXPR$3=[null:(VARCHAR(2147483647) CHARACTER SET "UTF-16LE", 
VARBINARY(2147483647)) MAP], EXPR$4=[false])
  :  +- LogicalValues(tuples=[[{ 0 }]])
  +- LogicalProject(EXPR$0=[_UTF-16LE'data 3'], EXPR$1=[3], 
EXPR$2=[2020-03-10 13:12:11.123:TIMESTAMP(3)], EXPR$3=[MAP(_UTF-16LE'k1', 
X'10':BINARY(1), _UTF-16LE'k2', X'20':BINARY(1))], EXPR$4=[true])
 +- LogicalValues(tuples=[[{ 0 }]])

== Optimized Physical Plan ==
Sink(table=[default_catalog.default_database.kafka], fields=[physical_1, 
physical_2, physical_3, headers, timestamp])
+- Union(all=[true], union=[physical_1, physical_2, physical_3, headers, 
timestamp])
   :- Calc(select=[_UTF-16LE'data 1' AS physical_1, 1 AS physical_2, true AS 
physical_3, CAST(CAST(MAP(_UTF-16LE'k1', X'c0ffee':VARBINARY(3), _UTF-16LE'k2', 
X'babe':VARBINARY(3)) AS (CHAR(2) CHARACTER SET "UTF-16LE" NOT NULL, BINARY(1) 
NOT NULL) MAP) AS (VARCHAR(2147483647) CHARACTER SET "UTF-16LE", 
VARBINARY(2147483647)) MAP) AS headers, CAST(2020-03-08 
12:12:11.123:TIMESTAMP_WITH_LOCAL_TIME_ZONE(3) AS 
TIMESTAMP_WITH_LOCAL_TIME_ZONE(3)) AS timestamp])
   :  +- Values(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])
   :- Calc(select=[_UTF-16LE'data 2' AS physical_1, 2 AS physical_2, false AS 
physical_3, null:(VARCHAR(2147483647) CHARACTER SET "UTF-16LE", 
VARBINARY(2147483647)) MAP AS headers, CAST(2020-03-09 
12:12:11.123:TIMESTAMP_WITH_LOCAL_TIME_ZONE(3) AS 
TIMESTAMP_WITH_LOCAL_TIME_ZONE(3)) AS timestamp])
   :  +- Values(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])
   +- Calc(select=[_UTF-16LE'data 3' AS physical_1, 3 AS physical_2, true AS 
physical_3, CAST(MAP(_UTF-16LE'k1', X'10':BINARY(1), _UTF-16LE'k2', 
X'20':BINARY(1)) AS (VARCHAR(2147483647) CHARACTER SET "UTF-16LE", 
VARBINARY(2147483647)) MAP) AS headers, CAST(2020-03-10 
12:12:11.123:TIMESTAMP_WITH_LOCAL_TIME_ZONE(3) AS 
TIMESTAMP_WITH_LOCAL_TIME_ZONE(3)) AS timestamp])
  +- Values(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])

== Optimized Execution Plan ==
Sink(table=[default_catalog.default_database.kafka], fields=[physical_1, 
physical_2, physical_3, headers, timestamp])
+- Union(all=[true], union=[physical_1, physical_2, physical_3, headers, 
timestamp])
   :- Calc(select=['data 1' AS physical_1, 1 AS physical_2, true AS physical_3, 

[jira] [Created] (FLINK-26548) the source parallelism is not set correctly with AdaptiveBatchScheduler

[zl (Jira)]

zl created FLINK-26548:
--

 Summary: the source parallelism is not set correctly with 
AdaptiveBatchScheduler
 Key: FLINK-26548
 URL: https://issues.apache.org/jira/browse/FLINK-26548
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 1.15.0
Reporter: zl
 Attachments: image-2022-03-09-19-00-18-396.png

When running *_org.apache.flink.table.tpcds.TpcdsTestProgram_* with 
{_}*AdaptiveBatchScheduler*{_}, I ran into a problem：the num of records sent by 
the source operator is always 1, and the parallelism of source operator is also 
1 even I set *_jobmanager.adaptive-batch-scheduler.default-source-parallelism_* 
to 8.

!image-2022-03-09-19-00-18-396.png!

After some research, I found that the operator A is not the actual file reader, 
it just splits files and assigns splits to downstream tasks for further 
processing, and the operator B is the actual file reader task. Here, the 
parallelism of operator B is 64, and the records sent by operator A is 1, this 
means, operator A assigned all splits to a task of operator B, {*}_the other 63 
tasks of operator B is idle_{*}, it is unreasonable.

In this case,  the parallelism of operator B should be 
*_jobmanager.adaptive-batch-scheduler.default-source-parallelism_*  and the num 
of records sent by operator A also should be 
{*}_jobmanager.adaptive-batch-scheduler.default-source-parallelism_{*}.

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26547) Accepting slots without a matching requirement lead to unfulfillable requirements

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-26547:


 Summary: Accepting slots without a matching requirement lead to 
unfulfillable requirements
 Key: FLINK-26547
 URL: https://issues.apache.org/jira/browse/FLINK-26547
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.15.0






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26546) Extract Observer Interface

[Matyas Orhidi (Jira)]

Matyas Orhidi created FLINK-26546:
-

 Summary: Extract Observer Interface
 Key: FLINK-26546
 URL: https://issues.apache.org/jira/browse/FLINK-26546
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Reporter: Matyas Orhidi


Similarly to the Reconciler Interface we should extract the Observer interface.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)