[jira] [Created] (FLINK-21553) WindowDistinctAggregateITCase#testHopWindow_Cube is unstable

[Jark Wu (Jira)]

Jark Wu created FLINK-21553:
---

 Summary: WindowDistinctAggregateITCase#testHopWindow_Cube is 
unstable
 Key: FLINK-21553
 URL: https://issues.apache.org/jira/browse/FLINK-21553
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Jark Wu
 Fix For: 1.13.0


See 
https://dev.azure.com/imjark/Flink/_build/results?buildId=422=logs=d1352042-8a7d-50b6-3946-a85d176b7981=b2322052-d503-5552-81e2-b3a532a1d7e8




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21552) The managed memory was not released if exception was thrown in createPythonExecutionEnvironment

[Dian Fu (Jira)]

Dian Fu created FLINK-21552:
---

 Summary: The managed memory was not released if exception was 
thrown in createPythonExecutionEnvironment
 Key: FLINK-21552
 URL: https://issues.apache.org/jira/browse/FLINK-21552
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.12.0
Reporter: Dian Fu
Assignee: Dian Fu
 Fix For: 1.12.3


If there is exception thrown in createPythonExecutionEnvironment, the job will 
failed with the following exception:
{code}
org.apache.flink.runtime.memory.MemoryAllocationException: Could not created 
the shared memory resource of size 611948962. Not enough memory left to reserve 
from the slot's managed memory.
at 
org.apache.flink.runtime.memory.MemoryManager.lambda$getSharedMemoryResourceForManagedMemory$5(MemoryManager.java:536)
at 
org.apache.flink.runtime.memory.SharedResources.createResource(SharedResources.java:126)
at 
org.apache.flink.runtime.memory.SharedResources.getOrAllocateSharedResource(SharedResources.java:72)
at 
org.apache.flink.runtime.memory.MemoryManager.getSharedMemoryResourceForManagedMemory(MemoryManager.java:555)
at 
org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.open(BeamPythonFunctionRunner.java:250)
at 
org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.open(AbstractPythonFunctionOperator.java:113)
at 
org.apache.flink.table.runtime.operators.python.AbstractStatelessFunctionOperator.open(AbstractStatelessFunctionOperator.java:116)
at 
org.apache.flink.table.runtime.operators.python.scalar.AbstractPythonScalarFunctionOperator.open(AbstractPythonScalarFunctionOperator.java:88)
at 
org.apache.flink.table.runtime.operators.python.scalar.AbstractRowDataPythonScalarFunctionOperator.open(AbstractRowDataPythonScalarFunctionOperator.java:70)
at 
org.apache.flink.table.runtime.operators.python.scalar.RowDataPythonScalarFunctionOperator.open(RowDataPythonScalarFunctionOperator.java:59)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:428)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:543)
at 
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:93)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:533)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
at java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.flink.runtime.memory.MemoryReservationException: Could 
not allocate 611948962 bytes, only 0 bytes are remaining. This usually 
indicates that you are requesting more memory than you have reserved. However, 
when running an old JVM version it can also be caused by slow garbage 
collection. Try to upgrade to Java 8u72 or higher if running on an old Java 
version.
at 
org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:170)
at 
org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:84)
at 
org.apache.flink.runtime.memory.MemoryManager.reserveMemory(MemoryManager.java:423)
at 
org.apache.flink.runtime.memory.MemoryManager.lambda$getSharedMemoryResourceForManagedMemory$5(MemoryManager.java:534)
... 17 more
{code}

The reason is that the reserved managed memory was not added back to the 
MemoryManager when Job failed because of exceptions thrown in 
createPythonExecutionEnvironment. This causes that there is no managed memory 
to allocate during failover.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.12.2, release candidate #2

[Kurt Young]

+1 (binding)

- We mainly checked the patch of FLINK-20663 [1] and confirmed there is no
OutOfManagedMemory error anymore.

[1] https://issues.apache.org/jira/browse/FLINK-20663

Best,
Kurt


On Tue, Mar 2, 2021 at 12:41 PM Yu Li  wrote:

> +1 (binding)
>
> - Checked the diff between 1.12.1 and 1.12.2-rc2: OK (
> https://github.com/apache/flink/compare/release-1.12.1...release-1.12.2-rc2
> )
>   - jackson version has been bumped to 2.10.5.1 through FLINK-21020 and all
> NOTICE files updated correctly
>   - beanutils version has been bumped to 1.9.4 through FLINK-21123 and all
> NOTICE files updated correctly
>   - testcontainer version has been bumped to 1.15.1 through FLINK-21277 and
> no NOTICE files impact
>   - japicmp version has been bumped to 1.12.1 and no NOTICE files impact
> - Checked release notes: OK
> - Checked sums and signatures: OK
> - Maven clean install from source: OK
> - Checked the jars in the staging repo: OK
> - Checked the website updates: OK (minor: corrected fix version of
> FLINK-21515  to make
> sure the website PR consistent with release note)
>
> Note: there's a vulnerability suspicion against 1.12.2-rc2 reported in
> user-zh mailing list [1] w/o enough evidence/information. Have asked the
> reporter to do more testing to confirm and I don't think it's a blocker for
> the release, but just a note here in case anyone has a different opinion.
>
> Thanks a lot for managing the new RC!
>
> Best Regards,
> Yu
>
> [1] http://apache-flink.147419.n8.nabble.com/flink-1-12-2-rc2-td11023.html
>
> On Tue, 2 Mar 2021 at 01:51, Piotr Nowojski  wrote:
>
> > +1 (binding)
> >
> > For the RC2 I have additionally confirmed that "stop-with-savepoint", and
> > "stop-with-savepoint --drain" seems to be working.
> >
> > Piotrek
> >
> > pon., 1 mar 2021 o 11:18 Matthias Pohl 
> > napisał(a):
> >
> > > Thanks for managing release 1.12.2, Yuan & Roman.
> > >
> > > +1 (non-binding)
> > >
> > > - Verified checksums and GPG of artifacts in [1]
> > > - Build the sources locally without errors
> > > - Started a local standalone cluster and deployed WordCount without
> > > problems (no suspicious logs identified)
> > > - Verified FLINK-21030 [2] by running the example jobs from the
> > > FLINK-21030-related SavepointITCase tests
> > >
> > > Best,
> > > Matthias
> > >
> > > [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.2-rc2/
> > > [2] https://issues.apache.org/jira/browse/FLINK-21030
> > >
> > > On Sun, Feb 28, 2021 at 2:41 PM Yuan Mei 
> wrote:
> > >
> > > > Hey Roman,
> > > >
> > > > Thank you very much for preparing RC2.
> > > >
> > > > +1 from my side.
> > > >
> > > > 1. Verified Checksums and GPG signatures.
> > > > 2. Verified that the source archives do not contain any binaries.
> > > > 3. Successfully Built the source with Maven.
> > > > 4. Started a local Flink cluster, ran the streaming WordCount example
> > > with
> > > > WebUI,
> > > > checked the output and JM/TM log, no suspicious output/log.
> > > > 5. Repeat Step 4 with the binary release as well, no suspicious
> > > output/log.
> > > > 6. Checked for source and binary release to make sure both an Apache
> > > > License file and a NOTICE file are included.
> > > > 7. Manually verified that no pom file changes between 1.12.2-rc1 and
> > > > 1.12.2-rc2; no obvious license problem.
> > > > 8. Review the release PR for RC2 updates, and double confirmed the
> > > > change-list for 1.12.2.
> > > >
> > > > Best,
> > > > Yuan
> > > >
> > > > On Sat, Feb 27, 2021 at 7:19 AM Roman Khachatryan 
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Please review and vote on the release candidate #1 for the version
> > > > 1.12.2,
> > > > > as follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > > > >
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > > * JIRA release notes [1],
> > > > > * the official Apache source release and binary convenience
> releases
> > to
> > > > be
> > > > > deployed to dist.apache.org [2], which are signed with the key
> with
> > > > > fingerprint 0D545F264D2DFDEBFD4E038F97B4625E2FCF517C [3],
> > > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > > * source code tag "release-1.12.2-rc2" [5],
> > > > > * website pull request listing the new release and adding
> > announcement
> > > > blog
> > > > > post [6].
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12349502=12315522
> > > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.2-rc2/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > > >
> > >
> 

[jira] [Created] (FLINK-21551) FlinkKafkaProducerITCase.testHappyPath fail

[Guowei Ma (Jira)]

Guowei Ma created FLINK-21551:
-

 Summary: FlinkKafkaProducerITCase.testHappyPath fail
 Key: FLINK-21551
 URL: https://issues.apache.org/jira/browse/FLINK-21551
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.11.3
Reporter: Guowei Ma


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13955=logs=961f8f81-6b52-53df-09f6-7291a2e4af6a=2f99feaa-7a9b-5916-4c1c-5e61f395079e
{code:java}
[ERROR] Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 47.008 
s <<< FAILURE! - in 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase [ERROR] 
testHappyPath(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
 Time elapsed: 12.865 s <<< ERROR! java.util.NoSuchElementException at 
org.apache.kafka.common.utils.AbstractIterator.next(AbstractIterator.java:52) 
at 
org.apache.flink.shaded.guava18.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:302)
 at 
org.apache.flink.shaded.guava18.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:289)
 at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.assertRecord(FlinkKafkaProducerITCase.java:212)
 at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.testHappyPath(FlinkKafkaProducerITCase.java:91)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
 at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) at 
java.lang.Thread.run(Thread.java:748) [INFO] [INFO] Results: [INFO] [ERROR] 
Errors: [ERROR] FlinkKafkaProducerITCase.testHappyPath:91->assertRecord:212 » 
NoSuchElement
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21550) ZooKeeperHaServicesTest.testSimpleClose fail

[Guowei Ma (Jira)]

Guowei Ma created FLINK-21550:
-

 Summary: ZooKeeperHaServicesTest.testSimpleClose fail
 Key: FLINK-21550
 URL: https://issues.apache.org/jira/browse/FLINK-21550
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.12.3
Reporter: Guowei Ma


{code:java}
[ERROR] 
testSimpleClose(org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServicesTest)
 Time elapsed: 9.265 s <<< ERROR! java.util.concurrent.TimeoutException: 
Listener was not notified about a new leader within 2000ms at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:151)
 at 
org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:136)
 at 
org.apache.flink.runtime.leaderelection.TestingRetrievalBase.waitForNewLeader(TestingRetrievalBase.java:53)
 at 
org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServicesTest.runCleanupTest(ZooKeeperHaServicesTest.java:195)
 at 
org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServicesTest.testSimpleClose(ZooKeeperHaServicesTest.java:100)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498) at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at 
org.junit.rules.RunRules.evaluate(RunRules.java:20) at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release 1.12.2, release candidate #2

[Yu Li]

+1 (binding)

- Checked the diff between 1.12.1 and 1.12.2-rc2: OK (
https://github.com/apache/flink/compare/release-1.12.1...release-1.12.2-rc2)
  - jackson version has been bumped to 2.10.5.1 through FLINK-21020 and all
NOTICE files updated correctly
  - beanutils version has been bumped to 1.9.4 through FLINK-21123 and all
NOTICE files updated correctly
  - testcontainer version has been bumped to 1.15.1 through FLINK-21277 and
no NOTICE files impact
  - japicmp version has been bumped to 1.12.1 and no NOTICE files impact
- Checked release notes: OK
- Checked sums and signatures: OK
- Maven clean install from source: OK
- Checked the jars in the staging repo: OK
- Checked the website updates: OK (minor: corrected fix version of
FLINK-21515  to make
sure the website PR consistent with release note)

Note: there's a vulnerability suspicion against 1.12.2-rc2 reported in
user-zh mailing list [1] w/o enough evidence/information. Have asked the
reporter to do more testing to confirm and I don't think it's a blocker for
the release, but just a note here in case anyone has a different opinion.

Thanks a lot for managing the new RC!

Best Regards,
Yu

[1] http://apache-flink.147419.n8.nabble.com/flink-1-12-2-rc2-td11023.html

On Tue, 2 Mar 2021 at 01:51, Piotr Nowojski  wrote:

> +1 (binding)
>
> For the RC2 I have additionally confirmed that "stop-with-savepoint", and
> "stop-with-savepoint --drain" seems to be working.
>
> Piotrek
>
> pon., 1 mar 2021 o 11:18 Matthias Pohl 
> napisał(a):
>
> > Thanks for managing release 1.12.2, Yuan & Roman.
> >
> > +1 (non-binding)
> >
> > - Verified checksums and GPG of artifacts in [1]
> > - Build the sources locally without errors
> > - Started a local standalone cluster and deployed WordCount without
> > problems (no suspicious logs identified)
> > - Verified FLINK-21030 [2] by running the example jobs from the
> > FLINK-21030-related SavepointITCase tests
> >
> > Best,
> > Matthias
> >
> > [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.2-rc2/
> > [2] https://issues.apache.org/jira/browse/FLINK-21030
> >
> > On Sun, Feb 28, 2021 at 2:41 PM Yuan Mei  wrote:
> >
> > > Hey Roman,
> > >
> > > Thank you very much for preparing RC2.
> > >
> > > +1 from my side.
> > >
> > > 1. Verified Checksums and GPG signatures.
> > > 2. Verified that the source archives do not contain any binaries.
> > > 3. Successfully Built the source with Maven.
> > > 4. Started a local Flink cluster, ran the streaming WordCount example
> > with
> > > WebUI,
> > > checked the output and JM/TM log, no suspicious output/log.
> > > 5. Repeat Step 4 with the binary release as well, no suspicious
> > output/log.
> > > 6. Checked for source and binary release to make sure both an Apache
> > > License file and a NOTICE file are included.
> > > 7. Manually verified that no pom file changes between 1.12.2-rc1 and
> > > 1.12.2-rc2; no obvious license problem.
> > > 8. Review the release PR for RC2 updates, and double confirmed the
> > > change-list for 1.12.2.
> > >
> > > Best,
> > > Yuan
> > >
> > > On Sat, Feb 27, 2021 at 7:19 AM Roman Khachatryan 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #1 for the version
> > > 1.12.2,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release and binary convenience releases
> to
> > > be
> > > > deployed to dist.apache.org [2], which are signed with the key with
> > > > fingerprint 0D545F264D2DFDEBFD4E038F97B4625E2FCF517C [3],
> > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > * source code tag "release-1.12.2-rc2" [5],
> > > > * website pull request listing the new release and adding
> announcement
> > > blog
> > > > post [6].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12349502=12315522
> > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.2-rc2/
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1414/
> > > > [5] https://github.com/apache/flink/releases/tag/release-1.12.2-rc2
> > > > [6] https://github.com/apache/flink-web/pull/418
> > > >
> > > > Regards,
> > > > Roman
> > > >
> >
>

Re: [DISCUSS]FLIP-163: SQL Client Improvements

[Jark Wu]

I prefer option#2 and I think this can make everyone happy.

Best,
Jark

On Mon, 1 Mar 2021 at 18:22, Shengkai Fang  wrote:

> Hi, everyone.
>
> After the long discussion, I am fine with both choices. But I prefer the
> second option that applies to both table modules and sql client. Just as
> Timo said, the option `table.dml-sync` can improve the SQL script
> portability. Users don't need to modify the script and execute the script
> in different platforms e.g gateway.
>
> What do you think? CC Timo, Jark, Leonard.
>
> Best,
> Shengkai.
>
> Kurt Young  于2021年3月1日周一 下午5:11写道：
>
> > I'm +1 for either:
> > 1. introduce a sql client specific option, or
> > 2. Introduce a table config option and make it apply to both table
> module &
> > sql client.
> >
> > It would be the FLIP owner's call to decide.
> >
> > Best,
> > Kurt
> >
> >
> > On Mon, Mar 1, 2021 at 3:25 PM Timo Walther  wrote:
> >
> > > We could also think about reading this config option in Table API. The
> > > effect would be to call `await()` directly in an execute call. I could
> > > also imagine this to be useful esp. when you fire a lot of insert into
> > > queries. We had the case before that users where confused that the
> > > execution happens asynchronously, such an option could prevent this to
> > > happen again.
> > >
> > > Regards,
> > > Timo
> > >
> > > On 01.03.21 05:14, Kurt Young wrote:
> > > > I also asked some users about their opinion that if we introduce some
> > > > config prefixed with "table" but doesn't
> > > > have affection with methods in Table API and SQL. All of them are
> kind
> > of
> > > > shocked by such question, asking
> > > > why we would do anything like this.
> > > >
> > > > This kind of reaction actually doesn't surprise me a lot, so I jumped
> > in
> > > > and challenged this config option even
> > > > after the FLIP had already been accepted.
> > > >
> > > > If we only have to define the execution behavior for multiple
> > statements
> > > in
> > > > SQL client, we should only introduce
> > > > a config option which would tell users it's affection scope by its
> > name.
> > > > Prefixing with "table" is definitely not a good
> > > > idea here.
> > > >
> > > > Best,
> > > > Kurt
> > > >
> > > >
> > > > On Fri, Feb 26, 2021 at 9:39 PM Leonard Xu 
> wrote:
> > > >
> > > >> Hi, all
> > > >>
> > > >> Look like there’s only one divergence about option [ table |
> > sql-client
> > > >> ].dml-sync in this thread, correct me if I’m wrong.
> > > >>
> > > >> 1. Leaving the context of this thread, from a user's perspective,
> > > >> the table.xx configurations should take effect in Table API & SQL,
> > > >> the sql-client.xx configurations should only take effect in
> > sql-client.
> > > >>   In my(the user's) opinion, other explanations are
> counterintuitive.
> > > >>
> > > >> 2.  It should be pointed out that both all existed table.xx
> > > configurations
> > > >> like table.exec.state.ttl, table.optimizer.agg-phase-strategy,
> > > >> table.local-time-zone,etc..  and the proposed sql-client.xx
> > > configurations
> > > >> like sql-client.verbose, sql-client.execution.max-table-result.rows
> > > >> comply with this convention.
> > > >>
> > > >> 3. Considering the portability to support different CLI tools
> > > (sql-client,
> > > >> sql-gateway, etc.), I prefer table.dml-sync.
> > > >>
> > > >> In addition, I think sql-client/sql-gateway/other CLI tools can be
> > > placed
> > > >> out of flink-table module even in an external project, this should
> not
> > > >> affect our conclusion.
> > > >>
> > > >>
> > > >> Hope this can help you.
> > > >>
> > > >>
> > > >> Best,
> > > >> Leonard
> > > >>
> > > >>
> > > >>
> > > >>> 在 2021年2月25日，18:51，Shengkai Fang  写道：
> > > >>>
> > > >>> Hi, everyone.
> > > >>>
> > > >>> I do some summaries about the discussion about the option. If the
> > > summary
> > > >>> has errors, please correct me.
> > > >>>
> > > >>> `table.dml-sync`:
> > > >>> - take effect for `executeMultiSql` and sql client
> > > >>> - benefit: SQL script portability. One script for all platforms.
> > > >>> - drawback: Don't work for `TableEnvironment#executeSql`.
> > > >>>
> > > >>> `table.multi-dml-sync`:
> > > >>> - take effect for `executeMultiSql` and sql client
> > > >>> - benefit: SQL script portability
> > > >>> - drawback: It's confused when the sql script has one dml statement
> > but
> > > >>> need to set option `table.multi-dml-sync`
> > > >>>
> > > >>> `client.dml-sync`:
> > > >>> - take effect for sql client only
> > > >>> - benefit: clear definition.
> > > >>> - drawback: Every platform needs to define its own option. Bad SQL
> > > script
> > > >>> portability.
> > > >>>
> > > >>> Just as Jark said, I think the `table.dml-sync` is a good choice if
> > we
> > > >> can
> > > >>> extend its scope and make this option works for `executeSql`.
> > > >>> It's straightforward and users can use this option now in table
> api.
> > > The
> > > >>> drawback is the  `TableResult#await` plays the same role as the

[VOTE] FLIP-162: Consistent Flink SQL time function behavior

[Leonard Xu]

Hi all,

I would like to start the vote for FLIP-162 [1], which has been discussed and
reached a consensus in the discussion thread [2].

Please vote +1 to approve the FLIP, or -1 with a comment.

The vote will be open until March 5th (72h), unless there is an objection or 
not enough votes.

Best,
Leonard

[1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-162%3A+Consistent+Flink+SQL+time+function+behavior
 

[2]http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-162-Consistent-Flink-SQL-time-function-behavior-tc48116.html
 

[jira] [Created] (FLINK-21549) Support json serialization/deserialization for the push-down results

[godfrey he (Jira)]

godfrey he created FLINK-21549:
--

 Summary: Support json serialization/deserialization for the 
push-down results
 Key: FLINK-21549
 URL: https://issues.apache.org/jira/browse/FLINK-21549
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: godfrey he
 Fix For: 1.13.0


Currently, Flink supports different kinds of push-down: projection, filter, 
watermark, limit, partition, reading-metadata. The push-down result should also 
be serialized to the json plan and then we can get the correct ExecNode plan 
from the json plan. To avoid modifying each connector, we can store the 
push-down result into the TableSourceTable during the Calcite optimization 
phase, and store the push-down result into DynamicTableSourceSpec in the 
ExecNode graph.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-165: Operator's Flame Graphs

[Alexander Fedulov]

Thanks Henry, I have some issues with subscribing with our domain (it is an
alias).

@All, this thread is a duplicate caused by some technical issues, sorry for
that. Please ignore it and use the previous one with the same title instead
for the discussion:

http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-165-Operator-s-Flame-Graphs-td49097.html

Best,
Alexander



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

Re: [DISCUSS] FLIP-165: Operator's Flame Graphs

[Henry Saputra]

Hi Alexander,

I had to moderate and accept your email to dev@ list. Could you subscribe
to dev@ list for Apache Flink [1] to continue getting updates from your
discussion thread?

Thanks,

Henry

[1] https://flink.apache.org/community.html#mailing-lists

On Mon, Mar 1, 2021 at 3:42 PM Alexander Fedulov 
wrote:

> Hi All,
>
> I would like to start a discussion for FLIP-165: Operator's Flame Graphs
> [1]
>
> A Flame Graph [2] is a visualization that is very effective for providing
> answers to the questions like:
> - Which methods are currently consuming CPU resources?
> - How CPU utilization by one method compares to the others?
> - Which series of calls on the stack led to executing a particular method?
>
> I have already opened a PR [3] that represents the implementation approach
> proposed in the FLIP. It supports both on-CPU and off-CPU [4] Flame Graphs.
>
> Looking forward to your feedback.
>
> P.S: I would like to give kudos to David Moravek for his prototyping work
> [5] on this feature. Although the proposed implementation significantly
> diverges from his prototype on the Flink side, the work done on connecting
> the d3-flame-graph library to the right data structure retrieved from Flink
> was instrumental for enabling this feature.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-165%3A+Operator%27s+Flame+Graphs
> [2] http://www.brendangregg.com/flamegraphs.html
> [3] https://github.com/apache/flink/pull/15054
> [4] http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
> [5]
>
> https://issues.apache.org/jira/browse/FLINK-13550?focusedCommentId=17083026=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17083026
>
>
> Best,
> --
>
> Alexander Fedulov | Solutions Architect
>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward - The Apache Flink Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
>
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
> Wehner
>
>
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>

[DISCUSS] FLIP-165: Operator's Flame Graphs

[Alexander Fedulov]

Hi All,

I would like to start a discussion for FLIP-165: Operator's Flame Graphs [1]

A Flame Graph [2] is a visualization that is very effective for providing
answers to the questions like:
- Which methods are currently consuming CPU resources?
- How CPU utilization by one method compares to the others?
- Which series of calls on the stack led to executing a particular method?

I have already opened a PR [3] that represents the implementation approach
proposed in the FLIP. It supports both on-CPU and off-CPU [4] Flame Graphs.

Looking forward to your feedback.

P.S: I would like to give kudos to David Moravek for his prototyping work
[5] on this feature. Although the proposed implementation significantly
diverges from his prototype on the Flink side, the work done on connecting
the d3-flame-graph library to the right data structure retrieved from Flink
was instrumental for enabling this feature.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-165%3A+Operator%27s+Flame+Graphs
[2] http://www.brendangregg.com/flamegraphs.html
[3] https://github.com/apache/flink/pull/15054
[4] http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
[5]
https://issues.apache.org/jira/browse/FLINK-13550?focusedCommentId=17083026=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17083026

Best,

--

Alexander Fedulov | Solutions Architect



Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner

[DISCUSS] FLIP-165: Operator's Flame Graphs

[Alexander Fedulov]

Hi All,

I would like to start a discussion for FLIP-165: Operator's Flame Graphs [1]

A Flame Graph [2] is a visualization that is very effective for providing
answers to the questions like:
- Which methods are currently consuming CPU resources?
- How CPU utilization by one method compares to the others?
- Which series of calls on the stack led to executing a particular method?

I have already opened a PR [3] that represents the implementation approach
proposed in the FLIP. It supports both on-CPU and off-CPU [4] Flame Graphs.

Looking forward to your feedback.

P.S: I would like to give kudos to David Moravek for his prototyping work
[5] on this feature. Although the proposed implementation significantly
diverges from his prototype on the Flink side, the work done on connecting
the d3-flame-graph library to the right data structure retrieved from Flink
was instrumental for enabling this feature.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-165%3A+Operator%27s+Flame+Graphs
[2] http://www.brendangregg.com/flamegraphs.html
[3] https://github.com/apache/flink/pull/15054
[4] http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html
[5]
https://issues.apache.org/jira/browse/FLINK-13550?focusedCommentId=17083026=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17083026


Best,
--

Alexander Fedulov | Solutions Architect


Follow us @VervericaData

--

Join Flink Forward - The Apache Flink Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--

Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner





--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

Re: Watermarks when reading from file

[Dominik Wosiński]

Hey Till,
You were obviously right, my bad here. My math was incorrect. The correct
reasoning is that indeed first 5 days of october will be added to the
window number 1 and the rest of days will end up in the second window.
Solved!

Thanks a lotte,
Best Regards,
Dom.

Re: Watermarks when reading from file

[Dominik Wosiński]

Hey,
Thanks for the answer, as I've mentioned in the email the data range is
only 30 days, for the tests I've used  the data from october so I basically
have timestamps starting at midningt of 1st october 2020 and finishing at
23:59 30 october 2020, so if I understand correctly this shouldn't cause
the double windowing, but correct me if I am wrong here.

Best Regards,
Dom.

Re: [DISCUSS] Apache Flink Jira Process

[Roman Khachatryan]

Hi,

Thanks for the proposal Konstantin,
I like the ideas expressed there.

I am a bit concerned about the new issue type "Technical Debt". In contrast
to other issue types, it doesn't imply that someone will likely work on
that. So it can linger until the bot closes it.
Probably we need some rules requiring a person opening such a ticket to
have an intention to work on it in the near future?
Another approach would be some wiki space.

As for the trivial priority, I would remove it and (use labels where
appropriate) as you suggested.

Regards,
Roman


On Mon, Mar 1, 2021 at 11:53 AM Konstantin Knauf 
wrote:

> Hi Dawid,
>
> Thanks for the feedback. Do you think we should simply get rid of the
> "Trivial" priority then and use the "starter" label more aggressively?
>
> Best,
>
> Konstantin
>
> On Mon, Mar 1, 2021 at 11:44 AM Dawid Wysakowicz 
> wrote:
>
> > Hi Konstantin,
> >
> > I also like the idea.
> >
> > Two comments:
> >
> > * you describe the "Trivial" priority as one that needs to be
> > implemented immediately. First of all it is not used to often, but I
> > think the way it works now is similar with a "starter" label. Tasks that
> > are not bugs, are easy to implement and we think they are fine to be
> > taken by newcomers. Therefore they do not fall in my mind into
> > "immediately" category.
> >
> > * I would still deprioritise test instabilities. I think there shouldn't
> > be a problem with that. We do post links to all failures therefore it
> > will automatically priortise the tasks according to failure frequencies.
> >
> > Best,
> >
> > Dawid
> >
> > On 01/03/2021 09:38, Konstantin Knauf wrote:
> > > Hi Xintong,
> > >
> > > yes, such labels would make a lot of sense. I added a sentence to the
> > > document.
> > >
> > > Thanks,
> > >
> > > Konstantin
> > >
> > > On Mon, Mar 1, 2021 at 8:51 AM Xintong Song 
> > wrote:
> > >
> > >> Thanks for driving this discussion, Konstantin.
> > >>
> > >> I like the idea of having a bot reminding reporter/assignee/watchers
> > about
> > >> inactive tickets and if needed downgrade/close them automatically.
> > >>
> > >> My two cents:
> > >> We may have labels like "downgraded-by-bot" / "closed-by-bot", so that
> > it's
> > >> easier to filter and review tickets updated by the bot.
> > >> We may want to review such tickets (e.g., monthly) in case a valid
> > ticket
> > >> failed to draw the attention of relevant committers and the reporter
> > >> doesn't know who to ping.
> > >>
> > >> Thank you~
> > >>
> > >> Xintong Song
> > >>
> > >>
> > >>
> > >> On Sat, Feb 27, 2021 at 1:37 AM Till Rohrmann 
> > >> wrote:
> > >>
> > >>> Thanks for starting this discussion Konstantin. I like your proposal
> > and
> > >>> also the idea of automating the tedious parts of it via a bot.
> > >>>
> > >>> Cheers,
> > >>> Till
> > >>>
> > >>> On Fri, Feb 26, 2021 at 4:17 PM Konstantin Knauf 
> > >>> wrote:
> > >>>
> >  Dear Flink Community,
> > 
> >  I would like to start a discussion on improving and to some extent
> > >> simply
> >  defining the way we work with Jira. Some aspects have been
> discussed a
> >  while back [1], but I would like to go a bit beyond that with the
> > >>> following
> >  goals in mind:
> > 
> > 
> > -
> > 
> > clearer communication and expectation management with the
> community
> > -
> > 
> >    a user or contributor should be able to judge the urgency of a
> > >>> ticket
> >    by its priority
> >    -
> > 
> >    if a ticket is assigned to someone the expectation that
> someone
> > >> is
> >    working on it should hold
> >    -
> > 
> > generally reduce noise in Jira
> > -
> > 
> > reduce overhead of committers to ask about status updates of
> > contributions or bug reports
> > -
> > 
> >    “Are you still working on this?”
> >    -
> > 
> >    “Are you still interested in this?”
> >    -
> > 
> >    “Does this still happen on Flink 1.x?”
> >    -
> > 
> >    “Are you still experiencing this issue?”
> >    -
> > 
> >    “What is the status of the implementation”?
> >    -
> > 
> > while still encouraging users to add new tickets and to leave
> > >> feedback
> > about existing tickets
> > 
> > 
> >  Please see the full proposal here:
> > 
> > 
> > >>
> >
> https://docs.google.com/document/d/19VmykDSn4BHgsCNTXtN89R7xea8e3cUIl-uivW8L6W8/edit#
> >  .
> > 
> >  The idea would be to discuss this proposal in this thread. If we
> come
> > >> to
> > >>> a
> >  conclusion, I'd document the proposal in the wiki [2] and we would
> > then
> >  vote on it (approval by "Lazy Majority").
> > 
> >  Cheers,
> > 
> >  Konstantin
> > 
> >  [1]
> > 
> > 
> > >>
> >
> 

[jira] [Created] (FLINK-21548) keyBy operation produces skewed event distribution with low-cardinality keys

[Iaroslav Zeigerman (Jira)]

Iaroslav Zeigerman created FLINK-21548:
--

 Summary: keyBy operation produces skewed event distribution with 
low-cardinality keys
 Key: FLINK-21548
 URL: https://issues.apache.org/jira/browse/FLINK-21548
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream, Runtime / Coordination, Runtime / Task
Affects Versions: 1.12.1, 1.11.0
Reporter: Iaroslav Zeigerman
 Attachments: Screen Shot 2021-03-01 at 10.52.31 AM.png, Screen Shot 
2021-03-01 at 10.54.42 AM.png, Screen Shot 2021-03-01 at 10.57.33 AM.png

When the cardinality of keys matches the existing parallelism not all 
downstream tasks are utilized in the downstream operator. Even those that are 
utilized are not utilized evenly.

For example if I have 500 unique keys [0, 500) only 313 downstream tasks (out 
of 500) will receive any records at all. 
This behavior can easily be reproduced with the following test case:
{code:scala}
import org.apache.flink.runtime.state.KeyGroupRangeAssignment
import scala.util.Random

object Test {

  val parallelism = 500
  val recordsNum  = 100

  def run(): Unit = {
val recordIds = (0 to recordsNum).map(_ % parallelism)
val tasks = recordIds.map(selectTask)

println(s"Total unique keys: ${recordIds.toSet.size}")
println(s"Key distribution: 
${recordIds.groupBy(identity).mapValues(_.size).toVector.sortBy(-_._2)}")
println("===")
println(s"Tasks involved: ${tasks.toSet.size}")
println(s"Record distribution by task: 
${tasks.groupBy(identity).mapValues(_.size).toVector.sortBy(-_._2)}")
  }

  def selectTask(key: Int): Int =
KeyGroupRangeAssignment.assignToKeyGroup(
  key,
  parallelism
)
}
{code}
Which produces the following results:
{noformat}
Total unique keys: 500
Key distribution: Vector((0,2001), (69,2000), ..., (232,2000), (100,2000))
===
Tasks involved: 313
Record distribution by task: Vector((147,1), (248,1), ..., (232,2000), 
(100,2000))
{noformat}
Record distribution visualized:
 !Screen Shot 2021-03-01 at 10.52.31 AM.png!

I have determined that in order to achieve the utilization of all tasks the 
number of unique keys should be at least 5 times of the parallelism value. The 
relation between number of unique keys and a fraction of utilized tasks appear 
to be exponential:
 !Screen Shot 2021-03-01 at 10.54.42 AM.png!  

But with 5x number of keys the skew is still quite significant:
!Screen Shot 2021-03-01 at 10.57.33 AM.png!

Given that keys used in my test are integer values for which `hashCode` returns 
the value itself I tend to believe that the skew is caused by the Flink's 
murmur hash implementation which is used 
[here|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyGroupRangeAssignment.java#L76].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Watermarks when reading from file

[Till Rohrmann]

Hi Dominik,

I think the problem could be that TumblingTimeWindows don't start with the
timestamp of the first arriving event but start at a multiple of the window
length. So when defining a 90 day tumbling window you define a window from
0 - 89, 90 - 179,  If your data ranges from day 79 - 109, then it would
fall into two windows.

Cheers,
Till

On Mon, Mar 1, 2021 at 5:34 PM Dominik Wosiński  wrote:

> Hey,
> I have a question regarding DataStream created from multiple files in s3. I
> have several files in AWS s3, say the path is s3://files/, and then there
> are several folders for different days, so in the end the full paths look
> like : s3://files/day=1/file.parquet, s3://files/day=2/file.parquet. I
> wanted to read all the files and sort them via some specific value.
>
> I thought that I could use the fact that the Long.MAX watermark is
> generated, so I've decided to use event time window of size larger than the
> data in files.
>
> So, I have something like:
>
> val inputFormat =new ParquetAvroInputFormat[TestData](new Path(
>   ("s3a://files/")))
> inputFormat.setNestedFileEnumeration(true)
> val ste = StreamExecutionEnvironment.createLocalEnvironment(1)
> ste.createInput(inputFormat)
>   .assignTimestampsAndWatermarks(
>  new OutOfOrdernessWatermarkStrategy[TestData](3000, _.getTimestamp))
>   .keyBy(_.getKey)
>   .timeWindow(Time.days(90))
>   .sideOutputLateData(sideOutput)
>   .process(new ProcessWindowFunction[TestData, TestData, String,
> TimeWindow] {
> override def process(key: String, context: Context,
>  elements: Iterable[TestData],
>  out: Collector[TestData]): Unit = {
>   println("Processing: " + elements.toList.size + " for key:" + key)
>   elements.toSeq.sortBy(_.getTimestamp)
> .foreach(out.collect(_))
> }
>   })
>
>
>
>
>
>
> *ParquetAvroInputFormat and OutOfOrdernessWatermarkStrategy are my classes
> but those do not cause the issue described here.*
> The data in files is kept for 30 days, so there is no way that window will
> be closed before the files are closed and *Long.Max* timestamp generated.
>
> Now, the problem I am observing is that I would expect to see one message
> printed per key, since the parallelism is one. But for some reason I am
> observing that for some of the keys(most of them really) there are two
> windows created*. *I have  30 unique keys and each key contains around 1M
> records. And The output I can see is more or less like that:
>
> 1. Several messages about Switching to Random IO seek policy
> 2. Print for most of the keys present in the dataset (but the counts are
> quite small, most of them around 100k, some as small as few hundred)
> 3. More Switching to Random IO seek policy
> 4. Print again for some keys, but now the counts are much higher.
>
> So, the total count of all processed values is correct. It's just I am
> interested why the window gets invoked twice.
>
> Thanks in advance,
> Best Regards,
> Dom.
>

Re: [DISCUSS] Enabling more dynamic, or metadata-driven behaviors

[Till Rohrmann]

Hi Maciej,

The Flink community highly appreciates any kind of feedback and improvement
suggestions. W/o going into more detail for the problems you've
encountered, it is very likely that other people have run into something
similar as well. Hence, it is a good idea to share these limitations and to
discuss possible solutions for them. If you already have a working solution
then this could be a good starting point for the discussion and in the best
case we can take the solution as it is. What I would suggest is to create
for each problem a separate JIRA issue and start a separate discussion if
required.

One thing to note is that the community is working towards the feature
freeze for Flink 1.13. Due to this it can happen that people might respond
a bit later.

Cheers,
Till

On Mon, Mar 1, 2021 at 12:58 PM Maciej Obuchowski <
obuchowski.mac...@gmail.com> wrote:

> While working on project that's strongly metadata-driven I've had to
> overcome several deficiencies, or assumptions in Flink code. Project
> involves reading from hundreds (possibly thousands) kafka topics, each
> containing avro messages with different schemas. Then, various
> transformations and aggregations are applied to that data. Then
> transformed and aggregated data is written to several sinks - JDBC,
> file, etc. On of the goals is making simple changes possible, like
> adding topics, changing transformations or schemas without writing
> code - so all is metadata driven.
>
> Some of the things I've encountered:
>
> It's not possible to read avro messages from kafka without somehow
> providing reader schema from user code. It's simply impractical to
> keep 1000s of schemas (and flink's AvroDeserializationSchemas) around,
> or even worse - Kafka consumers per topic/schema.
> Solution was to use custom deserialization schema similar to this approach
>
> https://stackoverflow.com/questions/58849635/is-it-possible-to-deserialize-avro-messageconsuming-message-from-kafka-without
>
> Another one was regarding serialization. This might be possible, but I
> haven't found way to serialize avro's generic data types without Kryo
> fallback, which hurts performance. I've resorted to manually
> serializing record to bytes[] and deserializing it in the next task.
> Also, see this mail thread where Lasse had similar case with Jackson's
> objects.
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Jackson-object-serialisations-td41691.html
>
> Third one was JDBC sink behavior. Currently, it assumes that it will
> be used to insert data to one table, provided statically. This one has
> the worst implication, because sink consumes one database connection,
> which are quite expensive when we're talking about hundreds of them.
> In this case there was no other way than forking Flink and providing
> another implementation of JdbcBatchStatementExecutor, that can create
> statements for multiple tables.
>
> After this lenghty introduction, my question is basically: do Flink
> developers and community welcome further discussion and contributrions
> aimed at easing those, and similar pain points regarding more
> "dynamic" behavior of Flink? I'm willing to contribute, but don't want
> to just throw code over the wall if no one else is interested in using
> it.
>
> Thanks,
> Maciej
>

Re: [VOTE] Release 1.12.2, release candidate #2

[Piotr Nowojski]

+1 (binding)

For the RC2 I have additionally confirmed that "stop-with-savepoint", and
"stop-with-savepoint --drain" seems to be working.

Piotrek

pon., 1 mar 2021 o 11:18 Matthias Pohl  napisał(a):

> Thanks for managing release 1.12.2, Yuan & Roman.
>
> +1 (non-binding)
>
> - Verified checksums and GPG of artifacts in [1]
> - Build the sources locally without errors
> - Started a local standalone cluster and deployed WordCount without
> problems (no suspicious logs identified)
> - Verified FLINK-21030 [2] by running the example jobs from the
> FLINK-21030-related SavepointITCase tests
>
> Best,
> Matthias
>
> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.2-rc2/
> [2] https://issues.apache.org/jira/browse/FLINK-21030
>
> On Sun, Feb 28, 2021 at 2:41 PM Yuan Mei  wrote:
>
> > Hey Roman,
> >
> > Thank you very much for preparing RC2.
> >
> > +1 from my side.
> >
> > 1. Verified Checksums and GPG signatures.
> > 2. Verified that the source archives do not contain any binaries.
> > 3. Successfully Built the source with Maven.
> > 4. Started a local Flink cluster, ran the streaming WordCount example
> with
> > WebUI,
> > checked the output and JM/TM log, no suspicious output/log.
> > 5. Repeat Step 4 with the binary release as well, no suspicious
> output/log.
> > 6. Checked for source and binary release to make sure both an Apache
> > License file and a NOTICE file are included.
> > 7. Manually verified that no pom file changes between 1.12.2-rc1 and
> > 1.12.2-rc2; no obvious license problem.
> > 8. Review the release PR for RC2 updates, and double confirmed the
> > change-list for 1.12.2.
> >
> > Best,
> > Yuan
> >
> > On Sat, Feb 27, 2021 at 7:19 AM Roman Khachatryan 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #1 for the version
> > 1.12.2,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release and binary convenience releases to
> > be
> > > deployed to dist.apache.org [2], which are signed with the key with
> > > fingerprint 0D545F264D2DFDEBFD4E038F97B4625E2FCF517C [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag "release-1.12.2-rc2" [5],
> > > * website pull request listing the new release and adding announcement
> > blog
> > > post [6].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > [1]
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12349502=12315522
> > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.2-rc2/
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1414/
> > > [5] https://github.com/apache/flink/releases/tag/release-1.12.2-rc2
> > > [6] https://github.com/apache/flink-web/pull/418
> > >
> > > Regards,
> > > Roman
> > >
>

[jira] [Created] (FLINK-21547) Fix improper log level in TwoPhaseCommitSinkFunction

[tim yu (Jira)]

tim yu created FLINK-21547:
--

 Summary: Fix improper log level in TwoPhaseCommitSinkFunction
 Key: FLINK-21547
 URL: https://issues.apache.org/jira/browse/FLINK-21547
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Reporter: tim yu


{code:java}
LOG.info(
"{} - checkpoint {} complete, committing transaction {} 
from checkpoint {}",
name(),
checkpointId,
pendingTransaction,
pendingTransactionCheckpointId);

logWarningIfTimeoutAlmostReached(pendingTransaction);
try {
commit(pendingTransaction.handle);
} catch (Throwable t) {
if (firstError == null) {
firstError = t;
}
}

LOG.debug("{} - committed checkpoint transaction {}", name(), 
pendingTransaction);
{code}
I think "committing transaction ..." should be the same log level as " 
committed checkpoint transaction ...".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Watermarks when reading from file

[Dominik Wosiński]

Hey,
I have a question regarding DataStream created from multiple files in s3. I
have several files in AWS s3, say the path is s3://files/, and then there
are several folders for different days, so in the end the full paths look
like : s3://files/day=1/file.parquet, s3://files/day=2/file.parquet. I
wanted to read all the files and sort them via some specific value.

I thought that I could use the fact that the Long.MAX watermark is
generated, so I've decided to use event time window of size larger than the
data in files.

So, I have something like:

val inputFormat =new ParquetAvroInputFormat[TestData](new Path(
  ("s3a://files/")))
inputFormat.setNestedFileEnumeration(true)
val ste = StreamExecutionEnvironment.createLocalEnvironment(1)
ste.createInput(inputFormat)
  .assignTimestampsAndWatermarks(
 new OutOfOrdernessWatermarkStrategy[TestData](3000, _.getTimestamp))
  .keyBy(_.getKey)
  .timeWindow(Time.days(90))
  .sideOutputLateData(sideOutput)
  .process(new ProcessWindowFunction[TestData, TestData, String, TimeWindow] {
override def process(key: String, context: Context,
 elements: Iterable[TestData],
 out: Collector[TestData]): Unit = {
  println("Processing: " + elements.toList.size + " for key:" + key)
  elements.toSeq.sortBy(_.getTimestamp)
.foreach(out.collect(_))
}
  })






*ParquetAvroInputFormat and OutOfOrdernessWatermarkStrategy are my classes
but those do not cause the issue described here.*
The data in files is kept for 30 days, so there is no way that window will
be closed before the files are closed and *Long.Max* timestamp generated.

Now, the problem I am observing is that I would expect to see one message
printed per key, since the parallelism is one. But for some reason I am
observing that for some of the keys(most of them really) there are two
windows created*. *I have  30 unique keys and each key contains around 1M
records. And The output I can see is more or less like that:

1. Several messages about Switching to Random IO seek policy
2. Print for most of the keys present in the dataset (but the counts are
quite small, most of them around 100k, some as small as few hundred)
3. More Switching to Random IO seek policy
4. Print again for some keys, but now the counts are much higher.

So, the total count of all processed values is correct. It's just I am
interested why the window gets invoked twice.

Thanks in advance,
Best Regards,
Dom.

Re: [VOTE] FLIP-151: Incremental snapshots for heap-based state backend

[Piotr Nowojski]

Thanks Roman for coming up with this proposal and driving this topic:

+1 (binding) from my side

Piotrek

pon., 1 mar 2021 o 10:12 Roman Khachatryan  napisał(a):

> Hi everyone,
>
> since the discussion [1] about FLIP-151 [2] seems to have reached a
> consensus, I'd like to start a formal vote for the FLIP.
>
> Please vote +1 to approve the FLIP, or -1 with a comment. The vote will be
> open at least until Wednesday, Mar 3rd.
>
> [1] https://s.apache.org/flip-151-discussion
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend
>
> Regards,
> Roman
>

Re: [DISCUSS] Splitting User support mailing list

[Roman Khachatryan]

@tzuli...@apache.org 
> instead of splitting into “sub-lists”, we should simply have dedicated
“sub-topic maintainers” assigned.
I think this could also work, but some mails may fall between the filters.

@i...@ververica.com 
I guess the previous decision about StateFun ML was made in a bit different
context: no other sub-lists and no data about the list.

Regards,
Roman


On Mon, Mar 1, 2021 at 2:59 PM Igal Shilman  wrote:

> Hi Roman,
>
> Regarding StateFun having a separate mailing list, I'm ok with it going
> either-way, however when we first contributed
> the project there was already a discussion about having a separate mailing
> list for StateFun [1] and the feedback was
> having StateFun be part of the regular mailing list.
>
>
> [1] https://www.mail-archive.com/dev@flink.apache.org/msg31464.html
>
> On Mon, Mar 1, 2021 at 12:25 PM Tzu-Li (Gordon) Tai 
> wrote:
>
>> Hi,
>>
>> I feel that the issues Roman has pointed out so far, is less a problem of
>> all topics (SQL / PyFlink / StateFun) being on the same list, and more a
>> problem that we are missing dedicated groups of “user support shepherds”
>> who are specifically responsible for individual topics on a day-to-day
>> basis.
>>
>> In the distant past, we used to assign shepherds for individual components
>> in Flink.
>> Perhaps something similar to that, but specifically for daily user mailing
>> lists support, is already sufficient to solve the mentioned problems.
>> So essentially, instead of splitting into “sub-lists”, we should simply
>> have dedicated “sub-topic maintainers” assigned.
>>
>> For example, for myself, I set a filter on my email client to look
>> specifically for “Stateful Functions / StateFun” mentions, and tag it
>> appropriately.
>> This already allows me to concentrate on StateFun questions, without
>> losing
>> the exposure to other things happening in the wider Flink project.
>> As far as I can tell, except for some more tricky questions, the
>> turnaround
>> time for StateFun user questions has been ok so far.
>>
>> What do you think?
>>
>> Cheers,
>> Gordon
>>
>> On Mon, Mar 1, 2021 at 6:56 PM Roman Khachatryan 
>> wrote:
>>
>> > Thanks for your replies!
>> >
>> > @Konstantin Knauf 
>> > > Why do you think the quality and speed of answers would improve with
>> > dedicated lists?
>> > If there is a question on something that you are not an expert in; then
>> you
>> > either have to
>> > - pull in someone who is more experienced in it (more time on hops,
>> esp. if
>> > the pulled in person isn't available)
>> > - or learn it and answer yourself (more time on learning and still
>> higher
>> > chance of missing something)
>> >
>> > @Timo Walther  and @Dawid Wysakowicz
>> > 
>> > > I fear that we are creating potential silos where a team doesn't know
>> > > what is going on in the other teams.
>> > I think some specialization is unavoidable in a big project like Flink
>> or
>> > Linux (which also has separate lists).
>> > And user support ML doesn't seem to me the right tool to deal with it.
>> >
>> > @Dawid Wysakowicz 
>> > > Personally I don't find it problematic. I often find the subjects
>> quite
>> > > descriptive, they often include tags or mention which API they refer
>> to.
>> > Yes, but that only means that the sender would already know the "right"
>> > list.
>> >
>> > @Konstantin Knauf  and @j...@apache.org <
>> > j...@apache.org>
>> >
>> > I agree that there are crosscutting areas; and also a chance of sending
>> a
>> > message to the wrong topic.
>> > But splitting doesn't change anything here: if a SQL question for
>> example
>> > is asked on StateFun ML then
>> > we still have the options above (plus an option to redirect user to the
>> > other list).
>> >
>> > Regards,
>> > Roman
>> >
>> >
>> > On Mon, Mar 1, 2021 at 11:30 AM Dawid Wysakowicz <
>> dwysakow...@apache.org>
>> > wrote:
>> >
>> > > As others I'd also rather be -1 on splitting (even splitting out the
>> > > statefun).
>> > >
>> > > Personally I don't find it problematic. I often find the subjects
>> quite
>> > > descriptive, they often include tags or mention which API they refer
>> to.
>> > > If they don't I am quite sure having separate sub-lists would not help
>> > > in those cases anyway. I agree with the others that splitting the list
>> > > would make the cross communication harder and create knowledge silos.
>> > >
>> > > It would also incur more requirements on users which already often
>> find
>> > > ML counter intuitive (See e.g. the discussion about adding a Flink
>> slack)
>> > >
>> > > Best,
>> > >
>> > > Dawid
>> > >
>> > > On 01/03/2021 11:20, Timo Walther wrote:
>> > > > I would vote -0 here.
>> > > >
>> > > > I fear that we are creating potential silos where a team doesn't
>> know
>> > > > what is going on in the other teams.
>> > > >
>> > > > Regards,
>> > > > Timo
>> > > >
>> > > >
>> > > > On 01.03.21 10:47, Jark Wu wrote:
>> > > >> I also have some concerns about splitting python and sql.
>> > > >> Because I have seen some 

Re: [DISCUSS] FLIP-162: Consistent Flink SQL time function behavior

[Leonard Xu]

> I'm fine with your proposal. But once we see users asking for better unified 
> semantics, we should not hesitate to introduce an option to give them more 
> flexibility.

Yes, I agree that we should introduce the option once we received feedback 
requirement from user input.  I will update this tip to FLIP-162 future plan 
section as well.

If all of us have no more opinions, I’d like start a VOTE thread.


Best,
Leonard


> On 01.03.21 12:59, Leonard Xu wrote:
>> Thanks Kurt and Timo for the feedbacks.
 I prefer to not introduce such config until we have to. Leonard's proposal
 already makes almost all users happy thus I think we can still wait.
>> I could understand Kurt’s concern that we don't need rush to introduce this 
>> option util we have to, Especially we don’t sure the right behavior of time 
>> function SQL standard about streaming part(SQL standard only contains batch 
>> part ), it may change in the future.
>>> However, one concern I would like to raise is still the bounded stream 
>>> processing. Users will not have the possibility to use query-start 
>>> semantics. For example, if users would like to use match_recognize on a CSV 
>>> file, they cannot use query-start
>>> timestamps.
>> I also think Timo’s concern that bounded cases may need query-start is 
>> reasonable in some user cases. Although it’s only a few scenes at present 
>> from my side, it will change in the future too.
>> As a tradeoff, I propose we could follow my last proposal as a conservative 
>> plan in the first step,
>> and then introduce the if there’re enough user requirement/feedback that 
>> they need the power to control the time function evaluation,
>> What do you think?
>> Best,
>> Leonard
 Best,
 Kurt
 On Mon, Mar 1, 2021 at 3:58 PM Timo Walther  wrote:
> and btw it is interesting to notice that AWS seems to do the approach
> that I suggested first.
> 
> All functions are SQL standard compliant, and only dedicated functions
> with a prefix such as CURRENT_ROW_TIMESTAMP divert from the standard.
> 
> Regards,
> Timo
> 
> On 01.03.21 08:45, Timo Walther wrote:
>> How about we simply go for your first approach by having [query-start,
>> row, auto] as configuration parameters where [auto] is the default?
>> 
>> This sounds like a good consensus where everyone is happy, no?
>> 
>> This also allows user to restore the old per-row behavior for all
>> functions that we had before Flink 1.13.
>> 
>> Regards,
>> Timo
>> 
>> 
>> On 26.02.21 11:10, Leonard Xu wrote:
>>> Thanks Joe for the great investigation.
>>> 
>>> 
 • Generally urging for semantics (batch > time of first query
 issued, streaming > row level).
 I discussed the thing now with Timo & Stephan:
 • It seems to go towards a config parameter, either [query-start,
 row]  or [query-start, row, auto] and what is the default?
 • The main question seems to be: are we pushing the default
 towards streaming. (probably related the insert into behaviour in the
 sql client).
>>> 
>>> 
>>> It looks like opinions in this thread and user inputs agreed that:
>>> batch should use time of first query, streaming should use row level.
>>> Based on these, we should keep row level for streaming and query start
>>> for batch just like the config parameter value [auto].
>>> 
>>> Currently Flink keeps row level for time function in both batch and
>>> streaming job, thus we only need to update the behavior in batch.
>>> 
>>> I tend to not expose an obscure configuration to users especially it
>>> is semantics-related.
>>> 
>>> 1.We can make [auto] as a default agreement，for current Flink
>>> streaming users，they feel nothing has changed，for current Flink
>>> batch users，they feel Flink batch is corrected to other good batch
>>> engines as well as SQL standard. We can also provide a function
>>> CURRENT_ROW_TIMESTAMP[1] for Flink batch users who want row level time
>>> function.
>>> 
>>> 2. CURRENT_ROW_TIMESTAMP can also be used in Flink streaming, it has
>>> clear semantics, we can encourage users to use it.
>>> 
>>> In this way, We don’t have to introduce an obscure configuration
>>> prematurely while making all users happy
>>> 
>>> How do you think?
>>> 
>>> Best,
>>> Leonard
>>> [1]
>>> 
> https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-current-row-timestamp.html
>>> 
>>> 
>>> 
>>> 
 Hope this helps,
 
 Thanks,
 Joe
 
> On 19.02.2021, at 10:25, Leonard Xu  wrote:
> 
> Hi, Joe
> 
> Thanks for volunteering to investigate the user data on this topic.
> Do you
> have any progress here?
> 
> Thanks,

Re: [DISCUSS] Splitting User support mailing list

[Igal Shilman]

Hi Roman,

Regarding StateFun having a separate mailing list, I'm ok with it going
either-way, however when we first contributed
the project there was already a discussion about having a separate mailing
list for StateFun [1] and the feedback was
having StateFun be part of the regular mailing list.


[1] https://www.mail-archive.com/dev@flink.apache.org/msg31464.html

On Mon, Mar 1, 2021 at 12:25 PM Tzu-Li (Gordon) Tai 
wrote:

> Hi,
>
> I feel that the issues Roman has pointed out so far, is less a problem of
> all topics (SQL / PyFlink / StateFun) being on the same list, and more a
> problem that we are missing dedicated groups of “user support shepherds”
> who are specifically responsible for individual topics on a day-to-day
> basis.
>
> In the distant past, we used to assign shepherds for individual components
> in Flink.
> Perhaps something similar to that, but specifically for daily user mailing
> lists support, is already sufficient to solve the mentioned problems.
> So essentially, instead of splitting into “sub-lists”, we should simply
> have dedicated “sub-topic maintainers” assigned.
>
> For example, for myself, I set a filter on my email client to look
> specifically for “Stateful Functions / StateFun” mentions, and tag it
> appropriately.
> This already allows me to concentrate on StateFun questions, without losing
> the exposure to other things happening in the wider Flink project.
> As far as I can tell, except for some more tricky questions, the turnaround
> time for StateFun user questions has been ok so far.
>
> What do you think?
>
> Cheers,
> Gordon
>
> On Mon, Mar 1, 2021 at 6:56 PM Roman Khachatryan  wrote:
>
> > Thanks for your replies!
> >
> > @Konstantin Knauf 
> > > Why do you think the quality and speed of answers would improve with
> > dedicated lists?
> > If there is a question on something that you are not an expert in; then
> you
> > either have to
> > - pull in someone who is more experienced in it (more time on hops, esp.
> if
> > the pulled in person isn't available)
> > - or learn it and answer yourself (more time on learning and still higher
> > chance of missing something)
> >
> > @Timo Walther  and @Dawid Wysakowicz
> > 
> > > I fear that we are creating potential silos where a team doesn't know
> > > what is going on in the other teams.
> > I think some specialization is unavoidable in a big project like Flink or
> > Linux (which also has separate lists).
> > And user support ML doesn't seem to me the right tool to deal with it.
> >
> > @Dawid Wysakowicz 
> > > Personally I don't find it problematic. I often find the subjects quite
> > > descriptive, they often include tags or mention which API they refer
> to.
> > Yes, but that only means that the sender would already know the "right"
> > list.
> >
> > @Konstantin Knauf  and @j...@apache.org <
> > j...@apache.org>
> >
> > I agree that there are crosscutting areas; and also a chance of sending a
> > message to the wrong topic.
> > But splitting doesn't change anything here: if a SQL question for example
> > is asked on StateFun ML then
> > we still have the options above (plus an option to redirect user to the
> > other list).
> >
> > Regards,
> > Roman
> >
> >
> > On Mon, Mar 1, 2021 at 11:30 AM Dawid Wysakowicz  >
> > wrote:
> >
> > > As others I'd also rather be -1 on splitting (even splitting out the
> > > statefun).
> > >
> > > Personally I don't find it problematic. I often find the subjects quite
> > > descriptive, they often include tags or mention which API they refer
> to.
> > > If they don't I am quite sure having separate sub-lists would not help
> > > in those cases anyway. I agree with the others that splitting the list
> > > would make the cross communication harder and create knowledge silos.
> > >
> > > It would also incur more requirements on users which already often find
> > > ML counter intuitive (See e.g. the discussion about adding a Flink
> slack)
> > >
> > > Best,
> > >
> > > Dawid
> > >
> > > On 01/03/2021 11:20, Timo Walther wrote:
> > > > I would vote -0 here.
> > > >
> > > > I fear that we are creating potential silos where a team doesn't know
> > > > what is going on in the other teams.
> > > >
> > > > Regards,
> > > > Timo
> > > >
> > > >
> > > > On 01.03.21 10:47, Jark Wu wrote:
> > > >> I also have some concerns about splitting python and sql.
> > > >> Because I have seen some SQL questions users reported but is related
> > to
> > > >> deployment or state backend.
> > > >>
> > > >> Best,
> > > >> Jark
> > > >>
> > > >> On Mon, 1 Mar 2021 at 17:15, Konstantin Knauf <
> > konstan...@ververica.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> Hi Roman,
> > > >>>
> > > >>> I slightly +1 for a list dedicated to Statefun users, but -1 for
> > > >>> splitting
> > > >>> up the rest. I think there are still a lot of crosscutting concerns
> > > >>> between
> > > >>> Python, DataStream, Table API and SQL where users of another API
> can
> > > >>> also
> > > >>> help out, too. It also requires 

[jira] [Created] (FLINK-21546) Upgrade io.netty netty-codec in Flink (four findings)

[Adam Roberts (Jira)]

Adam Roberts created FLINK-21546:


 Summary: Upgrade io.netty netty-codec in Flink (four findings)
 Key: FLINK-21546
 URL: https://issues.apache.org/jira/browse/FLINK-21546
 Project: Flink
  Issue Type: Bug
Reporter: Adam Roberts


Hi everyone, have been raising plenty of JIRAs after doing a Twistlock 
container scan for Flink 1.11.3 and Hadoop 3.3.1 snapshot, for Flink itself (so 
without using Hadoop) I've noticed the following libraries in use 
(unfortunately I don't get a path where, but somewhere in Flink they must be, 
or in a dependent jar?).

 

{{{"fixed in 
4.1.46","packageName":"io.netty_netty-codec","packageVersion":"4.1.34.Final"}}}

{{{"fixed in 
4.1.44","packageName":"io.netty_netty-codec","packageVersion":"4.1.34.Final"}}}

{{{"fixed in 
4.1.44","packageName":"io.netty_netty-codec","packageVersion":"4.1.34.Final"}}}{{}}

{{{fixed in 
4.1.42.Final","packageName":"io.netty_netty-codec","packageVersion":"4.1.34.Final"}}}

{{}}

https://issues.apache.org/jira/browse/HADOOP-17556 may be useful as well

Could we move up to Netty 4.1.46 (or something even newer?) across everything 
Flink's using? Again, I apologise for not having the paths to figure out what 
exactly is using it, but perhaps folks working directly with Flink may have a 
clue? Thanks

{{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21545) Running Kerberized YARN per-job on Docker test stalls on azure

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-21545:


 Summary: Running Kerberized YARN per-job on Docker test stalls on 
azure
 Key: FLINK-21545
 URL: https://issues.apache.org/jira/browse/FLINK-21545
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes, Tests
Affects Versions: 1.13.0
Reporter: Dawid Wysakowicz


For some reason the test started taking 10x more time than before

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13921=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13920=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13918=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13919=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529

{code}
[PASS] 'Running Kerberized YARN per-job on Docker test (default input)' passed 
after 40 minutes and 34 seconds! Test exited with exit code 0.
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21544) Upgrade Jackson databind version from 2.10.1 used in, at least, Flink Python jar

[Adam Roberts (Jira)]

Adam Roberts created FLINK-21544:


 Summary: Upgrade Jackson databind version from 2.10.1 used in, at 
least, Flink Python jar
 Key: FLINK-21544
 URL: https://issues.apache.org/jira/browse/FLINK-21544
 Project: Flink
  Issue Type: Bug
Reporter: Adam Roberts


Hi everyone, in a similar manner to 
https://issues.apache.org/jira/browse/HADOOP-17555 I have done a Twistlock 
container scan and am looking at any dependencies we can upgrade to remediate 
any security issues that may be present.

 

One such contender is this: 

{{\{
"version": "2.10.1",
"name": "com.fasterxml.jackson.core_jackson-databind",
"path": "/opt/flink/opt/flink-python_2.11-1.11.3.jar"},}}

{{}}

and so I'm wondering if we can upgrade this version to, say, 2.10.5.1, 2.12.1, 
or 2.11.4? Major bug because - surely CVEs in 2.10.1; it is quite old now as 
well (see 
[https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core/2.10.1)]

{{}}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21543) when using FIFO compaction, I found sst being deleted on the first checkpoint

[xiaogang zhou (Jira)]

xiaogang zhou created FLINK-21543:
-

 Summary: when using FIFO compaction, I found sst being deleted on 
the first checkpoint
 Key: FLINK-21543
 URL: https://issues.apache.org/jira/browse/FLINK-21543
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: xiaogang zhou


2021/03/01-18:51:01.202049 7f59042fc700 (Original Log Time 
2021/03/01-18:51:01.200883) [/compaction/compaction_picker_fifo.cc:107] 
[_timer_state/processing_user-timers] FIFO compaction: picking file 1710 with 
creation time 0 for deletion

 

the configuration is like 

currentOptions.setCompactionStyle(getCompactionStyle());
 currentOptions.setLevel0FileNumCompactionTrigger(8);
// currentOptions.setMaxTableFilesSizeFIFO(MemorySize.parse("2gb").getBytes());
 CompactionOptionsFIFO compactionOptionsFIFO = new CompactionOptionsFIFO();
 compactionOptionsFIFO.setMaxTableFilesSize(MemorySize.parse("8gb").getBytes());
 compactionOptionsFIFO.setAllowCompaction(true);

 

the rocksdb version is 


 io.github.myasuka
 frocksdbjni
 6.10.2-ververica-3.0


 

I think the problem is caused by manifest file is not uploaded by flink. Can 
any one suggest how i can skip this problem?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-162: Consistent Flink SQL time function behavior

[Timo Walther]

It is true that your proposal is kind of a conservative plan.

I'm fine with your proposal. But once we see users asking for better 
unified semantics, we should not hesitate to introduce an option to give 
them more flexibility.


Regards,
Timo


On 01.03.21 12:59, Leonard Xu wrote:

Thanks Kurt and Timo for the feedbacks.



I prefer to not introduce such config until we have to. Leonard's proposal
already makes almost all users happy thus I think we can still wait.


I could understand Kurt’s concern that we don't need rush to introduce this 
option util we have to, Especially we don’t sure the right behavior of time 
function SQL standard about streaming part(SQL standard only contains batch 
part ), it may change in the future.



However, one concern I would like to raise is still the bounded stream 
processing. Users will not have the possibility to use query-start semantics. 
For example, if users would like to use match_recognize on a CSV file, they 
cannot use query-start
timestamps.


I also think Timo’s concern that bounded cases may need query-start is 
reasonable in some user cases. Although it’s only a few scenes at present from 
my side, it will change in the future too.

As a tradeoff, I propose we could follow my last proposal as a conservative 
plan in the first step,

and then introduce the if there’re enough user requirement/feedback that they 
need the power to control the time function evaluation,

What do you think?

Best,
Leonard






Best,
Kurt
On Mon, Mar 1, 2021 at 3:58 PM Timo Walther  wrote:

and btw it is interesting to notice that AWS seems to do the approach
that I suggested first.

All functions are SQL standard compliant, and only dedicated functions
with a prefix such as CURRENT_ROW_TIMESTAMP divert from the standard.

Regards,
Timo

On 01.03.21 08:45, Timo Walther wrote:

How about we simply go for your first approach by having [query-start,
row, auto] as configuration parameters where [auto] is the default?

This sounds like a good consensus where everyone is happy, no?

This also allows user to restore the old per-row behavior for all
functions that we had before Flink 1.13.

Regards,
Timo


On 26.02.21 11:10, Leonard Xu wrote:

Thanks Joe for the great investigation.



 • Generally urging for semantics (batch > time of first query
issued, streaming > row level).
I discussed the thing now with Timo & Stephan:
 • It seems to go towards a config parameter, either [query-start,
row]  or [query-start, row, auto] and what is the default?
 • The main question seems to be: are we pushing the default
towards streaming. (probably related the insert into behaviour in the
sql client).



It looks like opinions in this thread and user inputs agreed that:
batch should use time of first query, streaming should use row level.
Based on these, we should keep row level for streaming and query start
for batch just like the config parameter value [auto].

Currently Flink keeps row level for time function in both batch and
streaming job, thus we only need to update the behavior in batch.

I tend to not expose an obscure configuration to users especially it
is semantics-related.

1.We can make [auto] as a default agreement，for current Flink
streaming users，they feel nothing has changed，for current Flink
batch users，they feel Flink batch is corrected to other good batch
engines as well as SQL standard. We can also provide a function
CURRENT_ROW_TIMESTAMP[1] for Flink batch users who want row level time
function.

2. CURRENT_ROW_TIMESTAMP can also be used in Flink streaming, it has
clear semantics, we can encourage users to use it.

In this way, We don’t have to introduce an obscure configuration
prematurely while making all users happy

How do you think?

Best,
Leonard
[1]


https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-current-row-timestamp.html






Hope this helps,

Thanks,
Joe


On 19.02.2021, at 10:25, Leonard Xu  wrote:

Hi, Joe

Thanks for volunteering to investigate the user data on this topic.
Do you
have any progress here?

Thanks,
Leonard

On Thu, Feb 4, 2021 at 3:08 PM Johannes Moser
 wrote:


Hello,

I will work with some users to get data on that.

Thanks, Joe


On 03.02.2021, at 14:58, Stephan Ewen  wrote:

Hi all!

A quick thought on this thread: We see a typical stalemate here,
as in so
many discussions recently.
One developer prefers it this way, another one another way. Both

have

pro/con arguments, it takes a lot of time from everyone, still
there is
little progress in the discussion.

Ultimately, this can only be decided by talking to the users. And it
would also be the best way to ensure that what we build is the
intuitive
and expected way for users.
The less the users are into the deep aspects of Flink SQL, the

better

they

can mirror what a common user would expect (a power user will

anyways

figure it out).
Let's find a person to drive that, spell it out in the FLIP as
"semantics
TBD", and focus on the implementation 

Re: [DISCUSS] FLIP-162: Consistent Flink SQL time function behavior

[Leonard Xu]

Thanks Kurt and Timo for the feedbacks.


>> I prefer to not introduce such config until we have to. Leonard's proposal
>> already makes almost all users happy thus I think we can still wait.

I could understand Kurt’s concern that we don't need rush to introduce this 
option util we have to, Especially we don’t sure the right behavior of time 
function SQL standard about streaming part(SQL standard only contains batch 
part ), it may change in the future.


> However, one concern I would like to raise is still the bounded stream 
> processing. Users will not have the possibility to use query-start semantics. 
> For example, if users would like to use match_recognize on a CSV file, they 
> cannot use query-start
> timestamps.

I also think Timo’s concern that bounded cases may need query-start is 
reasonable in some user cases. Although it’s only a few scenes at present from 
my side, it will change in the future too. 

As a tradeoff, I propose we could follow my last proposal as a conservative 
plan in the first step, 

and then introduce the if there’re enough user requirement/feedback that they 
need the power to control the time function evaluation, 

What do you think?

Best,
Leonard





>> Best,
>> Kurt
>> On Mon, Mar 1, 2021 at 3:58 PM Timo Walther  wrote:
>>> and btw it is interesting to notice that AWS seems to do the approach
>>> that I suggested first.
>>> 
>>> All functions are SQL standard compliant, and only dedicated functions
>>> with a prefix such as CURRENT_ROW_TIMESTAMP divert from the standard.
>>> 
>>> Regards,
>>> Timo
>>> 
>>> On 01.03.21 08:45, Timo Walther wrote:
 How about we simply go for your first approach by having [query-start,
 row, auto] as configuration parameters where [auto] is the default?
 
 This sounds like a good consensus where everyone is happy, no?
 
 This also allows user to restore the old per-row behavior for all
 functions that we had before Flink 1.13.
 
 Regards,
 Timo
 
 
 On 26.02.21 11:10, Leonard Xu wrote:
> Thanks Joe for the great investigation.
> 
> 
>> • Generally urging for semantics (batch > time of first query
>> issued, streaming > row level).
>> I discussed the thing now with Timo & Stephan:
>> • It seems to go towards a config parameter, either [query-start,
>> row]  or [query-start, row, auto] and what is the default?
>> • The main question seems to be: are we pushing the default
>> towards streaming. (probably related the insert into behaviour in the
>> sql client).
> 
> 
> It looks like opinions in this thread and user inputs agreed that:
> batch should use time of first query, streaming should use row level.
> Based on these, we should keep row level for streaming and query start
> for batch just like the config parameter value [auto].
> 
> Currently Flink keeps row level for time function in both batch and
> streaming job, thus we only need to update the behavior in batch.
> 
> I tend to not expose an obscure configuration to users especially it
> is semantics-related.
> 
> 1.We can make [auto] as a default agreement，for current Flink
> streaming users，they feel nothing has changed，for current Flink
> batch users，they feel Flink batch is corrected to other good batch
> engines as well as SQL standard. We can also provide a function
> CURRENT_ROW_TIMESTAMP[1] for Flink batch users who want row level time
> function.
> 
> 2. CURRENT_ROW_TIMESTAMP can also be used in Flink streaming, it has
> clear semantics, we can encourage users to use it.
> 
> In this way, We don’t have to introduce an obscure configuration
> prematurely while making all users happy
> 
> How do you think?
> 
> Best,
> Leonard
> [1]
> 
>>> https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-current-row-timestamp.html
> 
> 
> 
> 
>> Hope this helps,
>> 
>> Thanks,
>> Joe
>> 
>>> On 19.02.2021, at 10:25, Leonard Xu  wrote:
>>> 
>>> Hi, Joe
>>> 
>>> Thanks for volunteering to investigate the user data on this topic.
>>> Do you
>>> have any progress here?
>>> 
>>> Thanks,
>>> Leonard
>>> 
>>> On Thu, Feb 4, 2021 at 3:08 PM Johannes Moser
>>>  wrote:
>>> 
 Hello,
 
 I will work with some users to get data on that.
 
 Thanks, Joe
 
> On 03.02.2021, at 14:58, Stephan Ewen  wrote:
> 
> Hi all!
> 
> A quick thought on this thread: We see a typical stalemate here,
> as in so
> many discussions recently.
> One developer prefers it this way, another one another way. Both
>>> have
> pro/con arguments, it takes a lot of time from everyone, still
> there is
> little progress in the discussion.
> 

[DISCUSS] Enabling more dynamic, or metadata-driven behaviors

[Maciej Obuchowski]

While working on project that's strongly metadata-driven I've had to
overcome several deficiencies, or assumptions in Flink code. Project
involves reading from hundreds (possibly thousands) kafka topics, each
containing avro messages with different schemas. Then, various
transformations and aggregations are applied to that data. Then
transformed and aggregated data is written to several sinks - JDBC,
file, etc. On of the goals is making simple changes possible, like
adding topics, changing transformations or schemas without writing
code - so all is metadata driven.

Some of the things I've encountered:

It's not possible to read avro messages from kafka without somehow
providing reader schema from user code. It's simply impractical to
keep 1000s of schemas (and flink's AvroDeserializationSchemas) around,
or even worse - Kafka consumers per topic/schema.
Solution was to use custom deserialization schema similar to this approach
https://stackoverflow.com/questions/58849635/is-it-possible-to-deserialize-avro-messageconsuming-message-from-kafka-without

Another one was regarding serialization. This might be possible, but I
haven't found way to serialize avro's generic data types without Kryo
fallback, which hurts performance. I've resorted to manually
serializing record to bytes[] and deserializing it in the next task.
Also, see this mail thread where Lasse had similar case with Jackson's
objects.
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Jackson-object-serialisations-td41691.html

Third one was JDBC sink behavior. Currently, it assumes that it will
be used to insert data to one table, provided statically. This one has
the worst implication, because sink consumes one database connection,
which are quite expensive when we're talking about hundreds of them.
In this case there was no other way than forking Flink and providing
another implementation of JdbcBatchStatementExecutor, that can create
statements for multiple tables.

After this lenghty introduction, my question is basically: do Flink
developers and community welcome further discussion and contributrions
aimed at easing those, and similar pain points regarding more
"dynamic" behavior of Flink? I'm willing to contribute, but don't want
to just throw code over the wall if no one else is interested in using
it.

Thanks,
Maciej

[jira] [Created] (FLINK-21542) Add documentation for supporting INSERT INTO specific columns

[Jark Wu (Jira)]

Jark Wu created FLINK-21542:
---

 Summary: Add documentation for supporting INSERT INTO specific 
columns
 Key: FLINK-21542
 URL: https://issues.apache.org/jira/browse/FLINK-21542
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / API
Reporter: Jark Wu
 Fix For: 1.13.0


We have supported INSERT INTO specific columns in FLINK-18726, but no add 
documentation yet. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Splitting User support mailing list

[Tzu-Li (Gordon) Tai]

Hi,

I feel that the issues Roman has pointed out so far, is less a problem of
all topics (SQL / PyFlink / StateFun) being on the same list, and more a
problem that we are missing dedicated groups of “user support shepherds”
who are specifically responsible for individual topics on a day-to-day
basis.

In the distant past, we used to assign shepherds for individual components
in Flink.
Perhaps something similar to that, but specifically for daily user mailing
lists support, is already sufficient to solve the mentioned problems.
So essentially, instead of splitting into “sub-lists”, we should simply
have dedicated “sub-topic maintainers” assigned.

For example, for myself, I set a filter on my email client to look
specifically for “Stateful Functions / StateFun” mentions, and tag it
appropriately.
This already allows me to concentrate on StateFun questions, without losing
the exposure to other things happening in the wider Flink project.
As far as I can tell, except for some more tricky questions, the turnaround
time for StateFun user questions has been ok so far.

What do you think?

Cheers,
Gordon

On Mon, Mar 1, 2021 at 6:56 PM Roman Khachatryan  wrote:

> Thanks for your replies!
>
> @Konstantin Knauf 
> > Why do you think the quality and speed of answers would improve with
> dedicated lists?
> If there is a question on something that you are not an expert in; then you
> either have to
> - pull in someone who is more experienced in it (more time on hops, esp. if
> the pulled in person isn't available)
> - or learn it and answer yourself (more time on learning and still higher
> chance of missing something)
>
> @Timo Walther  and @Dawid Wysakowicz
> 
> > I fear that we are creating potential silos where a team doesn't know
> > what is going on in the other teams.
> I think some specialization is unavoidable in a big project like Flink or
> Linux (which also has separate lists).
> And user support ML doesn't seem to me the right tool to deal with it.
>
> @Dawid Wysakowicz 
> > Personally I don't find it problematic. I often find the subjects quite
> > descriptive, they often include tags or mention which API they refer to.
> Yes, but that only means that the sender would already know the "right"
> list.
>
> @Konstantin Knauf  and @j...@apache.org <
> j...@apache.org>
>
> I agree that there are crosscutting areas; and also a chance of sending a
> message to the wrong topic.
> But splitting doesn't change anything here: if a SQL question for example
> is asked on StateFun ML then
> we still have the options above (plus an option to redirect user to the
> other list).
>
> Regards,
> Roman
>
>
> On Mon, Mar 1, 2021 at 11:30 AM Dawid Wysakowicz 
> wrote:
>
> > As others I'd also rather be -1 on splitting (even splitting out the
> > statefun).
> >
> > Personally I don't find it problematic. I often find the subjects quite
> > descriptive, they often include tags or mention which API they refer to.
> > If they don't I am quite sure having separate sub-lists would not help
> > in those cases anyway. I agree with the others that splitting the list
> > would make the cross communication harder and create knowledge silos.
> >
> > It would also incur more requirements on users which already often find
> > ML counter intuitive (See e.g. the discussion about adding a Flink slack)
> >
> > Best,
> >
> > Dawid
> >
> > On 01/03/2021 11:20, Timo Walther wrote:
> > > I would vote -0 here.
> > >
> > > I fear that we are creating potential silos where a team doesn't know
> > > what is going on in the other teams.
> > >
> > > Regards,
> > > Timo
> > >
> > >
> > > On 01.03.21 10:47, Jark Wu wrote:
> > >> I also have some concerns about splitting python and sql.
> > >> Because I have seen some SQL questions users reported but is related
> to
> > >> deployment or state backend.
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> On Mon, 1 Mar 2021 at 17:15, Konstantin Knauf <
> konstan...@ververica.com
> > >
> > >> wrote:
> > >>
> > >>> Hi Roman,
> > >>>
> > >>> I slightly +1 for a list dedicated to Statefun users, but -1 for
> > >>> splitting
> > >>> up the rest. I think there are still a lot of crosscutting concerns
> > >>> between
> > >>> Python, DataStream, Table API and SQL where users of another API can
> > >>> also
> > >>> help out, too. It also requires users to think about which lists to
> > >>> subscribe/write to, instead of simply subscribing to one list.
> > >>>
> > >>> Why do you think the quality and speed of answers would improve with
> > >>> dedicated lists?
> > >>>
> > >>> Best,
> > >>>
> > >>> Konstantin
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Mon, Mar 1, 2021 at 10:09 AM xiao...@ysstech.com
> > >>> 
> > >>> wrote:
> > >>>
> >  Hi Roman,
> > 
> >  This is a very good idea. I will look forward to the official
> >  setting up
> >  "sub-lists" as soon as possible and sharing development experience
> and
> >  problems with friends in a certain field.
> > 
> >  Regards,
> >  

Re: [DISCUSS] Splitting User support mailing list

[Roman Khachatryan]

Thanks for your replies!

@Konstantin Knauf 
> Why do you think the quality and speed of answers would improve with
dedicated lists?
If there is a question on something that you are not an expert in; then you
either have to
- pull in someone who is more experienced in it (more time on hops, esp. if
the pulled in person isn't available)
- or learn it and answer yourself (more time on learning and still higher
chance of missing something)

@Timo Walther  and @Dawid Wysakowicz

> I fear that we are creating potential silos where a team doesn't know
> what is going on in the other teams.
I think some specialization is unavoidable in a big project like Flink or
Linux (which also has separate lists).
And user support ML doesn't seem to me the right tool to deal with it.

@Dawid Wysakowicz 
> Personally I don't find it problematic. I often find the subjects quite
> descriptive, they often include tags or mention which API they refer to.
Yes, but that only means that the sender would already know the "right"
list.

@Konstantin Knauf  and @j...@apache.org 

I agree that there are crosscutting areas; and also a chance of sending a
message to the wrong topic.
But splitting doesn't change anything here: if a SQL question for example
is asked on StateFun ML then
we still have the options above (plus an option to redirect user to the
other list).

Regards,
Roman


On Mon, Mar 1, 2021 at 11:30 AM Dawid Wysakowicz 
wrote:

> As others I'd also rather be -1 on splitting (even splitting out the
> statefun).
>
> Personally I don't find it problematic. I often find the subjects quite
> descriptive, they often include tags or mention which API they refer to.
> If they don't I am quite sure having separate sub-lists would not help
> in those cases anyway. I agree with the others that splitting the list
> would make the cross communication harder and create knowledge silos.
>
> It would also incur more requirements on users which already often find
> ML counter intuitive (See e.g. the discussion about adding a Flink slack)
>
> Best,
>
> Dawid
>
> On 01/03/2021 11:20, Timo Walther wrote:
> > I would vote -0 here.
> >
> > I fear that we are creating potential silos where a team doesn't know
> > what is going on in the other teams.
> >
> > Regards,
> > Timo
> >
> >
> > On 01.03.21 10:47, Jark Wu wrote:
> >> I also have some concerns about splitting python and sql.
> >> Because I have seen some SQL questions users reported but is related to
> >> deployment or state backend.
> >>
> >> Best,
> >> Jark
> >>
> >> On Mon, 1 Mar 2021 at 17:15, Konstantin Knauf  >
> >> wrote:
> >>
> >>> Hi Roman,
> >>>
> >>> I slightly +1 for a list dedicated to Statefun users, but -1 for
> >>> splitting
> >>> up the rest. I think there are still a lot of crosscutting concerns
> >>> between
> >>> Python, DataStream, Table API and SQL where users of another API can
> >>> also
> >>> help out, too. It also requires users to think about which lists to
> >>> subscribe/write to, instead of simply subscribing to one list.
> >>>
> >>> Why do you think the quality and speed of answers would improve with
> >>> dedicated lists?
> >>>
> >>> Best,
> >>>
> >>> Konstantin
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Mar 1, 2021 at 10:09 AM xiao...@ysstech.com
> >>> 
> >>> wrote:
> >>>
>  Hi Roman,
> 
>  This is a very good idea. I will look forward to the official
>  setting up
>  "sub-lists" as soon as possible and sharing development experience and
>  problems with friends in a certain field.
> 
>  Regards,
>  yue
> 
> 
> 
>  xiao...@ysstech.com
> 
>  From: Roman Khachatryan
>  Date: 2021-03-01 16:48
>  To: dev
>  Subject: [DISCUSS] Splitting User support mailing list
>  Hi everyone,
> 
>  I'd like to propose to extract several "sub-lists" from our user
>  mailing
>  list (u...@flink.apache.org).
> 
>  For example,
>  - user-sql@flink.a.o (Python)
>  - user-statefun@f.a.o (StateFun)
>  - user-py@f.a.o. (SQL/TableAPI)
>  And u...@flink.apache.org will remain the main or "default" list.
> 
>  That would improve the quality and speed of the answers and allow
>  developers to concentrate on the relevant topics.
> 
>  At the downside, this would lessen the exposure to the various Flink
> >>> areas
>  for lists maintainers.
> 
>  What do you think?
> 
>  Regards,
>  Roman
> 
> >>>
> >>>
> >>> --
> >>>
> >>> Konstantin Knauf | Head of Product
> >>>
> >>> +49 160 91394525
> >>>
> >>>
> >>> Follow us @VervericaData Ververica 
> >>>
> >>>
> >>> --
> >>>
> >>> Join Flink Forward  - The Apache Flink
> >>> Conference
> >>>
> >>> Stream Processing | Event Driven | Real Time
> >>>
> >>> --
> >>>
> >>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> >>>
> >>> --
> >>> Ververica GmbH
> >>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> >>> 

Re: [DISCUSS] Apache Flink Jira Process

[Konstantin Knauf]

Hi Dawid,

Thanks for the feedback. Do you think we should simply get rid of the
"Trivial" priority then and use the "starter" label more aggressively?

Best,

Konstantin

On Mon, Mar 1, 2021 at 11:44 AM Dawid Wysakowicz 
wrote:

> Hi Konstantin,
>
> I also like the idea.
>
> Two comments:
>
> * you describe the "Trivial" priority as one that needs to be
> implemented immediately. First of all it is not used to often, but I
> think the way it works now is similar with a "starter" label. Tasks that
> are not bugs, are easy to implement and we think they are fine to be
> taken by newcomers. Therefore they do not fall in my mind into
> "immediately" category.
>
> * I would still deprioritise test instabilities. I think there shouldn't
> be a problem with that. We do post links to all failures therefore it
> will automatically priortise the tasks according to failure frequencies.
>
> Best,
>
> Dawid
>
> On 01/03/2021 09:38, Konstantin Knauf wrote:
> > Hi Xintong,
> >
> > yes, such labels would make a lot of sense. I added a sentence to the
> > document.
> >
> > Thanks,
> >
> > Konstantin
> >
> > On Mon, Mar 1, 2021 at 8:51 AM Xintong Song 
> wrote:
> >
> >> Thanks for driving this discussion, Konstantin.
> >>
> >> I like the idea of having a bot reminding reporter/assignee/watchers
> about
> >> inactive tickets and if needed downgrade/close them automatically.
> >>
> >> My two cents:
> >> We may have labels like "downgraded-by-bot" / "closed-by-bot", so that
> it's
> >> easier to filter and review tickets updated by the bot.
> >> We may want to review such tickets (e.g., monthly) in case a valid
> ticket
> >> failed to draw the attention of relevant committers and the reporter
> >> doesn't know who to ping.
> >>
> >> Thank you~
> >>
> >> Xintong Song
> >>
> >>
> >>
> >> On Sat, Feb 27, 2021 at 1:37 AM Till Rohrmann 
> >> wrote:
> >>
> >>> Thanks for starting this discussion Konstantin. I like your proposal
> and
> >>> also the idea of automating the tedious parts of it via a bot.
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Fri, Feb 26, 2021 at 4:17 PM Konstantin Knauf 
> >>> wrote:
> >>>
>  Dear Flink Community,
> 
>  I would like to start a discussion on improving and to some extent
> >> simply
>  defining the way we work with Jira. Some aspects have been discussed a
>  while back [1], but I would like to go a bit beyond that with the
> >>> following
>  goals in mind:
> 
> 
> -
> 
> clearer communication and expectation management with the community
> -
> 
>    a user or contributor should be able to judge the urgency of a
> >>> ticket
>    by its priority
>    -
> 
>    if a ticket is assigned to someone the expectation that someone
> >> is
>    working on it should hold
>    -
> 
> generally reduce noise in Jira
> -
> 
> reduce overhead of committers to ask about status updates of
> contributions or bug reports
> -
> 
>    “Are you still working on this?”
>    -
> 
>    “Are you still interested in this?”
>    -
> 
>    “Does this still happen on Flink 1.x?”
>    -
> 
>    “Are you still experiencing this issue?”
>    -
> 
>    “What is the status of the implementation”?
>    -
> 
> while still encouraging users to add new tickets and to leave
> >> feedback
> about existing tickets
> 
> 
>  Please see the full proposal here:
> 
> 
> >>
> https://docs.google.com/document/d/19VmykDSn4BHgsCNTXtN89R7xea8e3cUIl-uivW8L6W8/edit#
>  .
> 
>  The idea would be to discuss this proposal in this thread. If we come
> >> to
> >>> a
>  conclusion, I'd document the proposal in the wiki [2] and we would
> then
>  vote on it (approval by "Lazy Majority").
> 
>  Cheers,
> 
>  Konstantin
> 
>  [1]
> 
> 
> >>
> https://lists.apache.org/thread.html/rd34fb695d371c2bf0cbd1696ce190bac35dd78f29edd8c60d0c7ee71%40%3Cdev.flink.apache.org%3E
>  [2]
> 
> 
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLINK+Jira+field+definitions
>  --
> 
>  Konstantin Knauf
> 
>  https://twitter.com/snntrable
> 
>  https://github.com/knaufk
> 
> >
>
>

-- 

Konstantin Knauf | Head of Product

+49 160 91394525


Follow us @VervericaData Ververica 


--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner

Re: [DISCUSS] Apache Flink Jira Process

[Dawid Wysakowicz]

Hi Konstantin,

I also like the idea.

Two comments:

* you describe the "Trivial" priority as one that needs to be
implemented immediately. First of all it is not used to often, but I
think the way it works now is similar with a "starter" label. Tasks that
are not bugs, are easy to implement and we think they are fine to be
taken by newcomers. Therefore they do not fall in my mind into
"immediately" category.

* I would still deprioritise test instabilities. I think there shouldn't
be a problem with that. We do post links to all failures therefore it
will automatically priortise the tasks according to failure frequencies.

Best,

Dawid

On 01/03/2021 09:38, Konstantin Knauf wrote:
> Hi Xintong,
>
> yes, such labels would make a lot of sense. I added a sentence to the
> document.
>
> Thanks,
>
> Konstantin
>
> On Mon, Mar 1, 2021 at 8:51 AM Xintong Song  wrote:
>
>> Thanks for driving this discussion, Konstantin.
>>
>> I like the idea of having a bot reminding reporter/assignee/watchers about
>> inactive tickets and if needed downgrade/close them automatically.
>>
>> My two cents:
>> We may have labels like "downgraded-by-bot" / "closed-by-bot", so that it's
>> easier to filter and review tickets updated by the bot.
>> We may want to review such tickets (e.g., monthly) in case a valid ticket
>> failed to draw the attention of relevant committers and the reporter
>> doesn't know who to ping.
>>
>> Thank you~
>>
>> Xintong Song
>>
>>
>>
>> On Sat, Feb 27, 2021 at 1:37 AM Till Rohrmann 
>> wrote:
>>
>>> Thanks for starting this discussion Konstantin. I like your proposal and
>>> also the idea of automating the tedious parts of it via a bot.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Fri, Feb 26, 2021 at 4:17 PM Konstantin Knauf 
>>> wrote:
>>>
 Dear Flink Community,

 I would like to start a discussion on improving and to some extent
>> simply
 defining the way we work with Jira. Some aspects have been discussed a
 while back [1], but I would like to go a bit beyond that with the
>>> following
 goals in mind:


-

clearer communication and expectation management with the community
-

   a user or contributor should be able to judge the urgency of a
>>> ticket
   by its priority
   -

   if a ticket is assigned to someone the expectation that someone
>> is
   working on it should hold
   -

generally reduce noise in Jira
-

reduce overhead of committers to ask about status updates of
contributions or bug reports
-

   “Are you still working on this?”
   -

   “Are you still interested in this?”
   -

   “Does this still happen on Flink 1.x?”
   -

   “Are you still experiencing this issue?”
   -

   “What is the status of the implementation”?
   -

while still encouraging users to add new tickets and to leave
>> feedback
about existing tickets


 Please see the full proposal here:


>> https://docs.google.com/document/d/19VmykDSn4BHgsCNTXtN89R7xea8e3cUIl-uivW8L6W8/edit#
 .

 The idea would be to discuss this proposal in this thread. If we come
>> to
>>> a
 conclusion, I'd document the proposal in the wiki [2] and we would then
 vote on it (approval by "Lazy Majority").

 Cheers,

 Konstantin

 [1]


>> https://lists.apache.org/thread.html/rd34fb695d371c2bf0cbd1696ce190bac35dd78f29edd8c60d0c7ee71%40%3Cdev.flink.apache.org%3E
 [2]


>> https://cwiki.apache.org/confluence/display/FLINK/FLINK+Jira+field+definitions
 --

 Konstantin Knauf

 https://twitter.com/snntrable

 https://github.com/knaufk

>



OpenPGP_signature
Description: OpenPGP digital signature

Re: [DISCUSS] Splitting User support mailing list

[Dawid Wysakowicz]

As others I'd also rather be -1 on splitting (even splitting out the
statefun).

Personally I don't find it problematic. I often find the subjects quite
descriptive, they often include tags or mention which API they refer to.
If they don't I am quite sure having separate sub-lists would not help
in those cases anyway. I agree with the others that splitting the list
would make the cross communication harder and create knowledge silos.

It would also incur more requirements on users which already often find
ML counter intuitive (See e.g. the discussion about adding a Flink slack)

Best,

Dawid

On 01/03/2021 11:20, Timo Walther wrote:
> I would vote -0 here.
>
> I fear that we are creating potential silos where a team doesn't know
> what is going on in the other teams.
>
> Regards,
> Timo
>
>
> On 01.03.21 10:47, Jark Wu wrote:
>> I also have some concerns about splitting python and sql.
>> Because I have seen some SQL questions users reported but is related to
>> deployment or state backend.
>>
>> Best,
>> Jark
>>
>> On Mon, 1 Mar 2021 at 17:15, Konstantin Knauf 
>> wrote:
>>
>>> Hi Roman,
>>>
>>> I slightly +1 for a list dedicated to Statefun users, but -1 for
>>> splitting
>>> up the rest. I think there are still a lot of crosscutting concerns
>>> between
>>> Python, DataStream, Table API and SQL where users of another API can
>>> also
>>> help out, too. It also requires users to think about which lists to
>>> subscribe/write to, instead of simply subscribing to one list.
>>>
>>> Why do you think the quality and speed of answers would improve with
>>> dedicated lists?
>>>
>>> Best,
>>>
>>> Konstantin
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Mar 1, 2021 at 10:09 AM xiao...@ysstech.com
>>> 
>>> wrote:
>>>
 Hi Roman,

 This is a very good idea. I will look forward to the official
 setting up
 "sub-lists" as soon as possible and sharing development experience and
 problems with friends in a certain field.

 Regards,
 yue



 xiao...@ysstech.com

 From: Roman Khachatryan
 Date: 2021-03-01 16:48
 To: dev
 Subject: [DISCUSS] Splitting User support mailing list
 Hi everyone,

 I'd like to propose to extract several "sub-lists" from our user
 mailing
 list (u...@flink.apache.org).

 For example,
 - user-sql@flink.a.o (Python)
 - user-statefun@f.a.o (StateFun)
 - user-py@f.a.o. (SQL/TableAPI)
 And u...@flink.apache.org will remain the main or "default" list.

 That would improve the quality and speed of the answers and allow
 developers to concentrate on the relevant topics.

 At the downside, this would lessen the exposure to the various Flink
>>> areas
 for lists maintainers.

 What do you think?

 Regards,
 Roman

>>>
>>>
>>> -- 
>>>
>>> Konstantin Knauf | Head of Product
>>>
>>> +49 160 91394525
>>>
>>>
>>> Follow us @VervericaData Ververica 
>>>
>>>
>>> -- 
>>>
>>> Join Flink Forward  - The Apache Flink
>>> Conference
>>>
>>> Stream Processing | Event Driven | Real Time
>>>
>>> -- 
>>>
>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>
>>> -- 
>>> Ververica GmbH
>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>> Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl
>>> Anton
>>> Wehner
>>>
>>
>



OpenPGP_signature
Description: OpenPGP digital signature

Re: [DISCUSS] Splitting User support mailing list

[Timo Walther]

I would vote -0 here.

I fear that we are creating potential silos where a team doesn't know 
what is going on in the other teams.


Regards,
Timo


On 01.03.21 10:47, Jark Wu wrote:

I also have some concerns about splitting python and sql.
Because I have seen some SQL questions users reported but is related to
deployment or state backend.

Best,
Jark

On Mon, 1 Mar 2021 at 17:15, Konstantin Knauf 
wrote:


Hi Roman,

I slightly +1 for a list dedicated to Statefun users, but -1 for splitting
up the rest. I think there are still a lot of crosscutting concerns between
Python, DataStream, Table API and SQL where users of another API can also
help out, too. It also requires users to think about which lists to
subscribe/write to, instead of simply subscribing to one list.

Why do you think the quality and speed of answers would improve with
dedicated lists?

Best,

Konstantin





On Mon, Mar 1, 2021 at 10:09 AM xiao...@ysstech.com 
wrote:


Hi Roman,

This is a very good idea. I will look forward to the official setting up
"sub-lists" as soon as possible and sharing development experience and
problems with friends in a certain field.

Regards,
yue



xiao...@ysstech.com

From: Roman Khachatryan
Date: 2021-03-01 16:48
To: dev
Subject: [DISCUSS] Splitting User support mailing list
Hi everyone,

I'd like to propose to extract several "sub-lists" from our user mailing
list (u...@flink.apache.org).

For example,
- user-sql@flink.a.o (Python)
- user-statefun@f.a.o (StateFun)
- user-py@f.a.o. (SQL/TableAPI)
And u...@flink.apache.org will remain the main or "default" list.

That would improve the quality and speed of the answers and allow
developers to concentrate on the relevant topics.

At the downside, this would lessen the exposure to the various Flink

areas

for lists maintainers.

What do you think?

Regards,
Roman




--

Konstantin Knauf | Head of Product

+49 160 91394525


Follow us @VervericaData Ververica 


--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner

Re: [DISCUSS]FLIP-163: SQL Client Improvements

[Shengkai Fang]

Hi, everyone.

After the long discussion, I am fine with both choices. But I prefer the
second option that applies to both table modules and sql client. Just as
Timo said, the option `table.dml-sync` can improve the SQL script
portability. Users don't need to modify the script and execute the script
in different platforms e.g gateway.

What do you think? CC Timo, Jark, Leonard.

Best,
Shengkai.

Kurt Young  于2021年3月1日周一 下午5:11写道：

> I'm +1 for either:
> 1. introduce a sql client specific option, or
> 2. Introduce a table config option and make it apply to both table module &
> sql client.
>
> It would be the FLIP owner's call to decide.
>
> Best,
> Kurt
>
>
> On Mon, Mar 1, 2021 at 3:25 PM Timo Walther  wrote:
>
> > We could also think about reading this config option in Table API. The
> > effect would be to call `await()` directly in an execute call. I could
> > also imagine this to be useful esp. when you fire a lot of insert into
> > queries. We had the case before that users where confused that the
> > execution happens asynchronously, such an option could prevent this to
> > happen again.
> >
> > Regards,
> > Timo
> >
> > On 01.03.21 05:14, Kurt Young wrote:
> > > I also asked some users about their opinion that if we introduce some
> > > config prefixed with "table" but doesn't
> > > have affection with methods in Table API and SQL. All of them are kind
> of
> > > shocked by such question, asking
> > > why we would do anything like this.
> > >
> > > This kind of reaction actually doesn't surprise me a lot, so I jumped
> in
> > > and challenged this config option even
> > > after the FLIP had already been accepted.
> > >
> > > If we only have to define the execution behavior for multiple
> statements
> > in
> > > SQL client, we should only introduce
> > > a config option which would tell users it's affection scope by its
> name.
> > > Prefixing with "table" is definitely not a good
> > > idea here.
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Fri, Feb 26, 2021 at 9:39 PM Leonard Xu  wrote:
> > >
> > >> Hi, all
> > >>
> > >> Look like there’s only one divergence about option [ table |
> sql-client
> > >> ].dml-sync in this thread, correct me if I’m wrong.
> > >>
> > >> 1. Leaving the context of this thread, from a user's perspective,
> > >> the table.xx configurations should take effect in Table API & SQL,
> > >> the sql-client.xx configurations should only take effect in
> sql-client.
> > >>   In my(the user's) opinion, other explanations are counterintuitive.
> > >>
> > >> 2.  It should be pointed out that both all existed table.xx
> > configurations
> > >> like table.exec.state.ttl, table.optimizer.agg-phase-strategy,
> > >> table.local-time-zone,etc..  and the proposed sql-client.xx
> > configurations
> > >> like sql-client.verbose, sql-client.execution.max-table-result.rows
> > >> comply with this convention.
> > >>
> > >> 3. Considering the portability to support different CLI tools
> > (sql-client,
> > >> sql-gateway, etc.), I prefer table.dml-sync.
> > >>
> > >> In addition, I think sql-client/sql-gateway/other CLI tools can be
> > placed
> > >> out of flink-table module even in an external project, this should not
> > >> affect our conclusion.
> > >>
> > >>
> > >> Hope this can help you.
> > >>
> > >>
> > >> Best,
> > >> Leonard
> > >>
> > >>
> > >>
> > >>> 在 2021年2月25日，18:51，Shengkai Fang  写道：
> > >>>
> > >>> Hi, everyone.
> > >>>
> > >>> I do some summaries about the discussion about the option. If the
> > summary
> > >>> has errors, please correct me.
> > >>>
> > >>> `table.dml-sync`:
> > >>> - take effect for `executeMultiSql` and sql client
> > >>> - benefit: SQL script portability. One script for all platforms.
> > >>> - drawback: Don't work for `TableEnvironment#executeSql`.
> > >>>
> > >>> `table.multi-dml-sync`:
> > >>> - take effect for `executeMultiSql` and sql client
> > >>> - benefit: SQL script portability
> > >>> - drawback: It's confused when the sql script has one dml statement
> but
> > >>> need to set option `table.multi-dml-sync`
> > >>>
> > >>> `client.dml-sync`:
> > >>> - take effect for sql client only
> > >>> - benefit: clear definition.
> > >>> - drawback: Every platform needs to define its own option. Bad SQL
> > script
> > >>> portability.
> > >>>
> > >>> Just as Jark said, I think the `table.dml-sync` is a good choice if
> we
> > >> can
> > >>> extend its scope and make this option works for `executeSql`.
> > >>> It's straightforward and users can use this option now in table api.
> > The
> > >>> drawback is the  `TableResult#await` plays the same role as the
> option.
> > >> I
> > >>> don't think the drawback is really critical because many systems have
> > >>> commands play the same role with the different names.
> > >>>
> > >>> Best,
> > >>> Shengkai
> > >>>
> > >>> Timo Walther  于2021年2月25日周四 下午4:23写道：
> > >>>
> >  The `table.` prefix is meant to be a general option in the table
> >  ecosystem. Not necessarily attached to Table API or SQL Client.
> 

Re: [DISCUSS] FLIP-162: Consistent Flink SQL time function behavior

[Timo Walther]

I agree that Leonard's last proposal makes "almost all" users happy. 
However, a config option (as Joe said) would make "all" user happy 
because they have the power to choose.


I don't have a strong opinion on this proposal as it is bascially a 
mixture of both approaches:


1) "some magic using the mode" + 2) "dedicated per-row function"

However, one concern I would like to raise is still the bounded stream 
processing. Users will not have the possibility to use query-start 
semantics. For example, if users would like to use match_recognize on a 
CSV file, they cannot use query-start timestamps.


Regards,
Timo


On 01.03.21 10:06, Kurt Young wrote:

I'm +1 to Leonard's last proposal, which:
1. Keep CURRENT_TIMESTAMP row level behavior in streaming mode, and make it
evaluated at query start in batch mode.
2. Introduce CURRENT_ROW_TIMESTAMP for batch users who want such semantic.

I'm slightly -1 for introducing an option because we are handling a
semantic question to our user. Imagine in the future, we
are all crystal clear about the desired behavior, and SQL standard also
covers such streaming use case. Then we will suffer
from such config option, because users can always make Flink SQL have
strange behavior by setting this config to an undesired way.

I prefer to not introduce such config until we have to. Leonard's proposal
already makes almost all users happy thus I think
we can still wait.

Best,
Kurt


On Mon, Mar 1, 2021 at 3:58 PM Timo Walther  wrote:


and btw it is interesting to notice that AWS seems to do the approach
that I suggested first.

All functions are SQL standard compliant, and only dedicated functions
with a prefix such as CURRENT_ROW_TIMESTAMP divert from the standard.

Regards,
Timo

On 01.03.21 08:45, Timo Walther wrote:

How about we simply go for your first approach by having [query-start,
row, auto] as configuration parameters where [auto] is the default?

This sounds like a good consensus where everyone is happy, no?

This also allows user to restore the old per-row behavior for all
functions that we had before Flink 1.13.

Regards,
Timo


On 26.02.21 11:10, Leonard Xu wrote:

Thanks Joe for the great investigation.



 • Generally urging for semantics (batch > time of first query
issued, streaming > row level).
I discussed the thing now with Timo & Stephan:
 • It seems to go towards a config parameter, either [query-start,
row]  or [query-start, row, auto] and what is the default?
 • The main question seems to be: are we pushing the default
towards streaming. (probably related the insert into behaviour in the
sql client).



It looks like opinions in this thread and user inputs agreed that:
batch should use time of first query, streaming should use row level.
Based on these, we should keep row level for streaming and query start
for batch just like the config parameter value [auto].

Currently Flink keeps row level for time function in both batch and
streaming job, thus we only need to update the behavior in batch.

I tend to not expose an obscure configuration to users especially it
is semantics-related.

1.We can make [auto] as a default agreement，for current Flink
streaming users，they feel nothing has changed，for current Flink
batch users，they feel Flink batch is corrected to other good batch
engines as well as SQL standard. We can also provide a function
CURRENT_ROW_TIMESTAMP[1] for Flink batch users who want row level time
function.

2. CURRENT_ROW_TIMESTAMP can also be used in Flink streaming, it has
clear semantics, we can encourage users to use it.

In this way, We don’t have to introduce an obscure configuration
prematurely while making all users happy

How do you think?

Best,
Leonard
[1]


https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-current-row-timestamp.html






Hope this helps,

Thanks,
Joe


On 19.02.2021, at 10:25, Leonard Xu  wrote:

Hi, Joe

Thanks for volunteering to investigate the user data on this topic.
Do you
have any progress here?

Thanks,
Leonard

On Thu, Feb 4, 2021 at 3:08 PM Johannes Moser
 wrote:


Hello,

I will work with some users to get data on that.

Thanks, Joe


On 03.02.2021, at 14:58, Stephan Ewen  wrote:

Hi all!

A quick thought on this thread: We see a typical stalemate here,
as in so
many discussions recently.
One developer prefers it this way, another one another way. Both

have

pro/con arguments, it takes a lot of time from everyone, still
there is
little progress in the discussion.

Ultimately, this can only be decided by talking to the users. And it
would also be the best way to ensure that what we build is the
intuitive
and expected way for users.
The less the users are into the deep aspects of Flink SQL, the

better

they

can mirror what a common user would expect (a power user will

anyways

figure it out).
Let's find a person to drive that, spell it out in the FLIP as
"semantics
TBD", and focus on the implementation of the parts that are agreed
upon.

For interviewing the 

Re: [VOTE] Release 1.12.2, release candidate #2

[Matthias Pohl]

Thanks for managing release 1.12.2, Yuan & Roman.

+1 (non-binding)

- Verified checksums and GPG of artifacts in [1]
- Build the sources locally without errors
- Started a local standalone cluster and deployed WordCount without
problems (no suspicious logs identified)
- Verified FLINK-21030 [2] by running the example jobs from the
FLINK-21030-related SavepointITCase tests

Best,
Matthias

[1] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.2-rc2/
[2] https://issues.apache.org/jira/browse/FLINK-21030

On Sun, Feb 28, 2021 at 2:41 PM Yuan Mei  wrote:

> Hey Roman,
>
> Thank you very much for preparing RC2.
>
> +1 from my side.
>
> 1. Verified Checksums and GPG signatures.
> 2. Verified that the source archives do not contain any binaries.
> 3. Successfully Built the source with Maven.
> 4. Started a local Flink cluster, ran the streaming WordCount example with
> WebUI,
> checked the output and JM/TM log, no suspicious output/log.
> 5. Repeat Step 4 with the binary release as well, no suspicious output/log.
> 6. Checked for source and binary release to make sure both an Apache
> License file and a NOTICE file are included.
> 7. Manually verified that no pom file changes between 1.12.2-rc1 and
> 1.12.2-rc2; no obvious license problem.
> 8. Review the release PR for RC2 updates, and double confirmed the
> change-list for 1.12.2.
>
> Best,
> Yuan
>
> On Sat, Feb 27, 2021 at 7:19 AM Roman Khachatryan 
> wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version
> 1.12.2,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release and binary convenience releases to
> be
> > deployed to dist.apache.org [2], which are signed with the key with
> > fingerprint 0D545F264D2DFDEBFD4E038F97B4625E2FCF517C [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-1.12.2-rc2" [5],
> > * website pull request listing the new release and adding announcement
> blog
> > post [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12349502=12315522
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.12.2-rc2/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1414/
> > [5] https://github.com/apache/flink/releases/tag/release-1.12.2-rc2
> > [6] https://github.com/apache/flink-web/pull/418
> >
> > Regards,
> > Roman
> >

[jira] [Created] (FLINK-21541) Update the existing K8s application e2e test to also verify pod template

[Yang Wang (Jira)]

Yang Wang created FLINK-21541:
-

 Summary: Update the existing K8s application e2e test to also 
verify pod template
 Key: FLINK-21541
 URL: https://issues.apache.org/jira/browse/FLINK-21541
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes, Tests
Affects Versions: 1.13.0
Reporter: Yang Wang


After then we could ensure that the devs don't accidentally break this feature 
w/o us noticing it (e.g. calling the file existence check for the specified 
file in the pods).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Splitting User support mailing list

[Jark Wu]

I also have some concerns about splitting python and sql.
Because I have seen some SQL questions users reported but is related to
deployment or state backend.

Best,
Jark

On Mon, 1 Mar 2021 at 17:15, Konstantin Knauf 
wrote:

> Hi Roman,
>
> I slightly +1 for a list dedicated to Statefun users, but -1 for splitting
> up the rest. I think there are still a lot of crosscutting concerns between
> Python, DataStream, Table API and SQL where users of another API can also
> help out, too. It also requires users to think about which lists to
> subscribe/write to, instead of simply subscribing to one list.
>
> Why do you think the quality and speed of answers would improve with
> dedicated lists?
>
> Best,
>
> Konstantin
>
>
>
>
>
> On Mon, Mar 1, 2021 at 10:09 AM xiao...@ysstech.com 
> wrote:
>
> > Hi Roman,
> >
> > This is a very good idea. I will look forward to the official setting up
> > "sub-lists" as soon as possible and sharing development experience and
> > problems with friends in a certain field.
> >
> > Regards,
> > yue
> >
> >
> >
> > xiao...@ysstech.com
> >
> > From: Roman Khachatryan
> > Date: 2021-03-01 16:48
> > To: dev
> > Subject: [DISCUSS] Splitting User support mailing list
> > Hi everyone,
> >
> > I'd like to propose to extract several "sub-lists" from our user mailing
> > list (u...@flink.apache.org).
> >
> > For example,
> > - user-sql@flink.a.o (Python)
> > - user-statefun@f.a.o (StateFun)
> > - user-py@f.a.o. (SQL/TableAPI)
> > And u...@flink.apache.org will remain the main or "default" list.
> >
> > That would improve the quality and speed of the answers and allow
> > developers to concentrate on the relevant topics.
> >
> > At the downside, this would lessen the exposure to the various Flink
> areas
> > for lists maintainers.
> >
> > What do you think?
> >
> > Regards,
> > Roman
> >
>
>
> --
>
> Konstantin Knauf | Head of Product
>
> +49 160 91394525
>
>
> Follow us @VervericaData Ververica 
>
>
> --
>
> Join Flink Forward  - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
> Wehner
>

[jira] [Created] (FLINK-21540) finegrained_resource_management tests hang on azure

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-21540:


 Summary:  finegrained_resource_management tests hang on azure
 Key: FLINK-21540
 URL: https://issues.apache.org/jira/browse/FLINK-21540
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.13.0
Reporter: Dawid Wysakowicz


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13905=logs=b0a398c0-685b-599c-eb57-c8c2a771138e=d13f554f-d4b9-50f8-30ee-d49c6fb0b3cc



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Splitting User support mailing list

[Konstantin Knauf]

Hi Roman,

I slightly +1 for a list dedicated to Statefun users, but -1 for splitting
up the rest. I think there are still a lot of crosscutting concerns between
Python, DataStream, Table API and SQL where users of another API can also
help out, too. It also requires users to think about which lists to
subscribe/write to, instead of simply subscribing to one list.

Why do you think the quality and speed of answers would improve with
dedicated lists?

Best,

Konstantin





On Mon, Mar 1, 2021 at 10:09 AM xiao...@ysstech.com 
wrote:

> Hi Roman,
>
> This is a very good idea. I will look forward to the official setting up
> "sub-lists" as soon as possible and sharing development experience and
> problems with friends in a certain field.
>
> Regards,
> yue
>
>
>
> xiao...@ysstech.com
>
> From: Roman Khachatryan
> Date: 2021-03-01 16:48
> To: dev
> Subject: [DISCUSS] Splitting User support mailing list
> Hi everyone,
>
> I'd like to propose to extract several "sub-lists" from our user mailing
> list (u...@flink.apache.org).
>
> For example,
> - user-sql@flink.a.o (Python)
> - user-statefun@f.a.o (StateFun)
> - user-py@f.a.o. (SQL/TableAPI)
> And u...@flink.apache.org will remain the main or "default" list.
>
> That would improve the quality and speed of the answers and allow
> developers to concentrate on the relevant topics.
>
> At the downside, this would lessen the exposure to the various Flink areas
> for lists maintainers.
>
> What do you think?
>
> Regards,
> Roman
>


-- 

Konstantin Knauf | Head of Product

+49 160 91394525


Follow us @VervericaData Ververica 


--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Yip Park Tung Jason, Jinwei (Kevin) Zhang, Karl Anton
Wehner

[jira] [Created] (FLINK-21539) 'SQL Client end-to-end test (Blink planner) Elasticsearch (v6.3.1)' fails during download

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-21539:


 Summary: 'SQL Client end-to-end test (Blink planner) Elasticsearch 
(v6.3.1)' fails during download
 Key: FLINK-21539
 URL: https://issues.apache.org/jira/browse/FLINK-21539
 Project: Flink
  Issue Type: Bug
  Components: Connectors / ElasticSearch, Table SQL / Ecosystem, Tests
Affects Versions: 1.11.3
Reporter: Dawid Wysakowicz


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13906=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=03dbd840-5430-533d-d1a7-05d0ebe03873

{code}
Downloading Elasticsearch from 
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.1.tar.gz 
...
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0
  0 87.1M0   9970 0   3298  0  7:42:02 --:--:--  7:42:02  3290
 17 87.1M   17 15.6M0 0  11.6M  0  0:00:07  0:00:01  0:00:06 11.6M
 31 87.1M   31 27.8M0 0  12.1M  0  0:00:07  0:00:02  0:00:05 12.1M
 49 87.1M   49 42.9M0 0  12.9M  0  0:00:06  0:00:03  0:00:03 12.9M
 67 87.1M   67 58.9M0 0  13.5M  0  0:00:06  0:00:04  0:00:02 13.5M
 87 87.1M   87 75.8M0 0  14.3M  0  0:00:06  0:00:05  0:00:01 15.1M
 87 87.1M   87 75.9M0 0  14.2M  0  0:00:06  0:00:05  0:00:01 15.1M
curl: (56) GnuTLS recv error (-110): The TLS connection was non-properly 
terminated.
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed

  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 
0curl: (7) Failed to connect to localhost port 9200: Connection refused
[FAIL] Test script contains errors.
Checking for errors...
No errors in log files.
Checking for exceptions...
No exceptions in log files.
Checking for non-empty .out files...
grep: 
/home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*.out:
 No such file or directory
No non-empty .out files.
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] FLIP-151: Incremental snapshots for heap-based state backend

[Roman Khachatryan]

Hi everyone,

since the discussion [1] about FLIP-151 [2] seems to have reached a
consensus, I'd like to start a formal vote for the FLIP.

Please vote +1 to approve the FLIP, or -1 with a comment. The vote will be
open at least until Wednesday, Mar 3rd.

[1] https://s.apache.org/flip-151-discussion
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-151%3A+Incremental+snapshots+for+heap-based+state+backend

Regards,
Roman

[jira] [Created] (FLINK-21538) Elasticsearch6DynamicSinkITCase.testWritingDocuments fails when submitting job

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-21538:


 Summary: Elasticsearch6DynamicSinkITCase.testWritingDocuments 
fails when submitting job
 Key: FLINK-21538
 URL: https://issues.apache.org/jira/browse/FLINK-21538
 Project: Flink
  Issue Type: Bug
  Components: Connectors / ElasticSearch, Runtime / Coordination
Affects Versions: 1.12.1
Reporter: Dawid Wysakowicz


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13868=logs=3d12d40f-c62d-5ec4-6acc-0efe94cc3e89=5d6e4255-0ea8-5e2a-f52c-c881b7872361

{code}
2021-02-27T00:16:06.9493539Z 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-02-27T00:16:06.9494494Zat 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-02-27T00:16:06.9495733Zat 
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:117)
2021-02-27T00:16:06.9496596Zat 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-02-27T00:16:06.9497354Zat 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-02-27T00:16:06.9525795Zat 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-02-27T00:16:06.9526744Zat 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-02-27T00:16:06.9527784Zat 
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-02-27T00:16:06.9528552Zat 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-02-27T00:16:06.9529271Zat 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-02-27T00:16:06.9530013Zat 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-02-27T00:16:06.9530482Zat 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-02-27T00:16:06.9531068Zat 
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1046)
2021-02-27T00:16:06.9531544Zat 
akka.dispatch.OnComplete.internal(Future.scala:264)
2021-02-27T00:16:06.9531908Zat 
akka.dispatch.OnComplete.internal(Future.scala:261)
2021-02-27T00:16:06.9532449Zat 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-02-27T00:16:06.9532860Zat 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-02-27T00:16:06.9533245Zat 
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2021-02-27T00:16:06.9533721Zat 
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-02-27T00:16:06.9534225Zat 
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:68)
2021-02-27T00:16:06.9534697Zat 
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:284)
2021-02-27T00:16:06.9535217Zat 
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:284)
2021-02-27T00:16:06.9535718Zat 
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:284)
2021-02-27T00:16:06.9536127Zat 
akka.pattern.PromiseActorRef.$bang(AskSupport.scala:573)
2021-02-27T00:16:06.9536861Zat 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
2021-02-27T00:16:06.9537394Zat 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
2021-02-27T00:16:06.9537916Zat 
scala.concurrent.Future.$anonfun$andThen$1(Future.scala:532)
2021-02-27T00:16:06.9605804Zat 
scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
2021-02-27T00:16:06.9606794Zat 
scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
2021-02-27T00:16:06.9607642Zat 
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
2021-02-27T00:16:06.9608419Zat 
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
2021-02-27T00:16:06.9609252Zat 
akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
2021-02-27T00:16:06.9610024Zat 
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
2021-02-27T00:16:06.9613676Zat 
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
2021-02-27T00:16:06.9615526Zat 
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
2021-02-27T00:16:06.9616727Zat 
akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
2021-02-27T00:16:06.9617826Zat 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
2021-02-27T00:16:06.9618940Zat 
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
2021-02-27T00:16:06.9620109Z   

Re: [DISCUSS]FLIP-163: SQL Client Improvements

[Kurt Young]

I'm +1 for either:
1. introduce a sql client specific option, or
2. Introduce a table config option and make it apply to both table module &
sql client.

It would be the FLIP owner's call to decide.

Best,
Kurt


On Mon, Mar 1, 2021 at 3:25 PM Timo Walther  wrote:

> We could also think about reading this config option in Table API. The
> effect would be to call `await()` directly in an execute call. I could
> also imagine this to be useful esp. when you fire a lot of insert into
> queries. We had the case before that users where confused that the
> execution happens asynchronously, such an option could prevent this to
> happen again.
>
> Regards,
> Timo
>
> On 01.03.21 05:14, Kurt Young wrote:
> > I also asked some users about their opinion that if we introduce some
> > config prefixed with "table" but doesn't
> > have affection with methods in Table API and SQL. All of them are kind of
> > shocked by such question, asking
> > why we would do anything like this.
> >
> > This kind of reaction actually doesn't surprise me a lot, so I jumped in
> > and challenged this config option even
> > after the FLIP had already been accepted.
> >
> > If we only have to define the execution behavior for multiple statements
> in
> > SQL client, we should only introduce
> > a config option which would tell users it's affection scope by its name.
> > Prefixing with "table" is definitely not a good
> > idea here.
> >
> > Best,
> > Kurt
> >
> >
> > On Fri, Feb 26, 2021 at 9:39 PM Leonard Xu  wrote:
> >
> >> Hi, all
> >>
> >> Look like there’s only one divergence about option [ table | sql-client
> >> ].dml-sync in this thread, correct me if I’m wrong.
> >>
> >> 1. Leaving the context of this thread, from a user's perspective,
> >> the table.xx configurations should take effect in Table API & SQL,
> >> the sql-client.xx configurations should only take effect in sql-client.
> >>   In my(the user's) opinion, other explanations are counterintuitive.
> >>
> >> 2.  It should be pointed out that both all existed table.xx
> configurations
> >> like table.exec.state.ttl, table.optimizer.agg-phase-strategy,
> >> table.local-time-zone,etc..  and the proposed sql-client.xx
> configurations
> >> like sql-client.verbose, sql-client.execution.max-table-result.rows
> >> comply with this convention.
> >>
> >> 3. Considering the portability to support different CLI tools
> (sql-client,
> >> sql-gateway, etc.), I prefer table.dml-sync.
> >>
> >> In addition, I think sql-client/sql-gateway/other CLI tools can be
> placed
> >> out of flink-table module even in an external project, this should not
> >> affect our conclusion.
> >>
> >>
> >> Hope this can help you.
> >>
> >>
> >> Best,
> >> Leonard
> >>
> >>
> >>
> >>> 在 2021年2月25日，18:51，Shengkai Fang  写道：
> >>>
> >>> Hi, everyone.
> >>>
> >>> I do some summaries about the discussion about the option. If the
> summary
> >>> has errors, please correct me.
> >>>
> >>> `table.dml-sync`:
> >>> - take effect for `executeMultiSql` and sql client
> >>> - benefit: SQL script portability. One script for all platforms.
> >>> - drawback: Don't work for `TableEnvironment#executeSql`.
> >>>
> >>> `table.multi-dml-sync`:
> >>> - take effect for `executeMultiSql` and sql client
> >>> - benefit: SQL script portability
> >>> - drawback: It's confused when the sql script has one dml statement but
> >>> need to set option `table.multi-dml-sync`
> >>>
> >>> `client.dml-sync`:
> >>> - take effect for sql client only
> >>> - benefit: clear definition.
> >>> - drawback: Every platform needs to define its own option. Bad SQL
> script
> >>> portability.
> >>>
> >>> Just as Jark said, I think the `table.dml-sync` is a good choice if we
> >> can
> >>> extend its scope and make this option works for `executeSql`.
> >>> It's straightforward and users can use this option now in table api.
> The
> >>> drawback is the  `TableResult#await` plays the same role as the option.
> >> I
> >>> don't think the drawback is really critical because many systems have
> >>> commands play the same role with the different names.
> >>>
> >>> Best,
> >>> Shengkai
> >>>
> >>> Timo Walther  于2021年2月25日周四 下午4:23写道：
> >>>
>  The `table.` prefix is meant to be a general option in the table
>  ecosystem. Not necessarily attached to Table API or SQL Client. That's
>  why SQL Client is also located in the `flink-table` module.
> 
>  My main concern is the SQL script portability. Declaring the
> sync/async
>  behavior will happen in many SQL scripts. And users should be easily
>  switch from SQL Client to some commercial product without the need of
>  changing the script again.
> 
>  Sure, we can change from `sql-client.dml-sync` to `table.dml-sync`
> later
>  but that would mean introducing future confusion. An app name (what
>  `sql-client` kind of is) should not be part of a config option key if
>  other apps will need the same kind of option.
> 
>  Regards,
>  Timo
> 
> 
>  On 

Re: [DISCUSS] Splitting User support mailing list

[xiao...@ysstech.com]

Hi Roman,

This is a very good idea. I will look forward to the official setting up 
"sub-lists" as soon as possible and sharing development experience and problems 
with friends in a certain field.

Regards,
yue



xiao...@ysstech.com
 
From: Roman Khachatryan
Date: 2021-03-01 16:48
To: dev
Subject: [DISCUSS] Splitting User support mailing list
Hi everyone,
 
I'd like to propose to extract several "sub-lists" from our user mailing
list (u...@flink.apache.org).
 
For example,
- user-sql@flink.a.o (Python)
- user-statefun@f.a.o (StateFun)
- user-py@f.a.o. (SQL/TableAPI)
And u...@flink.apache.org will remain the main or "default" list.
 
That would improve the quality and speed of the answers and allow
developers to concentrate on the relevant topics.
 
At the downside, this would lessen the exposure to the various Flink areas
for lists maintainers.
 
What do you think?
 
Regards,
Roman

Re: [DISCUSS] FLIP-162: Consistent Flink SQL time function behavior

[Kurt Young]

I'm +1 to Leonard's last proposal, which:
1. Keep CURRENT_TIMESTAMP row level behavior in streaming mode, and make it
evaluated at query start in batch mode.
2. Introduce CURRENT_ROW_TIMESTAMP for batch users who want such semantic.

I'm slightly -1 for introducing an option because we are handling a
semantic question to our user. Imagine in the future, we
are all crystal clear about the desired behavior, and SQL standard also
covers such streaming use case. Then we will suffer
from such config option, because users can always make Flink SQL have
strange behavior by setting this config to an undesired way.

I prefer to not introduce such config until we have to. Leonard's proposal
already makes almost all users happy thus I think
we can still wait.

Best,
Kurt


On Mon, Mar 1, 2021 at 3:58 PM Timo Walther  wrote:

> and btw it is interesting to notice that AWS seems to do the approach
> that I suggested first.
>
> All functions are SQL standard compliant, and only dedicated functions
> with a prefix such as CURRENT_ROW_TIMESTAMP divert from the standard.
>
> Regards,
> Timo
>
> On 01.03.21 08:45, Timo Walther wrote:
> > How about we simply go for your first approach by having [query-start,
> > row, auto] as configuration parameters where [auto] is the default?
> >
> > This sounds like a good consensus where everyone is happy, no?
> >
> > This also allows user to restore the old per-row behavior for all
> > functions that we had before Flink 1.13.
> >
> > Regards,
> > Timo
> >
> >
> > On 26.02.21 11:10, Leonard Xu wrote:
> >> Thanks Joe for the great investigation.
> >>
> >>
> >>> • Generally urging for semantics (batch > time of first query
> >>> issued, streaming > row level).
> >>> I discussed the thing now with Timo & Stephan:
> >>> • It seems to go towards a config parameter, either [query-start,
> >>> row]  or [query-start, row, auto] and what is the default?
> >>> • The main question seems to be: are we pushing the default
> >>> towards streaming. (probably related the insert into behaviour in the
> >>> sql client).
> >>
> >>
> >> It looks like opinions in this thread and user inputs agreed that:
> >> batch should use time of first query, streaming should use row level.
> >> Based on these, we should keep row level for streaming and query start
> >> for batch just like the config parameter value [auto].
> >>
> >> Currently Flink keeps row level for time function in both batch and
> >> streaming job, thus we only need to update the behavior in batch.
> >>
> >> I tend to not expose an obscure configuration to users especially it
> >> is semantics-related.
> >>
> >> 1.We can make [auto] as a default agreement，for current Flink
> >> streaming users，they feel nothing has changed，for current Flink
> >> batch users，they feel Flink batch is corrected to other good batch
> >> engines as well as SQL standard. We can also provide a function
> >> CURRENT_ROW_TIMESTAMP[1] for Flink batch users who want row level time
> >> function.
> >>
> >> 2. CURRENT_ROW_TIMESTAMP can also be used in Flink streaming, it has
> >> clear semantics, we can encourage users to use it.
> >>
> >> In this way, We don’t have to introduce an obscure configuration
> >> prematurely while making all users happy
> >>
> >> How do you think?
> >>
> >> Best,
> >> Leonard
> >> [1]
> >>
> https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-current-row-timestamp.html
> >>
> >>
> >>
> >>
> >>> Hope this helps,
> >>>
> >>> Thanks,
> >>> Joe
> >>>
>  On 19.02.2021, at 10:25, Leonard Xu  wrote:
> 
>  Hi, Joe
> 
>  Thanks for volunteering to investigate the user data on this topic.
>  Do you
>  have any progress here?
> 
>  Thanks,
>  Leonard
> 
>  On Thu, Feb 4, 2021 at 3:08 PM Johannes Moser
>   wrote:
> 
> > Hello,
> >
> > I will work with some users to get data on that.
> >
> > Thanks, Joe
> >
> >> On 03.02.2021, at 14:58, Stephan Ewen  wrote:
> >>
> >> Hi all!
> >>
> >> A quick thought on this thread: We see a typical stalemate here,
> >> as in so
> >> many discussions recently.
> >> One developer prefers it this way, another one another way. Both
> have
> >> pro/con arguments, it takes a lot of time from everyone, still
> >> there is
> >> little progress in the discussion.
> >>
> >> Ultimately, this can only be decided by talking to the users. And it
> >> would also be the best way to ensure that what we build is the
> >> intuitive
> >> and expected way for users.
> >> The less the users are into the deep aspects of Flink SQL, the
> better
> > they
> >> can mirror what a common user would expect (a power user will
> anyways
> >> figure it out).
> >> Let's find a person to drive that, spell it out in the FLIP as
> >> "semantics
> >> TBD", and focus on the implementation of the parts that are agreed
> >> upon.
> >>
> >> For 

[DISCUSS] Splitting User support mailing list

[Roman Khachatryan]

Hi everyone,

I'd like to propose to extract several "sub-lists" from our user mailing
list (u...@flink.apache.org).

For example,
- user-sql@flink.a.o (Python)
- user-statefun@f.a.o (StateFun)
- user-py@f.a.o. (SQL/TableAPI)
And u...@flink.apache.org will remain the main or "default" list.

That would improve the quality and speed of the answers and allow
developers to concentrate on the relevant topics.

At the downside, this would lessen the exposure to the various Flink areas
for lists maintainers.

What do you think?

Regards,
Roman

Re: [DISCUSS] Apache Flink Jira Process

[Konstantin Knauf]

Hi Xintong,

yes, such labels would make a lot of sense. I added a sentence to the
document.

Thanks,

Konstantin

On Mon, Mar 1, 2021 at 8:51 AM Xintong Song  wrote:

> Thanks for driving this discussion, Konstantin.
>
> I like the idea of having a bot reminding reporter/assignee/watchers about
> inactive tickets and if needed downgrade/close them automatically.
>
> My two cents:
> We may have labels like "downgraded-by-bot" / "closed-by-bot", so that it's
> easier to filter and review tickets updated by the bot.
> We may want to review such tickets (e.g., monthly) in case a valid ticket
> failed to draw the attention of relevant committers and the reporter
> doesn't know who to ping.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Sat, Feb 27, 2021 at 1:37 AM Till Rohrmann 
> wrote:
>
> > Thanks for starting this discussion Konstantin. I like your proposal and
> > also the idea of automating the tedious parts of it via a bot.
> >
> > Cheers,
> > Till
> >
> > On Fri, Feb 26, 2021 at 4:17 PM Konstantin Knauf 
> > wrote:
> >
> > > Dear Flink Community,
> > >
> > > I would like to start a discussion on improving and to some extent
> simply
> > > defining the way we work with Jira. Some aspects have been discussed a
> > > while back [1], but I would like to go a bit beyond that with the
> > following
> > > goals in mind:
> > >
> > >
> > >-
> > >
> > >clearer communication and expectation management with the community
> > >-
> > >
> > >   a user or contributor should be able to judge the urgency of a
> > ticket
> > >   by its priority
> > >   -
> > >
> > >   if a ticket is assigned to someone the expectation that someone
> is
> > >   working on it should hold
> > >   -
> > >
> > >generally reduce noise in Jira
> > >-
> > >
> > >reduce overhead of committers to ask about status updates of
> > >contributions or bug reports
> > >-
> > >
> > >   “Are you still working on this?”
> > >   -
> > >
> > >   “Are you still interested in this?”
> > >   -
> > >
> > >   “Does this still happen on Flink 1.x?”
> > >   -
> > >
> > >   “Are you still experiencing this issue?”
> > >   -
> > >
> > >   “What is the status of the implementation”?
> > >   -
> > >
> > >while still encouraging users to add new tickets and to leave
> feedback
> > >about existing tickets
> > >
> > >
> > > Please see the full proposal here:
> > >
> > >
> >
> https://docs.google.com/document/d/19VmykDSn4BHgsCNTXtN89R7xea8e3cUIl-uivW8L6W8/edit#
> > > .
> > >
> > > The idea would be to discuss this proposal in this thread. If we come
> to
> > a
> > > conclusion, I'd document the proposal in the wiki [2] and we would then
> > > vote on it (approval by "Lazy Majority").
> > >
> > > Cheers,
> > >
> > > Konstantin
> > >
> > > [1]
> > >
> > >
> >
> https://lists.apache.org/thread.html/rd34fb695d371c2bf0cbd1696ce190bac35dd78f29edd8c60d0c7ee71%40%3Cdev.flink.apache.org%3E
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLINK+Jira+field+definitions
> > >
> > > --
> > >
> > > Konstantin Knauf
> > >
> > > https://twitter.com/snntrable
> > >
> > > https://github.com/knaufk
> > >
> >
>


-- 

Konstantin Knauf

https://twitter.com/snntrable

https://github.com/knaufk

[jira] [Created] (FLINK-21537) SavepointITCase fails on azure

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-21537:


 Summary: SavepointITCase fails on azure
 Key: FLINK-21537
 URL: https://issues.apache.org/jira/browse/FLINK-21537
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.13.0
Reporter: Dawid Wysakowicz


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13866=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=a0a633b8-47ef-5c5a-2806-3c13b9e48228

{code}
2021-02-26T23:25:17.5041521Z [ERROR] 
testStopWithSavepointFailingInSnapshotCreation(org.apache.flink.test.checkpointing.SavepointITCase)
  Time elapsed: 0.359 s  <<< FAILURE!
2021-02-26T23:25:17.5042304Z java.lang.AssertionError
2021-02-26T23:25:17.5042938Zat org.junit.Assert.fail(Assert.java:86)
2021-02-26T23:25:17.5043637Zat org.junit.Assert.assertTrue(Assert.java:41)
2021-02-26T23:25:17.5046831Zat org.junit.Assert.assertTrue(Assert.java:52)
2021-02-26T23:25:17.5047567Zat 
org.apache.flink.test.checkpointing.SavepointITCase.lambda$assertInSnapshotCreationFailure$4(SavepointITCase.java:604)
2021-02-26T23:25:17.5048194Zat 
org.apache.flink.test.checkpointing.SavepointITCase.testStopWithFailingSourceInOnePipeline(SavepointITCase.java:684)
2021-02-26T23:25:17.5049245Zat 
org.apache.flink.test.checkpointing.SavepointITCase.testStopWithSavepointFailingInSnapshotCreation(SavepointITCase.java:564)
2021-02-26T23:25:17.5049751Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-02-26T23:25:17.5050168Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-02-26T23:25:17.5051317Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-02-26T23:25:17.5052136Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2021-02-26T23:25:17.5052931Zat 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-02-26T23:25:17.5053700Zat 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-02-26T23:25:17.5054466Zat 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-02-26T23:25:17.5055163Zat 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-02-26T23:25:17.5055865Zat 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
2021-02-26T23:25:17.5056560Zat 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2021-02-26T23:25:17.5057240Zat 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2021-02-26T23:25:17.5057906Zat 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2021-02-26T23:25:17.5058488Zat 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-02-26T23:25:17.5059193Zat 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-02-26T23:25:17.5059935Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-02-26T23:25:17.5060685Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-02-26T23:25:17.5061305Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-02-26T23:25:17.5061940Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-02-26T23:25:17.5062717Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-02-26T23:25:17.5063355Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-02-26T23:25:17.5064011Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-02-26T23:25:17.5064648Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-02-26T23:25:17.5065227Zat 
org.junit.runners.Suite.runChild(Suite.java:128)
2021-02-26T23:25:17.5065750Zat 
org.junit.runners.Suite.runChild(Suite.java:27)
2021-02-26T23:25:17.5066719Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-02-26T23:25:17.5067330Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-02-26T23:25:17.5067988Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-02-26T23:25:17.5068659Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-02-26T23:25:17.5069424Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-02-26T23:25:17.5070052Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-02-26T23:25:17.5070710Zat 
org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
2021-02-26T23:25:17.5071469Zat 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
2021-02-26T23:25:17.5072312Zat 
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)

[jira] [Created] (FLINK-21536) AdaptiveSchedulerSlotSharingITCase.testSchedulingOfJobRequiringSlotSharing fails on azure

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-21536:


 Summary: 
AdaptiveSchedulerSlotSharingITCase.testSchedulingOfJobRequiringSlotSharing 
fails on azure
 Key: FLINK-21536
 URL: https://issues.apache.org/jira/browse/FLINK-21536
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Dawid Wysakowicz


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13866=logs=59c257d0-c525-593b-261d-e96a86f1926b=b93980e3-753f-5433-6a19-13747adae66a

{code}
2021-02-27T01:48:44.4515164Z 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-02-27T01:48:44.4516118Zat 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-02-27T01:48:44.4517159Zat 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerSlotSharingITCase.runJob(AdaptiveSchedulerSlotSharingITCase.java:83)
2021-02-27T01:48:44.4518613Zat 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveSchedulerSlotSharingITCase.testSchedulingOfJobRequiringSlotSharing(AdaptiveSchedulerSlotSharingITCase.java:71)
2021-02-27T01:48:44.4519475Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-02-27T01:48:44.4519940Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-02-27T01:48:44.4520482Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-02-27T01:48:44.4520937Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2021-02-27T01:48:44.4521387Zat 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-02-27T01:48:44.4524658Zat 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-02-27T01:48:44.4525556Zat 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-02-27T01:48:44.4526056Zat 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-02-27T01:48:44.4526543Zat 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2021-02-27T01:48:44.4527006Zat 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2021-02-27T01:48:44.4527420Zat 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-02-27T01:48:44.4528013Zat 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-02-27T01:48:44.4528638Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-02-27T01:48:44.4529133Zat 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-02-27T01:48:44.4529570Zat 
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-02-27T01:48:44.4529997Zat 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-02-27T01:48:44.4530432Zat 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-02-27T01:48:44.4531025Zat 
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-02-27T01:48:44.4531434Zat 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-02-27T01:48:44.4532035Zat 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
2021-02-27T01:48:44.4532465Zat 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2021-02-27T01:48:44.4532866Zat 
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-02-27T01:48:44.4533318Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2021-02-27T01:48:44.4533827Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2021-02-27T01:48:44.4534346Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2021-02-27T01:48:44.4534850Zat 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
2021-02-27T01:48:44.4535721Zat 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2021-02-27T01:48:44.4536257Zat 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2021-02-27T01:48:44.4536761Zat 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2021-02-27T01:48:44.4537393Zat 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2021-02-27T01:48:44.4538093Z Caused by: 
org.apache.flink.runtime.client.JobExecutionException: Not enough resources 
available for scheduling.
2021-02-27T01:48:44.4538877Zat 
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.lambda$determineParallelismAndAssignResources$18(AdaptiveScheduler.java:585)
2021-02-27T01:48:44.4539420Zat 
java.util.Optional.orElseThrow(Optional.java:290)
2021-02-27T01:48:44.4539942Zat 

[jira] [Created] (FLINK-21535) UnalignedCheckpointITCase.execute failed with "OutOfMemoryError: Java heap space"

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-21535:


 Summary: UnalignedCheckpointITCase.execute failed with 
"OutOfMemoryError: Java heap space"
 Key: FLINK-21535
 URL: https://issues.apache.org/jira/browse/FLINK-21535
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.13.0
Reporter: Dawid Wysakowicz


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13866=logs=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3=a99e99c7-21cd-5a1f-7274-585e62b72f56

{code}
2021-02-27T02:11:41.5659201Z 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-02-27T02:11:41.5659947Zat 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-02-27T02:11:41.5660794Zat 
org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:137)
2021-02-27T02:11:41.5661618Zat 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
2021-02-27T02:11:41.5662356Zat 
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
2021-02-27T02:11:41.5663104Zat 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-02-27T02:11:41.5664016Zat 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-02-27T02:11:41.5664817Zat 
org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:237)
2021-02-27T02:11:41.5665638Zat 
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
2021-02-27T02:11:41.5666405Zat 
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
2021-02-27T02:11:41.5667609Zat 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
2021-02-27T02:11:41.5668358Zat 
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
2021-02-27T02:11:41.5669218Zat 
org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1066)
2021-02-27T02:11:41.5669928Zat 
akka.dispatch.OnComplete.internal(Future.scala:264)
2021-02-27T02:11:41.5670540Zat 
akka.dispatch.OnComplete.internal(Future.scala:261)
2021-02-27T02:11:41.5671268Zat 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
2021-02-27T02:11:41.5671881Zat 
akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
2021-02-27T02:11:41.5672512Zat 
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-02-27T02:11:41.5673219Zat 
org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
2021-02-27T02:11:41.5674085Zat 
scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
2021-02-27T02:11:41.5674794Zat 
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
2021-02-27T02:11:41.5675466Zat 
akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
2021-02-27T02:11:41.5676181Zat 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
2021-02-27T02:11:41.5676977Zat 
akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
2021-02-27T02:11:41.5677717Zat 
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
2021-02-27T02:11:41.5678409Zat 
scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
2021-02-27T02:11:41.5679071Zat 
scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
2021-02-27T02:11:41.5679776Zat 
akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
2021-02-27T02:11:41.5680576Zat 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
2021-02-27T02:11:41.5681383Zat 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-02-27T02:11:41.5682167Zat 
akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
2021-02-27T02:11:41.5683040Zat 
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
2021-02-27T02:11:41.5683759Zat 
akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
2021-02-27T02:11:41.5684493Zat 
akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
2021-02-27T02:11:41.5685238Zat 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
2021-02-27T02:11:41.5686193Zat 
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
2021-02-27T02:11:41.5686901Zat 
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
2021-02-27T02:11:41.5687621Zat 
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)

[jira] [Created] (FLINK-21534) Test jars are uploaded twice during Flink maven deployment

[Robert Metzger (Jira)]

Robert Metzger created FLINK-21534:
--

 Summary: Test jars are uploaded twice during Flink maven deployment
 Key: FLINK-21534
 URL: https://issues.apache.org/jira/browse/FLINK-21534
 Project: Flink
  Issue Type: Bug
  Components: Build System
Affects Versions: 1.13.0
Reporter: Robert Metzger


Here's an example of a snapshot build: 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13905=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7

You can see that the same jar is uploaded twice:
{code}
2021-02-28T21:07:05.2904685Z Uploading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/1.13-SNAPSHOT/flink-runtime_2.11-1.13-20210228.210704-95-tests.jar

2021-02-28T21:07:05.5738429Z Uploaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/1.13-SNAPSHOT/flink-runtime_2.11-1.13-20210228.210704-95-tests.jar
 (4714 KB at 16596.2 KB/sec)


2021-02-28T21:07:05.6506506Z Uploading: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/1.13-SNAPSHOT/flink-runtime_2.11-1.13-20210228.210704-95-tests.jar

2021-02-28T21:07:07.1976748Z Uploaded: 
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime_2.11/1.13-SNAPSHOT/flink-runtime_2.11-1.13-20210228.210704-95-tests.jar
 (4714 KB at 3046.7 KB/sec)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21533) Kafka011ITCase#testAllDeletes fails on azure

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-21533:


 Summary: Kafka011ITCase#testAllDeletes fails on azure
 Key: FLINK-21533
 URL: https://issues.apache.org/jira/browse/FLINK-21533
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Tests
Affects Versions: 1.11.3
Reporter: Dawid Wysakowicz


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13867=logs=3d12d40f-c62d-5ec4-6acc-0efe94cc3e89=e4f347ab-2a29-5d7c-3685-b0fcd2b6b051

{code}
2021-02-26T22:27:56.9286925Z [ERROR] 
testAllDeletes(org.apache.flink.streaming.connectors.kafka.Kafka011ITCase)  
Time elapsed: 3.228 s  <<< ERROR!
2021-02-26T22:27:56.9287994Z 
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
2021-02-26T22:27:56.9288805Zat 
org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
2021-02-26T22:27:56.9290091Zat 
org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:762)
2021-02-26T22:27:56.9290978Zat 
org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:77)
2021-02-26T22:27:56.9291926Zat 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1651)
2021-02-26T22:27:56.9293538Zat 
org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runAllDeletesTest(KafkaConsumerTestBase.java:1649)
2021-02-26T22:27:56.9294944Zat 
org.apache.flink.streaming.connectors.kafka.Kafka011ITCase.testAllDeletes(Kafka011ITCase.java:130)
2021-02-26T22:27:56.9295702Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-02-26T22:27:56.9296370Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-02-26T22:27:56.9299360Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-02-26T22:27:56.9299955Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2021-02-26T22:27:56.9300402Zat 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-02-26T22:27:56.9300897Zat 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-02-26T22:27:56.9301387Zat 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-02-26T22:27:56.9301851Zat 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-02-26T22:27:56.9302471Zat 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
2021-02-26T22:27:56.9325899Zat 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
2021-02-26T22:27:56.9327852Zat 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2021-02-26T22:27:56.9328934Zat java.lang.Thread.run(Thread.java:748)
2021-02-26T22:27:56.9329795Z Caused by: org.apache.flink.runtime.JobException: 
Recovery is suppressed by NoRestartBackoffTimeStrategy
2021-02-26T22:27:56.9330778Zat 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118)
2021-02-26T22:27:56.9331904Zat 
org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:80)
2021-02-26T22:27:56.9333126Zat 
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:206)
2021-02-26T22:27:56.9334090Zat 
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:197)
2021-02-26T22:27:56.9335043Zat 
org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:189)
2021-02-26T22:27:56.9335946Zat 
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:639)
2021-02-26T22:27:56.9336834Zat 
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:397)
2021-02-26T22:27:56.9337698Zat 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-02-26T22:27:56.9338398Zat 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-02-26T22:27:56.9339236Zat 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-02-26T22:27:56.9339949Zat 
java.lang.reflect.Method.invoke(Method.java:498)
2021-02-26T22:27:56.9340681Zat 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:306)
2021-02-26T22:27:56.9341537Zat 
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:213)
2021-02-26T22:27:56.9342486Zat 
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
2021-02-26T22:27:56.9343296Zat