[jira] [Created] (FLINK-22108) Ephemeral socket address was checkpointed to state and restored back in CollectSinkOperatorCoordinator
Kezhu Wang created FLINK-22108:
--
Summary: Ephemeral socket address was checkpointed to state and
restored back in CollectSinkOperatorCoordinator
Key: FLINK-22108
URL: https://issues.apache.org/jira/browse/FLINK-22108
Project: Flink
Issue Type: Bug
Components: API / DataStream
Affects Versions: 1.13.0
Reporter: Kezhu Wang
{{CollectSinkOperatorCoordinator}} checkpointed its {{address}} field to state
and restored back. That field is listener address of {{CollectSinkFunction}}.
After {{resetToCheckpoint}} (eg. global failover}}, {{address}} is meaningless.
If client request comes before {{CollectSinkAddressEvent}}, it will use that
meaningless address for connection. In best situation, error happens, and
nothing hurts. In bad situation, no one knows where the restored address points
to now.
cc [~TsReaper] [~ykt836]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22107) Include antlr into hive connector uber jars
Rui Li created FLINK-22107: -- Summary: Include antlr into hive connector uber jars Key: FLINK-22107 URL: https://issues.apache.org/jira/browse/FLINK-22107 Project: Flink Issue Type: Sub-task Components: Connectors / Hive Reporter: Rui Li Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22106) ResultType of GeneratedExpression in StringCallGen should be compatible with their definition in FlinkSqlOperatorTable
Caizhi Weng created FLINK-22106:
---
Summary: ResultType of GeneratedExpression in StringCallGen should
be compatible with their definition in FlinkSqlOperatorTable
Key: FLINK-22106
URL: https://issues.apache.org/jira/browse/FLINK-22106
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.13.0
Reporter: Caizhi Weng
Fix For: 1.13.0
Add the following test case to {{TableEnvironmentITCase}} to reproduce this bug:
{code:scala}
@Test
def myTest(): Unit = {
tEnv.executeSql(
"""
|CREATE TABLE my_source (
| a VARCHAR(10)
|) WITH (
| 'connector'='values',
| 'bounded'='true'
|)
|""".stripMargin)
tEnv.explainSql("SELECT ifnull(substring(a, 2, 5), 'null') FROM my_source")
}
{code}
The exception stack is
{code}
org.apache.flink.table.planner.codegen.CodeGenException: Mismatch of function's
argument data type 'VARCHAR(10)' and actual argument type 'STRING'.
at
org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$$anonfun$verifyArgumentTypes$1.apply(BridgingFunctionGenUtil.scala:323)
at
org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$$anonfun$verifyArgumentTypes$1.apply(BridgingFunctionGenUtil.scala:320)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.verifyArgumentTypes(BridgingFunctionGenUtil.scala:320)
at
org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.generateFunctionAwareCallWithDataType(BridgingFunctionGenUtil.scala:95)
at
org.apache.flink.table.planner.codegen.calls.BridgingFunctionGenUtil$.generateFunctionAwareCall(BridgingFunctionGenUtil.scala:65)
at
org.apache.flink.table.planner.codegen.calls.BridgingSqlFunctionCallGen.generate(BridgingSqlFunctionCallGen.scala:73)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateCallExpression(ExprCodeGenerator.scala:832)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:529)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.visitCall(ExprCodeGenerator.scala:56)
at org.apache.calcite.rex.RexCall.accept(RexCall.java:174)
at
org.apache.flink.table.planner.codegen.ExprCodeGenerator.generateExpression(ExprCodeGenerator.scala:155)
at
org.apache.flink.table.planner.codegen.CalcCodeGenerator$$anonfun$2.apply(CalcCodeGenerator.scala:141)
at
org.apache.flink.table.planner.codegen.CalcCodeGenerator$$anonfun$2.apply(CalcCodeGenerator.scala:141)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
org.apache.flink.table.planner.codegen.CalcCodeGenerator$.produceProjectionCode$1(CalcCodeGenerator.scala:141)
at
org.apache.flink.table.planner.codegen.CalcCodeGenerator$.generateProcessCode(CalcCodeGenerator.scala:167)
at
org.apache.flink.table.planner.codegen.CalcCodeGenerator$.generateCalcOperator(CalcCodeGenerator.scala:50)
at
org.apache.flink.table.planner.codegen.CalcCodeGenerator.generateCalcOperator(CalcCodeGenerator.scala)
at
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecCalc.translateToPlanInternal(CommonExecCalc.java:94)
at
org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:127)
at
org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:71)
at
org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:70)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at
[jira] [Created] (FLINK-22105) testForceAlignedCheckpointResultingInPriorityEvents unstable
Robert Metzger created FLINK-22105:
--
Summary: testForceAlignedCheckpointResultingInPriorityEvents
unstable
Key: FLINK-22105
URL: https://issues.apache.org/jira/browse/FLINK-22105
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.13.0
Reporter: Robert Metzger
Fix For: 1.13.0
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=9021=logs=9dc1b5dc-bcfa-5f83-eaa7-0cb181ddc267=ab910030-93db-52a7-74a3-34a0addb481b
{code}
2021-04-01T19:29:55.2392858Z [ERROR] Tests run: 10, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 1.921 s <<< FAILURE! - in
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorTest
2021-04-01T19:29:55.2396751Z [ERROR]
testForceAlignedCheckpointResultingInPriorityEvents(org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorTest)
Time elapsed: 0.02 s <<< ERROR!
2021-04-01T19:29:55.2397415Z java.lang.RuntimeException: unable to send request
to worker
2021-04-01T19:29:55.2397956Zat
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriterImpl.enqueue(ChannelStateWriterImpl.java:228)
2021-04-01T19:29:55.2398603Zat
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriterImpl.finishOutput(ChannelStateWriterImpl.java:183)
2021-04-01T19:29:55.2399310Zat
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:298)
2021-04-01T19:29:55.2400104Zat
org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorTest.testForceAlignedCheckpointResultingInPriorityEvents(SubtaskCheckpointCoordinatorTest.java:215)
2021-04-01T19:29:55.2400746Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2021-04-01T19:29:55.2401202Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2021-04-01T19:29:55.2401746Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2021-04-01T19:29:55.2402237Zat
java.lang.reflect.Method.invoke(Method.java:498)
2021-04-01T19:29:55.2402722Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2021-04-01T19:29:55.2403270Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2021-04-01T19:29:55.2403818Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2021-04-01T19:29:55.2404354Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2021-04-01T19:29:55.2404854Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2021-04-01T19:29:55.2405359Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2021-04-01T19:29:55.2405896Zat
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2021-04-01T19:29:55.2406393Zat
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2021-04-01T19:29:55.2406855Zat
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2021-04-01T19:29:55.2407331Zat
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2021-04-01T19:29:55.2407806Zat
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2021-04-01T19:29:55.2408279Zat
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2021-04-01T19:29:55.2408907Zat
org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2021-04-01T19:29:55.2409403Zat
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
2021-04-01T19:29:55.2409954Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
2021-04-01T19:29:55.2410524Zat
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
2021-04-01T19:29:55.2411318Zat
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
2021-04-01T19:29:55.2411880Zat
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2021-04-01T19:29:55.2412467Zat
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2021-04-01T19:29:55.2413006Zat
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2021-04-01T19:29:55.2413519Zat
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2021-04-01T19:29:55.2414082Z Caused by: java.lang.IllegalArgumentException:
writer not found while processing request: writeOutput 0
2021-04-01T19:29:55.2414726Zat
org.apache.flink.runtime.checkpoint.channel.CheckpointInProgressRequest.onWriterMissing(ChannelStateWriteRequest.java:223)
2021-04-01T19:29:55.2415463Zat
Re: [VOTE] Apache Flink Stateful Functions 3.0.0, release candidate #1
+1 (binding) Checked sums and signatures: OK Checked RAT and end-to-end tests: OK Checked version in pom/README/setup.py files: OK Checked release notes: OK Checked docker PR: OK Thanks for driving this release, Gordon! Best Regards, Yu On Fri, 2 Apr 2021 at 09:22, Seth Wiesman wrote: > +1 (non-binding) > > - Built from source and executed end to end tests > - Checked licenses and signatures > - Deployed remote Java SDK to gke cluster > - Took savepoint and statefully rescaled > > Seth > > On Thu, Apr 1, 2021 at 9:05 AM Konstantin Knauf wrote: > > > +1 (non-binding) > > > > - mvn clean install -Prun-e2e-tests (java 8) from source > > - python3 -m unittest tests > > - spin up Statefun Cluster on EKS with an image built from the > Dockerfiles > > of [1] > > - run Python & Java Greeter example on AWS Lambda > > - read through documentation (opened [2] to fix some tpoys) > > > > [1] https://github.com/apache/flink-statefun-docker/pull/13 > > [2] https://github.com/apache/flink-statefun/pull/219 > > > > On Thu, Apr 1, 2021 at 6:46 AM Tzu-Li (Gordon) Tai > > wrote: > > > > > +1 (binding) > > > > > > - verified signatures and hashes > > > - NOTICE and LICENSE files in statefun-flink-distribution, > > > statefun-protobuf-shaded, and statefun-sdk-java looks sane > > > - maven clean install -Prun-e2e-tests (java 8) from source > > > - ran all examples and tutorials in apache/flink-statefun-playground > with > > > the new artifacts > > > - Ran my SDK verifier utility [1] against the new Java and Python SDKs. > > > > > > Cheers, > > > Gordon > > > > > > [1] https://github.com/tzulitai/statefun-sdk-verifier > > > > > > On Wed, Mar 31, 2021 at 8:50 PM Igal Shilman > wrote: > > > > > > > Thanks Gordon for managing the release! > > > > > > > > +1 (non binding) from my side: > > > > > > > > Here are the results of my testing: > > > > - verified the signatures > > > > - verified that the source distribution doesn't contain any binary > > files > > > > - ran mvn clean install -Prun-e2e-tests with java8 > > > > - ran the smoke test that sends 100 million messages locally. > > > > - extended the smoke test to include the remote sdks (1 function in > the > > > > Java SDK, 1 function in the Python SDK), and it passes. > > > > - deployed to kubernetes with minio as an S3 replacement. > > > > > > > > > > > > On Tue, Mar 30, 2021 at 12:29 PM Tzu-Li (Gordon) Tai < > > > tzuli...@apache.org> > > > > wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > Please review and vote on the release candidate #1 for the version > > > 3.0.0 > > > > of > > > > > Apache Flink Stateful Functions, as follows: > > > > > [ ] +1, Approve the release > > > > > [ ] -1, Do not approve the release (please provide specific > comments) > > > > > > > > > > **Testing Guideline** > > > > > > > > > > You can find here [1] a page in the project wiki on instructions > for > > > > > testing. > > > > > To cast a vote, it is not necessary to perform all listed checks, > > > > > but please mention which checks you have performed when voting. > > > > > > > > > > **Release Overview** > > > > > > > > > > As an overview, the release consists of the following: > > > > > a) Stateful Functions canonical source distribution, to be deployed > > to > > > > the > > > > > release repository at dist.apache.org > > > > > b) Stateful Functions Python SDK distributions to be deployed to > PyPI > > > > > c) Maven artifacts to be deployed to the Maven Central Repository > > > > > d) New Dockerfiles for the release > > > > > > > > > > **Staging Areas to Review** > > > > > > > > > > The staging areas containing the above mentioned artifacts are as > > > > follows, > > > > > for your review: > > > > > * All artifacts for a) and b) can be found in the corresponding dev > > > > > repository at dist.apache.org [2] > > > > > * All artifacts for c) can be found at the Apache Nexus Repository > > [3] > > > > > > > > > > All artifacts are signed with the key > > > > > 1C1E2394D3194E1944613488F320986D35C33D6A [4] > > > > > > > > > > Other links for your review: > > > > > * JIRA release notes [5] > > > > > * source code tag “release-3.0.0-rc1” [6] > > > > > * PR for the new Dockerfiles [7] > > > > > > > > > > **Vote Duration** > > > > > > > > > > The voting time will run for at least 72 hours. I’m targeting this > > vote > > > > to > > > > > last until April. 2nd, 12pm CET. > > > > > It is adopted by majority approval, with at least 3 PMC affirmative > > > > votes. > > > > > > > > > > Thanks, > > > > > Gordon > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release > > > > > [2] > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-statefun-3.0.0-rc1/ > > > > > [3] > > > > > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1415/ > > > > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS > > > > > [5] > > > > > > > > > > > > > > > > > > > >
[jira] [Created] (FLINK-22104) SqlClientTest.testExecuteSqlFile fail
Guowei Ma created FLINK-22104:
-
Summary: SqlClientTest.testExecuteSqlFile fail
Key: FLINK-22104
URL: https://issues.apache.org/jira/browse/FLINK-22104
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Affects Versions: 1.13.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15996=logs=0e05564a-179a-5e65-3527-d468802ba62b=0f3e9f39-333c-5f88-6bce-3254b2cb21e7=8654
{code:java}
java.lang.AssertionError:
Expected: a string containing "LOAD MODULE\t\tLoad a module. Syntax: 'LOAD
MODULE [WITH ('' = '' [, '' = '', ...])];'"
but: was "No default environment specified.
Searching for
'/tmp/junit2344703144390059316/conf/sql-client-defaults.yaml'...not found.
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22103) testNumberOfBuiltinFunctions fail
Guowei Ma created FLINK-22103:
-
Summary: testNumberOfBuiltinFunctions fail
Key: FLINK-22103
URL: https://issues.apache.org/jira/browse/FLINK-22103
Project: Flink
Issue Type: Bug
Components: Connectors / Hive
Affects Versions: 1.13.0
Reporter: Guowei Ma
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15996=logs=8fd975ef-f478-511d-4997-6f15fe8a1fd3=ac0fa443-5d45-5a6b-3597-0310ecc1d2ab=24281
{code:java}
[ERROR] Tests run: 10, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.867
s <<< FAILURE! - in org.apache.flink.table.module.hive.HiveModuleTest
[ERROR]
testNumberOfBuiltinFunctions(org.apache.flink.table.module.hive.HiveModuleTest)
Time elapsed: 0.006 s <<< FAILURE!
java.lang.AssertionError: expected:<222> but was:<220>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at
org.apache.flink.table.module.hive.HiveModuleTest.verifyNumBuiltInFunctions(HiveModuleTest.java:89)
at
org.apache.flink.table.module.hive.HiveModuleTest.testNumberOfBuiltinFunctions(HiveModuleTest.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22102) Throw meaningful exceptions for features we don't support
Rui Li created FLINK-22102: -- Summary: Throw meaningful exceptions for features we don't support Key: FLINK-22102 URL: https://issues.apache.org/jira/browse/FLINK-22102 Project: Flink Issue Type: Sub-task Components: Connectors / Hive Reporter: Rui Li Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
