Re: [BULK]Re: [SURVEY] Remove Mesos support
+1 Best, Yangze Guo On Mon, Mar 29, 2021 at 11:31 AM Xintong Song wrote: > > +1 > It's already a matter of fact for a while that we no longer port new features > to the Mesos deployment. > > Thank you~ > > Xintong Song > > > > On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann wrote: >> >> +1 for officially deprecating this component for the 1.13 release. >> >> Cheers, >> Till >> >> On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf wrote: >>> >>> Hi Matthias, >>> >>> Thank you for following up on this. +1 to officially deprecate Mesos in the >>> code and documentation, too. It will be confusing for users if this >>> diverges from the roadmap. >>> >>> Cheers, >>> >>> Konstantin >>> >>> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl >>> wrote: Hi everyone, considering the upcoming release of Flink 1.13, I wanted to revive the discussion about the Mesos support ones more. Mesos is also already listed as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align the documentation accordingly to make it more explicit? What do you think? Best, Matthias [1] https://flink.apache.org/roadmap.html#feature-radar On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann wrote: > Hi Oleksandr, > > yes you are right. The biggest problem is at the moment the lack of test > coverage and thereby confidence to make changes. We have some e2e tests > which you can find here [1]. These tests are, however, quite coarse > grained > and are missing a lot of cases. One idea would be to add a Mesos e2e test > based on Flink's end-to-end test framework [2]. I think what needs to be > done there is to add a Mesos resource and a way to submit jobs to a Mesos > cluster to write e2e tests. > > [1] https://github.com/apache/flink/tree/master/flink-jepsen > [2] > https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common > > Cheers, > Till > > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi < > o.nitavs...@criteo.com> wrote: > >> Hello Xintong, >> >> Thanks for the insights and support. >> >> Browsing the Mesos backlog and didn't identify anything critical, which >> is left there. >> >> I see that there are were quite a lot of contributions to the Flink >> Mesos >> in the recent version: >> https://github.com/apache/flink/commits/master/flink-mesos. >> We plan to validate the current Flink master (or release 1.12 branch) >> our >> Mesos setup. In case of any issues, we will try to propose changes. >> My feeling is that our test results shouldn't affect the Flink 1.12 >> release cycle. And if any potential commits will land into the 1.12.1 it >> should be totally fine. >> >> In the future, we would be glad to help you guys with any >> maintenance-related questions. One of the highest priorities around this >> component seems to be the development of the full e2e test. >> >> Kind Regards >> Oleksandr Nitavskyi >> >> From: Xintong Song >> Sent: Tuesday, October 27, 2020 7:14 AM >> To: dev ; user >> Cc: Piyush Narang >> Subject: [BULK]Re: [SURVEY] Remove Mesos support >> >> Hi Piyush, >> >> Thanks a lot for sharing the information. It would be a great relief >> that >> you are good with Flink on Mesos as is. >> >> As for the jira issues, I believe the most essential ones should have >> already been resolved. You may find some remaining open issues here [1], >> but not all of them are necessary if we decide to keep Flink on Mesos >> as is. >> >> At the moment and in the short future, I think helps are mostly needed >> on >> testing the upcoming release 1.12 with Mesos use cases. The community is >> currently actively preparing the new release, and hopefully we could >> come >> up with a release candidate early next month. It would be greatly >> appreciated if you fork as experienced Flink on Mesos users can help >> with >> verifying the release candidates. >> >> >> Thank you~ >> >> Xintong Song >> >> [1] >> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open >> < >>
[jira] [Created] (FLINK-22005) SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1)
Guowei Ma created FLINK-22005: - Summary: SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1) Key: FLINK-22005 URL: https://issues.apache.org/jira/browse/FLINK-22005 Project: Flink Issue Type: Bug Components: Table SQL / Client Affects Versions: 1.13.0 Reporter: Guowei Ma The test fail because of Waiting for Elasticsearch records indefinitely. https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15583=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=19826 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [BULK]Re: [SURVEY] Remove Mesos support
+1 It's already a matter of fact for a while that we no longer port new features to the Mesos deployment. Thank you~ Xintong Song On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann wrote: > +1 for officially deprecating this component for the 1.13 release. > > Cheers, > Till > > On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf > wrote: > >> Hi Matthias, >> >> Thank you for following up on this. +1 to officially deprecate Mesos in >> the code and documentation, too. It will be confusing for users if this >> diverges from the roadmap. >> >> Cheers, >> >> Konstantin >> >> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl >> wrote: >> >>> Hi everyone, >>> considering the upcoming release of Flink 1.13, I wanted to revive the >>> discussion about the Mesos support ones more. Mesos is also already >>> listed >>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align >>> the >>> documentation accordingly to make it more explicit? >>> >>> What do you think? >>> >>> Best, >>> Matthias >>> >>> [1] https://flink.apache.org/roadmap.html#feature-radar >>> >>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann >>> wrote: >>> >>> > Hi Oleksandr, >>> > >>> > yes you are right. The biggest problem is at the moment the lack of >>> test >>> > coverage and thereby confidence to make changes. We have some e2e tests >>> > which you can find here [1]. These tests are, however, quite coarse >>> grained >>> > and are missing a lot of cases. One idea would be to add a Mesos e2e >>> test >>> > based on Flink's end-to-end test framework [2]. I think what needs to >>> be >>> > done there is to add a Mesos resource and a way to submit jobs to a >>> Mesos >>> > cluster to write e2e tests. >>> > >>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen >>> > [2] >>> > >>> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common >>> > >>> > Cheers, >>> > Till >>> > >>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi < >>> > o.nitavs...@criteo.com> wrote: >>> > >>> >> Hello Xintong, >>> >> >>> >> Thanks for the insights and support. >>> >> >>> >> Browsing the Mesos backlog and didn't identify anything critical, >>> which >>> >> is left there. >>> >> >>> >> I see that there are were quite a lot of contributions to the Flink >>> Mesos >>> >> in the recent version: >>> >> https://github.com/apache/flink/commits/master/flink-mesos. >>> >> We plan to validate the current Flink master (or release 1.12 branch) >>> our >>> >> Mesos setup. In case of any issues, we will try to propose changes. >>> >> My feeling is that our test results shouldn't affect the Flink 1.12 >>> >> release cycle. And if any potential commits will land into the 1.12.1 >>> it >>> >> should be totally fine. >>> >> >>> >> In the future, we would be glad to help you guys with any >>> >> maintenance-related questions. One of the highest priorities around >>> this >>> >> component seems to be the development of the full e2e test. >>> >> >>> >> Kind Regards >>> >> Oleksandr Nitavskyi >>> >> >>> >> From: Xintong Song >>> >> Sent: Tuesday, October 27, 2020 7:14 AM >>> >> To: dev ; user >>> >> Cc: Piyush Narang >>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support >>> >> >>> >> Hi Piyush, >>> >> >>> >> Thanks a lot for sharing the information. It would be a great relief >>> that >>> >> you are good with Flink on Mesos as is. >>> >> >>> >> As for the jira issues, I believe the most essential ones should have >>> >> already been resolved. You may find some remaining open issues here >>> [1], >>> >> but not all of them are necessary if we decide to keep Flink on Mesos >>> as is. >>> >> >>> >> At the moment and in the short future, I think helps are mostly >>> needed on >>> >> testing the upcoming release 1.12 with Mesos use cases. The community >>> is >>> >> currently actively preparing the new release, and hopefully we could >>> come >>> >> up with a release candidate early next month. It would be greatly >>> >> appreciated if you fork as experienced Flink on Mesos users can help >>> with >>> >> verifying the release candidates. >>> >> >>> >> >>> >> Thank you~ >>> >> >>> >> Xintong Song >>> >> >>> >> [1] >>> >> >>> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open >>> >> < >>> >> >>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D=0 >>> >> > >>> >> >>> >> On Tue, Oct 27, 2020 at
[jira] [Created] (FLINK-22004) Translate Flink Roadmap to Chinese.
Yuan Mei created FLINK-22004: Summary: Translate Flink Roadmap to Chinese. Key: FLINK-22004 URL: https://issues.apache.org/jira/browse/FLINK-22004 Project: Flink Issue Type: Task Components: Documentation Reporter: Yuan Mei https://flink.apache.org/roadmap.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22002) AggregateReduceGroupingITCase.testSingleAggOnTable_HashAgg_WithLocalAgg fail because of submitting task time-out.
Guowei Ma created FLINK-22002: - Summary: AggregateReduceGroupingITCase.testSingleAggOnTable_HashAgg_WithLocalAgg fail because of submitting task time-out. Key: FLINK-22002 URL: https://issues.apache.org/jira/browse/FLINK-22002 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.12.2 Reporter: Guowei Ma https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15634=logs=955770d3-1fed-5a0a-3db6-0c7554c910cb=14447d61-56b4-5000-80c1-daa459247f6a=6424 {code:java} org.apache.flink.table.planner.runtime.batch.sql.agg.AggregateReduceGroupingITCase 2021-03-29T00:27:25.3406344Z [ERROR] testSingleAggOnTable_HashAgg_WithLocalAgg(org.apache.flink.table.planner.runtime.batch.sql.agg.AggregateReduceGroupingITCase) Time elapsed: 21.908 s <<< ERROR! 2021-03-29T00:27:25.3407190Z java.lang.RuntimeException: Failed to fetch next result 2021-03-29T00:27:25.3407792Zat org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109) 2021-03-29T00:27:25.3408502Zat org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80) 2021-03-29T00:27:25.3409188Zat org.apache.flink.table.planner.sinks.SelectTableSinkBase$RowIteratorWrapper.hasNext(SelectTableSinkBase.java:117) 2021-03-29T00:27:25.3416724Zat org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:350) 2021-03-29T00:27:25.3417510Zat java.util.Iterator.forEachRemaining(Iterator.java:115) 2021-03-29T00:27:25.3418416Zat org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:108) 2021-03-29T00:27:25.3419031Zat org.apache.flink.table.planner.runtime.utils.BatchTestBase.executeQuery(BatchTestBase.scala:298) 2021-03-29T00:27:25.3419657Zat org.apache.flink.table.planner.runtime.utils.BatchTestBase.check(BatchTestBase.scala:138) 2021-03-29T00:27:25.3420638Zat org.apache.flink.table.planner.runtime.utils.BatchTestBase.checkResult(BatchTestBase.scala:104) 2021-03-29T00:27:25.3421384Zat org.apache.flink.table.planner.runtime.batch.sql.agg.AggregateReduceGroupingITCase.testSingleAggOnTable(AggregateReduceGroupingITCase.scala:182) 2021-03-29T00:27:25.3422284Zat org.apache.flink.table.planner.runtime.batch.sql.agg.AggregateReduceGroupingITCase.testSingleAggOnTable_HashAgg_WithLocalAgg(AggregateReduceGroupingITCase.scala:135) 2021-03-29T00:27:25.3422975Zat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2021-03-29T00:27:25.3423504Zat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 2021-03-29T00:27:25.3424298Zat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2021-03-29T00:27:25.3425229Zat java.lang.reflect.Method.invoke(Method.java:498) 2021-03-29T00:27:25.3426107Zat org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) 2021-03-29T00:27:25.3426756Zat org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) 2021-03-29T00:27:25.3427743Zat org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) 2021-03-29T00:27:25.3428520Zat org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) 2021-03-29T00:27:25.3429128Zat org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) 2021-03-29T00:27:25.3429715Zat org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) 2021-03-29T00:27:25.3433435Zat org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) 2021-03-29T00:27:25.3433977Zat org.junit.rules.RunRules.evaluate(RunRules.java:20) 2021-03-29T00:27:25.3434476Zat org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) 2021-03-29T00:27:25.3435607Zat org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) 2021-03-29T00:27:25.3436460Zat org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) 2021-03-29T00:27:25.3437054Zat org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) 2021-03-29T00:27:25.3437673Zat org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) 2021-03-29T00:27:25.3438765Zat org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) 2021-03-29T00:27:25.3439362Zat org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) 2021-03-29T00:27:25.3440504Zat org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) 2021-03-29T00:27:25.3441100Zat org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) 2021-03-29T00:27:25.3441673Zat org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) 2021-03-29T00:27:25.3442205Zat
[jira] [Created] (FLINK-22001) Exceptions from JobMaster initialization are not forwarded to the user
Robert Metzger created FLINK-22001: -- Summary: Exceptions from JobMaster initialization are not forwarded to the user Key: FLINK-22001 URL: https://issues.apache.org/jira/browse/FLINK-22001 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.13.0 Reporter: Robert Metzger Steps to reproduce: Set up a streaming job with an invalid parallelism configuration, for example: {code} .setParallelism(15).setMaxParallelism(1); {code} This should report the following exception to the user: {code} Caused by: org.apache.flink.runtime.JobException: Vertex Window(GlobalWindows(), DeltaTrigger, TimeEvictor, ComparableAggregator, PassThroughWindowFunction)'s parallelism (15) is higher than the max parallelism (1). Please lower the parallelism or increase the max parallelism. at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:160) at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.attachJobGraph(DefaultExecutionGraph.java:781) at org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:193) at org.apache.flink.runtime.scheduler.DefaultExecutionGraphFactory.createAndRestoreExecutionGraph(DefaultExecutionGraphFactory.java:106) at org.apache.flink.runtime.scheduler.SchedulerBase.createAndRestoreExecutionGraph(SchedulerBase.java:252) at org.apache.flink.runtime.scheduler.SchedulerBase.(SchedulerBase.java:185) at org.apache.flink.runtime.scheduler.DefaultScheduler.(DefaultScheduler.java:119) at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:132) at org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:110) at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:340) at org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:317) at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:94) at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:39) at org.apache.flink.runtime.jobmaster.JobManagerRunnerImpl.startJobMasterServiceSafely(JobManagerRunnerImpl.java:363) ... 13 more {code} However, what the user sees is {code} 2021-03-28 20:32:33,935 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Job 419f60eac551619fc1081c670ced3649 reached globally terminal state FAILED. ... 2021-03-28 20:32:33,974 INFO org.apache.flink.runtime.dispatcher.StandaloneDispatcher [] - Stopped dispatcher akka://flink/user/rpc/dispatcher_2. 2021-03-28 20:32:33,977 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Stopping Akka RPC service. Exception in thread "main" org.apache.flink.util.FlinkException: Failed to execute job 'CarTopSpeedWindowingExample'. at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:1975) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1853) at org.apache.flink.streaming.api.environment.LocalStreamEnvironment.execute(LocalStreamEnvironment.java:69) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1839) at org.apache.flink.streaming.examples.windowing.TopSpeedWindowing.main(TopSpeedWindowing.java:101) Caused by: java.lang.RuntimeException: Error while waiting for job to be initialized at org.apache.flink.client.ClientUtils.waitUntilJobInitializationFinished(ClientUtils.java:160) at org.apache.flink.client.program.PerJobMiniClusterFactory.lambda$submitJob$2(PerJobMiniClusterFactory.java:83) at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:73) at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616) at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591) at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at
[jira] [Created] (FLINK-22000) Set a default character set in InputStreamReader to solve potential garbled problems
dbgp2021 created FLINK-22000: Summary: Set a default character set in InputStreamReader to solve potential garbled problems Key: FLINK-22000 URL: https://issues.apache.org/jira/browse/FLINK-22000 Project: Flink Issue Type: Improvement Reporter: dbgp2021 When a InputStreamReader is used, the parameter setting of a default character set is recommended to solve potential garbled problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21999) The logic about whether Checkpoint is enabled.
ZhangWei created FLINK-21999: Summary: The logic about whether Checkpoint is enabled. Key: FLINK-21999 URL: https://issues.apache.org/jira/browse/FLINK-21999 Project: Flink Issue Type: Bug Components: Runtime / Checkpointing Reporter: ZhangWei org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder#isCheckpointingEnabled assumes checkpoint enabled when JobCheckpointingSettings is not null. While this is not enough, we must also guarantee the checkpoint interval is between [MINIMAL_CHECKPOINT_TIME, Long.MaxValue). That is like the JobGraph#isCheckpointingEnabled does. In current implement, when we do not set checkpoint interval, leaving it the default value -1, the interval will be changed to Long.MaxValue. Thus DefaultExecutionGraphBuilder#isCheckpointingEnabled will return true. That is not correct. in addition, there are different classes assume checkpoint enabled with different interval range. 1. CheckpointConfig -> (0,Long.MaxValue*]*. 2. JobGraph -> (0,Long.MaxValue) This is not consistent. And the correct range is [MINIMAL_CHECKPOINT_TIME, Long.MaxValue). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21998) Copy more code from hive and move them to a dedicated package
Rui Li created FLINK-21998: -- Summary: Copy more code from hive and move them to a dedicated package Key: FLINK-21998 URL: https://issues.apache.org/jira/browse/FLINK-21998 Project: Flink Issue Type: Sub-task Components: Connectors / Hive Reporter: Rui Li Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)