[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
flinkbot edited a comment on pull request #17344: URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696 ## CI report: * 70104a7e1bb8b322a8044143de0eb700a761794c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472) * 4ad29008075dc93463c07ce098cd7b1086104d7b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24493) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
flinkbot edited a comment on pull request #17344: URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696 ## CI report: * 70104a7e1bb8b322a8044143de0eb700a761794c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472) * 4ad29008075dc93463c07ce098cd7b1086104d7b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14544: [FLINK-20845] Drop Scala 2.11 support
flinkbot edited a comment on pull request #14544: URL: https://github.com/apache/flink/pull/14544#issuecomment-753633967 ## CI report: * d03f03be594cbe205f649142c24d64cb99734479 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24488) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-23647) UnalignedCheckpointStressITCase crashed on azure
[
https://issues.apache.org/jira/browse/FLINK-23647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-23647:
---
Labels: stale-assigned test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> UnalignedCheckpointStressITCase crashed on azure
>
>
> Key: FLINK-23647
> URL: https://issues.apache.org/jira/browse/FLINK-23647
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Affects Versions: 1.14.0
>Reporter: Roman Khachatryan
>Assignee: Arvid Heise
>Priority: Major
> Labels: stale-assigned, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21539=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=4855
> When testing DFS changelog implementation in FLINK-23279 and enabling it for
> all tests,
> UnalignedCheckpointStressITCase crashed with the following exception
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 18.433 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase
> [ERROR]
> runStressTest(org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase)
> Time elapsed: 17.663 s <<< ERROR!
> java.io.UncheckedIOException: java.nio.file.NoSuchFileException:
> /tmp/junit7860347244680665820/435237 d57439f2ceadfedba74dadd6fa/chk-16
>at
> java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:88)
>at java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:104)
>at java.util.Iterator.forEachRemaining(Iterator.java:115)
>at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
>at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
>at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
>at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
>at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:546)
>at
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.discoverRetainedCheckpoint(UnalignedCheckpointStressITCase.java:288)
>at
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.runAndTakeExternalCheckpoint(UnalignedCheckpointStressITCase.java:261)
>at
> org.apache.flink.test.checkpointing.UnalignedCheckpointStressITCase.runStressTest(UnalignedCheckpointStressITCase.java:157)
>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.lang.reflect.Method.invoke(Method.java:498)
>at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
>at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
>at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
>at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
>at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>at
[jira] [Updated] (FLINK-20661) Elasticsearch6DynamicSinkITCase.testWritingDocuments test failed with ConnectException
[
https://issues.apache.org/jira/browse/FLINK-20661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-20661:
---
Labels: auto-deprioritized-major test-stability (was: stale-major
test-stability)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Elasticsearch6DynamicSinkITCase.testWritingDocuments test failed with
> ConnectException
> --
>
> Key: FLINK-20661
> URL: https://issues.apache.org/jira/browse/FLINK-20661
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch
>Affects Versions: 1.11.0, 1.13.1
>Reporter: Huang Xingbo
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10989=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=eb5f4d19-2d2d-5856-a4ce-acf5f904a994]
> {code:java}
> 2020-12-17T22:52:41.2992508Z [ERROR] Tests run: 4, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 38.878 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSinkITCase
> 2020-12-17T22:52:41.2999076Z [ERROR]
> testWritingDocuments(org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSinkITCase)
> Time elapsed: 16.409 s <<< ERROR!
> 2020-12-17T22:52:41.3008441Z
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2020-12-17T22:52:41.3009290Z at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> 2020-12-17T22:52:41.3048924Z at
> org.apache.flink.client.program.PerJobMiniClusterFactory$PerJobMiniClusterJobClient.lambda$getJobExecutionResult$2(PerJobMiniClusterFactory.java:186)
> 2020-12-17T22:52:41.3058938Z at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> 2020-12-17T22:52:41.3067969Z at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> 2020-12-17T22:52:41.3080564Z at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2020-12-17T22:52:41.3098938Z at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> 2020-12-17T22:52:41.3128311Z at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:229)
> 2020-12-17T22:52:41.3141102Z at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> 2020-12-17T22:52:41.3168389Z at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> 2020-12-17T22:52:41.3178382Z at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2020-12-17T22:52:41.3179506Z at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> 2020-12-17T22:52:41.3180433Z at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:892)
> 2020-12-17T22:52:41.3181380Z at
> akka.dispatch.OnComplete.internal(Future.scala:264)
> 2020-12-17T22:52:41.3182138Z at
> akka.dispatch.OnComplete.internal(Future.scala:261)
> 2020-12-17T22:52:41.3182903Z at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
> 2020-12-17T22:52:41.3183893Z at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
> 2020-12-17T22:52:41.3184690Z at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> 2020-12-17T22:52:41.3185566Z at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:74)
> 2020-12-17T22:52:41.3186546Z at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> 2020-12-17T22:52:41.3187525Z at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> 2020-12-17T22:52:41.3188735Z at
> akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
> 2020-12-17T22:52:41.3189570Z at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:22)
> 2020-12-17T22:52:41.3190827Z at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
> 2020-12-17T22:52:41.3191576Z at
> scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:436)
> 2020-12-17T22:52:41.3192235Z at
> scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:435)
> 2020-12-17T22:52:41.3192897Z at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> 2020-12-17T22:52:41.3193788Z at
>
[jira] [Updated] (FLINK-23425) The impact of cpu cores on test results for StreamingJobGraphGeneratorTest#testSlotSharingResourceConfigurationWithDefaultSlotSharingGroup
[ https://issues.apache.org/jira/browse/FLINK-23425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-23425: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 60 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > The impact of cpu cores on test results for > StreamingJobGraphGeneratorTest#testSlotSharingResourceConfigurationWithDefaultSlotSharingGroup > > --- > > Key: FLINK-23425 > URL: https://issues.apache.org/jira/browse/FLINK-23425 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: tartarus >Priority: Major > Labels: pull-request-available, stale-major > > By default, LocalStreamEnvironment will use the number of cpu > cores(Runtime.getRuntime().availableProcessors()) as the default parallelism; > In our company, we will use a 1-core docker container to run the test. > Then > StreamingJobGraphGeneratorTest#testSlotSharingResourceConfigurationWithDefaultSlotSharingGroup > failed. > because the parallelism of map operator is 1, so the number of Vertex is 1, > not expected 2 anymore. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-23928) Test the buffer debloating
[
https://issues.apache.org/jira/browse/FLINK-23928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-23928:
---
Labels: release-testing stale-assigned (was: release-testing)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> Test the buffer debloating
> --
>
> Key: FLINK-23928
> URL: https://issues.apache.org/jira/browse/FLINK-23928
> Project: Flink
> Issue Type: Sub-task
>Reporter: Anton Kalashnikov
>Assignee: Chesnay Schepler
>Priority: Major
> Labels: release-testing, stale-assigned
> Fix For: 1.14.0
>
>
> # {color:#6a8759}Configure the buffer debloat by setting
> `taskmanager.network.memory.buffer-debloat.enabled` to true{color}
> # {color:#6a8759}Start the job with a long checkpoint time{color}
> # {color:#6a8759}Check the alignment time for each subtask it should be near
> taskmanager.network.memory.buffer-debloat.target{color}
> # {color:#6a8759}Check the checkpoint time it should be near taskCount *
> taskmanager.network.memory.buffer-debloat.target{color}
> # {color:#6a8759}Try to change
> taskmanager.network.memory.buffer-debloat.target. Checkponint time should be
> changed correspondetly.{color}
> # {color:#6a8759}Check metrics
> {color}{color:#6a8759}estimatedTimeToConsumerBuffersMs and
> {color}{color:#6a8759}debloatedBufferSize which also should depend on the
> current debloat configuration.
> {color}
> {color:#6a8759} {color}
> {color:#6a8759}Other configuration which can be changed for test.{color}
> * {color:#6a8759}taskmanager.memory.min-segment-size(if this would be equal
> to taskmanager.memory.segment-size then the behaviour should be same as with
> disabled buffer debloat)
> {color}
> * {color:#6a8759}taskmanager.network.memory.buffer-debloat.period{color}
> * {color:#6a8759}taskmanager.network.memory.buffer-debloat.samples{color}
> *
> {color:#6a8759}taskmanager.network.memory.buffer-debloat.threshold-percentages{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17354: [FLINK-24200][streaming] Calculating maximum alignment time rather than using the constant value
flinkbot edited a comment on pull request #17354: URL: https://github.com/apache/flink/pull/17354#issuecomment-926820469 ## CI report: * 69a3d2f47018a277458a8798e02caa7e3abb5779 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24487) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14544: [FLINK-20845] Drop Scala 2.11 support
flinkbot edited a comment on pull request #14544: URL: https://github.com/apache/flink/pull/14544#issuecomment-753633967 ## CI report: * b1fe24bab5f3a3588e594ed41932c41cc87bd069 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21445) * d03f03be594cbe205f649142c24d64cb99734479 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24488) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17353: [FLINK-24373][docs] Remove BETA Tag from FLIP-27 Source Docs
flinkbot edited a comment on pull request #17353: URL: https://github.com/apache/flink/pull/17353#issuecomment-926797798 ## CI report: * 77814421493a2f8ec6d06a55ec74823b574694b6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24486) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14544: [FLINK-20845] Drop Scala 2.11 support
flinkbot edited a comment on pull request #14544: URL: https://github.com/apache/flink/pull/14544#issuecomment-753633967 ## CI report: * b1fe24bab5f3a3588e594ed41932c41cc87bd069 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=21445) * d03f03be594cbe205f649142c24d64cb99734479 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
flinkbot edited a comment on pull request #17352: URL: https://github.com/apache/flink/pull/17352#issuecomment-926627700 ## CI report: * 471fd7fb6373ea0f2ebfe19d5ee1f7e097bc25b8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24483) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] NickBurkard commented on pull request #14544: [FLINK-20845] Drop Scala 2.11 support
NickBurkard commented on pull request #14544: URL: https://github.com/apache/flink/pull/14544#issuecomment-926845102 @MartijnVisser yep it should be unblocked now! I rebased with `master`, will let the CI suite run again to verify changes are good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17354: [FLINK-24200][streaming] Calculating maximum alignment time rather than using the constant value
flinkbot edited a comment on pull request #17354: URL: https://github.com/apache/flink/pull/17354#issuecomment-926820469 ## CI report: * 69a3d2f47018a277458a8798e02caa7e3abb5779 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24487) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17354: [FLINK-24200][streaming] Calculating maximum alignment time rather than using the constant value
flinkbot commented on pull request #17354: URL: https://github.com/apache/flink/pull/17354#issuecomment-926820469 ## CI report: * 69a3d2f47018a277458a8798e02caa7e3abb5779 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17353: [FLINK-24373][docs] Remove BETA Tag from FLIP-27 Source Docs
flinkbot edited a comment on pull request #17353: URL: https://github.com/apache/flink/pull/17353#issuecomment-926797798 ## CI report: * 77814421493a2f8ec6d06a55ec74823b574694b6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24486) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17353: [FLINK-24373][docs] Remove BETA Tag from FLIP-27 Source Docs
flinkbot commented on pull request #17353: URL: https://github.com/apache/flink/pull/17353#issuecomment-926797798 ## CI report: * 77814421493a2f8ec6d06a55ec74823b574694b6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17354: [FLINK-24200][streaming] Calculating maximum alignment time rather than using the constant value
flinkbot commented on pull request #17354: URL: https://github.com/apache/flink/pull/17354#issuecomment-926796300 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 69a3d2f47018a277458a8798e02caa7e3abb5779 (Fri Sep 24 17:20:27 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24200) CheckpointBarrierTrackerTest.testTwoLastBarriersOneByOne fails on azure
[
https://issues.apache.org/jira/browse/FLINK-24200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24200:
---
Labels: pull-request-available test-stability (was: test-stability)
> CheckpointBarrierTrackerTest.testTwoLastBarriersOneByOne fails on azure
> ---
>
> Key: FLINK-24200
> URL: https://issues.apache.org/jira/browse/FLINK-24200
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.5
>Reporter: Xintong Song
>Assignee: Anton Kalashnikov
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.12.6
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23719=logs=f0ac5c25-1168-55a5-07ff-0e88223afed9=0dbaca5d-7c38-52e6-f4fe-2fb69ccb3ada=8983
> {code}
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 0.893 s <<< FAILURE! - in
> org.apache.flink.streaming.runtime.io.CheckpointBarrierTrackerTest
> [ERROR]
> testTwoLastBarriersOneByOne(org.apache.flink.streaming.runtime.io.CheckpointBarrierTrackerTest)
> Time elapsed: 0.093 s <<< FAILURE!
> java.lang.AssertionError:
> Expected: a value less than or equal to <30L>
> but: <33L> was greater than <30L>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at
> org.apache.flink.streaming.runtime.io.CheckpointBarrierTrackerTest.testTwoLastBarriersOneByOne(CheckpointBarrierTrackerTest.java:616)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] akalash opened a new pull request #17354: [FLINK-24200][streaming] Calculating maximum alignment time rather than using the constant value
akalash opened a new pull request #17354: URL: https://github.com/apache/flink/pull/17354 …an using the constant value ## What is the purpose of the change *This PR fixes the test* ## Brief change log - *Calculating maximum alignment time rather than using the constant value* ## Verifying this change This is a test ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] StephanEwen merged pull request #17348: [BP-1.14][hotfix][docs] Remove flink-connector-base as required dependency fro…
StephanEwen merged pull request #17348: URL: https://github.com/apache/flink/pull/17348 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] StephanEwen merged pull request #17347: [hotfix][docs] Remove flink-connector-base as required dependency fro…
StephanEwen merged pull request #17347: URL: https://github.com/apache/flink/pull/17347 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17353: [FLINK-24373][docs] Remove BETA Tag from FLIP-27 Source Docs
flinkbot commented on pull request #17353: URL: https://github.com/apache/flink/pull/17353#issuecomment-926783072 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 77814421493a2f8ec6d06a55ec74823b574694b6 (Fri Sep 24 16:59:28 UTC 2021) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
flinkbot edited a comment on pull request #17352: URL: https://github.com/apache/flink/pull/17352#issuecomment-926627700 ## CI report: * 3af1ce2d14d61482fd59c55cd5cc9451cb1545ed Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24482) * 471fd7fb6373ea0f2ebfe19d5ee1f7e097bc25b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24483) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17351: [FLINK-24371][datastream] Call preCommit on SinkWriter although no committer is available
flinkbot edited a comment on pull request #17351: URL: https://github.com/apache/flink/pull/17351#issuecomment-926586520 ## CI report: * 1e785b4db987fa9006c04077a5f5eb2050a338ab Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24481) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24230) Buffer debloating microbenchmark for single gate
[ https://issues.apache.org/jira/browse/FLINK-24230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-24230: --- Labels: pull-request-available (was: ) > Buffer debloating microbenchmark for single gate > > > Key: FLINK-24230 > URL: https://issues.apache.org/jira/browse/FLINK-24230 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.14.0 >Reporter: Anton Kalashnikov >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Currently, there are no microbenchmarks that check buffer debloating > effectiveness. The idea is to create one which will measure the checkpoint > time. The benchmark should be similar to `UnalignedCheckpointTimeBenchmark` > but unlike the `UnalignedCheckpointTimeBenchmark` where we see the effect of > `Buffer debloat` only for extremely small values like 1ms for > BUFFER_DEBLOAT_TARGET. This benchmark should provide a more reliable way to > check the different implementations of `Buffer debloat` it can be reached by > increasing at least record size and checkpoint interval. The main target is > to have how long will it take to do the checkpoint during backpressure when > all buffers are full. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-benchmarks] dawidwys opened a new pull request #32: [FLINK-24230] Buffer debloating microbenchmark for single gate
dawidwys opened a new pull request #32: URL: https://github.com/apache/flink-benchmarks/pull/32 Added a test that first does an environment setup. 1. Submits a job 2. Waits for it being backpressured 3. Waits some extra time for buffer debloating to stabilize The above steps are not measured. After that we trigger a single checkpoint and measure the time for it to complete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #17353: [FLINK-24373][docs] Remove BETA Tag from FLIP-27 Source Docs
sjwiesman commented on pull request #17353: URL: https://github.com/apache/flink/pull/17353#issuecomment-926773281 cc @fapaul @wuchong or @leonardBang I feel pretty confident I deleted the relevant section from the translated page but would either of you mind double checking -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24373) Remove BETA Tag from FLIP-27 Source Docs
[ https://issues.apache.org/jira/browse/FLINK-24373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-24373: --- Labels: pull-request-available (was: ) > Remove BETA Tag from FLIP-27 Source Docs > > > Key: FLINK-24373 > URL: https://issues.apache.org/jira/browse/FLINK-24373 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman opened a new pull request #17353: [FLINK-24373][docs] Remove BETA Tag from FLIP-27 Source Docs
sjwiesman opened a new pull request #17353: URL: https://github.com/apache/flink/pull/17353 ## What is the purpose of the change Remove BETA Tag from FLIP-27 Source Docs. We will backport this to 1.14 release branch. ## Does this pull request potentially affect one of the following parts: Not Relevant ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-16504) Add a AWS DynamoDB sink
[ https://issues.apache.org/jira/browse/FLINK-16504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419874#comment-17419874 ] Yuri Gusev edited comment on FLINK-16504 at 9/24/21, 4:24 PM: -- Hi [~gzheng], We finished integration tests implementation, need to rebase on latest and we will submit PR Tuesday next week. For now we only implemented async batching sink (for streaming api), but other implementations will follow. As a workaround you can use our older implementations here: https://github.com/klarna-incubator/flink-connector-dynamodb was (Author: gusev): Hi [~gzheng], We finished integration tests implementation, need to rebase on latest and we will submit PR Tuesday next week. For now we only implemented async batching sink, but other implementations will follow. As a workaround you can use our older implementations here: https://github.com/klarna-incubator/flink-connector-dynamodb > Add a AWS DynamoDB sink > --- > > Key: FLINK-16504 > URL: https://issues.apache.org/jira/browse/FLINK-16504 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Reporter: Robert Metzger >Assignee: Yuri Gusev >Priority: Minor > Labels: pull-request-available > > I'm adding this ticket to track the amount of demand for this connector. > Please comment on this ticket, if you are looking for a DynamoDB connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16504) Add a AWS DynamoDB sink
[ https://issues.apache.org/jira/browse/FLINK-16504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419874#comment-17419874 ] Yuri Gusev edited comment on FLINK-16504 at 9/24/21, 4:23 PM: -- Hi [~gzheng], We finished integration tests implementation, need to rebase on latest and we will submit PR Tuesday next week. For now we only implemented async batching sink, but other implementations will follow. As a workaround you can use our older implementations here: https://github.com/klarna-incubator/flink-connector-dynamodb was (Author: gusev): Hi [~gzheng], We finished integration tests implementation, meed to rebase on latest and we will submit PR Tuesday next week. For now we only implemented async batching sink, but other implementations will follow. As a workaround you can use our older implementations here: https://github.com/klarna-incubator/flink-connector-dynamodb > Add a AWS DynamoDB sink > --- > > Key: FLINK-16504 > URL: https://issues.apache.org/jira/browse/FLINK-16504 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Reporter: Robert Metzger >Assignee: Yuri Gusev >Priority: Minor > Labels: pull-request-available > > I'm adding this ticket to track the amount of demand for this connector. > Please comment on this ticket, if you are looking for a DynamoDB connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16504) Add a AWS DynamoDB sink
[ https://issues.apache.org/jira/browse/FLINK-16504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419874#comment-17419874 ] Yuri Gusev commented on FLINK-16504: Hi [~gzheng], We finished integration tests implementation, meed to rebase on latest and we will submit PR Tuesday next week. For now we only implemented async batching sink, but other implementations will follow. As a workaround you can use our older implementations here: https://github.com/klarna-incubator/flink-connector-dynamodb > Add a AWS DynamoDB sink > --- > > Key: FLINK-16504 > URL: https://issues.apache.org/jira/browse/FLINK-16504 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Reporter: Robert Metzger >Assignee: Yuri Gusev >Priority: Minor > Labels: pull-request-available > > I'm adding this ticket to track the amount of demand for this connector. > Please comment on this ticket, if you are looking for a DynamoDB connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
flinkbot edited a comment on pull request #17352: URL: https://github.com/apache/flink/pull/17352#issuecomment-926627700 ## CI report: * 3af1ce2d14d61482fd59c55cd5cc9451cb1545ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24482) * 471fd7fb6373ea0f2ebfe19d5ee1f7e097bc25b8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24483) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24373) Remove BETA Tag from FLIP-27 Source Docs
Seth Wiesman created FLINK-24373: Summary: Remove BETA Tag from FLIP-27 Source Docs Key: FLINK-24373 URL: https://issues.apache.org/jira/browse/FLINK-24373 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Seth Wiesman Assignee: Seth Wiesman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
flinkbot edited a comment on pull request #17352: URL: https://github.com/apache/flink/pull/17352#issuecomment-926627700 ## CI report: * 3af1ce2d14d61482fd59c55cd5cc9451cb1545ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24482) * 471fd7fb6373ea0f2ebfe19d5ee1f7e097bc25b8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on a change in pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
RocMarshal commented on a change in pull request #17352:
URL: https://github.com/apache/flink/pull/17352#discussion_r715700515
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -1176,6 +1178,31 @@ public TableResult executeInternal(Operation operation) {
.getTableIdentifier()
.asSerializableString()));
}
+} else if (operation instanceof ShowCreateViewOperation) {
+ShowCreateViewOperation showCreateViewOperation =
(ShowCreateViewOperation) operation;
+Optional result =
+
catalogManager.getTable(showCreateViewOperation.getViewIdentifier());
+if (result.isPresent()) {
Review comment:
OK. I'll update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on a change in pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
RocMarshal commented on a change in pull request #17352:
URL: https://github.com/apache/flink/pull/17352#discussion_r715696263
##
File path:
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/catalog/CatalogViewITCase.scala
##
@@ -18,24 +18,24 @@
package org.apache.flink.table.planner.catalog
-import org.apache.flink.table.api.config.{ExecutionConfigOptions,
TableConfigOptions}
+import com.google.common.collect.Lists
+import org.apache.flink.table.api.config.ExecutionConfigOptions
import org.apache.flink.table.api.internal.TableEnvironmentImpl
import org.apache.flink.table.api.{EnvironmentSettings, TableEnvironment}
import
org.apache.flink.table.planner.factories.utils.TestCollectionTableFactory
import org.apache.flink.test.util.AbstractTestBase
import org.apache.flink.types.Row
-
+import org.apache.flink.util.CollectionUtil
import org.junit.Assert.assertEquals
import org.junit.rules.ExpectedException
import org.junit.runner.RunWith
import org.junit.runners.Parameterized
import org.junit.{Before, Rule, Test}
import java.util
-
import scala.collection.JavaConversions._
-/** Test cases for view related DDLs. */
+/** Test cases for view related DDLs. TODO roc todo */
Review comment:
Good catch. Thank you. I'll remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on a change in pull request #17349: [FLINK-24370][documentation] Added documentation for Asynchronous Sin…
sjwiesman commented on a change in pull request #17349:
URL: https://github.com/apache/flink/pull/17349#discussion_r715670611
##
File path: docs/content/docs/connectors/datastream/async_base.md
##
@@ -0,0 +1,135 @@
+---
+title: Async Base
+weight: 5
+type: docs
+aliases:
+ - /dev/connectors/async_base.html
+ - /apis/streaming/connectors/async_base.html
Review comment:
This is a new page right? Aliases are for when we move existing pages to
not break links. Please remove.
##
File path: docs/content/docs/connectors/datastream/async_base.md
##
@@ -0,0 +1,135 @@
+---
+title: Async Base
+weight: 5
+type: docs
+aliases:
+ - /dev/connectors/async_base.html
+ - /apis/streaming/connectors/async_base.html
+---
+
+
+# Asynchronous Sink Base
+
+The basic functionalities of [sinks]({{< ref
"docs/connectors/datastream/overview" >}}) are quite similar. They batch events
according to user defined buffering hints, sign requests and send them to the
respective endpoint, retry unsuccessful or throttled requests, and participate
in checkpointing.
+
+Such functionalities have been implemented in an abstract sink. Adding support
for a new destination now just means creating a lightweight shim that only
implements the specific interfaces of the destination using a client that
supports async requests. Having a common abstraction will reduce the effort
required to maintain all these individual sinks. It will also make it much
easier and faster to create integrations with additional destinations.
Moreover, improvements or bug fixes to the core of the sink will benefit all
implementations that are based on it.
+
+{{< hint warning >}}
+**Attention** The sink is designed to participate in checkpointing to provide
at-least once semantics, but it is limited to destinations that provide a
client that supports async requests.
+{{< /hint >}}
+
+The design of the sink focuses on extensibility and a broad support of
destinations. The core of the sink is kept generic and free of any connector
specific dependencies.
+
+You will find information on how to implement a sink based on this design in
the rest of this document.
+
+### Dependency
+
+To use this connector, add the following dependency to your project:
+
+{{< artifact flink-connector-base withScalaVersion >}}
Review comment:
```suggestion
{{< artifact flink-connector-base >}}
```
The connector base doesn't have a scala suffix in its pom.
##
File path: docs/content/docs/connectors/datastream/async_base.md
##
@@ -0,0 +1,135 @@
+---
+title: Async Base
+weight: 5
+type: docs
+aliases:
+ - /dev/connectors/async_base.html
+ - /apis/streaming/connectors/async_base.html
+---
+
+
+# Asynchronous Sink Base
+
Review comment:
I think we need to be very clear that this is not a page most users need
to read.
```suggestion
{{< hint info >}}
This page describes Flink’s Asynchronous Sink Base which simplifies the
development of sinks with `at-least-once` delivery semantics.
If you are looking for pre-defined connectors, please check the [Connector
Docs]({{< ref "docs/connectors/datastream/overview" >}}).
{{< /hint >}}
```
##
File path: docs/content/docs/connectors/datastream/async_base.md
##
@@ -0,0 +1,135 @@
+---
+title: Async Base
+weight: 5
+type: docs
+aliases:
+ - /dev/connectors/async_base.html
+ - /apis/streaming/connectors/async_base.html
+---
+
+
+# Asynchronous Sink Base
+
+The basic functionalities of [sinks]({{< ref
"docs/connectors/datastream/overview" >}}) are quite similar. They batch events
according to user defined buffering hints, sign requests and send them to the
respective endpoint, retry unsuccessful or throttled requests, and participate
in checkpointing.
+
+Such functionalities have been implemented in an abstract sink. Adding support
for a new destination now just means creating a lightweight shim that only
implements the specific interfaces of the destination using a client that
supports async requests. Having a common abstraction will reduce the effort
required to maintain all these individual sinks. It will also make it much
easier and faster to create integrations with additional destinations.
Moreover, improvements or bug fixes to the core of the sink will benefit all
implementations that are based on it.
+
+{{< hint warning >}}
+**Attention** The sink is designed to participate in checkpointing to provide
at-least once semantics, but it is limited to destinations that provide a
client that supports async requests.
+{{< /hint >}}
+
+The design of the sink focuses on extensibility and a broad support of
destinations. The core of the sink is kept generic and free of any connector
specific dependencies.
+
+You will find information on how to implement a sink based on this design in
the rest of this document.
+
+### Dependency
+
+To use this connector, add the following dependency to your project:
+
[GitHub] [flink] flinkbot edited a comment on pull request #17245: [FLINK-23316][Table SQL/Ecosystem] Add tests for custom PartitionCommitPolicy
flinkbot edited a comment on pull request #17245: URL: https://github.com/apache/flink/pull/17245#issuecomment-917600794 ## CI report: * a0309bd0dc7b478d7f761ee5cbbc4d214e576c2a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24479) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
RocMarshal edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-925446816 Hi, @MartijnVisser @Airblader @wuchong @twalthr , I made some changes based on your suggestions. Could you help me to review it ? Thank you so much for your attention. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
flinkbot edited a comment on pull request #17352: URL: https://github.com/apache/flink/pull/17352#issuecomment-926627700 ## CI report: * 3af1ce2d14d61482fd59c55cd5cc9451cb1545ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24482) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17350: [FLINK-24359][table-runtime] Use ResolvedSchema in AbstractFileSystemTable
Airblader commented on a change in pull request #17350:
URL: https://github.com/apache/flink/pull/17350#discussion_r715632847
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/filesystem/AbstractFileSystemTable.java
##
@@ -49,7 +54,11 @@
this.tableIdentifier = context.getObjectIdentifier();
this.tableOptions = new Configuration();
context.getCatalogTable().getOptions().forEach(tableOptions::setString);
-this.schema =
TableSchemaUtils.getPhysicalSchema(context.getCatalogTable().getSchema());
+this.schema = context.getCatalogTable().getResolvedSchema();
+this.physicalColumns =
Review comment:
This seems redundant, since `schema` contains this entirely.
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/filesystem/DeserializationSchemaAdapter.java
##
@@ -73,31 +73,32 @@
public DeserializationSchemaAdapter(
DeserializationSchema deserializationSchema,
-TableSchema schema,
+List physicalColumns,
Review comment:
We should just pass the (physical) `DataType` here and keep that. It
contains both names and types.
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/filesystem/AbstractFileSystemTable.java
##
@@ -59,15 +68,19 @@ ReadableConfig formatOptions(String identifier) {
return new DelegatingConfiguration(tableOptions, identifier + ".");
}
-DataType getFormatDataType() {
-TableSchema.Builder builder = TableSchema.builder();
-schema.getTableColumns()
-.forEach(
-column -> {
-if (!partitionKeys.contains(column.getName())) {
-builder.add(column);
-}
-});
-return builder.build().toRowDataType();
+DataType getRowDataTypeWithoutPartitionColumns() {
Review comment:
This can be entirely derived from `getRowDataType` by using
`getRowDataType().getLogicalType()` which is always `RowType` and has all the
information.
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/filesystem/FileSystemTableSource.java
##
@@ -164,7 +167,7 @@ private SourceProvider
createSourceProvider(BulkFormat
@Override
public TableSchema getSchema() {
-return schema;
+return TableSchema.fromResolvedSchema(schema);
Review comment:
So migrating this isn't needed to fix the issue you had encountered?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-24372) Deprecate Elasticsearch Sinkfunctions
Fabian Paul created FLINK-24372: --- Summary: Deprecate Elasticsearch Sinkfunctions Key: FLINK-24372 URL: https://issues.apache.org/jira/browse/FLINK-24372 Project: Flink Issue Type: Sub-task Components: Connectors / ElasticSearch Affects Versions: 1.15.0 Reporter: Fabian Paul Once all other tickets of FLINK-24323 are resolved we can mark the Elasticsearch sinks implementing sinkfunction as deprecated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Airblader commented on a change in pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
Airblader commented on a change in pull request #17352:
URL: https://github.com/apache/flink/pull/17352#discussion_r715618204
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -1441,6 +1468,59 @@ private String buildShowCreateTableRow(
return sb.toString();
}
+/** Show create view statement only for views. */
+private String buildShowCreateViewRow(
+ResolvedCatalogBaseTable view,
+ObjectIdentifier viewIdentifier,
+boolean isTemporary) {
+final String printIndent = " ";
+CatalogBaseTable.TableKind kind = view.getTableKind();
+if (kind != CatalogBaseTable.TableKind.VIEW) {
+throw new TableException(
+String.format(
+"SHOW CREATE VIEW does not support showing CREATE
TABLE statement with identifier %s.",
+viewIdentifier.asSerializableString()));
+}
+StringBuilder stringBuilder = new StringBuilder();
+ResolvedSchema schema = view.getResolvedSchema();
+if (view.getOrigin() instanceof QueryOperationCatalogView) {
+stringBuilder.append(
+String.format(
+"CREATE %sVIEW %s (%s",
+isTemporary ? "TEMPORARY " : "",
+viewIdentifier.asSerializableString(),
+System.lineSeparator()));
+// append columns
+stringBuilder.append(
+schema.getColumns().stream()
+.map(
+column ->
+String.format(
+"%s%s", printIndent,
getColumnString(column)))
+.collect(Collectors.joining("," +
System.lineSeparator(;
+stringBuilder.append("\n)");
+} else {
+stringBuilder.append(
+String.format(
+"CREATE %sVIEW %s%s as%s%s",
+isTemporary ? "TEMPORARY " : "",
+viewIdentifier.asSerializableString(),
+String.format(
+"(%s)",
+schema.getColumns().stream()
+.map(Column::getName)
Review comment:
```suggestion
.map(Column::getName)
.map(EncodingUtils::escapeIdentifier)
```
##
File path:
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/catalog/CatalogViewITCase.scala
##
@@ -18,24 +18,24 @@
package org.apache.flink.table.planner.catalog
-import org.apache.flink.table.api.config.{ExecutionConfigOptions,
TableConfigOptions}
+import com.google.common.collect.Lists
+import org.apache.flink.table.api.config.ExecutionConfigOptions
import org.apache.flink.table.api.internal.TableEnvironmentImpl
import org.apache.flink.table.api.{EnvironmentSettings, TableEnvironment}
import
org.apache.flink.table.planner.factories.utils.TestCollectionTableFactory
import org.apache.flink.test.util.AbstractTestBase
import org.apache.flink.types.Row
-
+import org.apache.flink.util.CollectionUtil
import org.junit.Assert.assertEquals
import org.junit.rules.ExpectedException
import org.junit.runner.RunWith
import org.junit.runners.Parameterized
import org.junit.{Before, Rule, Test}
import java.util
-
import scala.collection.JavaConversions._
-/** Test cases for view related DDLs. */
+/** Test cases for view related DDLs. TODO roc todo */
Review comment:
TODO?
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -1441,6 +1468,59 @@ private String buildShowCreateTableRow(
return sb.toString();
}
+/** Show create view statement only for views. */
+private String buildShowCreateViewRow(
+ResolvedCatalogBaseTable view,
+ObjectIdentifier viewIdentifier,
+boolean isTemporary) {
+final String printIndent = " ";
+CatalogBaseTable.TableKind kind = view.getTableKind();
+if (kind != CatalogBaseTable.TableKind.VIEW) {
+throw new TableException(
+String.format(
+"SHOW CREATE VIEW does not support showing CREATE
TABLE statement with identifier %s.",
Review comment:
I think we could make this more helpful for the user:
```suggestion
"SHOW CREATE VIEW is only supported for views,
but '%s' is a table. Please use SHOW CREATE TABLE instead.",
```
##
File path:
[GitHub] [flink] flinkbot commented on pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
flinkbot commented on pull request #17352: URL: https://github.com/apache/flink/pull/17352#issuecomment-926627700 ## CI report: * 3af1ce2d14d61482fd59c55cd5cc9451cb1545ed UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17350: [FLINK-24359][table-runtime] Use ResolvedSchema in AbstractFileSystemTable
flinkbot edited a comment on pull request #17350: URL: https://github.com/apache/flink/pull/17350#issuecomment-926550692 ## CI report: * 98fac3d3a2b3e3c79cbea8625625faee50b16de8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24480) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on pull request #4: URL: https://github.com/apache/flink-ml/pull/4#issuecomment-926624833 @becketqin After looking into the package path issue, I think the CI is actually working correctly. Here is what I find: 1) I have tried `mvn package` with that package path locally and it works. So the CI is working as expected. 2) Previously the file PipelineModel.java defines the `PipelineModel` class under the package path `org.apache.flink.ml.api.pipeline`. And all other classes (e.g.`Pipeline`) imports the `PipelineModel` class using the package path `org.apache.flink.ml.api.pipeline`. This could explain why `mvn package` works. 3) What puzzled us is the fact that the class `org.apache.flink.ml.api.pipeline.PipelineModel` is defined in the file with path `org/apache/flink/ml/api/core/PipelineModel.java`. However, after searching related information on Google, I find no document that says the package path of a class must be consistent with the path of the file that defines the class. So that setup is technically OK. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-24200) CheckpointBarrierTrackerTest.testTwoLastBarriersOneByOne fails on azure
[
https://issues.apache.org/jira/browse/FLINK-24200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anton Kalashnikov reassigned FLINK-24200:
-
Assignee: Anton Kalashnikov
> CheckpointBarrierTrackerTest.testTwoLastBarriersOneByOne fails on azure
> ---
>
> Key: FLINK-24200
> URL: https://issues.apache.org/jira/browse/FLINK-24200
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.5
>Reporter: Xintong Song
>Assignee: Anton Kalashnikov
>Priority: Major
> Labels: test-stability
> Fix For: 1.12.6
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=23719=logs=f0ac5c25-1168-55a5-07ff-0e88223afed9=0dbaca5d-7c38-52e6-f4fe-2fb69ccb3ada=8983
> {code}
> [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 0.893 s <<< FAILURE! - in
> org.apache.flink.streaming.runtime.io.CheckpointBarrierTrackerTest
> [ERROR]
> testTwoLastBarriersOneByOne(org.apache.flink.streaming.runtime.io.CheckpointBarrierTrackerTest)
> Time elapsed: 0.093 s <<< FAILURE!
> java.lang.AssertionError:
> Expected: a value less than or equal to <30L>
> but: <33L> was greater than <30L>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at
> org.apache.flink.streaming.runtime.io.CheckpointBarrierTrackerTest.testTwoLastBarriersOneByOne(CheckpointBarrierTrackerTest.java:616)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-ml] lindong28 commented on pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on pull request #4: URL: https://github.com/apache/flink-ml/pull/4#issuecomment-926613101 Thanks for the review @becketqin. I did `git clean -dfx && mvn package` locally on my local machine after fixing the package path. It works well without reporting any `flink-ml-examples-streaming` failure. Can you try again after doing `git clean -dfx`? Yes we can remove Scala dependency later. For now let's keep it because the Alink's infra used for KMeans (see https://github.com/zhipeng93/flink-ml/tree/alink) has Scala code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21582) COALESCE not works when cast a variable return null
[
https://issues.apache.org/jira/browse/FLINK-21582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419760#comment-17419760
]
Francesco Guardiani commented on FLINK-21582:
-
I tried the query in the issue description with the patch from
https://issues.apache.org/jira/browse/FLINK-23385 applied, and it still fails
because the cast function return value nullability is defined as follows:
{code:java}
.outputTypeStrategy(
nullableIfArgs(ConstantArgumentCount.to(0), TypeStrategies.argument(1)))
{code}
while it should probably be forceNullable for the way the cast function behaves
now. Still changing to forceNullable doesn't work, I'm investigating on it.
On a side note, shouldn't cast raise an exception more than returning NULL?
Perhaps should we consider changing this behaviour and introduce TRY_CAST?
> COALESCE not works when cast a variable return null
> ---
>
> Key: FLINK-21582
> URL: https://issues.apache.org/jira/browse/FLINK-21582
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.11.1
> Environment: Flink1.11.1
>
>Reporter: Jerry
>Priority: Minor
> Labels: auto-unassigned
> Fix For: 1.14.0
>
> Attachments: image-2021-03-03-19-48-21-306.png
>
>
> select COALESCE(cast('aa' as int), 0);return NULL
> select COALESCE(NULL, 0); return 0
> The first case works failed, but the second case works successful
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17351: [FLINK-24371][datastream] Call preCommit on SinkWriter although no committer is available
flinkbot edited a comment on pull request #17351: URL: https://github.com/apache/flink/pull/17351#issuecomment-926586520 ## CI report: * 1e785b4db987fa9006c04077a5f5eb2050a338ab Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24481) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17341: [hotfix][test] Refactor BuiltInFunctions IT Tests
flinkbot edited a comment on pull request #17341: URL: https://github.com/apache/flink/pull/17341#issuecomment-925901262 ## CI report: * f2c827b884fa646bae5f08c3d26a613a58a4ba3b UNKNOWN * 8e3d7edefa906fcb3950227aea7bfd0b8902aca7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24475) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r715594298
##
File path:
flink-ml-api/src/main/java/org/apache/flink/ml/api/core/PipelineModel.java
##
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.api.pipeline;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.ml.api.core.AlgoOperator;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.table.api.Table;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * A PipelineModel acts as a Model. It consists of an ordered list of stages,
each of which could be
+ * a Model, Transformer or AlgoOperator.
+ */
+@PublicEvolving
+public final class PipelineModel implements Model {
+private static final long serialVersionUID = 6184950154217411318L;
+private final List> stages;
+private final Params params = new Params();
+
+public PipelineModel(List> stages) {
+this.stages = stages;
+}
+
+/**
+ * Applies all stages in this PipelineModel on the input tables in order.
The output of one
+ * stage is used as the input of the next stage (if any). The output of
the last stage is
+ * returned as the result of this method.
+ *
+ * @param inputs a list of tables
+ * @return a list of tables
+ */
+@Override
+public Table[] transform(Table... inputs) {
+for (Stage stage : stages) {
+inputs = ((AlgoOperator) stage).transform(inputs);
+}
+return inputs;
+}
+
+@Override
+public void save(String path) throws IOException {
+throw new UnsupportedOperationException();
+}
+
+public static PipelineModel load(String path) throws IOException {
+throw new UnsupportedOperationException();
+}
+
+@Override
+public Params getParams() {
+return params;
+}
+
+/**
+ * Returns a list of all stages in this PipelineModel in order. The list
is immutable.
+ *
+ * @return an immutable list of transformers.
+ */
+@VisibleForTesting
+public List> getStages() {
Review comment:
Yes. It is changed to package private now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16606: [FLINK-21357][runtime/statebackend]Periodic materialization for generalized incremental checkpoints
flinkbot edited a comment on pull request #16606: URL: https://github.com/apache/flink/pull/16606#issuecomment-887431748 ## CI report: * 264be5cc6a0485171413099e8b64b9e917d06e85 UNKNOWN * 1b7da8565a2ab9560f1aad65007930c91945087f UNKNOWN * f77f6bd12ea5a6b1cf8f698c8b36bfab394d627b UNKNOWN * ddafd4a0f21b1805e614d471b1c9a8e01a39ba88 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24478) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on a change in pull request #4: URL: https://github.com/apache/flink-ml/pull/4#discussion_r715594171 ## File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/PipelineModel.java ## @@ -0,0 +1,86 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.api.pipeline; Review comment: The package path is fixed now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r715593065
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,101 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
each of which could be
+ * an Estimator, Model, Transformer or AlgoOperator.
*/
@PublicEvolving
-public final class Pipeline
-implements Estimator, Transformer,
Model {
-private static final long serialVersionUID = 1L;
-private final List stages = new ArrayList<>();
+public final class Pipeline implements Estimator {
+private static final long serialVersionUID = 6384850154817512318L;
+private final List> stages;
private final Params params = new Params();
-private int lastEstimatorIndex = -1;
-
-public Pipeline() {}
-
-public Pipeline(String pipelineJson) {
-this.loadJson(pipelineJson);
-}
-
-public Pipeline(List stages) {
-for (PipelineStage s : stages) {
-appendStage(s);
-}
-}
-
-// is the stage a simple Estimator or pipeline with Estimator
-private static boolean isStageNeedFit(PipelineStage stage) {
-return (stage instanceof Pipeline && ((Pipeline) stage).needFit())
-|| (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+public Pipeline(List> stages) {
+this.stages = stages;
}
/**
- * Appends a PipelineStage to the tail of this pipeline. Pipeline is
editable only via this
- * method. The PipelineStage must be Estimator, Transformer, Model or
Pipeline.
+ * Trains the pipeline to fit on the given tables.
*
- * @param stage the stage to be appended
- */
-public Pipeline appendStage(PipelineStage stage) {
-if (isStageNeedFit(stage)) {
-lastEstimatorIndex = stages.size();
-} else if (!(stage instanceof Transformer)) {
-throw new RuntimeException(
-"All PipelineStages should be Estimator or Transformer,
got:"
-+ stage.getClass().getSimpleName());
-}
-stages.add(stage);
-return this;
-}
-
-/**
- * Returns a list of all stages in this pipeline in order, the list is
immutable.
- *
- * @return an immutable list of all stages in this pipeline in order.
- */
-public List getStages() {
-return Collections.unmodifiableList(stages);
-}
-
-/**
- * Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
- * true, that means this pipeline contains an {@link Estimator} and thus
users must invoke
- * {@link #fit(TableEnvironment, Table)} before they can use this pipeline
as a {@link
- * Transformer}. Otherwise, the pipeline can be used as a {@link
Transformer} directly.
- *
- * @return {@code true} if this pipeline has an Estimator, {@code false}
otherwise
- */
-public boolean needFit() {
-return this.getIndexOfLastEstimator() >= 0;
-}
-
-public Params getParams() {
-return params;
-}
-
-// find the last Estimator or Pipeline that needs fit in
[GitHub] [flink-ml] lindong28 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r715591990
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,101 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
each of which could be
+ * an Estimator, Model, Transformer or AlgoOperator.
*/
@PublicEvolving
-public final class Pipeline
-implements Estimator, Transformer,
Model {
-private static final long serialVersionUID = 1L;
-private final List stages = new ArrayList<>();
+public final class Pipeline implements Estimator {
+private static final long serialVersionUID = 6384850154817512318L;
+private final List> stages;
private final Params params = new Params();
-private int lastEstimatorIndex = -1;
-
-public Pipeline() {}
-
-public Pipeline(String pipelineJson) {
-this.loadJson(pipelineJson);
-}
-
-public Pipeline(List stages) {
-for (PipelineStage s : stages) {
-appendStage(s);
-}
-}
-
-// is the stage a simple Estimator or pipeline with Estimator
-private static boolean isStageNeedFit(PipelineStage stage) {
-return (stage instanceof Pipeline && ((Pipeline) stage).needFit())
-|| (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+public Pipeline(List> stages) {
+this.stages = stages;
}
/**
- * Appends a PipelineStage to the tail of this pipeline. Pipeline is
editable only via this
- * method. The PipelineStage must be Estimator, Transformer, Model or
Pipeline.
+ * Trains the pipeline to fit on the given tables.
*
- * @param stage the stage to be appended
- */
-public Pipeline appendStage(PipelineStage stage) {
-if (isStageNeedFit(stage)) {
-lastEstimatorIndex = stages.size();
-} else if (!(stage instanceof Transformer)) {
-throw new RuntimeException(
-"All PipelineStages should be Estimator or Transformer,
got:"
-+ stage.getClass().getSimpleName());
-}
-stages.add(stage);
-return this;
-}
-
-/**
- * Returns a list of all stages in this pipeline in order, the list is
immutable.
- *
- * @return an immutable list of all stages in this pipeline in order.
- */
-public List getStages() {
-return Collections.unmodifiableList(stages);
-}
-
-/**
- * Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
- * true, that means this pipeline contains an {@link Estimator} and thus
users must invoke
- * {@link #fit(TableEnvironment, Table)} before they can use this pipeline
as a {@link
- * Transformer}. Otherwise, the pipeline can be used as a {@link
Transformer} directly.
- *
- * @return {@code true} if this pipeline has an Estimator, {@code false}
otherwise
- */
-public boolean needFit() {
-return this.getIndexOfLastEstimator() >= 0;
-}
-
-public Params getParams() {
-return params;
-}
-
-// find the last Estimator or Pipeline that needs fit in
[GitHub] [flink] flinkbot commented on pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
flinkbot commented on pull request #17352: URL: https://github.com/apache/flink/pull/17352#issuecomment-926602676 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 3af1ce2d14d61482fd59c55cd5cc9451cb1545ed (Fri Sep 24 12:55:28 UTC 2021) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal opened a new pull request #17352: [FLINK-10230][table] Support 'SHOW CREATE VIEW' syntax to print the query of a view
RocMarshal opened a new pull request #17352: URL: https://github.com/apache/flink/pull/17352 ## What is the purpose of the change *Support 'SHOW CREATE VIEW' syntax to print the query of a view* ## Brief change log *Support 'SHOW CREATE VIEW' syntax to print the query of a view* ## Verifying this change This change added tests and can be verified as follows: - org.apache.flink.table.planner.catalog.CatalogViewITCase ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (**yes** / no) - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17351: [FLINK-24371][datastream] Call preCommit on SinkWriter although no committer is available
flinkbot commented on pull request #17351: URL: https://github.com/apache/flink/pull/17351#issuecomment-926586520 ## CI report: * 1e785b4db987fa9006c04077a5f5eb2050a338ab UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17349: [FLINK-24370][documentation] Added documentation for Asynchronous Sin…
flinkbot edited a comment on pull request #17349: URL: https://github.com/apache/flink/pull/17349#issuecomment-926517209 ## CI report: * fb869b349b7a692bc81b4d6dc6598369750655de Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24477) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16606: [FLINK-21357][runtime/statebackend]Periodic materialization for generalized incremental checkpoints
flinkbot edited a comment on pull request #16606: URL: https://github.com/apache/flink/pull/16606#issuecomment-887431748 ## CI report: * 264be5cc6a0485171413099e8b64b9e917d06e85 UNKNOWN * 1b7da8565a2ab9560f1aad65007930c91945087f UNKNOWN * f77f6bd12ea5a6b1cf8f698c8b36bfab394d627b UNKNOWN * bfa91d2ed02578aa4efe31a8f44ee5ad1e38569f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24425) * ddafd4a0f21b1805e614d471b1c9a8e01a39ba88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24478) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24371) Support SinkWriter preCommit without the need of a committer
[ https://issues.apache.org/jira/browse/FLINK-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-24371: --- Labels: pull-request-available (was: ) > Support SinkWriter preCommit without the need of a committer > > > Key: FLINK-24371 > URL: https://issues.apache.org/jira/browse/FLINK-24371 > Project: Flink > Issue Type: Bug > Components: API / DataStream, Connectors / Common >Affects Versions: 1.14.0, 1.15.0 >Reporter: Fabian Paul >Assignee: Fabian Paul >Priority: Major > Labels: pull-request-available > > For some sinks i.e. elasticsearch we only want to listen for the snapshot > barrier once retrieved the sink flushes the buffered records. > These sinks do not snapshot any state thus do not implement the > `snapshotState()` method. > We already have a NoopCommitHandler which swallows the passed committables > but it should at least call `preCommit()` on the SinkWriter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #17351: [FLINK-24371][datastream] Call preCommit on SinkWriter although no committer is available
flinkbot commented on pull request #17351: URL: https://github.com/apache/flink/pull/17351#issuecomment-926571614 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 1e785b4db987fa9006c04077a5f5eb2050a338ab (Fri Sep 24 12:04:10 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17350: [FLINK-24359][table-runtime] Use ResolvedSchema in AbstractFileSystemTable
flinkbot edited a comment on pull request #17350: URL: https://github.com/apache/flink/pull/17350#issuecomment-926550692 ## CI report: * 98fac3d3a2b3e3c79cbea8625625faee50b16de8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24480) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17344: [FLINK-20895] [flink-table-planner] support local aggregate push down in table planner
flinkbot edited a comment on pull request #17344: URL: https://github.com/apache/flink/pull/17344#issuecomment-925952696 ## CI report: * 70104a7e1bb8b322a8044143de0eb700a761794c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24472) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17245: [FLINK-23316][Table SQL/Ecosystem] Add tests for custom PartitionCommitPolicy
flinkbot edited a comment on pull request #17245: URL: https://github.com/apache/flink/pull/17245#issuecomment-917600794 ## CI report: * bc2102f6f4ddb7084ea717e3745cd9a97dcf4469 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24342) * a0309bd0dc7b478d7f761ee5cbbc4d214e576c2a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24479) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16606: [FLINK-21357][runtime/statebackend]Periodic materialization for generalized incremental checkpoints
flinkbot edited a comment on pull request #16606: URL: https://github.com/apache/flink/pull/16606#issuecomment-887431748 ## CI report: * 264be5cc6a0485171413099e8b64b9e917d06e85 UNKNOWN * 1b7da8565a2ab9560f1aad65007930c91945087f UNKNOWN * f77f6bd12ea5a6b1cf8f698c8b36bfab394d627b UNKNOWN * bfa91d2ed02578aa4efe31a8f44ee5ad1e38569f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24425) * ddafd4a0f21b1805e614d471b1c9a8e01a39ba88 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fapaul opened a new pull request #17351: Flink 24371
fapaul opened a new pull request #17351: URL: https://github.com/apache/flink/pull/17351 ## What is the purpose of the change Before this change, if no committer was after the SinkWriter in the pipeline preCommit was never called. With this change, preCommit is now called as well but the return values are discarded as before. ## Brief change log - Call the passed lambda in NoopCommitHandler ## Verifying this change - Added a test to ensure that preCommit is called if the Sink is configured without a commiter ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22530) RuntimeException after subsequent windowed grouping in TableAPI
[
https://issues.apache.org/jira/browse/FLINK-22530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christopher Rost updated FLINK-22530:
-
Labels: (was: auto-deprioritized-major)
> RuntimeException after subsequent windowed grouping in TableAPI
> ---
>
> Key: FLINK-22530
> URL: https://issues.apache.org/jira/browse/FLINK-22530
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.0
>Reporter: Christopher Rost
>Priority: Minor
>
> After applying the following using the TableAPI v 1.12.0, an error is thrown:
> {code:java}
> java.lang.RuntimeException: Error while applying rule
> StreamExecGroupWindowAggregateRule(in:LOGICAL,out:STREAM_PHYSICAL), args
> [rel#505:FlinkLogicalWindowAggregate.LOGICAL.any.None:
> 0.[NONE].[NONE](input=RelSubset#504,group={1},window=TumblingGroupWindow('w2,
> w1_rowtime, 1),properties=EXPR$1)]{code}
> The code snippet to reproduce:
> {code:java}
> Table table2 = table1
> .window(Tumble.over(lit(10).seconds()).on($(EVENT_TIME)).as("w1"))
> .groupBy($(ID), $(LABEL), $("w1"))
> .select($(ID), $(LABEL), $("w1").rowtime().as("w1_rowtime"));
> // table2.execute().print(); --> work well
> Table table3 = table2
> .window(Tumble.over(lit(10).seconds()).on($("w1_rowtime")).as("w2"))
> .groupBy($(LABEL), $("w2"))
> .select(
> $(LABEL).as("super_label"),
> lit(1).count().as("super_count"),
> $("w2").rowtime().as("w2_rowtime")
> );
> // table3.execute().print(); //--> work well
>table3.select($("super_label"), $("w2_rowtime"))
> .execute().print(); // --> throws exception
> {code}
> It seems that the alias "w1_rowtime" is no longer available for further
> usages of table3, since the cause of the exception is:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: field [w1_rowtime] not found;
> input fields are: [vertex_id, vertex_label, EXPR$0
> {noformat}
> {{The complete trace:}}
> {code:java}
> java.lang.RuntimeException: Error while applying rule
> StreamExecGroupWindowAggregateRule(in:LOGICAL,out:STREAM_PHYSICAL), args
> [rel#197:FlinkLogicalWindowAggregate.LOGICAL.any.None:
> 0.[NONE].[NONE](input=RelSubset#196,group={1},window=TumblingGroupWindow('w2,
> w1_rowtime, 1),properties=EXPR$1)]at
> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:256)
> at
> org.apache.calcite.plan.volcano.IterativeRuleDriver.drive(IterativeRuleDriver.java:58)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:510)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:312)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:64)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:62)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram$$anonfun$optimize$1.apply(FlinkChainedProgram.scala:58)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:57)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeTree(StreamCommonSubGraphBasedOptimizer.scala:163)
> at
> org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.doOptimize(StreamCommonSubGraphBasedOptimizer.scala:79)
> at
> org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:77)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:286)
> at
> org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:165)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.translate(TableEnvironmentImpl.java:1267)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:703)
> at
>
[GitHub] [flink] flinkbot commented on pull request #17350: [FLINK-24359][table-runtime] Use ResolvedSchema in AbstractFileSystemTable
flinkbot commented on pull request #17350: URL: https://github.com/apache/flink/pull/17350#issuecomment-926550692 ## CI report: * 98fac3d3a2b3e3c79cbea8625625faee50b16de8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17245: [FLINK-23316][Table SQL/Ecosystem] Add tests for custom PartitionCommitPolicy
flinkbot edited a comment on pull request #17245: URL: https://github.com/apache/flink/pull/17245#issuecomment-917600794 ## CI report: * bc2102f6f4ddb7084ea717e3745cd9a97dcf4469 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24342) * a0309bd0dc7b478d7f761ee5cbbc4d214e576c2a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-24359) Migrate FileSystem connector to ResolvedSchema
[
https://issues.apache.org/jira/browse/FLINK-24359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser reassigned FLINK-24359:
--
Assignee: Francesco Guardiani
> Migrate FileSystem connector to ResolvedSchema
> --
>
> Key: FLINK-24359
> URL: https://issues.apache.org/jira/browse/FLINK-24359
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Ecosystem
> Environment: Flink 1.14-SNAPSHOT
>Reporter: Francesco Guardiani
>Assignee: Francesco Guardiani
>Priority: Major
> Labels: pull-request-available
>
> Filesystem connector uses the TableSchema deprecated APIs. This causes issues
> with Table APIs, because TableSchema#fromResolvedSchema(ResolvedSchema)
> requires the expressions to be serializable strings
> (ResolvedExpression#asSerializableString).
> For example:
> {code:java}
> TableDescriptor inputTable = TableDescriptor.forConnector("filesystem")
> .schema(
> Schema.newBuilder()
> .column("character", DataTypes.STRING())
> .column("latitude", DataTypes.STRING())
> .column("longitude", DataTypes.STRING())
> .column("time", DataTypes.TIMESTAMP(3))
> .watermark("time", $("time").minus(lit(2).seconds()))
> .build()
> )
> // Other options
> .build();
> {code}
> When used in a table pipeline, throws the following exception:
> {code:java}
> Caused by: org.apache.flink.table.api.TableException: Expression 'minus(time,
> 2000)' is not string serializable. Currently, only expressions that
> originated from a SQL expression have a well-defined string representation.
> at
> org.apache.flink.table.expressions.ResolvedExpression.asSerializableString(ResolvedExpression.java:51)
> at
> org.apache.flink.table.api.TableSchema.lambda$fromResolvedSchema$13(TableSchema.java:455)
> at
> java.base/java.util.Collections$SingletonList.forEach(Collections.java:4976)
> at
> org.apache.flink.table.api.TableSchema.fromResolvedSchema(TableSchema.java:451)
> at
> org.apache.flink.table.catalog.ResolvedCatalogBaseTable.getSchema(ResolvedCatalogBaseTable.java:54)
> at
> org.apache.flink.table.filesystem.AbstractFileSystemTable.(AbstractFileSystemTable.java:52)
> at
> org.apache.flink.table.filesystem.FileSystemTableSource.(FileSystemTableSource.java:91)
> at
> org.apache.flink.table.filesystem.FileSystemTableFactory.createDynamicTableSource(FileSystemTableFactory.java:74)
> at
> org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:145)
> {code}
> The same table definition using SQL works fine:
> {code:java}
> CREATE TABLE IF NOT EXISTS LocationEvents (
> `character` STRING,
> `latitude` STRING,
> `longitude` STRING,
> `time` TIMESTAMP(3),
> WATERMARK FOR `time` AS `time` - INTERVAL '5' MINUTES
> ) WITH (
> -- Load from filesystem
> 'connector' = 'filesystem',
> --- Other configs
> );
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17348: [BP-1.14][hotfix][docs] Remove flink-connector-base as required dependency fro…
flinkbot edited a comment on pull request #17348: URL: https://github.com/apache/flink/pull/17348#issuecomment-926447846 ## CI report: * d0d5f2e050ed54a1f8d9eaeb209ce4bb16a5ec34 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24474) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17349: [FLINK-24370][documentation] Added documentation for Asynchronous Sin…
flinkbot edited a comment on pull request #17349: URL: https://github.com/apache/flink/pull/17349#issuecomment-926517209 ## CI report: * fb869b349b7a692bc81b4d6dc6598369750655de Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24477) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17347: [hotfix][docs] Remove flink-connector-base as required dependency fro…
flinkbot edited a comment on pull request #17347: URL: https://github.com/apache/flink/pull/17347#issuecomment-926447737 ## CI report: * d7ae1c1b86aaa3e6a1f515671a3611d94fe83be8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24473) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17350: [FLINK-24359][table-runtime] Use ResolvedSchema in AbstractFileSystemTable
flinkbot commented on pull request #17350: URL: https://github.com/apache/flink/pull/17350#issuecomment-926536168 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 98fac3d3a2b3e3c79cbea8625625faee50b16de8 (Fri Sep 24 10:58:43 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-24359).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JackWangCS commented on a change in pull request #17245: [FLINK-23316][Table SQL/Ecosystem] Add tests for custom PartitionCommitPolicy
JackWangCS commented on a change in pull request #17245:
URL: https://github.com/apache/flink/pull/17245#discussion_r715516784
##
File path:
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveTableSinkITCase.java
##
@@ -441,6 +470,93 @@ private void checkSuccessFiles(String path) {
Assert.assertTrue(new File(new File(basePath, "e=11"),
"_MY_SUCCESS").exists());
}
+private void testStreamingWriteWithCustomPartitionCommitPolicy(
+String customPartitionCommitPolicyClassName) throws Exception {
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+env.setParallelism(1);
+env.enableCheckpointing(100);
+// avoid the job to restart infinitely
+env.setRestartStrategy(RestartStrategies.fixedDelayRestart(3, 1_000));
+
+StreamTableEnvironment tEnv =
HiveTestUtils.createTableEnvInStreamingMode(env);
+tEnv.registerCatalog(hiveCatalog.getName(), hiveCatalog);
+tEnv.useCatalog(hiveCatalog.getName());
+tEnv.getConfig().setSqlDialect(SqlDialect.HIVE);
+
+try {
+tEnv.executeSql("create database db1");
+tEnv.useDatabase("db1");
+
+// prepare source
+List data =
+Arrays.asList(
+Row.of(1, "a", "b", "2020-05-03", "7"),
+Row.of(2, "p", "q", "2020-05-03", "8"),
+Row.of(3, "x", "y", "2020-05-03", "9"),
+Row.of(4, "x", "y", "2020-05-03", "10"),
+Row.of(5, "x", "y", "2020-05-03", "11"));
+DataStream stream =
+env.addSource(
+new FiniteTestSource<>(data),
+new RowTypeInfo(
+Types.INT,
+Types.STRING,
+Types.STRING,
+Types.STRING,
+Types.STRING));
+tEnv.createTemporaryView("my_table", stream, $("a"), $("b"),
$("c"), $("d"), $("e"));
+
+// DDL
+tEnv.executeSql(
+"create external table sink_table (a int,b string,c string"
++ ") "
++ "partitioned by (d string,e string) "
++ " stored as textfile"
++ " TBLPROPERTIES ("
++ "'"
++ PARTITION_TIME_EXTRACTOR_TIMESTAMP_PATTERN.key()
Review comment:
I forgot to remove this properties, I will remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-24359) Migrate FileSystem connector to ResolvedSchema
[
https://issues.apache.org/jira/browse/FLINK-24359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-24359:
---
Labels: pull-request-available (was: )
> Migrate FileSystem connector to ResolvedSchema
> --
>
> Key: FLINK-24359
> URL: https://issues.apache.org/jira/browse/FLINK-24359
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Ecosystem
> Environment: Flink 1.14-SNAPSHOT
>Reporter: Francesco Guardiani
>Priority: Major
> Labels: pull-request-available
>
> Filesystem connector uses the TableSchema deprecated APIs. This causes issues
> with Table APIs, because TableSchema#fromResolvedSchema(ResolvedSchema)
> requires the expressions to be serializable strings
> (ResolvedExpression#asSerializableString).
> For example:
> {code:java}
> TableDescriptor inputTable = TableDescriptor.forConnector("filesystem")
> .schema(
> Schema.newBuilder()
> .column("character", DataTypes.STRING())
> .column("latitude", DataTypes.STRING())
> .column("longitude", DataTypes.STRING())
> .column("time", DataTypes.TIMESTAMP(3))
> .watermark("time", $("time").minus(lit(2).seconds()))
> .build()
> )
> // Other options
> .build();
> {code}
> When used in a table pipeline, throws the following exception:
> {code:java}
> Caused by: org.apache.flink.table.api.TableException: Expression 'minus(time,
> 2000)' is not string serializable. Currently, only expressions that
> originated from a SQL expression have a well-defined string representation.
> at
> org.apache.flink.table.expressions.ResolvedExpression.asSerializableString(ResolvedExpression.java:51)
> at
> org.apache.flink.table.api.TableSchema.lambda$fromResolvedSchema$13(TableSchema.java:455)
> at
> java.base/java.util.Collections$SingletonList.forEach(Collections.java:4976)
> at
> org.apache.flink.table.api.TableSchema.fromResolvedSchema(TableSchema.java:451)
> at
> org.apache.flink.table.catalog.ResolvedCatalogBaseTable.getSchema(ResolvedCatalogBaseTable.java:54)
> at
> org.apache.flink.table.filesystem.AbstractFileSystemTable.(AbstractFileSystemTable.java:52)
> at
> org.apache.flink.table.filesystem.FileSystemTableSource.(FileSystemTableSource.java:91)
> at
> org.apache.flink.table.filesystem.FileSystemTableFactory.createDynamicTableSource(FileSystemTableFactory.java:74)
> at
> org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:145)
> {code}
> The same table definition using SQL works fine:
> {code:java}
> CREATE TABLE IF NOT EXISTS LocationEvents (
> `character` STRING,
> `latitude` STRING,
> `longitude` STRING,
> `time` TIMESTAMP(3),
> WATERMARK FOR `time` AS `time` - INTERVAL '5' MINUTES
> ) WITH (
> -- Load from filesystem
> 'connector' = 'filesystem',
> --- Other configs
> );
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] slinkydeveloper opened a new pull request #17350: [FLINK-24359][table-runtime] Use ResolvedSchema in AbstractFileSystemTable
slinkydeveloper opened a new pull request #17350: URL: https://github.com/apache/flink/pull/17350 Signed-off-by: slinkydeveloper ## What is the purpose of the change Migrate `AbstractFileSystemTable` to use the new `ResolvedSchema` APIs ## Brief change log - Now `AbstractFileSystemTable` and inheritors use the new `ResolvedSchema` APIs. Some usage of the old `TableSchema` persists because it's required by the `FileSystemFormatFactory.ReaderContext` class, but it will be easy to remove once `FileSystemFormatFactory` will be pruned. ## Verifying this change This change is already covered by existing tests in `org.apache.flink.table.filesystem`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20895) Support LocalAggregatePushDown in Blink planner
[ https://issues.apache.org/jira/browse/FLINK-20895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419721#comment-17419721 ] Peng Yu commented on FLINK-20895: - Hi [~jark], the new pull request ([17344|https://github.com/apache/flink/pull/17344]) is ready for review. It is based on the old one ([14894|https://github.com/apache/flink/pull/14894]). And it keeps the commit log of [~shared_ptr]. > Support LocalAggregatePushDown in Blink planner > --- > > Key: FLINK-20895 > URL: https://issues.apache.org/jira/browse/FLINK-20895 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Sebastian Liu >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.14.0 > > > Will add related rule to support LocalAggregatePushDown in Blink planner -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-ml] becketqin commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
becketqin commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r715477933
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,101 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
each of which could be
+ * an Estimator, Model, Transformer or AlgoOperator.
*/
@PublicEvolving
-public final class Pipeline
-implements Estimator, Transformer,
Model {
-private static final long serialVersionUID = 1L;
-private final List stages = new ArrayList<>();
+public final class Pipeline implements Estimator {
+private static final long serialVersionUID = 6384850154817512318L;
+private final List> stages;
private final Params params = new Params();
-private int lastEstimatorIndex = -1;
-
-public Pipeline() {}
-
-public Pipeline(String pipelineJson) {
-this.loadJson(pipelineJson);
-}
-
-public Pipeline(List stages) {
-for (PipelineStage s : stages) {
-appendStage(s);
-}
-}
-
-// is the stage a simple Estimator or pipeline with Estimator
-private static boolean isStageNeedFit(PipelineStage stage) {
-return (stage instanceof Pipeline && ((Pipeline) stage).needFit())
-|| (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+public Pipeline(List> stages) {
+this.stages = stages;
}
/**
- * Appends a PipelineStage to the tail of this pipeline. Pipeline is
editable only via this
- * method. The PipelineStage must be Estimator, Transformer, Model or
Pipeline.
+ * Trains the pipeline to fit on the given tables.
*
- * @param stage the stage to be appended
- */
-public Pipeline appendStage(PipelineStage stage) {
-if (isStageNeedFit(stage)) {
-lastEstimatorIndex = stages.size();
-} else if (!(stage instanceof Transformer)) {
-throw new RuntimeException(
-"All PipelineStages should be Estimator or Transformer,
got:"
-+ stage.getClass().getSimpleName());
-}
-stages.add(stage);
-return this;
-}
-
-/**
- * Returns a list of all stages in this pipeline in order, the list is
immutable.
- *
- * @return an immutable list of all stages in this pipeline in order.
- */
-public List getStages() {
-return Collections.unmodifiableList(stages);
-}
-
-/**
- * Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
- * true, that means this pipeline contains an {@link Estimator} and thus
users must invoke
- * {@link #fit(TableEnvironment, Table)} before they can use this pipeline
as a {@link
- * Transformer}. Otherwise, the pipeline can be used as a {@link
Transformer} directly.
- *
- * @return {@code true} if this pipeline has an Estimator, {@code false}
otherwise
- */
-public boolean needFit() {
-return this.getIndexOfLastEstimator() >= 0;
-}
-
-public Params getParams() {
-return params;
-}
-
-// find the last Estimator or Pipeline that needs fit in
[jira] [Assigned] (FLINK-24371) Support SinkWriter preCommit without the need of a committer
[ https://issues.apache.org/jira/browse/FLINK-24371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fabian Paul reassigned FLINK-24371: --- Assignee: Fabian Paul > Support SinkWriter preCommit without the need of a committer > > > Key: FLINK-24371 > URL: https://issues.apache.org/jira/browse/FLINK-24371 > Project: Flink > Issue Type: Bug > Components: API / DataStream, Connectors / Common >Affects Versions: 1.14.0, 1.15.0 >Reporter: Fabian Paul >Assignee: Fabian Paul >Priority: Major > > For some sinks i.e. elasticsearch we only want to listen for the snapshot > barrier once retrieved the sink flushes the buffered records. > These sinks do not snapshot any state thus do not implement the > `snapshotState()` method. > We already have a NoopCommitHandler which swallows the passed committables > but it should at least call `preCommit()` on the SinkWriter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24371) Support SinkWriter preCommit without the need of a committer
Fabian Paul created FLINK-24371: --- Summary: Support SinkWriter preCommit without the need of a committer Key: FLINK-24371 URL: https://issues.apache.org/jira/browse/FLINK-24371 Project: Flink Issue Type: Bug Components: API / DataStream, Connectors / Common Affects Versions: 1.14.0, 1.15.0 Reporter: Fabian Paul For some sinks i.e. elasticsearch we only want to listen for the snapshot barrier once retrieved the sink flushes the buffered records. These sinks do not snapshot any state thus do not implement the `snapshotState()` method. We already have a NoopCommitHandler which swallows the passed committables but it should at least call `preCommit()` on the SinkWriter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-10911) Enable flink-scala-shell with Scala 2.12
[
https://issues.apache.org/jira/browse/FLINK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-10911.
Resolution: Won't Fix
> Enable flink-scala-shell with Scala 2.12
>
>
> Key: FLINK-10911
> URL: https://issues.apache.org/jira/browse/FLINK-10911
> Project: Flink
> Issue Type: Bug
> Components: Scala Shell
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
>
> Flink's {{flink-scala-shell}} module is not working with Scala 2.12.
> Therefore, it is currently excluded from the Scala 2.12 builds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #17349: [FLINK-24370][documentation] Added documentation for Asynchronous Sin…
flinkbot commented on pull request #17349: URL: https://github.com/apache/flink/pull/17349#issuecomment-926517209 ## CI report: * fb869b349b7a692bc81b4d6dc6598369750655de UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17341: [hotfix][test] Refactor BuiltInFunctions IT Tests
flinkbot edited a comment on pull request #17341: URL: https://github.com/apache/flink/pull/17341#issuecomment-925901262 ## CI report: * f2c827b884fa646bae5f08c3d26a613a58a4ba3b UNKNOWN * 1b6da15897e67037eb0b065776e8d67014011090 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24455) Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24470) * 8e3d7edefa906fcb3950227aea7bfd0b8902aca7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24475) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #14544: [FLINK-20845] Drop Scala 2.11 support
MartijnVisser commented on pull request #14544: URL: https://github.com/apache/flink/pull/14544#issuecomment-926509165 @NickBurkard Thanks to your work for dropping the Scala Shell via https://github.com/apache/flink/pull/17340, does this now also unblock you to complete this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17332: [FLINK-24349]Support customized Calalogs via JDBC
Airblader commented on a change in pull request #17332: URL: https://github.com/apache/flink/pull/17332#discussion_r715486971 ## File path: flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/JdbcDialect.java ## @@ -142,4 +143,16 @@ */ String getSelectFromStatement( String tableName, String[] selectFields, String[] conditionFields); + +/** Create catalog instance. */ +default AbstractJdbcCatalog createCatalog( +String catalogName, Review comment: Once a new public API is released, we cannot just change it anymore – because it is a breaking change for users, and we need to avoid breaking changes. This is different from internal implementations which we can change whenever we need to. In this case, the internal implementation doesn't make for a good public API, so we need to bring it into better shape. If we just open another issue, we cannot ensure that this issue would be resolved prior to the next release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18285) solution for 'For input string: “0x100” ' make bash command 'reset' fail to work
[
https://issues.apache.org/jira/browse/FLINK-18285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-18285.
--
Resolution: Won't Fix
Closed due to https://issues.apache.org/jira/browse/FLINK-24360
> solution for 'For input string: “0x100” ' make bash command 'reset' fail to
> work
>
>
> Key: FLINK-18285
> URL: https://issues.apache.org/jira/browse/FLINK-18285
> Project: Flink
> Issue Type: Improvement
> Components: Scala Shell
>Affects Versions: 1.10.1
>Reporter: appleyuchi
>Priority: Minor
> Labels: auto-deprioritized-major
>
> When I start a Scala shell with the following command:
> {{start-scala-shell.sh local}}
> {{}}
> {{[ERROR] Failed to construct terminal; falling back to unsupported
> java.lang.NumberFormatException: For input string: "0x100"}}
> Other questions and GitHub suggest that the solution is:
> {{export TERM=xterm-color}}
> However, this solution *disables the command “{{reset”}}*.
> After I set {{export TERM=xterm-color}} in ~/.bashrc or /etc/profile and
> source the file,
> then I type “{{reset”}}, *the command {{reset}} will fail to reset Terminal.*
> So, what's the perfect solution to solve this problem while keeping the
> command “{{reset”}} working?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-12596) Unify all kinds of cluster via ClusterClient in FlinkShell
[ https://issues.apache.org/jira/browse/FLINK-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-12596. -- Resolution: Won't Fix Closed due to https://issues.apache.org/jira/browse/FLINK-24360 > Unify all kinds of cluster via ClusterClient in FlinkShell > -- > > Key: FLINK-12596 > URL: https://issues.apache.org/jira/browse/FLINK-12596 > Project: Flink > Issue Type: Improvement > Components: Scala Shell >Affects Versions: 1.8.0 >Reporter: Jeff Zhang >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > > In FlinkShell, we treat different kinds of cluster mode, and use scala > pattern matching to handle that. > This is not necessary and no readable. We can unify them into ClusterClient > which support all the cluster modes > > [https://github.com/apache/flink/blob/master/flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala#L143] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-5744) Check remote connection in Flink-shell
[
https://issues.apache.org/jira/browse/FLINK-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-5744.
-
Resolution: Won't Fix
Closed due to https://issues.apache.org/jira/browse/FLINK-24360
> Check remote connection in Flink-shell
> --
>
> Key: FLINK-5744
> URL: https://issues.apache.org/jira/browse/FLINK-5744
> Project: Flink
> Issue Type: Improvement
> Components: Scala Shell
>Affects Versions: 1.3.0
>Reporter: Anton Solovev
>Priority: Not a Priority
>
> I wish to check connection to remote host before execution a program when
> starting flink-shell
> For example, right after {{bin/start-scala-shell.sh remote 35007}}
> it checks connection and will not start if it has errors
> Actualy I would like to change welcome pic to just "welcome to flink" like in
> ignite, spring or spark
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-10911) Enable flink-scala-shell with Scala 2.12
[
https://issues.apache.org/jira/browse/FLINK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17419698#comment-17419698
]
Martijn Visser commented on FLINK-10911:
Closed due to https://issues.apache.org/jira/browse/FLINK-24360
> Enable flink-scala-shell with Scala 2.12
>
>
> Key: FLINK-10911
> URL: https://issues.apache.org/jira/browse/FLINK-10911
> Project: Flink
> Issue Type: Bug
> Components: Scala Shell
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
>
> Flink's {{flink-scala-shell}} module is not working with Scala 2.12.
> Therefore, it is currently excluded from the Scala 2.12 builds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-2709) line editing in scala shell
[ https://issues.apache.org/jira/browse/FLINK-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-2709. - Resolution: Won't Fix Closed due to https://issues.apache.org/jira/browse/FLINK-24360 > line editing in scala shell > --- > > Key: FLINK-2709 > URL: https://issues.apache.org/jira/browse/FLINK-2709 > Project: Flink > Issue Type: New Feature > Components: Scala Shell >Reporter: Matthew Farrellee >Priority: Minor > Labels: auto-deprioritized-major > > it would be very helpful to be able to edit lines in the shell. for instance, > up/down arrow to navigate history and left/right to navigate a line. > bonus for history search and advanced single line editing (e.g. emacs > bindings) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-11204) flink start-scala-shell.sh do not work in security mode with kerberos authentication.
[
https://issues.apache.org/jira/browse/FLINK-11204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-11204.
--
Resolution: Won't Fix
Closed due to https://issues.apache.org/jira/browse/FLINK-24360
> flink start-scala-shell.sh do not work in security mode with kerberos
> authentication.
> -
>
> Key: FLINK-11204
> URL: https://issues.apache.org/jira/browse/FLINK-11204
> Project: Flink
> Issue Type: Bug
> Components: Scala Shell
>Affects Versions: 1.3.3, 1.4.2, 1.5.5, 1.6.2, 1.7.0
>Reporter: kelun wang
>Priority: Not a Priority
>
> Hello,
> When using flink start-scala-shell.sh in a cluster with kerberos credential,
> the script does not supports Kerberos authentication, errors like below will
> occur:
> 1) Fail to deploy Yarn cluster.
>
> {code:java}
> start-scala-shell.sh yarn -n 3
> Exception in thread "main" java.lang.RuntimeException: Error deploying the
> YARN cluster
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createCluster(FlinkYarnSessionCli.java:594)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.createCluster(FlinkYarnSessionCli.java:81)
> at
> org.apache.flink.api.scala.FlinkShell$.deployNewYarnCluster(FlinkShell.scala:256)
> at
> org.apache.flink.api.scala.FlinkShell$.fetchConnectionInfo(FlinkShell.scala:165)
> at org.apache.flink.api.scala.FlinkShell$.liftedTree1$1(FlinkShell.scala:189)
> at org.apache.flink.api.scala.FlinkShell$.startShell(FlinkShell.scala:188)
> at org.apache.flink.api.scala.FlinkShell$.main(FlinkShell.scala:137)
> at org.apache.flink.api.scala.FlinkShell.main(FlinkShell.scala)
> Caused by: java.lang.IllegalArgumentException: Can't get Kerberos realm
> at
> org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
> at
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:318)
> at
> org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:303)
> at
> org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:391)
> at
> org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:385)
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:384)
> at
> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:351)
> ... 8 more
> {code}
>
> 2)Fail to fetch deployed a Flink cluster, when using the following command :
> bin/start-scala-shell.sh yarn
>
> {code:java}
> def fetchDeployedYarnClusterInfo(
> configuration: Configuration,
> configurationDirectory: String) = {
> val args = ArrayBuffer[String](
> "-m", "yarn-cluster"
> )
> {code}
> when fething deployed yarn cluster, with param "-m yarn-cluster" it will
> create new one, but has no "-n", still fail.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #17349: [FLINK-24370][documentation] Added documentation for Asynchronous Sin…
flinkbot commented on pull request #17349: URL: https://github.com/apache/flink/pull/17349#issuecomment-926501118 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit fb869b349b7a692bc81b4d6dc6598369750655de (Fri Sep 24 09:56:36 UTC 2021) **Warnings:** * Documentation files were touched, but no `docs/content.zh/` files: Update Chinese documentation or file Jira ticket. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17341: [hotfix][test] Refactor BuiltInFunctions IT Tests
flinkbot edited a comment on pull request #17341: URL: https://github.com/apache/flink/pull/17341#issuecomment-925901262 ## CI report: * f2c827b884fa646bae5f08c3d26a613a58a4ba3b UNKNOWN * 1b6da15897e67037eb0b065776e8d67014011090 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24455) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24470) * 8e3d7edefa906fcb3950227aea7bfd0b8902aca7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24475) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] cuibo01 commented on a change in pull request #17332: [FLINK-24349]Support customized Calalogs via JDBC
cuibo01 commented on a change in pull request #17332: URL: https://github.com/apache/flink/pull/17332#discussion_r715475502 ## File path: flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/JdbcDialect.java ## @@ -142,4 +143,16 @@ */ String getSelectFromStatement( String tableName, String[] selectFields, String[] conditionFields); + +/** Create catalog instance. */ +default AbstractJdbcCatalog createCatalog( +String catalogName, Review comment: > Otherwise the other ticket will fall through the cracks and this gets released as-is. I'm not a committer, though, this is just my opinion. i am sorry, i dont understand why can not we merge the PR first and then create a new ticket? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] CrynetLogistics opened a new pull request #17349: [FLINK-24370][documentation] Added documentation for Asynchronous Sin…
CrynetLogistics opened a new pull request #17349: URL: https://github.com/apache/flink/pull/17349 ## What is the purpose of the change *User stories:* * As a Flink user, I’d like to have some documentation for implementing my sink based atop the Asynchronous Sink Base. *Context:* The AsyncSinkWriter has already been upstreamed as part of FLINK-24041. This is the accompanying documentation. ## Brief change log - *Added documentation to `docs/content/docs/connectors/datastream/async_base.md`* ## Verifying this change No changes to code ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? The whole PR is a documentation ;-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
