[jira] [Commented] (FLINK-4884) Eagerly Store MergingWindowSet in State in WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-4884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601207#comment-15601207 ] ASF GitHub Bot commented on FLINK-4884: --- GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/2685 [FLINK-4884] Eagerly Store MergingWindowSet in State in WindowOperator R: @kl0u for review, please You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink refactor-merging-window-set Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2685.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2685 commit f6803077c5f6e1609dd962946951c41086e8ff67 Author: Aljoscha Krettek Date: 2016-10-21T15:55:53Z [FLINK-4884] Eagerly Store MergingWindowSet in State in WindowOperator > Eagerly Store MergingWindowSet in State in WindowOperator > - > > Key: FLINK-4884 > URL: https://issues.apache.org/jira/browse/FLINK-4884 > Project: Flink > Issue Type: Sub-task > Components: Streaming >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2672: [FLINK-4866] [streaming] Make Trigger.clear() Abstract to...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2672 I merged this. Could you please close the issue? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2685: [FLINK-4884] Eagerly Store MergingWindowSet in Sta...
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/2685 [FLINK-4884] Eagerly Store MergingWindowSet in State in WindowOperator R: @kl0u for review, please You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink refactor-merging-window-set Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2685.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2685 commit f6803077c5f6e1609dd962946951c41086e8ff67 Author: Aljoscha Krettek Date: 2016-10-21T15:55:53Z [FLINK-4884] Eagerly Store MergingWindowSet in State in WindowOperator --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4866) Make Trigger.clear() Abstract to Enforce Implementation
[ https://issues.apache.org/jira/browse/FLINK-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601211#comment-15601211 ] ASF GitHub Bot commented on FLINK-4866: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2672 I merged this. Could you please close the issue? > Make Trigger.clear() Abstract to Enforce Implementation > --- > > Key: FLINK-4866 > URL: https://issues.apache.org/jira/browse/FLINK-4866 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek >Assignee: Jark Wu > Fix For: 1.2.0 > > > If the method is not abstract implementors of custom triggers will not > realise that it could be necessary and they will likely not clean up their > state/timers properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-4866) Make Trigger.clear() Abstract to Enforce Implementation
[ https://issues.apache.org/jira/browse/FLINK-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek resolved FLINK-4866. - Resolution: Fixed Fix Version/s: 1.2.0 Implemented in https://github.com/apache/flink/commit/2c20b39ff03b19363a14024836f99098b52611f5 > Make Trigger.clear() Abstract to Enforce Implementation > --- > > Key: FLINK-4866 > URL: https://issues.apache.org/jira/browse/FLINK-4866 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek >Assignee: Jark Wu > Fix For: 1.2.0 > > > If the method is not abstract implementors of custom triggers will not > realise that it could be necessary and they will likely not clean up their > state/timers properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4883) Prevent UDFs implementations through Scala singleton objects
[
https://issues.apache.org/jira/browse/FLINK-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601278#comment-15601278
]
Stefan Richter commented on FLINK-4883:
---
Ok, thanks! I will assign you to this.
> Prevent UDFs implementations through Scala singleton objects
>
>
> Key: FLINK-4883
> URL: https://issues.apache.org/jira/browse/FLINK-4883
> Project: Flink
> Issue Type: Bug
>Reporter: Stefan Richter
>
> Currently, user can create and use UDFs in Scala like this:
> {code}
> object FlatMapper extends RichCoFlatMapFunction[Long, String, (Long, String)]
> {
> ...
> }
> {code}
> However, this leads to problems as the UDF is now a singleton that Flink
> could use across several operator instances, which leads to job failures. We
> should detect and prevent the usage of singleton UDFs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4743) The sqrt/power function not accept the real data types.
[
https://issues.apache.org/jira/browse/FLINK-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601283#comment-15601283
]
Anton Solovev commented on FLINK-4743:
--
I reported bug in calcite https://issues.apache.org/jira/browse/CALCITE-1467
and solve flink issue through native pow method that in java.lang.Math
> The sqrt/power function not accept the real data types.
> ---
>
> Key: FLINK-4743
> URL: https://issues.apache.org/jira/browse/FLINK-4743
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.1.1
>Reporter: Anton Mushin
>Assignee: Anton Solovev
>
> At time calculate the sequences sql aggregate functions for real type column,
> got exception: No applicable constructor/method found for actual parameters
> "float, java.math.BigDecimal"
> And for column of integer type the problem does not occur.
> Code reproduce:
> {code}
> @Test
> def test():Unit={
> val env = ExecutionEnvironment.getExecutionEnvironment
> val tEnv = TableEnvironment.getTableEnvironment(env, config)
> val ds = env.fromElements(
> (1.0f, 1),
> (2.0f, 2)).toTable(tEnv)
> tEnv.registerTable("MyTable", ds)
> val sqlQuery = "SELECT " +
> "SQRT((SUM(a*a) - SUM(a)*SUM(a)/COUNT(a))/COUNT(a)) "+
> "from (select _1 as a from MyTable)"
> tEnv.sql(sqlQuery).toDataSet[Row].collect().foreach(x=>print(s"$x "))
> }
> {code}
> got exception:
> {noformat}
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.api.table.runtime.FunctionCompiler$class.compile(FunctionCompiler.scala:37)
> at
> org.apache.flink.api.table.runtime.FlatMapRunner.compile(FlatMapRunner.scala:28)
> at
> org.apache.flink.api.table.runtime.FlatMapRunner.open(FlatMapRunner.scala:42)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)
> at
> org.apache.flink.api.common.operators.base.FlatMapOperatorBase.executeOnCollections(FlatMapOperatorBase.java:62)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.executeUnaryOperator(CollectionExecutor.java:249)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:147)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:129)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.executeDataSink(CollectionExecutor.java:180)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:156)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:129)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:113)
> at
> org.apache.flink.api.java.CollectionEnvironment.execute(CollectionEnvironment.java:35)
> at
> org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:47)
> at
> org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:42)
> at
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:637)
> at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:547)
> at
> org.apache.flink.api.scala.batch.sql.AggregationsITCase.test(AggregationsITCase.scala:307)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(P
[GitHub] flink issue #2672: [FLINK-4866] [streaming] Make Trigger.clear() Abstract to...
Github user wuchong commented on the issue: https://github.com/apache/flink/pull/2672 Sure. I will close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4866) Make Trigger.clear() Abstract to Enforce Implementation
[ https://issues.apache.org/jira/browse/FLINK-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601290#comment-15601290 ] ASF GitHub Bot commented on FLINK-4866: --- Github user wuchong closed the pull request at: https://github.com/apache/flink/pull/2672 > Make Trigger.clear() Abstract to Enforce Implementation > --- > > Key: FLINK-4866 > URL: https://issues.apache.org/jira/browse/FLINK-4866 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek >Assignee: Jark Wu > Fix For: 1.2.0 > > > If the method is not abstract implementors of custom triggers will not > realise that it could be necessary and they will likely not clean up their > state/timers properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2672: [FLINK-4866] [streaming] Make Trigger.clear() Abst...
Github user wuchong closed the pull request at: https://github.com/apache/flink/pull/2672 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4866) Make Trigger.clear() Abstract to Enforce Implementation
[ https://issues.apache.org/jira/browse/FLINK-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601289#comment-15601289 ] ASF GitHub Bot commented on FLINK-4866: --- Github user wuchong commented on the issue: https://github.com/apache/flink/pull/2672 Sure. I will close this. > Make Trigger.clear() Abstract to Enforce Implementation > --- > > Key: FLINK-4866 > URL: https://issues.apache.org/jira/browse/FLINK-4866 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek >Assignee: Jark Wu > Fix For: 1.2.0 > > > If the method is not abstract implementors of custom triggers will not > realise that it could be necessary and they will likely not clean up their > state/timers properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4862) NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger
[
https://issues.apache.org/jira/browse/FLINK-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601324#comment-15601324
]
ASF GitHub Bot commented on FLINK-4862:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/2671
@mxm @manuzhang Yes, the timers will be ignored because there is a check in
`onEventTime()`.
The fix looks good! Will you go ahead and merge, @mxm?
> NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger
> --
>
> Key: FLINK-4862
> URL: https://issues.apache.org/jira/browse/FLINK-4862
> Project: Flink
> Issue Type: Bug
> Components: Streaming, Windowing Operators
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 1.2.0, 1.1.4
>
>
> h3. what's the error ?
> The following NPE error is thrown when EventTimeSessionWindows with
> ContinuousEventTimeTrigger is used.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.streaming.api.windowing.triggers.ContinuousEventTimeTrigger.clear(ContinuousEventTimeTrigger.java:91)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.clear(WindowOperator.java:768)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$2.merge(WindowOperator.java:310)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$2.merge(WindowOperator.java:297)
> at
> org.apache.flink.streaming.runtime.operators.windowing.MergingWindowSet.addWindow(MergingWindowSet.java:196)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:297)
> at
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:271)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> h3. how to reproduce ?
> use {{ContinuousEventTimeTrigger}} instead of the default
> {{EventTimeTrigger}} in [SessionWindowing |
> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/SessionWindowing.java#L84]
> example.
> h3. what's the cause ?
> When two session windows are being merged, the states of the two
> {{ContinuousEventTimeTrigger}} are merged as well and the new namespace is
> the merged window. Later when the context tries to delete {{Timer}} from the
> old trigger and looks up the timestamp by the old namespace, null value is
> returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2671: [FLINK-4862] fix Timer register in ContinuousEventTimeTri...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/2671 @mxm @manuzhang Yes, the timers will be ignored because there is a check in `onEventTime()`. The fix looks good! Will you go ahead and merge, @mxm? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-4890) FileInputFormatTest#testExcludeFiles fails on Windows OS
Chesnay Schepler created FLINK-4890:
---
Summary: FileInputFormatTest#testExcludeFiles fails on Windows OS
Key: FLINK-4890
URL: https://issues.apache.org/jira/browse/FLINK-4890
Project: Flink
Issue Type: Bug
Components: Batch Connectors and Input/Output Formats
Affects Versions: 1.1.3
Environment: Windows 10
Reporter: Chesnay Schepler
Running the mentioned test leads to an exception:
{code}
Illegal char <:> at index 4:
file:/C:/dev/cygwin64/tmp/junit3838395086498044255/another_file.bin
java.nio.file.InvalidPathException: Illegal char <:> at index 4:
file:/C:/dev/cygwin64/tmp/junit3838395086498044255/anot
her_file.bin
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at java.nio.file.Paths.get(Paths.java:84)
at
org.apache.flink.api.common.io.GlobFilePathFilter.filterPath(GlobFilePathFilter.java:95)
at
org.apache.flink.api.common.io.FileInputFormat.acceptFile(FileInputFormat.java:644)
at
org.apache.flink.api.common.io.FileInputFormat.addFilesInDir(FileInputFormat.java:600)
at
org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:476)
at
org.apache.flink.api.common.io.FileInputFormatTest.testReadMultiplePatterns(FileInputFormatTest.java:362)
{code}
The problem is that we are given a flink Path, which is then converted to a
String and given to the nio FIleSystem. The passed path is thus /C:/..., which
nio can't work with.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2646: [FLINK-4843] Test for FsCheckpointStateOutputStream::getP...
Github user uce commented on the issue: https://github.com/apache/flink/pull/2646 Merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4843) Introduce Test for FsCheckpointStateOutputStream::getPos
[ https://issues.apache.org/jira/browse/FLINK-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601402#comment-15601402 ] ASF GitHub Bot commented on FLINK-4843: --- Github user uce commented on the issue: https://github.com/apache/flink/pull/2646 Merging > Introduce Test for FsCheckpointStateOutputStream::getPos > > > Key: FLINK-4843 > URL: https://issues.apache.org/jira/browse/FLINK-4843 > Project: Flink > Issue Type: Test >Reporter: Stefan Richter >Assignee: Stefan Richter > > Introduce a test for FsCheckpointStateOutputStream::getPos, which is > currently not included in the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4891) Remove flink-contrib/flink-operator-stats
Ufuk Celebi created FLINK-4891:
--
Summary: Remove flink-contrib/flink-operator-stats
Key: FLINK-4891
URL: https://issues.apache.org/jira/browse/FLINK-4891
Project: Flink
Issue Type: Task
Reporter: Ufuk Celebi
Assignee: Ufuk Celebi
As per discussion on the mailing list, remove module
{{flink-contrib/flink-operator-stats}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4891) Remove flink-contrib/flink-operator-stats
[
https://issues.apache.org/jira/browse/FLINK-4891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601406#comment-15601406
]
Ufuk Celebi commented on FLINK-4891:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Removing-flink-contrib-flink-operator-stats-tt14096.html
> Remove flink-contrib/flink-operator-stats
> -
>
> Key: FLINK-4891
> URL: https://issues.apache.org/jira/browse/FLINK-4891
> Project: Flink
> Issue Type: Task
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
>
> As per discussion on the mailing list, remove module
> {{flink-contrib/flink-operator-stats}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-4891) Remove flink-contrib/flink-operator-stats
[
https://issues.apache.org/jira/browse/FLINK-4891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ufuk Celebi closed FLINK-4891.
--
Resolution: Implemented
Fix Version/s: 1.2.0
Done in d5b97b0 (master).
> Remove flink-contrib/flink-operator-stats
> -
>
> Key: FLINK-4891
> URL: https://issues.apache.org/jira/browse/FLINK-4891
> Project: Flink
> Issue Type: Task
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
> Fix For: 1.2.0
>
>
> As per discussion on the mailing list, remove module
> {{flink-contrib/flink-operator-stats}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-4833) Unstable test OperatorStatsAccumulatorTest.testAccumulatorHeavyHitterCountMinSketch
[ https://issues.apache.org/jira/browse/FLINK-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi closed FLINK-4833. -- Resolution: Implemented Fix Version/s: 1.2.0 Fixed in d5b97b0 (master) via removal of the module > Unstable test > OperatorStatsAccumulatorTest.testAccumulatorHeavyHitterCountMinSketch > --- > > Key: FLINK-4833 > URL: https://issues.apache.org/jira/browse/FLINK-4833 > Project: Flink > Issue Type: Bug > Components: Tests >Reporter: Kostas Kloudas > Labels: test-stability > Fix For: 1.2.0 > > > Some instances: > view-source:https://s3.amazonaws.com/archive.travis-ci.org/jobs/167801187/log.txt > view-source:https://s3.amazonaws.com/archive.travis-ci.org/jobs/167801191/log.txt > view-source:https://s3.amazonaws.com/archive.travis-ci.org/jobs/167801193/log.txt > view-source:https://s3.amazonaws.com/archive.travis-ci.org/jobs/167801195/log.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-4843) Introduce Test for FsCheckpointStateOutputStream::getPos
[ https://issues.apache.org/jira/browse/FLINK-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi closed FLINK-4843. -- Resolution: Implemented Fix Version/s: 1.2.0 Implemented in 2b31ec9 (master). > Introduce Test for FsCheckpointStateOutputStream::getPos > > > Key: FLINK-4843 > URL: https://issues.apache.org/jira/browse/FLINK-4843 > Project: Flink > Issue Type: Test >Reporter: Stefan Richter >Assignee: Stefan Richter > Fix For: 1.2.0 > > > Introduce a test for FsCheckpointStateOutputStream::getPos, which is > currently not included in the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2646: [FLINK-4843] Test for FsCheckpointStateOutputStrea...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2646 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4843) Introduce Test for FsCheckpointStateOutputStream::getPos
[ https://issues.apache.org/jira/browse/FLINK-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601418#comment-15601418 ] ASF GitHub Bot commented on FLINK-4843: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2646 > Introduce Test for FsCheckpointStateOutputStream::getPos > > > Key: FLINK-4843 > URL: https://issues.apache.org/jira/browse/FLINK-4843 > Project: Flink > Issue Type: Test >Reporter: Stefan Richter >Assignee: Stefan Richter > Fix For: 1.2.0 > > > Introduce a test for FsCheckpointStateOutputStream::getPos, which is > currently not included in the tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4235) ClassLoaderITCase.testDisposeSavepointWithCustomKvState timed out on Travis
[
https://issues.apache.org/jira/browse/FLINK-4235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601421#comment-15601421
]
Robert Metzger commented on FLINK-4235:
---
It seems like this happened again
{code}
testDisposeSavepointWithCustomKvState(org.apache.flink.test.classloading.ClassLoaderITCase)
Time elapsed: 100.018 sec <<< ERROR!
java.util.concurrent.TimeoutException: Futures timed out after [7263142827
nanoseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at
scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
at
scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:190)
at scala.concurrent.Await.result(package.scala)
at
org.apache.flink.test.classloading.ClassLoaderITCase.testDisposeSavepointWithCustomKvState(ClassLoaderITCase.java:300)
{code}
https://s3.amazonaws.com/archive.travis-ci.org/jobs/169700823/log.txt
> ClassLoaderITCase.testDisposeSavepointWithCustomKvState timed out on Travis
> ---
>
> Key: FLINK-4235
> URL: https://issues.apache.org/jira/browse/FLINK-4235
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Ufuk Celebi
> Labels: test-stability
> Fix For: 1.1.0
>
>
> The {{ClasLoaderITCase.testDisposeSavepointWithCustomKvState}} timed out on
> Travis. A potential fix could be to increase the timeouts for this test. When
> fixing this issue, then we should let the {{ClassLoaderITCase}} extend the
> {{TestLogger}}.
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/145882668/log.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4565) Support for SQL IN operator
[ https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601426#comment-15601426 ] Jark Wu commented on FLINK-4565: Hi [~nvasilishin], do you still plan to implement this issue ? Otherwise, I can have a try on this . > Support for SQL IN operator > --- > > Key: FLINK-4565 > URL: https://issues.apache.org/jira/browse/FLINK-4565 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Timo Walther >Assignee: Nikolay Vasilishin > > It seems that Flink SQL supports the uncorrelated sub-query IN operator. But > it should also be available in the Table API and tested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-4883) Prevent UDFs implementations through Scala singleton objects
[
https://issues.apache.org/jira/browse/FLINK-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Richter updated FLINK-4883:
--
Assignee: Renkai Ge
> Prevent UDFs implementations through Scala singleton objects
>
>
> Key: FLINK-4883
> URL: https://issues.apache.org/jira/browse/FLINK-4883
> Project: Flink
> Issue Type: Bug
>Reporter: Stefan Richter
>Assignee: Renkai Ge
>
> Currently, user can create and use UDFs in Scala like this:
> {code}
> object FlatMapper extends RichCoFlatMapFunction[Long, String, (Long, String)]
> {
> ...
> }
> {code}
> However, this leads to problems as the UDF is now a singleton that Flink
> could use across several operator instances, which leads to job failures. We
> should detect and prevent the usage of singleton UDFs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2686: [FLINK-4743] The sqrt/power function not accept th...
GitHub user tonycox opened a pull request: https://github.com/apache/flink/pull/2686 [FLINK-4743] The sqrt/power function not accept the real data types. Add supporting of variety types on "power" function. Add casting for Float and BigDecimal. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tonycox/flink FLINK-4743 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2686.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2686 commit e5cfd1106b3c69f599149b56eee2ede7936e1e63 Author: anton_solo...@epam.com Date: 2016-10-17T07:44:12Z [FLINK-4743] Add supporting of variety types on "power" function. Add casting for Float and BigDecimal. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4743) The sqrt/power function not accept the real data types.
[
https://issues.apache.org/jira/browse/FLINK-4743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601470#comment-15601470
]
ASF GitHub Bot commented on FLINK-4743:
---
GitHub user tonycox opened a pull request:
https://github.com/apache/flink/pull/2686
[FLINK-4743] The sqrt/power function not accept the real data types.
Add supporting of variety types on "power" function. Add casting for Float
and BigDecimal.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tonycox/flink FLINK-4743
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2686.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2686
commit e5cfd1106b3c69f599149b56eee2ede7936e1e63
Author: anton_solo...@epam.com
Date: 2016-10-17T07:44:12Z
[FLINK-4743] Add supporting of variety types on "power" function. Add
casting for Float and BigDecimal.
> The sqrt/power function not accept the real data types.
> ---
>
> Key: FLINK-4743
> URL: https://issues.apache.org/jira/browse/FLINK-4743
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.1.1
>Reporter: Anton Mushin
>Assignee: Anton Solovev
>
> At time calculate the sequences sql aggregate functions for real type column,
> got exception: No applicable constructor/method found for actual parameters
> "float, java.math.BigDecimal"
> And for column of integer type the problem does not occur.
> Code reproduce:
> {code}
> @Test
> def test():Unit={
> val env = ExecutionEnvironment.getExecutionEnvironment
> val tEnv = TableEnvironment.getTableEnvironment(env, config)
> val ds = env.fromElements(
> (1.0f, 1),
> (2.0f, 2)).toTable(tEnv)
> tEnv.registerTable("MyTable", ds)
> val sqlQuery = "SELECT " +
> "SQRT((SUM(a*a) - SUM(a)*SUM(a)/COUNT(a))/COUNT(a)) "+
> "from (select _1 as a from MyTable)"
> tEnv.sql(sqlQuery).toDataSet[Row].collect().foreach(x=>print(s"$x "))
> }
> {code}
> got exception:
> {noformat}
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.api.table.runtime.FunctionCompiler$class.compile(FunctionCompiler.scala:37)
> at
> org.apache.flink.api.table.runtime.FlatMapRunner.compile(FlatMapRunner.scala:28)
> at
> org.apache.flink.api.table.runtime.FlatMapRunner.open(FlatMapRunner.scala:42)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:38)
> at
> org.apache.flink.api.common.operators.base.FlatMapOperatorBase.executeOnCollections(FlatMapOperatorBase.java:62)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.executeUnaryOperator(CollectionExecutor.java:249)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:147)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:129)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.executeDataSink(CollectionExecutor.java:180)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:156)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:129)
> at
> org.apache.flink.api.common.operators.CollectionExecutor.execute(CollectionExecutor.java:113)
> at
> org.apache.flink.api.java.CollectionEnvironment.execute(CollectionEnvironment.java:35)
> at
> org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:47)
> at
> org.apache.flink.test.util.CollectionTestEnvironment.execute(CollectionTestEnvironment.java:42)
> at
> org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:637)
> at org.apache.flink.api.scala.DataSet.collect(DataSet.scala:547)
> at
> org.apache.flink.api.scala.batch.sql.AggregationsITCase.test(AggregationsITCase.scala:307)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.Framew
[jira] [Assigned] (FLINK-4762) Use plural in time interval units
[
https://issues.apache.org/jira/browse/FLINK-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-4762:
--
Assignee: Jark Wu
> Use plural in time interval units
> -
>
> Key: FLINK-4762
> URL: https://issues.apache.org/jira/browse/FLINK-4762
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> During the creation of FLIP-11 we decided to rename the time interval units.
> From {{minute}} to {{minutes}} and so on in Java and Scala Table API.
> {{12.minutes + 2.hours}} reads better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2644: [FLINK-4837] flink-streaming-akka source connector
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2644 We recently started collaborating with Apache Bahir (http://bahir.apache.org) to host additions. That makes it easier to add new connectors and to do faster releases of connectors. The Bahir website does not yet reflect that, because it started recently and the first release of Flink connectors is still pending. What do you think about adding this to Bahir as well? @rmetzger can probably help out here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4837) flink-streaming-akka source connector
[ https://issues.apache.org/jira/browse/FLINK-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601489#comment-15601489 ] ASF GitHub Bot commented on FLINK-4837: --- Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2644 We recently started collaborating with Apache Bahir (http://bahir.apache.org) to host additions. That makes it easier to add new connectors and to do faster releases of connectors. The Bahir website does not yet reflect that, because it started recently and the first release of Flink connectors is still pending. What do you think about adding this to Bahir as well? @rmetzger can probably help out here. > flink-streaming-akka source connector > - > > Key: FLINK-4837 > URL: https://issues.apache.org/jira/browse/FLINK-4837 > Project: Flink > Issue Type: New Feature > Components: Streaming Connectors >Reporter: Subhobrata Dey > > Hello, > This issue is created to propose the idea of having a flink-streaming-akka > source connector. > The source connector can be used to receive messages from an Akka feeder or > publisher actor & these messages can then be processed using flink streaming. > The source connector has the following features. > 1. It can supports several different message formats like iterable data, > bytes array & data with timestamp. > 2. It can send back acknowledgements to the feeder actor. > Thanks & regards, > Subhobrata -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4565) Support for SQL IN operator
[ https://issues.apache.org/jira/browse/FLINK-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601503#comment-15601503 ] Nikolay Vasilishin commented on FLINK-4565: --- Hi [~jark]. Yes, i'm working on it right now. > Support for SQL IN operator > --- > > Key: FLINK-4565 > URL: https://issues.apache.org/jira/browse/FLINK-4565 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Reporter: Timo Walther >Assignee: Nikolay Vasilishin > > It seems that Flink SQL supports the uncorrelated sub-query IN operator. But > it should also be available in the Table API and tested. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4651) Re-register processing time timers at the WindowOperator upon recovery.
[
https://issues.apache.org/jira/browse/FLINK-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601507#comment-15601507
]
Aljoscha Krettek commented on FLINK-4651:
-
We can either back port the "only one physical processing-time timer" change or
only make sure that we register timers in {{open()}}.
> Re-register processing time timers at the WindowOperator upon recovery.
> ---
>
> Key: FLINK-4651
> URL: https://issues.apache.org/jira/browse/FLINK-4651
> Project: Flink
> Issue Type: Bug
> Components: Windowing Operators
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Labels: windows
> Fix For: 1.2.0, 1.1.4
>
>
> Currently the {{WindowOperator}} checkpoints the processing time timers, but
> upon recovery it does not re-registers them with the {{TimeServiceProvider}}.
> To actually reprocess them it relies on another element that will come and
> register a new timer for a future point in time. Although this is a realistic
> assumption in long running jobs, we can remove this assumption by
> re-registering the restored timers with the {{TimeServiceProvider}} in the
> {{open()}} method of the {{WindowOperator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-3202) Make Timer/Trigger Service Scoped to Key/Namespace
[
https://issues.apache.org/jira/browse/FLINK-3202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek resolved FLINK-3202.
-
Resolution: Fixed
Fix Version/s: 1.2.0
Implemented a while back during the {{WindowOperator}} refactoring and while
adding support for keyed timers in user functions.
> Make Timer/Trigger Service Scoped to Key/Namespace
> --
>
> Key: FLINK-3202
> URL: https://issues.apache.org/jira/browse/FLINK-3202
> Project: Flink
> Issue Type: Sub-task
> Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
> Fix For: 1.2.0
>
>
> For the WindowOperator to be able to use the partitioned state abstraction
> timers/triggers also need to be scoped to the same key/namespace so that
> firing triggers can be matched to the correct window.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2653: [FLINK-4469] [table] Add support for user defined table f...
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2653 HI @wuchong, thanks for this feature. This looks really interesting :-) I haven't looked at the PR in detail yet. However, I would like to avoid customizing the parser if possible. I haven't tested it but would it be possible to express `OUTER APPLY` and `CROSS APPLY` with Calcite's `LATERAL` tables? `CROSS APPLY` corresponds to: ``` SELECT MyTable.*, t.s FROM MyTable, LATERAL TABLE(split(MyTable.a)) AS t(s) ``` `OUTER APPLY` could be: ``` SELECT MyTable.*, t.s FROM MyTable LEFT JOIN LATERAL TABLE(split(MyTable.a)) AS t(s) ``` Adding `crossApply` and `outerApply` for the Table API is fine with me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4469) Add support for user defined table function in Table API & SQL
[
https://issues.apache.org/jira/browse/FLINK-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601513#comment-15601513
]
ASF GitHub Bot commented on FLINK-4469:
---
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2653
HI @wuchong, thanks for this feature. This looks really interesting :-)
I haven't looked at the PR in detail yet. However, I would like to avoid
customizing the parser if possible. I haven't tested it but would it be
possible to express `OUTER APPLY` and `CROSS APPLY` with Calcite's `LATERAL`
tables?
`CROSS APPLY` corresponds to:
```
SELECT MyTable.*, t.s FROM MyTable, LATERAL TABLE(split(MyTable.a)) AS t(s)
```
`OUTER APPLY` could be:
```
SELECT MyTable.*, t.s
FROM MyTable LEFT JOIN LATERAL TABLE(split(MyTable.a)) AS t(s)
```
Adding `crossApply` and `outerApply` for the Table API is fine with me.
> Add support for user defined table function in Table API & SQL
> --
>
> Key: FLINK-4469
> URL: https://issues.apache.org/jira/browse/FLINK-4469
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Jark Wu
>Assignee: Jark Wu
>
> Normal user-defined functions, such as concat(), take in a single input row
> and output a single output row. In contrast, table-generating functions
> transform a single input row to multiple output rows. It is very useful in
> some cases, such as look up in HBase by rowkey and return one or more rows.
> Adding a user defined table function should:
> 1. inherit from UDTF class with specific generic type T
> 2. define one or more evel function.
> NOTE:
> 1. the eval method must be public and non-static.
> 2. eval should always return java.lang.Iterable or scala.collection.Iterable
> with the generic type T.
> 3. the generic type T is the row type returned by table function. Because of
> Java type erasure, we can’t extract T from the Iterable.
> 4. eval method can be overload. Blink will choose the best match eval method
> to call according to parameter types and number.
> {code}
> public class Word {
> public String word;
> public Integer length;
> }
> public class SplitStringUDTF extends UDTF {
> public Iterable eval(String str) {
> if (str == null) {
> return new ArrayList<>();
> } else {
> List list = new ArrayList<>();
> for (String s : str.split(",")) {
> Word word = new Word(s, s.length());
> list.add(word);
> }
> return list;
> }
> }
> }
> // in SQL
> tableEnv.registerFunction("split", new SplitStringUDTF())
> tableEnv.sql("SELECT a, b, t.* FROM MyTable CROSS APPLY split(c) AS t(w,l)")
> // in Java Table API
> tableEnv.registerFunction("split", new SplitStringUDTF())
> // rename split table columns to “w” and “l”
> table.crossApply("split(c)", "w, l")
> .select("a, b, w, l")
> // without renaming, we will use the origin field names in the POJO/case/...
> table.crossApply("split(c)")
> .select("a, b, word, length")
> // in Scala Table API
> val split = new SplitStringUDTF()
> table.crossApply(split('c), 'w, 'l)
> .select('a, 'b, 'w, 'l)
> // outerApply for outer join to a UDTF
> table.outerApply(split('c))
> .select('a, 'b, 'word, 'length)
> {code}
> Here we introduce CROSS/OUTER APPLY keywords to join table functions , which
> is used in SQL Server. We can discuss the API in the comment.
> Maybe the {{UDTF}} class should be replaced by {{TableFunction}} or something
> others, because we have introduced {{ScalarFunction}} for custom functions,
> we need to keep consistent. Although, I prefer {{UDTF}} rather than
> {{TableFunction}} as the former is more SQL-like and the latter maybe
> confused with DataStream functions.
> **This issue is blocked by CALCITE-1309, so we need to wait Calcite fix this
> and release.**
> See [1] for more information about UDTF design.
> [1]
> https://docs.google.com/document/d/15iVc1781dxYWm3loVQlESYvMAxEzbbuVFPZWBYuY1Ek/edit#
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4892) Snapshot TimerService using Key-Grouped State
Aljoscha Krettek created FLINK-4892: --- Summary: Snapshot TimerService using Key-Grouped State Key: FLINK-4892 URL: https://issues.apache.org/jira/browse/FLINK-4892 Project: Flink Issue Type: Improvement Components: Streaming Affects Versions: 1.2.0 Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek Fix For: 1.2.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4651) Re-register processing time timers at the WindowOperator upon recovery.
[
https://issues.apache.org/jira/browse/FLINK-4651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601517#comment-15601517
]
Stephan Ewen commented on FLINK-4651:
-
The big changes in the window operator (move timers, key groups) came after
that patch, I think. So we can probably backport it without too much effort.
> Re-register processing time timers at the WindowOperator upon recovery.
> ---
>
> Key: FLINK-4651
> URL: https://issues.apache.org/jira/browse/FLINK-4651
> Project: Flink
> Issue Type: Bug
> Components: Windowing Operators
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
> Labels: windows
> Fix For: 1.2.0, 1.1.4
>
>
> Currently the {{WindowOperator}} checkpoints the processing time timers, but
> upon recovery it does not re-registers them with the {{TimeServiceProvider}}.
> To actually reprocess them it relies on another element that will come and
> register a new timer for a future point in time. Although this is a realistic
> assumption in long running jobs, we can remove this assumption by
> re-registering the restored timers with the {{TimeServiceProvider}} in the
> {{open()}} method of the {{WindowOperator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4893) Use single RecordSerializer per RecordWriter
Ufuk Celebi created FLINK-4893:
--
Summary: Use single RecordSerializer per RecordWriter
Key: FLINK-4893
URL: https://issues.apache.org/jira/browse/FLINK-4893
Project: Flink
Issue Type: Improvement
Components: Network
Reporter: Ufuk Celebi
Instead of creating n instances of the serializer for each outgoing channel, we
can simply keep the references to the {{Buffer}} instances in the writer and
use a single record serializer instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4894) Don't block on buffer request after broadcastEvent
Ufuk Celebi created FLINK-4894: -- Summary: Don't block on buffer request after broadcastEvent Key: FLINK-4894 URL: https://issues.apache.org/jira/browse/FLINK-4894 Project: Flink Issue Type: Improvement Reporter: Ufuk Celebi After broadcasting an event (like the checkpoint barrier), the record writer might block on a buffer request although that buffer will only be needed on the next write on that channel. Instead of assuming that each serializer has a buffer set, we can change the logic in the writer to request the buffer when it requires one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2673: [FLINK-4864] [table] Shade Calcite dependency in flink-ta...
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2673 Thanks for investigating this and submitting a PR to Calcite, @wuchong! Maybe @rmetzger has an idea of what's going wrong with the shading? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4864) Shade Calcite dependency in flink-table
[ https://issues.apache.org/jira/browse/FLINK-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601535#comment-15601535 ] ASF GitHub Bot commented on FLINK-4864: --- Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2673 Thanks for investigating this and submitting a PR to Calcite, @wuchong! Maybe @rmetzger has an idea of what's going wrong with the shading? > Shade Calcite dependency in flink-table > --- > > Key: FLINK-4864 > URL: https://issues.apache.org/jira/browse/FLINK-4864 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Affects Versions: 1.2.0 >Reporter: Fabian Hueske >Assignee: Jark Wu > > The Table API has a dependency on Apache Calcite. > A user reported to have version conflicts when having a own Calcite > dependency in the classpath. > The solution would be to shade away the Calcite dependency (Calcite's > transitive dependencies are already shaded). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2521: [FLINK-4639] [table] Make Calcite features more pl...
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2521#discussion_r84659763
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/CalciteConfig.scala
---
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table
+
+import org.apache.calcite.sql.SqlOperatorTable
+import org.apache.calcite.sql.parser.SqlParser
+import org.apache.calcite.tools.RuleSet
+import org.apache.flink.util.Preconditions
+
+/**
+ * Builder for creating a Calcite configuration.
+ */
+class CalciteConfigBuilder {
+ private var replaceRuleSet: Option[RuleSet] = None
+ private var chainRuleSet: Option[RuleSet] = None
+
+ private var replaceSqlOperatorTable: Option[SqlOperatorTable] = None
+ private var chainSqlOperatorTable: Option[SqlOperatorTable] = None
+
+ private var replaceSqlParserConfig: Option[SqlParser.Config] = None
+
+ /**
+* Replaces the built-in rule set with the given rule set.
+*/
+ def replaceRuleSet(ruleSet: RuleSet): CalciteConfigBuilder = {
+Preconditions.checkNotNull(ruleSet)
+replaceRuleSet = Some(ruleSet)
+chainRuleSet = None
+this
+ }
+
+ /**
+* Appends the given rule set to the built-in rule set.
+*/
+ def addRuleSet(ruleSet: RuleSet): CalciteConfigBuilder = {
+Preconditions.checkNotNull(ruleSet)
+replaceRuleSet = None
+chainRuleSet = Some(ruleSet)
+this
+ }
+
+ /**
+* Replaces the built-in SQL operator table with the given table.
+*/
+ def replaceSqlOperatorTable(sqlOperatorTable: SqlOperatorTable):
CalciteConfigBuilder = {
+Preconditions.checkNotNull(sqlOperatorTable)
+replaceSqlOperatorTable = Some(sqlOperatorTable)
+chainSqlOperatorTable = None
+this
+ }
+
+ /**
+* Appends the given table to the built-in SQL operator table.
+*/
+ def addSqlOperatorTable(sqlOperatorTable: SqlOperatorTable):
CalciteConfigBuilder = {
+Preconditions.checkNotNull(sqlOperatorTable)
+replaceSqlOperatorTable = None
+chainSqlOperatorTable = Some(sqlOperatorTable)
--- End diff --
operator table would be replaced and not added if `addSqlOperatorTable` is
called twice.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-4639) Make Calcite features more pluggable
[
https://issues.apache.org/jira/browse/FLINK-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601578#comment-15601578
]
ASF GitHub Bot commented on FLINK-4639:
---
Github user fhueske commented on a diff in the pull request:
https://github.com/apache/flink/pull/2521#discussion_r84659763
--- Diff:
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/CalciteConfig.scala
---
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.table
+
+import org.apache.calcite.sql.SqlOperatorTable
+import org.apache.calcite.sql.parser.SqlParser
+import org.apache.calcite.tools.RuleSet
+import org.apache.flink.util.Preconditions
+
+/**
+ * Builder for creating a Calcite configuration.
+ */
+class CalciteConfigBuilder {
+ private var replaceRuleSet: Option[RuleSet] = None
+ private var chainRuleSet: Option[RuleSet] = None
+
+ private var replaceSqlOperatorTable: Option[SqlOperatorTable] = None
+ private var chainSqlOperatorTable: Option[SqlOperatorTable] = None
+
+ private var replaceSqlParserConfig: Option[SqlParser.Config] = None
+
+ /**
+* Replaces the built-in rule set with the given rule set.
+*/
+ def replaceRuleSet(ruleSet: RuleSet): CalciteConfigBuilder = {
+Preconditions.checkNotNull(ruleSet)
+replaceRuleSet = Some(ruleSet)
+chainRuleSet = None
+this
+ }
+
+ /**
+* Appends the given rule set to the built-in rule set.
+*/
+ def addRuleSet(ruleSet: RuleSet): CalciteConfigBuilder = {
+Preconditions.checkNotNull(ruleSet)
+replaceRuleSet = None
+chainRuleSet = Some(ruleSet)
+this
+ }
+
+ /**
+* Replaces the built-in SQL operator table with the given table.
+*/
+ def replaceSqlOperatorTable(sqlOperatorTable: SqlOperatorTable):
CalciteConfigBuilder = {
+Preconditions.checkNotNull(sqlOperatorTable)
+replaceSqlOperatorTable = Some(sqlOperatorTable)
+chainSqlOperatorTable = None
+this
+ }
+
+ /**
+* Appends the given table to the built-in SQL operator table.
+*/
+ def addSqlOperatorTable(sqlOperatorTable: SqlOperatorTable):
CalciteConfigBuilder = {
+Preconditions.checkNotNull(sqlOperatorTable)
+replaceSqlOperatorTable = None
+chainSqlOperatorTable = Some(sqlOperatorTable)
--- End diff --
operator table would be replaced and not added if `addSqlOperatorTable` is
called twice.
> Make Calcite features more pluggable
>
>
> Key: FLINK-4639
> URL: https://issues.apache.org/jira/browse/FLINK-4639
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Some users might want to extend the feature set of the Table API by adding or
> replacing Calcite optimizer rules, modifying the parser etc. It would be good
> to have means to hook into the Table API and change Calcite behavior. We
> should implement something like a {{CalciteConfigBuilder}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4639) Make Calcite features more pluggable
[
https://issues.apache.org/jira/browse/FLINK-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601597#comment-15601597
]
ASF GitHub Bot commented on FLINK-4639:
---
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/2521
PR looks good to me, except for some methods of the added `CalciteConfig`.
Will fix these issues and merge the PR.
> Make Calcite features more pluggable
>
>
> Key: FLINK-4639
> URL: https://issues.apache.org/jira/browse/FLINK-4639
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> Some users might want to extend the feature set of the Table API by adding or
> replacing Calcite optimizer rules, modifying the parser etc. It would be good
> to have means to hook into the Table API and change Calcite behavior. We
> should implement something like a {{CalciteConfigBuilder}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2521: [FLINK-4639] [table] Make Calcite features more pluggable
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2521 PR looks good to me, except for some methods of the added `CalciteConfig`. Will fix these issues and merge the PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-4630) add netty tcp/restful pushed source support
[ https://issues.apache.org/jira/browse/FLINK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shijinkui updated FLINK-4630: - Description: When source stream get start, listen a provided tcp port, receive stream data from user data source. This netty tcp source is keepping alive and end-to-end, that is from business system to flink worker directly. Such source service is needed in produce indeed. describe the source in detail below: 1. source run as a netty tcp server 2. user provide a tcp port, if the port is in used, increace the port number between 1024 to 65535. Source can parallel. 3. callback the provided url to report the real port to listen 4. user push streaming data to netty server, then collect the data to flink link to https://issues.apache.org/jira/browse/BAHIR-72 was: When source stream get start, listen a provided tcp port, receive stream data from user data source. This netty tcp source is keepping alive and end-to-end, that is from business system to flink worker directly. Such source service is needed in produce indeed. describe the source in detail below: 1. source run as a netty tcp server 2. user provide a tcp port, if the port is in used, increace the port number between 1024 to 65535. Source can parallel. 3. callback the provided url to report the real port to listen 4. user push streaming data to netty server, then collect the data to flink > add netty tcp/restful pushed source support > --- > > Key: FLINK-4630 > URL: https://issues.apache.org/jira/browse/FLINK-4630 > Project: Flink > Issue Type: New Feature > Components: Streaming Connectors >Reporter: shijinkui > > When source stream get start, listen a provided tcp port, receive stream data > from user data source. > This netty tcp source is keepping alive and end-to-end, that is from business > system to flink worker directly. > Such source service is needed in produce indeed. > describe the source in detail below: > 1.source run as a netty tcp server > 2.user provide a tcp port, if the port is in used, increace the port > number between 1024 to 65535. Source can parallel. > 3.callback the provided url to report the real port to listen > 4.user push streaming data to netty server, then collect the data to flink > link to https://issues.apache.org/jira/browse/BAHIR-72 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4837) flink-streaming-akka source connector
[ https://issues.apache.org/jira/browse/FLINK-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601606#comment-15601606 ] ASF GitHub Bot commented on FLINK-4837: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2644 Thank you for opening a pull request for adding a new streaming connector. I also think that this connector is a good candidate to go into Apache Bahir. You find the GitHub repository of Bahir here: https://github.com/apache/bahir-flink Let me know if you need help while opening a pull request at Bahir. I'm happy to review your pull request at Bahir! > flink-streaming-akka source connector > - > > Key: FLINK-4837 > URL: https://issues.apache.org/jira/browse/FLINK-4837 > Project: Flink > Issue Type: New Feature > Components: Streaming Connectors >Reporter: Subhobrata Dey > > Hello, > This issue is created to propose the idea of having a flink-streaming-akka > source connector. > The source connector can be used to receive messages from an Akka feeder or > publisher actor & these messages can then be processed using flink streaming. > The source connector has the following features. > 1. It can supports several different message formats like iterable data, > bytes array & data with timestamp. > 2. It can send back acknowledgements to the feeder actor. > Thanks & regards, > Subhobrata -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2644: [FLINK-4837] flink-streaming-akka source connector
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2644 Thank you for opening a pull request for adding a new streaming connector. I also think that this connector is a good candidate to go into Apache Bahir. You find the GitHub repository of Bahir here: https://github.com/apache/bahir-flink Let me know if you need help while opening a pull request at Bahir. I'm happy to review your pull request at Bahir! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2687: [FLINK-3123] [kafka] Allow custom specific start o...
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/2687
[FLINK-3123] [kafka] Allow custom specific start offsets for Kafka consumers
This PR is based on #2509, so only the last commit is relevant.
With this change, users can now specify specific start offsets for Kafka
consumers like this:
```
Map specificStartOffsets = new HashMap<>();
specificStartOffsets.put(new KafkaTopicPartition("topic", 0), 23L);
specificStartOffsets.put(new KafkaTopicPartition("topic", 1), 31L);
specificStartOffsets.put(new KafkaTopicPartition("topic", 2), 43L);
FlinkKafkaConsumer08 consumer = new FlinkKafkaConsumer08<>(...);
consumer.setStartFromSpecificOffsets(specificStartOffsets);
...
```
If a subscribed partition is not defined a specific offset (does not have a
corresponding entry in the `specificStartOffsets` map), then the startup
behaviour for that particular partition fallbacks to the default group offset
behaviour (look for offset in ZK / Kafka for that partition, or use
"auto.offset.reset" if none can be found).
An IT test `runStartFromSpecificOffsets()` is added for this functionality,
however is currently only enabled on the Kafka 0.8 consumer, because 0.9 and
0.10 tests have the same Kafka config problem mentioned in #2509. So, for now,
for versions 0.9 and 0.10, I have only manually tested this new functionality,
and it works correctly as described above.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-3123
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2687.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2687
commit eca9043ebd63ea201b14b129ce08a9f3ee78c49c
Author: Tzu-Li (Gordon) Tai
Date: 2016-09-17T13:41:50Z
[FLINK-4280][kafka] Explicit start position configuration for Kafka Consumer
commit 0703469e1880daa63bc5e92bae9920573659806d
Author: Tzu-Li (Gordon) Tai
Date: 2016-10-23T08:55:58Z
[FLINK-4280] Allow Kafka 0.10 to override 0.10-specific API calls
Methods calls to `seekToBeginning` and `seekToEnd` have breaking APIs
across 0.9 and 0.10, causing 0.10 IT tests to fail.
commit bdf6b76eb86c3d6a4a0bb84ab26beb13e84526b1
Author: Tzu-Li (Gordon) Tai
Date: 2016-10-24T05:24:10Z
[FLINK-4280] Add IT tests for explicit start position configuration
commit d8f5f976ef2e2e1d994a45468d7e9ef3b8bf0015
Author: Tzu-Li (Gordon) Tai
Date: 2016-10-24T08:57:05Z
[FLINK-4280] Add documentation for the new explicit start position methods
commit 098360fc797b78f15917aaf1b22d09c06a4d0a6c
Author: Tzu-Li (Gordon) Tai
Date: 2016-10-24T08:08:18Z
[FLINK-3123] [kafka] Allow custom specific start offsets for Kafka consumers
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-3123) Allow setting custom start-offsets for the Kafka consumer
[
https://issues.apache.org/jira/browse/FLINK-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601611#comment-15601611
]
ASF GitHub Bot commented on FLINK-3123:
---
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/2687
[FLINK-3123] [kafka] Allow custom specific start offsets for Kafka consumers
This PR is based on #2509, so only the last commit is relevant.
With this change, users can now specify specific start offsets for Kafka
consumers like this:
```
Map specificStartOffsets = new HashMap<>();
specificStartOffsets.put(new KafkaTopicPartition("topic", 0), 23L);
specificStartOffsets.put(new KafkaTopicPartition("topic", 1), 31L);
specificStartOffsets.put(new KafkaTopicPartition("topic", 2), 43L);
FlinkKafkaConsumer08 consumer = new FlinkKafkaConsumer08<>(...);
consumer.setStartFromSpecificOffsets(specificStartOffsets);
...
```
If a subscribed partition is not defined a specific offset (does not have a
corresponding entry in the `specificStartOffsets` map), then the startup
behaviour for that particular partition fallbacks to the default group offset
behaviour (look for offset in ZK / Kafka for that partition, or use
"auto.offset.reset" if none can be found).
An IT test `runStartFromSpecificOffsets()` is added for this functionality,
however is currently only enabled on the Kafka 0.8 consumer, because 0.9 and
0.10 tests have the same Kafka config problem mentioned in #2509. So, for now,
for versions 0.9 and 0.10, I have only manually tested this new functionality,
and it works correctly as described above.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-3123
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2687.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2687
commit eca9043ebd63ea201b14b129ce08a9f3ee78c49c
Author: Tzu-Li (Gordon) Tai
Date: 2016-09-17T13:41:50Z
[FLINK-4280][kafka] Explicit start position configuration for Kafka Consumer
commit 0703469e1880daa63bc5e92bae9920573659806d
Author: Tzu-Li (Gordon) Tai
Date: 2016-10-23T08:55:58Z
[FLINK-4280] Allow Kafka 0.10 to override 0.10-specific API calls
Methods calls to `seekToBeginning` and `seekToEnd` have breaking APIs
across 0.9 and 0.10, causing 0.10 IT tests to fail.
commit bdf6b76eb86c3d6a4a0bb84ab26beb13e84526b1
Author: Tzu-Li (Gordon) Tai
Date: 2016-10-24T05:24:10Z
[FLINK-4280] Add IT tests for explicit start position configuration
commit d8f5f976ef2e2e1d994a45468d7e9ef3b8bf0015
Author: Tzu-Li (Gordon) Tai
Date: 2016-10-24T08:57:05Z
[FLINK-4280] Add documentation for the new explicit start position methods
commit 098360fc797b78f15917aaf1b22d09c06a4d0a6c
Author: Tzu-Li (Gordon) Tai
Date: 2016-10-24T08:08:18Z
[FLINK-3123] [kafka] Allow custom specific start offsets for Kafka consumers
> Allow setting custom start-offsets for the Kafka consumer
> -
>
> Key: FLINK-3123
> URL: https://issues.apache.org/jira/browse/FLINK-3123
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
>Reporter: Robert Metzger
> Fix For: 1.0.0
>
>
> Currently, the Kafka consumer only allows to start reading from the earliest
> available offset or the current offset.
> Sometimes, users want to set a specific start offset themselves.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (FLINK-4630) add netty tcp/restful pushed source support
[ https://issues.apache.org/jira/browse/FLINK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger closed FLINK-4630. - Resolution: Won't Fix I've closed the issue here in favor of the issue at Bahir: https://issues.apache.org/jira/browse/BAHIR-72 > add netty tcp/restful pushed source support > --- > > Key: FLINK-4630 > URL: https://issues.apache.org/jira/browse/FLINK-4630 > Project: Flink > Issue Type: New Feature > Components: Streaming Connectors >Reporter: shijinkui > > When source stream get start, listen a provided tcp port, receive stream data > from user data source. > This netty tcp source is keepping alive and end-to-end, that is from business > system to flink worker directly. > Such source service is needed in produce indeed. > describe the source in detail below: > 1.source run as a netty tcp server > 2.user provide a tcp port, if the port is in used, increace the port > number between 1024 to 65535. Source can parallel. > 3.callback the provided url to report the real port to listen > 4.user push streaming data to netty server, then collect the data to flink > link to https://issues.apache.org/jira/browse/BAHIR-72 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-3123) Allow setting custom start-offsets for the Kafka consumer
[ https://issues.apache.org/jira/browse/FLINK-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai updated FLINK-3123: --- Fix Version/s: 1.2.0 > Allow setting custom start-offsets for the Kafka consumer > - > > Key: FLINK-3123 > URL: https://issues.apache.org/jira/browse/FLINK-3123 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Reporter: Robert Metzger >Assignee: Tzu-Li (Gordon) Tai > Fix For: 1.0.0, 1.2.0 > > > Currently, the Kafka consumer only allows to start reading from the earliest > available offset or the current offset. > Sometimes, users want to set a specific start offset themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (FLINK-3123) Allow setting custom start-offsets for the Kafka consumer
[ https://issues.apache.org/jira/browse/FLINK-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai reassigned FLINK-3123: -- Assignee: Tzu-Li (Gordon) Tai > Allow setting custom start-offsets for the Kafka consumer > - > > Key: FLINK-3123 > URL: https://issues.apache.org/jira/browse/FLINK-3123 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Reporter: Robert Metzger >Assignee: Tzu-Li (Gordon) Tai > Fix For: 1.0.0, 1.2.0 > > > Currently, the Kafka consumer only allows to start reading from the earliest > available offset or the current offset. > Sometimes, users want to set a specific start offset themselves. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4469) Add support for user defined table function in Table API & SQL
[
https://issues.apache.org/jira/browse/FLINK-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601682#comment-15601682
]
Shaoxuan Wang commented on FLINK-4469:
--
Thanks for checking this, Fabian.
I replied this thread earlier by email, but just realised that my email has not
been synchronised to this Jira. Yes, we already have an implementation which
leverages LATERAL TABLE for SQL.
Attached my early email:
"Jark, Giving this for the second thought, I feel we can just use LATERAL
TABLE (which is already supported by Calcite) instead of introducing
"CROSS/OUTER APPLY" . Let us sync offline and see how to revise the current PR."
> Add support for user defined table function in Table API & SQL
> --
>
> Key: FLINK-4469
> URL: https://issues.apache.org/jira/browse/FLINK-4469
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Jark Wu
>Assignee: Jark Wu
>
> Normal user-defined functions, such as concat(), take in a single input row
> and output a single output row. In contrast, table-generating functions
> transform a single input row to multiple output rows. It is very useful in
> some cases, such as look up in HBase by rowkey and return one or more rows.
> Adding a user defined table function should:
> 1. inherit from UDTF class with specific generic type T
> 2. define one or more evel function.
> NOTE:
> 1. the eval method must be public and non-static.
> 2. eval should always return java.lang.Iterable or scala.collection.Iterable
> with the generic type T.
> 3. the generic type T is the row type returned by table function. Because of
> Java type erasure, we can’t extract T from the Iterable.
> 4. eval method can be overload. Blink will choose the best match eval method
> to call according to parameter types and number.
> {code}
> public class Word {
> public String word;
> public Integer length;
> }
> public class SplitStringUDTF extends UDTF {
> public Iterable eval(String str) {
> if (str == null) {
> return new ArrayList<>();
> } else {
> List list = new ArrayList<>();
> for (String s : str.split(",")) {
> Word word = new Word(s, s.length());
> list.add(word);
> }
> return list;
> }
> }
> }
> // in SQL
> tableEnv.registerFunction("split", new SplitStringUDTF())
> tableEnv.sql("SELECT a, b, t.* FROM MyTable CROSS APPLY split(c) AS t(w,l)")
> // in Java Table API
> tableEnv.registerFunction("split", new SplitStringUDTF())
> // rename split table columns to “w” and “l”
> table.crossApply("split(c)", "w, l")
> .select("a, b, w, l")
> // without renaming, we will use the origin field names in the POJO/case/...
> table.crossApply("split(c)")
> .select("a, b, word, length")
> // in Scala Table API
> val split = new SplitStringUDTF()
> table.crossApply(split('c), 'w, 'l)
> .select('a, 'b, 'w, 'l)
> // outerApply for outer join to a UDTF
> table.outerApply(split('c))
> .select('a, 'b, 'word, 'length)
> {code}
> Here we introduce CROSS/OUTER APPLY keywords to join table functions , which
> is used in SQL Server. We can discuss the API in the comment.
> Maybe the {{UDTF}} class should be replaced by {{TableFunction}} or something
> others, because we have introduced {{ScalarFunction}} for custom functions,
> we need to keep consistent. Although, I prefer {{UDTF}} rather than
> {{TableFunction}} as the former is more SQL-like and the latter maybe
> confused with DataStream functions.
> **This issue is blocked by CALCITE-1309, so we need to wait Calcite fix this
> and release.**
> See [1] for more information about UDTF design.
> [1]
> https://docs.google.com/document/d/15iVc1781dxYWm3loVQlESYvMAxEzbbuVFPZWBYuY1Ek/edit#
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4469) Add support for user defined table function in Table API & SQL
[
https://issues.apache.org/jira/browse/FLINK-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601728#comment-15601728
]
Jark Wu commented on FLINK-4469:
Hi [~shaoxuan], Sorry, I didn't notice your reply. I have thought about using
LATERAL TABLE directly. However, actually, the {{OUTER APPLY}} corresponds to :
{code}
SELECT MyTable.*, t.s
FROM MyTable LEFT JOIN LATERAL TABLE(split(MyTable.a)) AS t(s)
ON TRUE
{code}
The {{ON TRUE}} part is necessary for {{LEFT JOIN}}, but it's trivial for
users. That's why I'm introducing "CROSS/OUT APPLY" which will simplify the SQL
a lot.
Hi [~fhueske] , I do not like the parser customization too. This will increase
our maintenance costs when upgrading Calcite. But I haven't found other ways to
do this. I will bring this to Calcite dev mailing list in these days to discuss
whether there is nicer solutions.
> Add support for user defined table function in Table API & SQL
> --
>
> Key: FLINK-4469
> URL: https://issues.apache.org/jira/browse/FLINK-4469
> Project: Flink
> Issue Type: New Feature
> Components: Table API & SQL
>Reporter: Jark Wu
>Assignee: Jark Wu
>
> Normal user-defined functions, such as concat(), take in a single input row
> and output a single output row. In contrast, table-generating functions
> transform a single input row to multiple output rows. It is very useful in
> some cases, such as look up in HBase by rowkey and return one or more rows.
> Adding a user defined table function should:
> 1. inherit from UDTF class with specific generic type T
> 2. define one or more evel function.
> NOTE:
> 1. the eval method must be public and non-static.
> 2. eval should always return java.lang.Iterable or scala.collection.Iterable
> with the generic type T.
> 3. the generic type T is the row type returned by table function. Because of
> Java type erasure, we can’t extract T from the Iterable.
> 4. eval method can be overload. Blink will choose the best match eval method
> to call according to parameter types and number.
> {code}
> public class Word {
> public String word;
> public Integer length;
> }
> public class SplitStringUDTF extends UDTF {
> public Iterable eval(String str) {
> if (str == null) {
> return new ArrayList<>();
> } else {
> List list = new ArrayList<>();
> for (String s : str.split(",")) {
> Word word = new Word(s, s.length());
> list.add(word);
> }
> return list;
> }
> }
> }
> // in SQL
> tableEnv.registerFunction("split", new SplitStringUDTF())
> tableEnv.sql("SELECT a, b, t.* FROM MyTable CROSS APPLY split(c) AS t(w,l)")
> // in Java Table API
> tableEnv.registerFunction("split", new SplitStringUDTF())
> // rename split table columns to “w” and “l”
> table.crossApply("split(c)", "w, l")
> .select("a, b, w, l")
> // without renaming, we will use the origin field names in the POJO/case/...
> table.crossApply("split(c)")
> .select("a, b, word, length")
> // in Scala Table API
> val split = new SplitStringUDTF()
> table.crossApply(split('c), 'w, 'l)
> .select('a, 'b, 'w, 'l)
> // outerApply for outer join to a UDTF
> table.outerApply(split('c))
> .select('a, 'b, 'word, 'length)
> {code}
> Here we introduce CROSS/OUTER APPLY keywords to join table functions , which
> is used in SQL Server. We can discuss the API in the comment.
> Maybe the {{UDTF}} class should be replaced by {{TableFunction}} or something
> others, because we have introduced {{ScalarFunction}} for custom functions,
> we need to keep consistent. Although, I prefer {{UDTF}} rather than
> {{TableFunction}} as the former is more SQL-like and the latter maybe
> confused with DataStream functions.
> **This issue is blocked by CALCITE-1309, so we need to wait Calcite fix this
> and release.**
> See [1] for more information about UDTF design.
> [1]
> https://docs.google.com/document/d/15iVc1781dxYWm3loVQlESYvMAxEzbbuVFPZWBYuY1Ek/edit#
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-3030) Enhance Dashboard to show Execution Attempts
[ https://issues.apache.org/jira/browse/FLINK-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601802#comment-15601802 ] ASF GitHub Bot commented on FLINK-3030: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2448 Okay, I'll run gulp when merging. > Enhance Dashboard to show Execution Attempts > > > Key: FLINK-3030 > URL: https://issues.apache.org/jira/browse/FLINK-3030 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Affects Versions: 0.10.0 >Reporter: Stephan Ewen >Assignee: Ivan Mushketyk > Fix For: 1.0.0 > > > Currently, the web dashboard shows only the latest execution attempt. We > should make all execution attempts and their accumulators available for > inspection. > The REST monitoring API supports this, so it should be a change only to the > frontend part. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4762) Use plural in time interval units
[
https://issues.apache.org/jira/browse/FLINK-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601800#comment-15601800
]
ASF GitHub Bot commented on FLINK-4762:
---
GitHub user wuchong opened a pull request:
https://github.com/apache/flink/pull/2688
[FLINK-4762] [table] Use plural in time interval units
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
Instead of renaming time interval units, I would like to add the plural
time interval units. This will not break APIs, and I think `1.hour +
10.minutes` reads better. The documentation has been updated too.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wuchong/flink time-plural-FLINK-4762
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2688.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2688
commit 3e2087b55ed6696d835712da4525b424997ca231
Author: Jark Wu
Date: 2016-10-24T09:48:53Z
[FLINK-4762] [table] Use plural in time interval units
> Use plural in time interval units
> -
>
> Key: FLINK-4762
> URL: https://issues.apache.org/jira/browse/FLINK-4762
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Jark Wu
>
> During the creation of FLIP-11 we decided to rename the time interval units.
> From {{minute}} to {{minutes}} and so on in Java and Scala Table API.
> {{12.minutes + 2.hours}} reads better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2448: [FLINK-3030][web frontend] Enhance dashboard to show exec...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2448 Okay, I'll run gulp when merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2688: [FLINK-4762] [table] Use plural in time interval u...
GitHub user wuchong opened a pull request:
https://github.com/apache/flink/pull/2688
[FLINK-4762] [table] Use plural in time interval units
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
Instead of renaming time interval units, I would like to add the plural
time interval units. This will not break APIs, and I think `1.hour +
10.minutes` reads better. The documentation has been updated too.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/wuchong/flink time-plural-FLINK-4762
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2688.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2688
commit 3e2087b55ed6696d835712da4525b424997ca231
Author: Jark Wu
Date: 2016-10-24T09:48:53Z
[FLINK-4762] [table] Use plural in time interval units
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-4623) Create Physical Execution Plan of a DataStream
[
https://issues.apache.org/jira/browse/FLINK-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601846#comment-15601846
]
Anton Solovev commented on FLINK-4623:
--
Do I need to create special getSqlExecutionPlan method for stream explain
org.apache.flink.api.table.explain.PlanJsonParser ?
> Create Physical Execution Plan of a DataStream
> --
>
> Key: FLINK-4623
> URL: https://issues.apache.org/jira/browse/FLINK-4623
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Anton Solovev
> Labels: starter
>
> The {{StreamTableEnvironment#explain(Table)}} command for tables of a
> {{StreamTableEnvironment}} only outputs the abstract syntax tree. It would be
> helpful if the {{explain}} method could also generate a string from the
> {{DataStream}} containing a physical execution plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (FLINK-4623) Create Physical Execution Plan of a DataStream
[
https://issues.apache.org/jira/browse/FLINK-4623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601846#comment-15601846
]
Anton Solovev edited comment on FLINK-4623 at 10/24/16 12:31 PM:
-
Do I need to create special getSqlExecutionPlan() method for stream explain in
org.apache.flink.api.table.explain.PlanJsonParser ?
was (Author: tonycox):
Do I need to create special getSqlExecutionPlan method for stream explain
org.apache.flink.api.table.explain.PlanJsonParser ?
> Create Physical Execution Plan of a DataStream
> --
>
> Key: FLINK-4623
> URL: https://issues.apache.org/jira/browse/FLINK-4623
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Anton Solovev
> Labels: starter
>
> The {{StreamTableEnvironment#explain(Table)}} command for tables of a
> {{StreamTableEnvironment}} only outputs the abstract syntax tree. It would be
> helpful if the {{explain}} method could also generate a string from the
> {{DataStream}} containing a physical execution plan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4864) Shade Calcite dependency in flink-table
[ https://issues.apache.org/jira/browse/FLINK-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601887#comment-15601887 ] ASF GitHub Bot commented on FLINK-4864: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2673 I tried reproducing the issue reported by @fhueske. The only issue I got was `Caused by: org.codehaus.commons.compiler.CompileException: Line 8, Column 13: Class "org.apache.calcite.rel.metadata.MetadataDef" not found`. I think that'll be fixed with CALCITE-1461. I guess @fhueske's issue is something local with the development environment. > Shade Calcite dependency in flink-table > --- > > Key: FLINK-4864 > URL: https://issues.apache.org/jira/browse/FLINK-4864 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Affects Versions: 1.2.0 >Reporter: Fabian Hueske >Assignee: Jark Wu > > The Table API has a dependency on Apache Calcite. > A user reported to have version conflicts when having a own Calcite > dependency in the classpath. > The solution would be to shade away the Calcite dependency (Calcite's > transitive dependencies are already shaded). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2673: [FLINK-4864] [table] Shade Calcite dependency in flink-ta...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2673 I tried reproducing the issue reported by @fhueske. The only issue I got was `Caused by: org.codehaus.commons.compiler.CompileException: Line 8, Column 13: Class "org.apache.calcite.rel.metadata.MetadataDef" not found`. I think that'll be fixed with CALCITE-1461. I guess @fhueske's issue is something local with the development environment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink issue #2673: [FLINK-4864] [table] Shade Calcite dependency in flink-ta...
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2673 Thanks for looking into this @rmetzger. I'll check again on my setup. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4864) Shade Calcite dependency in flink-table
[ https://issues.apache.org/jira/browse/FLINK-4864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601890#comment-15601890 ] ASF GitHub Bot commented on FLINK-4864: --- Github user fhueske commented on the issue: https://github.com/apache/flink/pull/2673 Thanks for looking into this @rmetzger. I'll check again on my setup. > Shade Calcite dependency in flink-table > --- > > Key: FLINK-4864 > URL: https://issues.apache.org/jira/browse/FLINK-4864 > Project: Flink > Issue Type: Improvement > Components: Table API & SQL >Affects Versions: 1.2.0 >Reporter: Fabian Hueske >Assignee: Jark Wu > > The Table API has a dependency on Apache Calcite. > A user reported to have version conflicts when having a own Calcite > dependency in the classpath. > The solution would be to shade away the Calcite dependency (Calcite's > transitive dependencies are already shaded). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2650: [FLINK-4563] [metrics] scope caching not adjusted for mul...
Github user ex00 commented on the issue: https://github.com/apache/flink/pull/2650 Hello. I pushed new changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4563) [metrics] scope caching not adjusted for multiple reporters
[ https://issues.apache.org/jira/browse/FLINK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601894#comment-15601894 ] ASF GitHub Bot commented on FLINK-4563: --- Github user ex00 commented on the issue: https://github.com/apache/flink/pull/2650 Hello. I pushed new changes. > [metrics] scope caching not adjusted for multiple reporters > --- > > Key: FLINK-4563 > URL: https://issues.apache.org/jira/browse/FLINK-4563 > Project: Flink > Issue Type: Bug > Components: Metrics >Affects Versions: 1.1.0 >Reporter: Chesnay Schepler >Assignee: Anton Mushin > > Every metric group contains a scope string, representing what entities > (job/task/etc.) a given metric belongs to, which is calculated on demand. > Before this string is cached a CharacterFilter is applied to it, which is > provided by the callee, usually a reporter. This was done since different > reporters have different requirements in regards to valid characters. The > filtered string is cached so that we don't have to refilter the string every > time. > This all works fine with a single reporter; with multiple however it is > completely broken as only the first filter is ever applied. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2487: [FLINK-4520][flink-siddhi] Integrate Siddhi as a light-we...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2487 Hi @haoch, thanks alot for this contribution. I recently started moving some of the streaming connectors of Flink to Apache Bahir, a community for extensions to Spark, Flink (and maybe others). You wrote in an earlier comment: > I think it's both ok to keep this in the core or as an separated project, but the concern is it maybe better for community development to centralize qualified libraries togather. I think Bahir is addressing this issue nicely. So far we added only streaming connectors to Bahir, but I would like to see libraries and other things build on top of Flink there as well. I'm a committer at Bahir and can help you to get the code in there. The Bahir repository is located here https://github.com/apache/bahir-flink By the way, the tests you've added are failing on our CI system. Can you look into it? https://s3.amazonaws.com/archive.travis-ci.org/jobs/166483919/log.txt --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4520) Integrate Siddhi as a lightweight CEP Library
[
https://issues.apache.org/jira/browse/FLINK-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601918#comment-15601918
]
ASF GitHub Bot commented on FLINK-4520:
---
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/2487
Hi @haoch, thanks alot for this contribution.
I recently started moving some of the streaming connectors of Flink to
Apache Bahir, a community for extensions to Spark, Flink (and maybe others).
You wrote in an earlier comment:
> I think it's both ok to keep this in the core or as an separated
project, but the concern is it maybe better for community development to
centralize qualified libraries togather.
I think Bahir is addressing this issue nicely. So far we added only
streaming connectors to Bahir, but I would like to see libraries and other
things build on top of Flink there as well.
I'm a committer at Bahir and can help you to get the code in there.
The Bahir repository is located here https://github.com/apache/bahir-flink
By the way, the tests you've added are failing on our CI system. Can you
look into it?
https://s3.amazonaws.com/archive.travis-ci.org/jobs/166483919/log.txt
> Integrate Siddhi as a lightweight CEP Library
> -
>
> Key: FLINK-4520
> URL: https://issues.apache.org/jira/browse/FLINK-4520
> Project: Flink
> Issue Type: New Feature
> Components: CEP
>Affects Versions: 1.2.0
>Reporter: Hao Chen
>Assignee: Hao Chen
> Labels: cep, library, patch-available
> Fix For: 1.2.0
>
>
> h1. flink-siddhi proposal
> h2. Abstraction
> Siddhi CEP is a lightweight and easy-to-use Open Source Complex Event
> Processing Engine (CEP) released as a Java Library under `Apache Software
> License v2.0`. Siddhi CEP processes events which are generated by various
> event sources, analyses them and notifies appropriate complex events
> according to the user specified queries.
> It would be very helpful for flink users (especially streaming application
> developer) to provide a library to run Siddhi CEP query directly in Flink
> streaming application.
> * http://wso2.com/products/complex-event-processor/
> * https://github.com/wso2/siddhi
> h2. Features
> * Integrate Siddhi CEP as an stream operator (i.e.
> `TupleStreamSiddhiOperator`), supporting rich CEP features like
> * Filter
> * Join
> * Aggregation
> * Group by
> * Having
> * Window
> * Conditions and Expressions
> * Pattern processing
> * Sequence processing
> * Event Tables
> ...
> * Provide easy-to-use Siddhi CEP API to integrate Flink DataStream API (See
> `SiddhiCEP` and `SiddhiStream`)
> * Register Flink DataStream associating native type information with
> Siddhi Stream Schema, supporting POJO,Tuple, Primitive Type, etc.
> * Connect with single or multiple Flink DataStreams with Siddhi CEP
> Execution Plan
> * Return output stream as DataStream with type intelligently inferred
> from Siddhi Stream Schema
> * Integrate siddhi runtime state management with Flink state (See
> `AbstractSiddhiOperator`)
> * Support siddhi plugin management to extend CEP functions. (See
> `SiddhiCEP#registerExtension`)
> h2. Test Cases
> * org.apache.flink.contrib.siddhi.SiddhiCEPITCase:
> https://github.com/haoch/flink/blob/FLINK-4520/flink-contrib/flink-siddhi/src/test/java/org/apache/flink/contrib/siddhi/SiddhiCEPITCase.java
> h2. Example
> {code}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> SiddhiCEP cep = SiddhiCEP.getSiddhiEnvironment(env);
> cep.registerExtension("custom:plus",CustomPlusFunctionExtension.class);
> cep.registerStream("inputStream1", input1, "id", "name",
> "price","timestamp");
> cep.registerStream("inputStream2", input2, "id", "name",
> "price","timestamp");
> DataStream> output = cep
> .from("inputStream1").union("inputStream2")
> .sql(
> "from every s1 = inputStream1[id == 2] "
> + " -> s2 = inputStream2[id == 3] "
> + "select s1.id as id_1, s1.name as name_1, s2.id as id_2, s2.name as
> name_2 , custom:plus(s1.price,s2.price) as price"
> + "insert into outputStream"
> )
> .returns("outputStream");
> env.execute();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4832) Count/Sum 0 elements
[
https://issues.apache.org/jira/browse/FLINK-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601955#comment-15601955
]
Aljoscha Krettek commented on FLINK-4832:
-
Counting {{0}} elements using a union with a dummy would only be possible on a
global aggregation, not on a keyed aggregation. For the keyed aggregation you
would have to insert a dummy for every key but what is "every key".
> Count/Sum 0 elements
>
>
> Key: FLINK-4832
> URL: https://issues.apache.org/jira/browse/FLINK-4832
> Project: Flink
> Issue Type: Improvement
> Components: Table API & SQL
>Reporter: Timo Walther
>
> Currently, the Table API is unable to count or sum up 0 elements. We should
> improve DataSet aggregations for this. Maybe by union the original DataSet
> with a dummy record or by using a MapPartition function. Coming up with a
> good design for this is also part of this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (FLINK-4894) Don't block on buffer request after broadcastEvent
[ https://issues.apache.org/jira/browse/FLINK-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ufuk Celebi reassigned FLINK-4894: -- Assignee: Ufuk Celebi > Don't block on buffer request after broadcastEvent > --- > > Key: FLINK-4894 > URL: https://issues.apache.org/jira/browse/FLINK-4894 > Project: Flink > Issue Type: Improvement >Reporter: Ufuk Celebi >Assignee: Ufuk Celebi > > After broadcasting an event (like the checkpoint barrier), the record writer > might block on a buffer request although that buffer will only be needed on > the next write on that channel. > Instead of assuming that each serializer has a buffer set, we can change the > logic in the writer to request the buffer when it requires one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (FLINK-4895) Drop support for Hadoop 1
Robert Metzger created FLINK-4895: - Summary: Drop support for Hadoop 1 Key: FLINK-4895 URL: https://issues.apache.org/jira/browse/FLINK-4895 Project: Flink Issue Type: Task Components: Build System Affects Versions: 1.2.0 Reporter: Robert Metzger As per this mailing list discussion: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-Hadoop-1-support-with-Flink-1-2-td9530.html the community agreed to drop support for Hadoop 1. The task includes - removing the hadoop-1 / hadoop-2 build profiles, - removing the scripts for generating hadoop-x poms - updating the release script - updating the nightly build script - updating the travis configuration file - updating the documentation - updating the website -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2491) Operators are not participating in state checkpointing in some cases
[
https://issues.apache.org/jira/browse/FLINK-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601982#comment-15601982
]
Aljoscha Krettek commented on FLINK-2491:
-
What source are you using, [~Revy1313]?
> Operators are not participating in state checkpointing in some cases
>
>
> Key: FLINK-2491
> URL: https://issues.apache.org/jira/browse/FLINK-2491
> Project: Flink
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 0.10.0
>Reporter: Robert Metzger
>Assignee: Márton Balassi
>Priority: Critical
> Fix For: 1.0.0
>
>
> While implementing a test case for the Kafka Consumer, I came across the
> following bug:
> Consider the following topology, with the operator parallelism in parentheses:
> Source (2) --> Sink (1).
> In this setup, the {{snapshotState()}} method is called on the source, but
> not on the Sink.
> The sink receives the generated data.
> The only one of the two sources is generating data.
> I've implemented a test case for this, you can find it here:
> https://github.com/rmetzger/flink/blob/para_checkpoint_bug/flink-tests/src/test/java/org/apache/flink/test/checkpointing/ParallelismChangeCheckpoinedITCase.java
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4888) instantiated job manager metrics missing important job statistics
[ https://issues.apache.org/jira/browse/FLINK-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15601991#comment-15601991 ] Robert Metzger commented on FLINK-4888: --- Thank you for working on this. I gave you "Contributor" permissions in our JIRA so that you can assign issues yourself. (I've already assigned this one to you) > instantiated job manager metrics missing important job statistics > -- > > Key: FLINK-4888 > URL: https://issues.apache.org/jira/browse/FLINK-4888 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.1.2 >Reporter: Philipp von dem Bussche >Assignee: Philipp von dem Bussche >Priority: Minor > > A jobmanager is currently (only) instantiated with the following metrics: > taskSlotsAvailable, taskSlotsTotal, numRegisteredTaskManagers and > numRunningJobs. Important other metrics would be numFailedJobs, > numCancelledJobs and numFinishedJobs. Also to get parity between JobManager > metrics and whats available via the REST API it would be good to have these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (FLINK-4888) instantiated job manager metrics missing important job statistics
[ https://issues.apache.org/jira/browse/FLINK-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-4888: -- Assignee: Philipp von dem Bussche > instantiated job manager metrics missing important job statistics > -- > > Key: FLINK-4888 > URL: https://issues.apache.org/jira/browse/FLINK-4888 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.1.2 >Reporter: Philipp von dem Bussche >Assignee: Philipp von dem Bussche >Priority: Minor > > A jobmanager is currently (only) instantiated with the following metrics: > taskSlotsAvailable, taskSlotsTotal, numRegisteredTaskManagers and > numRunningJobs. Important other metrics would be numFailedJobs, > numCancelledJobs and numFinishedJobs. Also to get parity between JobManager > metrics and whats available via the REST API it would be good to have these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-4852) ClassCastException when assigning Watermarks with TimeCharacteristic.ProcessingTime
[
https://issues.apache.org/jira/browse/FLINK-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-4852.
---
> ClassCastException when assigning Watermarks with
> TimeCharacteristic.ProcessingTime
> ---
>
> Key: FLINK-4852
> URL: https://issues.apache.org/jira/browse/FLINK-4852
> Project: Flink
> Issue Type: Bug
> Components: Streaming
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Maximilian Michels
>Assignee: Aljoscha Krettek
> Fix For: 1.2.0
>
>
> As per FLINK-3688 and FLINK-2936 this should already been resolved. Still,
> when emitting Watermarks and using processing time, you get the following
> ClassCastException:
> {noformat}
> Exception in thread "main"
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply$mcV$sp(JobManager.scala:822)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:768)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:768)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.RuntimeException: Could not forward element to next
> operator
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:376)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:358)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:346)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:329)
> at
> org.apache.flink.streaming.api.operators.StreamSource$NonTimestampContext.collect(StreamSource.java:161)
> at
> org.apache.flink.streaming.api.functions.source.StatefulSequenceSource.run(StatefulSequenceSource.java:68)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:80)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:53)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:56)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:266)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:585)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
> Could not forward element to next operator
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.emitWatermark(OperatorChain.java:343)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.emitWatermark(AbstractStreamOperator.java:340)
> at
> org.apache.flink.streaming.runtime.operators.TimestampsAndPunctuatedWatermarksOperator.processElement(TimestampsAndPunctuatedWatermarksOperator.java:58)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:373)
> ... 11 more
> Caused by: java.lang.RuntimeException:
> org.apache.flink.streaming.api.watermark.Watermark cannot be cast to
> org.apache.flink.streaming.runtime.streamrecord.StreamRecord
> at
> org.apache.flink.streaming.runtime.io.RecordWriterOutput.emitWatermark(RecordWriterOutput.java:93)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.emitWatermark(AbstractStreamOperator.java:340)
> at
> org.apache.flink.streaming.api.operators.StreamMap.processWatermark(StreamMap.java:44)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain$ChainingOutput.emitWatermark(OperatorChain.java:340)
> ... 14
[jira] [Closed] (FLINK-3674) Add an interface for Time aware User Functions
[
https://issues.apache.org/jira/browse/FLINK-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-3674.
---
> Add an interface for Time aware User Functions
> --
>
> Key: FLINK-3674
> URL: https://issues.apache.org/jira/browse/FLINK-3674
> Project: Flink
> Issue Type: New Feature
> Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>Assignee: Aljoscha Krettek
> Fix For: 1.2.0
>
>
> I suggest to add an interface that UDFs can implement, which will let them be
> notified upon watermark updates.
> Example usage:
> {code}
> public interface EventTimeFunction {
> void onWatermark(Watermark watermark);
> }
> public class MyMapper implements MapFunction,
> EventTimeFunction {
> private long currentEventTime = Long.MIN_VALUE;
> public String map(String value) {
> return value + " @ " + currentEventTime;
> }
> public void onWatermark(Watermark watermark) {
> currentEventTime = watermark.getTimestamp();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4853) Clean up JobManager registration at the ResourceManager
[
https://issues.apache.org/jira/browse/FLINK-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602007#comment-15602007
]
ASF GitHub Bot commented on FLINK-4853:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2657#discussion_r84689078
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
---
@@ -202,101 +205,125 @@ public void shutDown() throws Exception {
// RPC methods
//
- /**
-* Register a {@link JobMaster} at the resource manager.
-*
-* @param resourceManagerLeaderId The fencing token for the
ResourceManager leader
-* @param jobMasterAddressThe address of the JobMaster that
registers
-* @param jobID The Job ID of the JobMaster that
registers
-* @return Future registration response
-*/
@RpcMethod
- public Future registerJobMaster(
- final UUID resourceManagerLeaderId, final UUID
jobMasterLeaderId,
- final String jobMasterAddress, final JobID jobID) {
+ public Future registerJobManager(
+ final UUID resourceManagerLeaderId,
+ final UUID jobManagerLeaderId,
+ final String jobManagerAddress,
+ final JobID jobId) {
+
+ checkNotNull(resourceManagerLeaderId);
+ checkNotNull(jobManagerLeaderId);
+ checkNotNull(jobManagerAddress);
+ checkNotNull(jobId);
+
+ if (isValid(resourceManagerLeaderId)) {
+ if (!jobLeaderIdService.containsJob(jobId)) {
+ try {
+ jobLeaderIdService.addJob(jobId);
+ } catch (Exception e) {
+ // This should actually never happen
because, it should always be possible to add a new job
+ ResourceManagerException exception =
new ResourceManagerException("Could not add the job " +
+ jobId + " to the job id leader
service. This should never happen.", e);
+
+ onFatalErrorAsync(exception);
+
+ log.debug("Could not add job {} to job
leader id service.", jobId, e);
+ return
FlinkCompletableFuture.completedExceptionally(exception);
+ }
+ }
- checkNotNull(jobMasterAddress);
- checkNotNull(jobID);
+ log.info("Registering job manager {}@{} for job {}.",
jobManagerLeaderId, jobManagerAddress, jobId);
+
+ Future jobLeaderIdFuture;
- // create a leader retriever in case it doesn't exist
- final JobIdLeaderListener jobIdLeaderListener;
- if (leaderListeners.containsKey(jobID)) {
- jobIdLeaderListener = leaderListeners.get(jobID);
- } else {
try {
- LeaderRetrievalService jobMasterLeaderRetriever
=
-
highAvailabilityServices.getJobManagerLeaderRetriever(jobID);
- jobIdLeaderListener = new
JobIdLeaderListener(jobID, jobMasterLeaderRetriever);
+ jobLeaderIdFuture =
jobLeaderIdService.getLeaderId(jobId);
} catch (Exception e) {
- log.warn("Failed to start
JobMasterLeaderRetriever for job id {}", jobID, e);
+ // we cannot check the job leader id so let's
fail
+ // TODO: Maybe it's also ok to skip this check
in case that we cannot check the leader id
+ ResourceManagerException exception = new
ResourceManagerException("Cannot obtain the " +
+ "job leader id future to verify the
correct job leader.", e);
+
+ onFatalErrorAsync(exception);
- return
FlinkCompletableFuture.completed(
- new
RegistrationResponse.Decline("Failed to retrieve JobMasterLeaderRetriever"));
+ log.debug("Could not obtain the job leader id
future to verify the correct job leader.");
+ return
FlinkCompletableFuture.completedExceptionally(exception);
}
- leaderListeners.put(jobID, jobIdLeaderListener);
-
[GitHub] flink issue #2671: [FLINK-4862] fix Timer register in ContinuousEventTimeTri...
Github user mxm commented on the issue: https://github.com/apache/flink/pull/2671 Thanks for having a look @aljoscha. Thanks again for the PR @manuzhang. Merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request #2683: [FLINK-4888][metrics] instantiated job manager met...
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/2683#discussion_r84689406
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -1828,6 +1828,33 @@ class JobManager(
jobManagerMetricGroup.gauge[Long, Gauge[Long]]("numRunningJobs", new
Gauge[Long] {
override def getValue: Long = JobManager.this.currentJobs.size
})
+jobManagerMetricGroup.gauge[Long, Gauge[Long]]("numFailedJobs", new
Gauge[Long] {
+ override def getValue: Long = {
+ var failedJobs = 0
+ val ourJobs = createJobStatusOverview()
+ val future = (archive ?
RequestJobsOverview.getInstance())(timeout)
+ val archivedJobs : JobsOverview = Await.result(future,
timeout).asInstanceOf[JobsOverview]
+ failedJobs += ourJobs.getNumJobsFailed() +
archivedJobs.getNumJobsFailed()
+ failedJobs
+}})
+jobManagerMetricGroup.gauge[Long, Gauge[Long]]("numCancelledJobs", new
Gauge[Long] {
+ override def getValue: Long = {
+ var cancelledJobs = 0
+ val ourJobs = createJobStatusOverview()
+ val future = (archive ?
RequestJobsOverview.getInstance())(timeout)
+ val archivedJobs : JobsOverview = Await.result(future,
timeout).asInstanceOf[JobsOverview]
+ cancelledJobs += ourJobs.getNumJobsCancelled() +
archivedJobs.getNumJobsCancelled()
+ cancelledJobs
+}})
+jobManagerMetricGroup.gauge[Long, Gauge[Long]]("numFinishedJobs", new
Gauge[Long] {
+ override def getValue: Long = {
+ var finishedJobs = 0
+ val ourJobs = createJobStatusOverview()
+ val future = (archive ?
RequestJobsOverview.getInstance())(timeout)
+ val archivedJobs : JobsOverview = Await.result(future,
timeout).asInstanceOf[JobsOverview]
+ finishedJobs += ourJobs.getNumJobsFinished() +
archivedJobs.getNumJobsFinished()
+ finishedJobs
+}})
--- End diff --
@zentol What is your take on this change?
I'm uncertain if doing RPC calls in gauges is a good idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-4862) NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger
[
https://issues.apache.org/jira/browse/FLINK-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602012#comment-15602012
]
ASF GitHub Bot commented on FLINK-4862:
---
Github user mxm commented on the issue:
https://github.com/apache/flink/pull/2671
Thanks for having a look @aljoscha. Thanks again for the PR @manuzhang.
Merging.
> NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger
> --
>
> Key: FLINK-4862
> URL: https://issues.apache.org/jira/browse/FLINK-4862
> Project: Flink
> Issue Type: Bug
> Components: Streaming, Windowing Operators
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 1.2.0, 1.1.4
>
>
> h3. what's the error ?
> The following NPE error is thrown when EventTimeSessionWindows with
> ContinuousEventTimeTrigger is used.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.streaming.api.windowing.triggers.ContinuousEventTimeTrigger.clear(ContinuousEventTimeTrigger.java:91)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.clear(WindowOperator.java:768)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$2.merge(WindowOperator.java:310)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$2.merge(WindowOperator.java:297)
> at
> org.apache.flink.streaming.runtime.operators.windowing.MergingWindowSet.addWindow(MergingWindowSet.java:196)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:297)
> at
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:271)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> h3. how to reproduce ?
> use {{ContinuousEventTimeTrigger}} instead of the default
> {{EventTimeTrigger}} in [SessionWindowing |
> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/SessionWindowing.java#L84]
> example.
> h3. what's the cause ?
> When two session windows are being merged, the states of the two
> {{ContinuousEventTimeTrigger}} are merged as well and the new namespace is
> the merged window. Later when the context tries to delete {{Timer}} from the
> old trigger and looks up the timestamp by the old namespace, null value is
> returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4888) instantiated job manager metrics missing important job statistics
[
https://issues.apache.org/jira/browse/FLINK-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602014#comment-15602014
]
ASF GitHub Bot commented on FLINK-4888:
---
Github user rmetzger commented on a diff in the pull request:
https://github.com/apache/flink/pull/2683#discussion_r84689406
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
---
@@ -1828,6 +1828,33 @@ class JobManager(
jobManagerMetricGroup.gauge[Long, Gauge[Long]]("numRunningJobs", new
Gauge[Long] {
override def getValue: Long = JobManager.this.currentJobs.size
})
+jobManagerMetricGroup.gauge[Long, Gauge[Long]]("numFailedJobs", new
Gauge[Long] {
+ override def getValue: Long = {
+ var failedJobs = 0
+ val ourJobs = createJobStatusOverview()
+ val future = (archive ?
RequestJobsOverview.getInstance())(timeout)
+ val archivedJobs : JobsOverview = Await.result(future,
timeout).asInstanceOf[JobsOverview]
+ failedJobs += ourJobs.getNumJobsFailed() +
archivedJobs.getNumJobsFailed()
+ failedJobs
+}})
+jobManagerMetricGroup.gauge[Long, Gauge[Long]]("numCancelledJobs", new
Gauge[Long] {
+ override def getValue: Long = {
+ var cancelledJobs = 0
+ val ourJobs = createJobStatusOverview()
+ val future = (archive ?
RequestJobsOverview.getInstance())(timeout)
+ val archivedJobs : JobsOverview = Await.result(future,
timeout).asInstanceOf[JobsOverview]
+ cancelledJobs += ourJobs.getNumJobsCancelled() +
archivedJobs.getNumJobsCancelled()
+ cancelledJobs
+}})
+jobManagerMetricGroup.gauge[Long, Gauge[Long]]("numFinishedJobs", new
Gauge[Long] {
+ override def getValue: Long = {
+ var finishedJobs = 0
+ val ourJobs = createJobStatusOverview()
+ val future = (archive ?
RequestJobsOverview.getInstance())(timeout)
+ val archivedJobs : JobsOverview = Await.result(future,
timeout).asInstanceOf[JobsOverview]
+ finishedJobs += ourJobs.getNumJobsFinished() +
archivedJobs.getNumJobsFinished()
+ finishedJobs
+}})
--- End diff --
@zentol What is your take on this change?
I'm uncertain if doing RPC calls in gauges is a good idea.
> instantiated job manager metrics missing important job statistics
> --
>
> Key: FLINK-4888
> URL: https://issues.apache.org/jira/browse/FLINK-4888
> Project: Flink
> Issue Type: Improvement
> Components: Metrics
>Affects Versions: 1.1.2
>Reporter: Philipp von dem Bussche
>Assignee: Philipp von dem Bussche
>Priority: Minor
>
> A jobmanager is currently (only) instantiated with the following metrics:
> taskSlotsAvailable, taskSlotsTotal, numRegisteredTaskManagers and
> numRunningJobs. Important other metrics would be numFailedJobs,
> numCancelledJobs and numFinishedJobs. Also to get parity between JobManager
> metrics and whats available via the REST API it would be good to have these.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2683: [FLINK-4888][metrics] instantiated job manager metrics mi...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2683 Thank you for opening a pull request. I agree that we should expose all numbers we show in the web interface as a metric as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4888) instantiated job manager metrics missing important job statistics
[ https://issues.apache.org/jira/browse/FLINK-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602015#comment-15602015 ] ASF GitHub Bot commented on FLINK-4888: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2683 Thank you for opening a pull request. I agree that we should expose all numbers we show in the web interface as a metric as well. > instantiated job manager metrics missing important job statistics > -- > > Key: FLINK-4888 > URL: https://issues.apache.org/jira/browse/FLINK-4888 > Project: Flink > Issue Type: Improvement > Components: Metrics >Affects Versions: 1.1.2 >Reporter: Philipp von dem Bussche >Assignee: Philipp von dem Bussche >Priority: Minor > > A jobmanager is currently (only) instantiated with the following metrics: > taskSlotsAvailable, taskSlotsTotal, numRegisteredTaskManagers and > numRunningJobs. Important other metrics would be numFailedJobs, > numCancelledJobs and numFinishedJobs. Also to get parity between JobManager > metrics and whats available via the REST API it would be good to have these. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (FLINK-4866) Make Trigger.clear() Abstract to Enforce Implementation
[ https://issues.apache.org/jira/browse/FLINK-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-4866. --- > Make Trigger.clear() Abstract to Enforce Implementation > --- > > Key: FLINK-4866 > URL: https://issues.apache.org/jira/browse/FLINK-4866 > Project: Flink > Issue Type: Bug > Components: Streaming >Reporter: Aljoscha Krettek >Assignee: Jark Wu > Fix For: 1.2.0 > > > If the method is not abstract implementors of custom triggers will not > realise that it could be necessary and they will likely not clean up their > state/timers properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2671: [FLINK-4862] fix Timer register in ContinuousEvent...
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2671#discussion_r84689568
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers/ContinuousEventTimeTrigger.java
---
@@ -99,8 +111,12 @@ public boolean canMerge() {
}
@Override
- public TriggerResult onMerge(W window, OnMergeContext ctx) {
+ public TriggerResult onMerge(W window, OnMergeContext ctx) throws
Exception {
ctx.mergePartitionedState(stateDesc);
+ Long nextFireTimestamp =
ctx.getPartitionedState(stateDesc).get();
+ if (nextFireTimestamp != null) {
+ ctx.registerEventTimeTimer(nextFireTimestamp);
+ }
--- End diff --
Yes, you're right. It is actually handled correctly in `EventTimeTrigger`
but not for the continuous trigger.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-4862) NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger
[
https://issues.apache.org/jira/browse/FLINK-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602016#comment-15602016
]
ASF GitHub Bot commented on FLINK-4862:
---
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2671#discussion_r84689568
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers/ContinuousEventTimeTrigger.java
---
@@ -99,8 +111,12 @@ public boolean canMerge() {
}
@Override
- public TriggerResult onMerge(W window, OnMergeContext ctx) {
+ public TriggerResult onMerge(W window, OnMergeContext ctx) throws
Exception {
ctx.mergePartitionedState(stateDesc);
+ Long nextFireTimestamp =
ctx.getPartitionedState(stateDesc).get();
+ if (nextFireTimestamp != null) {
+ ctx.registerEventTimeTimer(nextFireTimestamp);
+ }
--- End diff --
Yes, you're right. It is actually handled correctly in `EventTimeTrigger`
but not for the continuous trigger.
> NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger
> --
>
> Key: FLINK-4862
> URL: https://issues.apache.org/jira/browse/FLINK-4862
> Project: Flink
> Issue Type: Bug
> Components: Streaming, Windowing Operators
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 1.2.0, 1.1.4
>
>
> h3. what's the error ?
> The following NPE error is thrown when EventTimeSessionWindows with
> ContinuousEventTimeTrigger is used.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.streaming.api.windowing.triggers.ContinuousEventTimeTrigger.clear(ContinuousEventTimeTrigger.java:91)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.clear(WindowOperator.java:768)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$2.merge(WindowOperator.java:310)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$2.merge(WindowOperator.java:297)
> at
> org.apache.flink.streaming.runtime.operators.windowing.MergingWindowSet.addWindow(MergingWindowSet.java:196)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:297)
> at
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:271)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> h3. how to reproduce ?
> use {{ContinuousEventTimeTrigger}} instead of the default
> {{EventTimeTrigger}} in [SessionWindowing |
> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/SessionWindowing.java#L84]
> example.
> h3. what's the cause ?
> When two session windows are being merged, the states of the two
> {{ContinuousEventTimeTrigger}} are merged as well and the new namespace is
> the merged window. Later when the context tries to delete {{Timer}} from the
> old trigger and looks up the timestamp by the old namespace, null value is
> returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2657: [FLINK-4853] [rm] Harden job manager registration ...
Github user mxm commented on a diff in the pull request:
https://github.com/apache/flink/pull/2657#discussion_r84689078
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
---
@@ -202,101 +205,125 @@ public void shutDown() throws Exception {
// RPC methods
//
- /**
-* Register a {@link JobMaster} at the resource manager.
-*
-* @param resourceManagerLeaderId The fencing token for the
ResourceManager leader
-* @param jobMasterAddressThe address of the JobMaster that
registers
-* @param jobID The Job ID of the JobMaster that
registers
-* @return Future registration response
-*/
@RpcMethod
- public Future registerJobMaster(
- final UUID resourceManagerLeaderId, final UUID
jobMasterLeaderId,
- final String jobMasterAddress, final JobID jobID) {
+ public Future registerJobManager(
+ final UUID resourceManagerLeaderId,
+ final UUID jobManagerLeaderId,
+ final String jobManagerAddress,
+ final JobID jobId) {
+
+ checkNotNull(resourceManagerLeaderId);
+ checkNotNull(jobManagerLeaderId);
+ checkNotNull(jobManagerAddress);
+ checkNotNull(jobId);
+
+ if (isValid(resourceManagerLeaderId)) {
+ if (!jobLeaderIdService.containsJob(jobId)) {
+ try {
+ jobLeaderIdService.addJob(jobId);
+ } catch (Exception e) {
+ // This should actually never happen
because, it should always be possible to add a new job
+ ResourceManagerException exception =
new ResourceManagerException("Could not add the job " +
+ jobId + " to the job id leader
service. This should never happen.", e);
+
+ onFatalErrorAsync(exception);
+
+ log.debug("Could not add job {} to job
leader id service.", jobId, e);
+ return
FlinkCompletableFuture.completedExceptionally(exception);
+ }
+ }
- checkNotNull(jobMasterAddress);
- checkNotNull(jobID);
+ log.info("Registering job manager {}@{} for job {}.",
jobManagerLeaderId, jobManagerAddress, jobId);
+
+ Future jobLeaderIdFuture;
- // create a leader retriever in case it doesn't exist
- final JobIdLeaderListener jobIdLeaderListener;
- if (leaderListeners.containsKey(jobID)) {
- jobIdLeaderListener = leaderListeners.get(jobID);
- } else {
try {
- LeaderRetrievalService jobMasterLeaderRetriever
=
-
highAvailabilityServices.getJobManagerLeaderRetriever(jobID);
- jobIdLeaderListener = new
JobIdLeaderListener(jobID, jobMasterLeaderRetriever);
+ jobLeaderIdFuture =
jobLeaderIdService.getLeaderId(jobId);
} catch (Exception e) {
- log.warn("Failed to start
JobMasterLeaderRetriever for job id {}", jobID, e);
+ // we cannot check the job leader id so let's
fail
+ // TODO: Maybe it's also ok to skip this check
in case that we cannot check the leader id
+ ResourceManagerException exception = new
ResourceManagerException("Cannot obtain the " +
+ "job leader id future to verify the
correct job leader.", e);
+
+ onFatalErrorAsync(exception);
- return
FlinkCompletableFuture.completed(
- new
RegistrationResponse.Decline("Failed to retrieve JobMasterLeaderRetriever"));
+ log.debug("Could not obtain the job leader id
future to verify the correct job leader.");
+ return
FlinkCompletableFuture.completedExceptionally(exception);
}
- leaderListeners.put(jobID, jobIdLeaderListener);
- }
+ Future jobMasterGatewayFuture =
getRpcService().connect(jobManagerAddress, JobMasterGateway.class);
- return getRpcService()
- .execute(new Callable() {
+ Future re
[GitHub] flink issue #2680: [FLINK-4876] Allow web interface to be bound to a specifi...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2680 Thank you for opening a PR for fixing this. +1 to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (FLINK-4876) Allow web interface to be bound to a specific ip/interface/inetHost
[ https://issues.apache.org/jira/browse/FLINK-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-4876: -- Component/s: Webfrontend > Allow web interface to be bound to a specific ip/interface/inetHost > --- > > Key: FLINK-4876 > URL: https://issues.apache.org/jira/browse/FLINK-4876 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Bram Vogelaar >Priority: Minor > > Currently the web interface automatically binds to all interfaces on 0.0.0.0. > IMHO there are some use cases to only bind to a specific ipadress, (e.g. > access through an authenticated proxy, not binding on the management or > backup interface) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4876) Allow web interface to be bound to a specific ip/interface/inetHost
[ https://issues.apache.org/jira/browse/FLINK-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602028#comment-15602028 ] ASF GitHub Bot commented on FLINK-4876: --- Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2680 Thank you for opening a PR for fixing this. +1 to merge. > Allow web interface to be bound to a specific ip/interface/inetHost > --- > > Key: FLINK-4876 > URL: https://issues.apache.org/jira/browse/FLINK-4876 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Affects Versions: 1.2.0, 1.1.2, 1.1.3 >Reporter: Bram Vogelaar >Priority: Minor > > Currently the web interface automatically binds to all interfaces on 0.0.0.0. > IMHO there are some use cases to only bind to a specific ipadress, (e.g. > access through an authenticated proxy, not binding on the management or > backup interface) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-4072) EventTimeWindowCheckpointingITCase.testSlidingTimeWindow fails on Travis
[
https://issues.apache.org/jira/browse/FLINK-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602031#comment-15602031
]
Aljoscha Krettek commented on FLINK-4072:
-
I saw this problem:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/170071654/log.txt
Not a failure but stuck, it seems.
> EventTimeWindowCheckpointingITCase.testSlidingTimeWindow fails on Travis
>
>
> Key: FLINK-4072
> URL: https://issues.apache.org/jira/browse/FLINK-4072
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: test-stability
>
> The test case {{EventTimeWindowCheckpointingITCase.testSlidingTimeWindow}}
> failed on Travis.
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/137498497/log.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-4879) class KafkaTableSource should be public just like KafkaTableSink
[
https://issues.apache.org/jira/browse/FLINK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger resolved FLINK-4879.
---
Resolution: Fixed
Fix Version/s: (was: 1.1.4)
1.2.0
Resolved for 1.2 in http://git-wip-us.apache.org/repos/asf/flink/commit/e3324372
> class KafkaTableSource should be public just like KafkaTableSink
>
>
> Key: FLINK-4879
> URL: https://issues.apache.org/jira/browse/FLINK-4879
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector, Table API & SQL
>Affects Versions: 1.1.1, 1.1.3
>Reporter: yuemeng
>Priority: Minor
> Fix For: 1.2.0
>
> Attachments: 0001-class-KafkaTableSource-should-be-public.patch
>
>
> *class KafkaTableSource should be public just like KafkaTableSink,by
> default,it's modifier is default ，and we cann't access out of it's package*,
> for example:
> {code}
> def createKafkaTableSource(
> topic: String,
> properties: Properties,
> deserializationSchema: DeserializationSchema[Row],
> fieldsNames: Array[String],
> typeInfo: Array[TypeInformation[_]]): KafkaTableSource = {
> if (deserializationSchema != null) {
> new Kafka09TableSource(topic, properties, deserializationSchema,
> fieldsNames, typeInfo)
> } else {
> new Kafka09JsonTableSource(topic, properties, fieldsNames, typeInfo)
> }
> }
> {code}
> Because of the class KafkaTableSource modifier is default,we cann't define
> this function result type with KafkaTableSource ,we must give the specific
> type.
> if some other kafka source extends KafkaTableSource ,and we don't sure which
> subclass of KafkaTableSource should be use,how can we specific the type?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4879) class KafkaTableSource should be public just like KafkaTableSink
[
https://issues.apache.org/jira/browse/FLINK-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602042#comment-15602042
]
ASF GitHub Bot commented on FLINK-4879:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/2678
> class KafkaTableSource should be public just like KafkaTableSink
>
>
> Key: FLINK-4879
> URL: https://issues.apache.org/jira/browse/FLINK-4879
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector, Table API & SQL
>Affects Versions: 1.1.1, 1.1.3
>Reporter: yuemeng
>Priority: Minor
> Fix For: 1.2.0
>
> Attachments: 0001-class-KafkaTableSource-should-be-public.patch
>
>
> *class KafkaTableSource should be public just like KafkaTableSink,by
> default,it's modifier is default ，and we cann't access out of it's package*,
> for example:
> {code}
> def createKafkaTableSource(
> topic: String,
> properties: Properties,
> deserializationSchema: DeserializationSchema[Row],
> fieldsNames: Array[String],
> typeInfo: Array[TypeInformation[_]]): KafkaTableSource = {
> if (deserializationSchema != null) {
> new Kafka09TableSource(topic, properties, deserializationSchema,
> fieldsNames, typeInfo)
> } else {
> new Kafka09JsonTableSource(topic, properties, fieldsNames, typeInfo)
> }
> }
> {code}
> Because of the class KafkaTableSource modifier is default,we cann't define
> this function result type with KafkaTableSource ,we must give the specific
> type.
> if some other kafka source extends KafkaTableSource ,and we don't sure which
> subclass of KafkaTableSource should be use,how can we specific the type?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2654: [hotfix] [kafka] Fix RackAwareMode instantiation in Kafka...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2654 Nice fix, thank you +1 to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (FLINK-4862) NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger
[
https://issues.apache.org/jira/browse/FLINK-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels closed FLINK-4862.
-
Resolution: Fixed
master: 45762162fe7c23fd921db1e0f826b2906bfe1dcd
release-1.1: 05a5f460b33828cc8a1e6a45d37b555facc7133f
> NPE on EventTimeSessionWindows with ContinuousEventTimeTrigger
> --
>
> Key: FLINK-4862
> URL: https://issues.apache.org/jira/browse/FLINK-4862
> Project: Flink
> Issue Type: Bug
> Components: Streaming, Windowing Operators
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Manu Zhang
>Assignee: Manu Zhang
> Fix For: 1.2.0, 1.1.4
>
>
> h3. what's the error ?
> The following NPE error is thrown when EventTimeSessionWindows with
> ContinuousEventTimeTrigger is used.
> {code}
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.streaming.api.windowing.triggers.ContinuousEventTimeTrigger.clear(ContinuousEventTimeTrigger.java:91)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.clear(WindowOperator.java:768)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$2.merge(WindowOperator.java:310)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$2.merge(WindowOperator.java:297)
> at
> org.apache.flink.streaming.runtime.operators.windowing.MergingWindowSet.addWindow(MergingWindowSet.java:196)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:297)
> at
> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:183)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:271)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:609)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> h3. how to reproduce ?
> use {{ContinuousEventTimeTrigger}} instead of the default
> {{EventTimeTrigger}} in [SessionWindowing |
> https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/SessionWindowing.java#L84]
> example.
> h3. what's the cause ?
> When two session windows are being merged, the states of the two
> {{ContinuousEventTimeTrigger}} are merged as well and the new namespace is
> the merged window. Later when the context tries to delete {{Timer}} from the
> old trigger and looks up the timestamp by the old namespace, null value is
> returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2626: [FLINK-4787] [runtime-web] Expose cancel-with-savepoint v...
Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/2626 Looks like the tests are failing. I quickly scrolled over the changes. +1 to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4787) Add REST API call for cancel-with-savepoints
[
https://issues.apache.org/jira/browse/FLINK-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15602060#comment-15602060
]
ASF GitHub Bot commented on FLINK-4787:
---
Github user rmetzger commented on the issue:
https://github.com/apache/flink/pull/2626
Looks like the tests are failing.
I quickly scrolled over the changes. +1 to merge.
> Add REST API call for cancel-with-savepoints
>
>
> Key: FLINK-4787
> URL: https://issues.apache.org/jira/browse/FLINK-4787
> Project: Flink
> Issue Type: Improvement
> Components: Webfrontend
>Reporter: Ufuk Celebi
>
> As a follow up to FLINK-4717, expose the cancel-with-savepoint command via
> the REST API.
> {code}
> /jobs/:jobid/cancel-with-savepoint/
> /jobs/:jobid/cancel-with-savepoint/:targetDirectory
> {code}
> The first command goes to the default savepoint directory, the second one
> uses the given target directory.
> The calls need to be async, as triggering a savepoint can take some time. For
> this, the handlers return a {{201 (Accepted)}} response with the location of
> the status, e.g. {{/jobs/:jobid/cancel-with-savepoint/in-progress/:id}}.
> The user has to check that location until the final savepoint path is
> returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2671: [FLINK-4862] fix Timer register in ContinuousEvent...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2671 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
