[jira] [Commented] (FLINK-8149) Replace usages of deprecated SerializationSchema
[
https://issues.apache.org/jira/browse/FLINK-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272291#comment-16272291
]
ASF GitHub Bot commented on FLINK-8149:
---
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/5069
LGTM, merging this ..
> Replace usages of deprecated SerializationSchema
>
>
> Key: FLINK-8149
> URL: https://issues.apache.org/jira/browse/FLINK-8149
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Affects Versions: 1.4.0
>Reporter: Hai Zhou UTC+8
>Assignee: Hai Zhou UTC+8
> Fix For: 1.5.0
>
>
> The deprecated {{SerializationSchema}} in {{flink-streaming-java}}, has been
> moved to {{flink-core}}.
> But, the deprecate {{SerializationSchema}} is still used in
> {{flink-connector-kinesis}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5069: [FLINK-8149][kinesis] Replace usages of deprecated Serial...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/5069 LGTM, merging this .. ---
[jira] [Comment Edited] (FLINK-8167) Failing WikipediaEditsSourceTest on Travis
[
https://issues.apache.org/jira/browse/FLINK-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272233#comment-16272233
]
Hai Zhou UTC+8 edited comment on FLINK-8167 at 11/30/17 6:59 AM:
-
I agree with "move some low popularity connectors to Apache Bahir". about this,
I mentioned a ticket FLINK-7510 before.
BTW, It is very common for test suites or build scripts to hang. Travis CI has
specific time limits for each job, and will stop. {{A job on travis-ci.org
takes longer than 50 minutes, will be stoped.}}
Currently, the "misc" profile mvn script executes very close to 50 minutes and
occasionally builds timeouts.
Do we have a better way to solve this problem?
was (Author: yew1eb):
I agree with "move some low popularity connectors to Apache Bahir". about this,
I mentioned a ticket FLINK-7510 before.
BTW, It is very common for test suites or build scripts to hang. Travis CI has
specific time limits for each job, and will stop. {{A job on travis-ci.org
takes longer than 50 minutes, will be stoped.}}
Currently, the "misc" profile mvn script executes very close to 50 minutes and
occasionally builds timeouts.
We have no better way to solve this problem?
> Failing WikipediaEditsSourceTest on Travis
> --
>
> Key: FLINK-8167
> URL: https://issues.apache.org/jira/browse/FLINK-8167
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Ufuk Celebi
>Priority: Critical
> Labels: test-stability
> Fix For: 1.5.0
>
>
> I've observed the {{WikipediaEditsSourceTest}} failing on Travis.
> https://travis-ci.org/tillrohrmann/flink/jobs/308351298
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8167) Failing WikipediaEditsSourceTest on Travis
[
https://issues.apache.org/jira/browse/FLINK-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272233#comment-16272233
]
Hai Zhou UTC+8 commented on FLINK-8167:
---
I agree with "move some low popularity connectors to Apache Bahir". about this,
I mentioned a ticket FLINK-7510 before.
BTW, It is very common for test suites or build scripts to hang. Travis CI has
specific time limits for each job, and will stop. {{A job on travis-ci.org
takes longer than 50 minutes, will be stoped.}}
Currently, the "misc" profile mvn script executes very close to 50 minutes and
occasionally builds timeouts.
We have no better way to solve this problem?
> Failing WikipediaEditsSourceTest on Travis
> --
>
> Key: FLINK-8167
> URL: https://issues.apache.org/jira/browse/FLINK-8167
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Ufuk Celebi
>Priority: Critical
> Labels: test-stability
> Fix For: 1.5.0
>
>
> I've observed the {{WikipediaEditsSourceTest}} failing on Travis.
> https://travis-ci.org/tillrohrmann/flink/jobs/308351298
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8158) Rowtime window inner join emits late data
[
https://issues.apache.org/jira/browse/FLINK-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272204#comment-16272204
]
ASF GitHub Bot commented on FLINK-8158:
---
Github user hequn8128 commented on the issue:
https://github.com/apache/flink/pull/5094
Thanks for your replies.
I agree with that it is valid to join late input data.What I concern is the
watermark has not been hold back correctly.
Take `testRowTimeJoinWithCommonBounds2` as an example.
1. Output watermark with timestamp 1000 (hold back with 5000 from 6000)
2. Output record1 with timestamp 6000 (valid)
3. Output record2 with timestamp 1000 (invalid)
The record2 is outputted invalidly because it's timestamp is equal to the
previous output watermark (1000). In my pr, I hold the input watermark back
with 5001 to make record2 valid. So the test will output with:
1. Output watermark with timestamp 999 (hold back with 5001 from 6000)
2. Output record1 with timestamp 6000 (valid)
3. Output record2 with timestamp 1000 (valid)
> Rowtime window inner join emits late data
> -
>
> Key: FLINK-8158
> URL: https://issues.apache.org/jira/browse/FLINK-8158
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
> Attachments: screenshot-1xxx.png
>
>
> When executing the join, the join operator needs to make sure that no late
> data is emitted. Currently, this achieved by holding back watermarks.
> However, the window border is not handled correctly. For the sql bellow:
> {quote}
> val sqlQuery =
> """
> SELECT t2.key, t2.id, t1.id
> FROM T1 as t1 join T2 as t2 ON
> t1.key = t2.key AND
> t1.rt BETWEEN t2.rt - INTERVAL '5' SECOND AND
> t2.rt + INTERVAL '1' SECOND
> """.stripMargin
> val data1 = new mutable.MutableList[(String, String, Long)]
> // for boundary test
> data1.+=(("A", "LEFT1", 6000L))
> val data2 = new mutable.MutableList[(String, String, Long)]
> data2.+=(("A", "RIGHT1", 6000L))
> {quote}
> Join will output a watermark with timestamp 1000, but if left comes with
> another data ("A", "LEFT1", 1000L), join will output a record with timestamp
> 1000 which equals previous watermark.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5094: [FLINK-8158] [table] Fix rowtime window inner join emits ...
Github user hequn8128 commented on the issue: https://github.com/apache/flink/pull/5094 Thanks for your replies. I agree with that it is valid to join late input data.What I concern is the watermark has not been hold back correctly. Take `testRowTimeJoinWithCommonBounds2` as an example. 1. Output watermark with timestamp 1000 (hold back with 5000 from 6000) 2. Output record1 with timestamp 6000 (valid) 3. Output record2 with timestamp 1000 (invalid) The record2 is outputted invalidly because it's timestamp is equal to the previous output watermark (1000). In my pr, I hold the input watermark back with 5001 to make record2 valid. So the test will output with: 1. Output watermark with timestamp 999 (hold back with 5001 from 6000) 2. Output record1 with timestamp 6000 (valid) 3. Output record2 with timestamp 1000 (valid) ---
[jira] [Updated] (FLINK-8175) remove flink-streaming-contrib and migrate its classes to flink-streaming-java/scala
[ https://issues.apache.org/jira/browse/FLINK-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-8175: Description: I propose removing flink-streaming-contrib from flink-contrib, and migrating its classes to flink-streaming-java/scala for the following reasons: - flink-streaming-contrib is so small that it only has 4 classes (3 java and 1 scala), and they don't need a dedicated jar for Flink to distribute and maintain it and for users to deal with the overhead of dependency management - the 4 classes in flink-streaming-contrib are logically more tied to flink-streaming-java/scala, and thus can be easily migrated - flink-contrib is already too crowded and noisy. It contains lots of sub modules with different purposes which confuse developers and users, and they lack a proper project hierarchy To take a step even forward, I would argue that even flink-contrib should be removed and all its submodules should be migrated to other top-level modules for the following reasons: 1) Apache Flink the whole project itself is a result of contributions from many developers, there's no reason to highlight some contributions in a dedicated module named 'contrib' 2) flink-contrib inherently doesn't have a good hierarchy to hold submodules was: I propose removing flink-streaming-contrib from flink-contrib, and migrating its classes to flink-streaming-java/scala for the following reasons: - flink-streaming-contrib is so small that it only has 4 classes (3 java and 1 scala), and they don't need a dedicated jar for Flink to distribute and maintain it and for users to deal with the overhead of dependency management - the 4 classes in flink-streaming-contrib are logically more tied to flink-streaming-java/scala, and thus can be easily migrated - flink-contrib is already too crowded and noisy. It contains lots of sub modules with different purposes which confuse developers and users, and they lack a proper project hierarchy To take a step even forward, I would argue that even flink-contrib should be removed and its submodules should be migrated to other top-level modules for the following reasons: 1) Apache Flink the whole project itself is a result of contributions from all developers, there's no reason to highlight some contributions in a dedicated module named 'contrib' 2) flink-contrib inherently doesn't have a good hierarchy to hold submodules > remove flink-streaming-contrib and migrate its classes to > flink-streaming-java/scala > > > Key: FLINK-8175 > URL: https://issues.apache.org/jira/browse/FLINK-8175 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.5.0 >Reporter: Bowen Li > > I propose removing flink-streaming-contrib from flink-contrib, and migrating > its classes to flink-streaming-java/scala for the following reasons: > - flink-streaming-contrib is so small that it only has 4 classes (3 java and > 1 scala), and they don't need a dedicated jar for Flink to distribute and > maintain it and for users to deal with the overhead of dependency management > - the 4 classes in flink-streaming-contrib are logically more tied to > flink-streaming-java/scala, and thus can be easily migrated > - flink-contrib is already too crowded and noisy. It contains lots of sub > modules with different purposes which confuse developers and users, and they > lack a proper project hierarchy > To take a step even forward, I would argue that even flink-contrib should be > removed and all its submodules should be migrated to other top-level modules > for the following reasons: 1) Apache Flink the whole project itself is a > result of contributions from many developers, there's no reason to highlight > some contributions in a dedicated module named 'contrib' 2) flink-contrib > inherently doesn't have a good hierarchy to hold submodules -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-8175) remove flink-streaming-contrib and migrate its classes to flink-streaming-java/scala
[ https://issues.apache.org/jira/browse/FLINK-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-8175: Description: I propose removing flink-streaming-contrib from flink-contrib, and migrating its classes to flink-streaming-java/scala for the following reasons: - flink-streaming-contrib is so small that it only has 4 classes (3 java and 1 scala), and they don't need a dedicated jar for Flink to distribute and maintain it and for users to deal with the overhead of dependency management - the 4 classes in flink-streaming-contrib are logically more tied to flink-streaming-java/scala, and thus can be easily migrated - flink-contrib is already too crowded and noisy. It contains lots of sub modules with different purposes which confuse developers and users, and they lack a proper project hierarchy To take a step even forward, I would argue that even flink-contrib should be removed and its submodules should be migrated to other top-level modules for the following reasons: 1) Apache Flink the whole project itself is a result of contributions from all developers, there's no reason to highlight some contributions in a dedicated module named 'contrib' 2) flink-contrib inherently doesn't have a good hierarchy to hold submodules was: I propose removing flink-streaming-contrib from flink-contrib, and migrating its classes to flink-streaming-java/scala for the following reasons: - flink-streaming-contrib is so small that it only has 4 classes (3 java and 1 scala), and they don't need a dedicated jar for Flink to distribute and maintain it and for users to deal with the overhead of dependency management - the 4 classes in flink-streaming-contrib are logically more tied to flink-streaming-java/scala, and thus can be easily migrated - flink-contrib is already too crowded and noisy. It contains lots of sub modules with different purposes which confuse developers and users, and they lack a proper project hierarchy According to the history of flink-streaming-contrib, it has things to do with storm-compatibility and I believe that's not blocking us from migrating it anymore. > remove flink-streaming-contrib and migrate its classes to > flink-streaming-java/scala > > > Key: FLINK-8175 > URL: https://issues.apache.org/jira/browse/FLINK-8175 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.5.0 >Reporter: Bowen Li > > I propose removing flink-streaming-contrib from flink-contrib, and migrating > its classes to flink-streaming-java/scala for the following reasons: > - flink-streaming-contrib is so small that it only has 4 classes (3 java and > 1 scala), and they don't need a dedicated jar for Flink to distribute and > maintain it and for users to deal with the overhead of dependency management > - the 4 classes in flink-streaming-contrib are logically more tied to > flink-streaming-java/scala, and thus can be easily migrated > - flink-contrib is already too crowded and noisy. It contains lots of sub > modules with different purposes which confuse developers and users, and they > lack a proper project hierarchy > To take a step even forward, I would argue that even flink-contrib should be > removed and its submodules should be migrated to other top-level modules for > the following reasons: 1) Apache Flink the whole project itself is a result > of contributions from all developers, there's no reason to highlight some > contributions in a dedicated module named 'contrib' 2) flink-contrib > inherently doesn't have a good hierarchy to hold submodules -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (FLINK-7294) mesos.resourcemanager.framework.role not working
[
https://issues.apache.org/jira/browse/FLINK-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272190#comment-16272190
]
Bhumika Bayani edited comment on FLINK-7294 at 11/30/17 5:34 AM:
-
Hi [~eronwright],
As per this link
http://mesos.apache.org/documentation/latest/roles/#default-role, the
unreserved resources are assigned to "*" role by default. So the fix should
have worked even for unreserved resources as per me. I remember testing this
with mesos 1.0.
If that is not the case, i.e. unreserved resources are not assigned to * role
by default, this fix wont work and we will need a fix that works for both
conditions.
was (Author: bbayani):
Hi [~eronwright],
As per this link
http://mesos.apache.org/documentation/latest/roles/#default-role, the
unreserved resources are assigned to "*" role by default. So the fix should
have worked even for unreserved resources as per me.
If that is not the case, i.e. unreserved resources are not assigned to * role
by default, this fix wont work and we will need a fix that works for both
conditions.
> mesos.resourcemanager.framework.role not working
>
>
> Key: FLINK-7294
> URL: https://issues.apache.org/jira/browse/FLINK-7294
> Project: Flink
> Issue Type: Bug
> Components: Mesos
>Affects Versions: 1.3.1
>Reporter: Bhumika Bayani
>Assignee: Eron Wright
>Priority: Critical
> Fix For: 1.4.0, 1.3.3
>
>
> I am using the above said setting in flink-conf.yaml
> e.g.
> mesos.resourcemanager.framework.role: mesos_role_tasks
> I see a flink-scheduler registered in mesos/frameworks tab with above said
> role.
> But the scheduler fails to launch any tasks inspite of getting
> resource-offers from mesos-agents with correct role.
> The error seen is:
> {code}
> 2017-07-28 13:23:00,683 INFO
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager -
> Mesos task taskmanager-03768 failed, with a TaskManager in launch or
> registration. State: TASK_ERROR Reason: REASON_TASK_INVALID (Task uses more
> resources cpus(\*):1; mem(\*):1024; ports(\*):[4006-4007] than available
> cpus(mesos_role_tasks):7.4; mem(mesos_role_tasks):45876;
> ports(mesos_role_tasks):[4002-4129, 4131-4380, 4382-4809, 4811-4957,
> 4959-4966, 4968-4979, 4981-5049, 31000-31196, 31198-31431, 31433-31607,
> 31609-32000]; ephemeral_storage(mesos_role_tasks):37662;
> efs_storage(mesos_role_tasks):8.79609e+12; disk(mesos_role_tasks):5115)
> {code}
> The request is made for resources with * role. We do not have mesos running
> anywhere with * role. Thus task manager never come up.
> Am I missing any configuration?
> I am using flink version 1.3.1
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7294) mesos.resourcemanager.framework.role not working
[
https://issues.apache.org/jira/browse/FLINK-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272190#comment-16272190
]
Bhumika Bayani commented on FLINK-7294:
---
Hi [~eronwright],
As per this link
http://mesos.apache.org/documentation/latest/roles/#default-role, the
unreserved resources are assigned to "*" role by default. So the fix should
have worked even for unreserved resources as per me.
If that is not the case, i.e. unreserved resources are not assigned to * role
by default, this fix wont work and we will need a fix that works for both
conditions.
> mesos.resourcemanager.framework.role not working
>
>
> Key: FLINK-7294
> URL: https://issues.apache.org/jira/browse/FLINK-7294
> Project: Flink
> Issue Type: Bug
> Components: Mesos
>Affects Versions: 1.3.1
>Reporter: Bhumika Bayani
>Assignee: Eron Wright
>Priority: Critical
> Fix For: 1.4.0, 1.3.3
>
>
> I am using the above said setting in flink-conf.yaml
> e.g.
> mesos.resourcemanager.framework.role: mesos_role_tasks
> I see a flink-scheduler registered in mesos/frameworks tab with above said
> role.
> But the scheduler fails to launch any tasks inspite of getting
> resource-offers from mesos-agents with correct role.
> The error seen is:
> {code}
> 2017-07-28 13:23:00,683 INFO
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager -
> Mesos task taskmanager-03768 failed, with a TaskManager in launch or
> registration. State: TASK_ERROR Reason: REASON_TASK_INVALID (Task uses more
> resources cpus(\*):1; mem(\*):1024; ports(\*):[4006-4007] than available
> cpus(mesos_role_tasks):7.4; mem(mesos_role_tasks):45876;
> ports(mesos_role_tasks):[4002-4129, 4131-4380, 4382-4809, 4811-4957,
> 4959-4966, 4968-4979, 4981-5049, 31000-31196, 31198-31431, 31433-31607,
> 31609-32000]; ephemeral_storage(mesos_role_tasks):37662;
> efs_storage(mesos_role_tasks):8.79609e+12; disk(mesos_role_tasks):5115)
> {code}
> The request is made for resources with * role. We do not have mesos running
> anywhere with * role. Thus task manager never come up.
> Am I missing any configuration?
> I am using flink version 1.3.1
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8167) Failing WikipediaEditsSourceTest on Travis
[
https://issues.apache.org/jira/browse/FLINK-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272182#comment-16272182
]
Bowen Li commented on FLINK-8167:
-
I also opened FLINK-8175. Let me know what you guys think
> Failing WikipediaEditsSourceTest on Travis
> --
>
> Key: FLINK-8167
> URL: https://issues.apache.org/jira/browse/FLINK-8167
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Ufuk Celebi
>Priority: Critical
> Labels: test-stability
> Fix For: 1.5.0
>
>
> I've observed the {{WikipediaEditsSourceTest}} failing on Travis.
> https://travis-ci.org/tillrohrmann/flink/jobs/308351298
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8175) remove flink-streaming-contrib and migrate its classes to flink-streaming-java/scala
Bowen Li created FLINK-8175: --- Summary: remove flink-streaming-contrib and migrate its classes to flink-streaming-java/scala Key: FLINK-8175 URL: https://issues.apache.org/jira/browse/FLINK-8175 Project: Flink Issue Type: Improvement Affects Versions: 1.5.0 Reporter: Bowen Li I propose removing flink-streaming-contrib from flink-contrib, and migrating its classes to flink-streaming-java/scala for the following reasons: - flink-streaming-contrib is so small that it only has 4 classes (3 java and 1 scala), and they don't need a dedicated jar for Flink to distribute and maintain it and for users to deal with the overhead of dependency management - the 4 classes in flink-streaming-contrib are logically more tied to flink-streaming-java/scala, and thus can be easily migrated - flink-contrib is already too crowded and noisy. It contains lots of sub modules with different purposes which confuse developers and users, and they lack a proper project hierarchy According to the history of flink-streaming-contrib, it has things to do with storm-compatibility and I believe that's not blocking us from migrating it anymore. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8167) Failing WikipediaEditsSourceTest on Travis
[
https://issues.apache.org/jira/browse/FLINK-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272162#comment-16272162
]
Bowen Li commented on FLINK-8167:
-
I'm wondering why wikiedit has to be part of Flink itself? I don't see much
necessity given its low popularity in real use case in Flink community.
Furthermore, flink-contrib is too crowded and noisy with lots of things which
are of different purposes, and those all result in the growing build time :(.
Can we move wikiedit to Apache Bahir?
> Failing WikipediaEditsSourceTest on Travis
> --
>
> Key: FLINK-8167
> URL: https://issues.apache.org/jira/browse/FLINK-8167
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Ufuk Celebi
>Priority: Critical
> Labels: test-stability
> Fix For: 1.5.0
>
>
> I've observed the {{WikipediaEditsSourceTest}} failing on Travis.
> https://travis-ci.org/tillrohrmann/flink/jobs/308351298
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7294) mesos.resourcemanager.framework.role not working
[
https://issues.apache.org/jira/browse/FLINK-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272120#comment-16272120
]
Eron Wright commented on FLINK-7294:
-
[~bbayani] I believe the fix we came up with is not correct. Looking at this
bug again, it appears that you were using a reservation (static or dynamic,
doesn't matter) and finding that Flink was unable to use _reserved _resources.
Unfortunately the fix has reversed the situation, and Flink is now unable to
use _unreserved _resources. We must find a solution that works for both
situations. Do you agree with my characterization? Thanks.
See FLINK-8174.
> mesos.resourcemanager.framework.role not working
>
>
> Key: FLINK-7294
> URL: https://issues.apache.org/jira/browse/FLINK-7294
> Project: Flink
> Issue Type: Bug
> Components: Mesos
>Affects Versions: 1.3.1
>Reporter: Bhumika Bayani
>Assignee: Eron Wright
>Priority: Critical
> Fix For: 1.4.0, 1.3.3
>
>
> I am using the above said setting in flink-conf.yaml
> e.g.
> mesos.resourcemanager.framework.role: mesos_role_tasks
> I see a flink-scheduler registered in mesos/frameworks tab with above said
> role.
> But the scheduler fails to launch any tasks inspite of getting
> resource-offers from mesos-agents with correct role.
> The error seen is:
> {code}
> 2017-07-28 13:23:00,683 INFO
> org.apache.flink.mesos.runtime.clusterframework.MesosFlinkResourceManager -
> Mesos task taskmanager-03768 failed, with a TaskManager in launch or
> registration. State: TASK_ERROR Reason: REASON_TASK_INVALID (Task uses more
> resources cpus(\*):1; mem(\*):1024; ports(\*):[4006-4007] than available
> cpus(mesos_role_tasks):7.4; mem(mesos_role_tasks):45876;
> ports(mesos_role_tasks):[4002-4129, 4131-4380, 4382-4809, 4811-4957,
> 4959-4966, 4968-4979, 4981-5049, 31000-31196, 31198-31431, 31433-31607,
> 31609-32000]; ephemeral_storage(mesos_role_tasks):37662;
> efs_storage(mesos_role_tasks):8.79609e+12; disk(mesos_role_tasks):5115)
> {code}
> The request is made for resources with * role. We do not have mesos running
> anywhere with * role. Thus task manager never come up.
> Am I missing any configuration?
> I am using flink version 1.3.1
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8174) Mesos RM unable to accept offers for unreserved resources
Eron Wright created FLINK-8174: --- Summary: Mesos RM unable to accept offers for unreserved resources Key: FLINK-8174 URL: https://issues.apache.org/jira/browse/FLINK-8174 Project: Flink Issue Type: Bug Components: Mesos Affects Versions: 1.4.0, 1.3.3 Reporter: Eron Wright Assignee: Eron Wright Priority: Blocker Fix For: 1.4.0 Flink has suffered a regression due to FLINK-7294. Any attempt to accept a resource offer that is based on unreserved resources will fail, because Flink (as of FLINK-7294) erroneously insists that the resource come from a prior reservation. Looking at the original issue, the problem may have been misdiagnosed. Ideally Flink should work with both reserved and unreserved resources, but the latter is a more common situation that is now broken. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8173) InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue.
[ https://issues.apache.org/jira/browse/FLINK-8173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271988#comment-16271988 ] Aegeaner commented on FLINK-8173: - Assignment conversion not possible from type "java.lang.CharSequence" to type "org.apache.avro.util.Utf8" > InvalidProgramException: Table program cannot be compiled. This is a bug. > Please file an issue. > --- > > Key: FLINK-8173 > URL: https://issues.apache.org/jira/browse/FLINK-8173 > Project: Flink > Issue Type: Bug >Reporter: Tao Xia > > It is a stream of Avro objects, simply select a String field and trying to > print out > val query = "SELECT nd_key FROM table1" > val result = tableEnv.sql(query) > tableEnv.toAppendStream[org.apache.avro.util.Utf8](result).print() > 11/29/2017 16:07:36 Source: Custom Source -> from: (accepted_cohort_id, > admin_id, after_submission, amount_paid, anonymous_id, application_id, > atom_key, bd_group_key, biz_geo, braavos_purchase_id, category, cohort_id, > concept_key, concept_rank, context, context_campaign, context_experiment, > coupon_code, course_key, course_rank, cta_destination, cta_location, > cta_message, cta_type, currency, decision_group_id, device_browser, > device_os, device_os_version, device_type, duration, evaluation_id, > event_type, fin_geo, in_collaboration_with, lab_id, lab_rank, label, > lesson_key, lesson_rank, locale, max_pause_duration, message, message_id, > module_key, module_rank, nd_key, nd_unit_id, nd_unit_rank, new_cohort_id, > notification_id, num_concepts_completed, num_interactions, > num_lessons_completed, old_cohort_id, part_key, part_rank, pause_duration, > pause_reason, payment_plan, payment_provider, points_earned, points_possible, > price, price_sheet, product_key, product_type, provider_charge_id, > provider_refund_id, quiz_type, referrer, refund_amount, requested_cohort_id, > results, scholarship_group_key, search_term, skill_level, subscription_id, > suspension_length, suspension_reason, technology, timestamp, total_concepts, > total_lessons, total_time_sec, type, unenroll_reason, user_id, user_locale, > user_response, variant, version, workspace_id, workspace_session, > workspace_type) -> select: (nd_key) -> to: Utf8 -> Sink: Unnamed(5/8) > switched to FAILED > org.apache.flink.api.common.InvalidProgramException: Table program cannot be > compiled. This is a bug. Please file an issue. > at > org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36) > at > org.apache.flink.table.runtime.CRowOutputMapRunner.compile(CRowOutputMapRunner.scala:33) > at > org.apache.flink.table.runtime.CRowOutputMapRunner.open(CRowOutputMapRunner.scala:48) > at > org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:111) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:376) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.codehaus.commons.compiler.CompileException: Line 790, Column > 15: Assignment conversion not possible from type "java.lang.CharSequence" to > type "org.apache.avro.util.Utf8" > at > org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11672) > at > org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10528) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2534) > at org.codehaus.janino.UnitCompiler.access$2600(UnitCompiler.java:212) > at > org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1459) > at > org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1443) > at > org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:3348) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1443) > at > org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1523) > at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3052) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1313) > at > org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1286) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:785) > at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:436) > at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:212) > at
[jira] [Created] (FLINK-8173) InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue.
Tao Xia created FLINK-8173: -- Summary: InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue. Key: FLINK-8173 URL: https://issues.apache.org/jira/browse/FLINK-8173 Project: Flink Issue Type: Bug Reporter: Tao Xia It is a stream of Avro objects, simply select a String field and trying to print out val query = "SELECT nd_key FROM table1" val result = tableEnv.sql(query) tableEnv.toAppendStream[org.apache.avro.util.Utf8](result).print() 11/29/2017 16:07:36 Source: Custom Source -> from: (accepted_cohort_id, admin_id, after_submission, amount_paid, anonymous_id, application_id, atom_key, bd_group_key, biz_geo, braavos_purchase_id, category, cohort_id, concept_key, concept_rank, context, context_campaign, context_experiment, coupon_code, course_key, course_rank, cta_destination, cta_location, cta_message, cta_type, currency, decision_group_id, device_browser, device_os, device_os_version, device_type, duration, evaluation_id, event_type, fin_geo, in_collaboration_with, lab_id, lab_rank, label, lesson_key, lesson_rank, locale, max_pause_duration, message, message_id, module_key, module_rank, nd_key, nd_unit_id, nd_unit_rank, new_cohort_id, notification_id, num_concepts_completed, num_interactions, num_lessons_completed, old_cohort_id, part_key, part_rank, pause_duration, pause_reason, payment_plan, payment_provider, points_earned, points_possible, price, price_sheet, product_key, product_type, provider_charge_id, provider_refund_id, quiz_type, referrer, refund_amount, requested_cohort_id, results, scholarship_group_key, search_term, skill_level, subscription_id, suspension_length, suspension_reason, technology, timestamp, total_concepts, total_lessons, total_time_sec, type, unenroll_reason, user_id, user_locale, user_response, variant, version, workspace_id, workspace_session, workspace_type) -> select: (nd_key) -> to: Utf8 -> Sink: Unnamed(5/8) switched to FAILED org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue. at org.apache.flink.table.codegen.Compiler$class.compile(Compiler.scala:36) at org.apache.flink.table.runtime.CRowOutputMapRunner.compile(CRowOutputMapRunner.scala:33) at org.apache.flink.table.runtime.CRowOutputMapRunner.open(CRowOutputMapRunner.scala:48) at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:111) at org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:376) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) at java.lang.Thread.run(Thread.java:748) Caused by: org.codehaus.commons.compiler.CompileException: Line 790, Column 15: Assignment conversion not possible from type "java.lang.CharSequence" to type "org.apache.avro.util.Utf8" at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11672) at org.codehaus.janino.UnitCompiler.assignmentConversion(UnitCompiler.java:10528) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2534) at org.codehaus.janino.UnitCompiler.access$2600(UnitCompiler.java:212) at org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1459) at org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1443) at org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:3348) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1443) at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1523) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3052) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1313) at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1286) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:785) at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:436) at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:212) at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:390) at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:385) at org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1405) at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:385) at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:357) at org.codehaus.j
[jira] [Commented] (FLINK-7907) Flink Metrics documentation missing Scala examples
[
https://issues.apache.org/jira/browse/FLINK-7907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271580#comment-16271580
]
ASF GitHub Bot commented on FLINK-7907:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5071#discussion_r153923624
--- Diff: docs/monitoring/metrics.md ---
@@ -94,8 +94,8 @@ Alternatively you can also use your own `Counter`
implementation:
{% highlight java %}
-public class MyMapper extends RichMapFunction {
- private Counter counter;
+RichMapFunction {
--- End diff --
looks like you removed more than intended?
> Flink Metrics documentation missing Scala examples
> --
>
> Key: FLINK-7907
> URL: https://issues.apache.org/jira/browse/FLINK-7907
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Colin Williams
>Assignee: Hai Zhou UTC+8
>Priority: Minor
>
> The Flink metrics documentation is missing Scala examples for many of the
> metrics types. To be consistent there should be Scala examples for all the
> types.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #5071: [FLINK-7907][docs] The metrics documentation missi...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5071#discussion_r153923624
--- Diff: docs/monitoring/metrics.md ---
@@ -94,8 +94,8 @@ Alternatively you can also use your own `Counter`
implementation:
{% highlight java %}
-public class MyMapper extends RichMapFunction {
- private Counter counter;
+RichMapFunction {
--- End diff --
looks like you removed more than intended?
---
[jira] [Commented] (FLINK-4812) Report Watermark metrics in all operators
[ https://issues.apache.org/jira/browse/FLINK-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271487#comment-16271487 ] ASF GitHub Bot commented on FLINK-4812: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5100 yes, we _could_ setup all IO metrics in the operator chain. But that will require some more changes there since the methods there never access 2 operators at the same time. For a chain A->B, to setup the output of A you need metrics for both A and B. You cannot be "clever" and re-use the output metrics of A as the input metrics for B, since this will cause odd results in forking chain scenarios where the input metrics for both chained operators change at the same time even though they are always processed sequentially. This would get even more funky if an operator decides that it's input/output metrics are always supposed to be identical, at which point you have metrics for 3 operators changing at the same time. But just thinking about modifying the `OperatorChain` sends shivers down my spine ❄️ > Report Watermark metrics in all operators > - > > Key: FLINK-4812 > URL: https://issues.apache.org/jira/browse/FLINK-4812 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.5.0 > > > As reported by a user, Flink does currently not export the current low > watermark for sources > (http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/currentLowWatermark-metric-not-reported-for-all-tasks-td13770.html). > This JIRA is for adding such a metric for the sources as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #5100: [FLINK-4812][metrics] Expose currentLowWatermark for all ...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5100 yes, we _could_ setup all IO metrics in the operator chain. But that will require some more changes there since the methods there never access 2 operators at the same time. For a chain A->B, to setup the output of A you need metrics for both A and B. You cannot be "clever" and re-use the output metrics of A as the input metrics for B, since this will cause odd results in forking chain scenarios where the input metrics for both chained operators change at the same time even though they are always processed sequentially. This would get even more funky if an operator decides that it's input/output metrics are always supposed to be identical, at which point you have metrics for 3 operators changing at the same time. But just thinking about modifying the `OperatorChain` sends shivers down my spine âï¸ ---
[jira] [Commented] (FLINK-8151) [Table] Map equality check to use entrySet equality
[ https://issues.apache.org/jira/browse/FLINK-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271417#comment-16271417 ] ASF GitHub Bot commented on FLINK-8151: --- Github user walterddr commented on the issue: https://github.com/apache/flink/pull/5070 @twalthr I think my previous unit test method was not correct. Since TableAPI always use `java.util.HashMap` per flink documentation. I think it is safe to directly use `$leftTerm.equals($rightTerm)` and it should always be using `java.util.AbstractMap.equals` implementation. Do you think I should add extra check or do you think the `isMap(resultType)` should be suffice? > [Table] Map equality check to use entrySet equality > --- > > Key: FLINK-8151 > URL: https://issues.apache.org/jira/browse/FLINK-8151 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Reporter: Rong Rong >Assignee: Rong Rong > > Following up with FLINK-8038. The equality check currently is broken. Plan to > support element-wise equality check by always using the base class: > "java.util.Map.equals" method. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #5070: [FLINK-8151][table]Map equality check to use entrySet equ...
Github user walterddr commented on the issue: https://github.com/apache/flink/pull/5070 @twalthr I think my previous unit test method was not correct. Since TableAPI always use `java.util.HashMap` per flink documentation. I think it is safe to directly use `$leftTerm.equals($rightTerm)` and it should always be using `java.util.AbstractMap.equals` implementation. Do you think I should add extra check or do you think the `isMap(resultType)` should be suffice? ---
[jira] [Updated] (FLINK-8151) [Table] Map equality check to use entrySet equality
[ https://issues.apache.org/jira/browse/FLINK-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rong Rong updated FLINK-8151: - Summary: [Table] Map equality check to use entrySet equality (was: [Table] Clean up Map equality check) > [Table] Map equality check to use entrySet equality > --- > > Key: FLINK-8151 > URL: https://issues.apache.org/jira/browse/FLINK-8151 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Reporter: Rong Rong >Assignee: Rong Rong > > Following up with FLINK-8038. The equality check currently is broken. Plan to > support element-wise equality check by always using the base class: > "java.util.Map.equals" method. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-8151) [Table] Clean up Map equality check
[ https://issues.apache.org/jira/browse/FLINK-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rong Rong updated FLINK-8151: - Description: Following up with FLINK-8038. The equality check currently is broken. Plan to support element-wise equality check by always using the base class: "java.util.Map.equals" method. was:Following up with FLINK-8038. The equality check is not working as Map does not support element-wise equality. Suggest to remove it. > [Table] Clean up Map equality check > --- > > Key: FLINK-8151 > URL: https://issues.apache.org/jira/browse/FLINK-8151 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Reporter: Rong Rong >Assignee: Rong Rong > > Following up with FLINK-8038. The equality check currently is broken. Plan to > support element-wise equality check by always using the base class: > "java.util.Map.equals" method. > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (FLINK-8151) [Table] Clean up Map equality check
[ https://issues.apache.org/jira/browse/FLINK-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rong Rong updated FLINK-8151: - Summary: [Table] Clean up Map equality check (was: [Table] removing map value equality check) > [Table] Clean up Map equality check > --- > > Key: FLINK-8151 > URL: https://issues.apache.org/jira/browse/FLINK-8151 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Reporter: Rong Rong >Assignee: Rong Rong > > Following up with FLINK-8038. The equality check is not working as Map does > not support element-wise equality. Suggest to remove it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-7595) Removing stateless task from task chain breaks savepoint restore
[
https://issues.apache.org/jira/browse/FLINK-7595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271279#comment-16271279
]
ASF GitHub Bot commented on FLINK-7595:
---
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/5103
[FLINK-7595] [Savepoints] Allow removing stateless operators
This is a fixed version of #4651 for 1.4. It adds some checkstyle fixes and
modifies the SavepointLoaderTest to actually have a stateful task. The test is
currently only passing due to the broken behavior.
## What is the purpose of the change
This PR reverts a regression where stateless operators could no longer be
removed from a job when loading a savepoint without setting the
`--allowNonRestoredState` flag. The check now explicitly checks whether the
state of an operator, that could not be mapped to the new program, is empty.
## Brief change log
* Modify `SavepointLoader` to check whether the unmapped state is actually
empty
* Modify `AbstractOperatorRestoreTestBase` to allow subclasses to set the
`--allowNonRestoredState` flag
* Add a modified version of `ChainLengthDecreaseTest` to prevent this issue
from re-emerging.
## Verifying this change
This change added tests and can be verified as follows:
Run `ChainLengthStatelessDecreaseTest`. Alternatively, run the reproducer
from the JIRA before and after the change.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
This should be merged to 1.3 and master. Note that for 1.3 it may be
necessary to backport the `OperatorSubtaskState#hasState()` method.
@StefanRRichter @uce
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 7595_14
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5103.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5103
commit 083d43b0365705c6d6355d5609da6b812d3ac909
Author: zentol
Date: 2017-09-06T13:38:20Z
[FLINK-7595] [Savepoints] Allow removing stateless operators
commit 2e6bad3ee473aeb76c323d826d1f57b51f2968a2
Author: zentol
Date: 2017-11-29T17:57:51Z
checkstyle
commit d0674e8c135ab195ed91ef0ebbcb82f9e9aec79e
Author: zentol
Date: 2017-11-29T18:14:18Z
make task stateful in SavepointLoaderTest
> Removing stateless task from task chain breaks savepoint restore
>
>
> Key: FLINK-7595
> URL: https://issues.apache.org/jira/browse/FLINK-7595
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Ufuk Celebi
>Assignee: Chesnay Schepler
> Attachments: ChainedTaskRemoveTest.java
>
>
> When removing a stateless operator from a 2-task chain where the head
> operator is stateful breaks savepoint restore with
> {code}
> Caused by: java.lang.IllegalStateException: Failed to rollback to savepoint
> /var/folders/py/s_1l8vln6f19ygc77m8c4zhrgn/T/junit1167397515334838028/junit8006766303945373008/savepoint-cb0bcf-3cfa67865ac0.
> Cannot map savepoint state...
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #5103: [FLINK-7595] [Savepoints] Allow removing stateless...
GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/5103 [FLINK-7595] [Savepoints] Allow removing stateless operators This is a fixed version of #4651 for 1.4. It adds some checkstyle fixes and modifies the SavepointLoaderTest to actually have a stateful task. The test is currently only passing due to the broken behavior. ## What is the purpose of the change This PR reverts a regression where stateless operators could no longer be removed from a job when loading a savepoint without setting the `--allowNonRestoredState` flag. The check now explicitly checks whether the state of an operator, that could not be mapped to the new program, is empty. ## Brief change log * Modify `SavepointLoader` to check whether the unmapped state is actually empty * Modify `AbstractOperatorRestoreTestBase` to allow subclasses to set the `--allowNonRestoredState` flag * Add a modified version of `ChainLengthDecreaseTest` to prevent this issue from re-emerging. ## Verifying this change This change added tests and can be verified as follows: Run `ChainLengthStatelessDecreaseTest`. Alternatively, run the reproducer from the JIRA before and after the change. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This should be merged to 1.3 and master. Note that for 1.3 it may be necessary to backport the `OperatorSubtaskState#hasState()` method. @StefanRRichter @uce You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 7595_14 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5103.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5103 commit 083d43b0365705c6d6355d5609da6b812d3ac909 Author: zentol Date: 2017-09-06T13:38:20Z [FLINK-7595] [Savepoints] Allow removing stateless operators commit 2e6bad3ee473aeb76c323d826d1f57b51f2968a2 Author: zentol Date: 2017-11-29T17:57:51Z checkstyle commit d0674e8c135ab195ed91ef0ebbcb82f9e9aec79e Author: zentol Date: 2017-11-29T18:14:18Z make task stateful in SavepointLoaderTest ---
[jira] [Commented] (FLINK-8151) [Table] removing map value equality check
[ https://issues.apache.org/jira/browse/FLINK-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271238#comment-16271238 ] ASF GitHub Bot commented on FLINK-8151: --- Github user walterddr commented on the issue: https://github.com/apache/flink/pull/5070 Ah. I see. `AbstractMap.equals` look very much similar to `Arrays.equals`. I will add it in and some more testings. thanks @fhueske for the great tips > [Table] removing map value equality check > - > > Key: FLINK-8151 > URL: https://issues.apache.org/jira/browse/FLINK-8151 > Project: Flink > Issue Type: Bug > Components: Table API & SQL >Reporter: Rong Rong >Assignee: Rong Rong > > Following up with FLINK-8038. The equality check is not working as Map does > not support element-wise equality. Suggest to remove it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #5070: [FLINK-8151][table]Remove Map type equality comparison.
Github user walterddr commented on the issue: https://github.com/apache/flink/pull/5070 Ah. I see. `AbstractMap.equals` look very much similar to `Arrays.equals`. I will add it in and some more testings. thanks @fhueske for the great tips ---
[jira] [Comment Edited] (FLINK-5506) Java 8 - CommunityDetection.java:158 - java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271183#comment-16271183
]
Christos Hadjinikolis edited comment on FLINK-5506 at 11/29/17 5:42 PM:
Same issue here. I am running:
{{{color:red}return {color:#205081}graph{color}.run(new
CommunityDetection<>(MAX_ITERATIONS_FOR_SEGMENTATION, DELTA)){color}}}
where my *graph *is of type:{{{color:#205081}Graph{color}}}.
The *graph *is generated with:
{{{color:#14892c}Graph.fromDataSet(env.fromCollection(vertices),
env.fromCollection(edges), env);{color}}}
where *vertices* are:{{{color:#14892c} List>
vertices{color}}}
and edges:{{ {color:#14892c}Set> edges{color}}}
It is worth noting that I am intentionally creating a vertices with distinct
Long ids to create List> vertices. After running the code
I am getting the below *nullPointerException*:
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.NullPointerException
at
*{color:#d04437}org.apache.flink.graph.library.CommunityDetection$VertexLabelUpdater.updateVertex(CommunityDetection.java:158{color}*)
at
org.apache.flink.graph.spargel.ScatterGatherIteration$GatherUdfSimpleVV.coGroup(ScatterGatherIteration.java:389)
at
org.apache.flink.runtime.operators.CoGroupWithSolutionSetSecondDriver.run(CoGroupWithSolutionSetSecondDriver.java:218)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:490)
at
org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run(AbstractIterativeTask.java:146)
at
org.apache.flink.runtime.iterative.task.IterationTailTask.run(IterationTailTask.java:107)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
Is there a quick fix for this? e.g. initialising vertices directly using their
ids as both their ids and values as per Vasia's advice? Has anyone solved this?
Anything else I can try before this is resolved?
was (Author: hadjinik):
Same issue here. I am running:
{{{color:red}return {color:#205081}graph{color}.run(new
CommunityDetection<>(MAX_ITERATIONS_FOR_SEGMENTATION, DELTA)){color}}}
where my *graph *is of type:{{{color:#205081}Graph{color}}}.
The *graph *is generated with:
{{{color:#14892c}Graph.fromDataSet(env.fromCollection(vertices),
env.fromCollection(edges), env);{color}}}
where *vertices* are:{{{color:#14892c} List>
vertices{color}}}
and edges:{{ {color:#14892c}Set> edges{color}}}
It is worth noting that I am intentionally creating a vertices with distinct
Long ids to create List> vertices. After running the code
I am getting the below *nullPointerException*:
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doE
[jira] [Comment Edited] (FLINK-5506) Java 8 - CommunityDetection.java:158 - java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271183#comment-16271183
]
Christos Hadjinikolis edited comment on FLINK-5506 at 11/29/17 5:38 PM:
Same issue here. I am running:
{{{color:red}return {color:#205081}graph{color}.run(new
CommunityDetection<>(MAX_ITERATIONS_FOR_SEGMENTATION, DELTA)){color}}}
where my *graph *is of type:{{{color:#205081}Graph{color}}}.
The *graph *is generated with:
{{{color:#14892c}Graph.fromDataSet(env.fromCollection(vertices),
env.fromCollection(edges), env);{color}}}
where *vertices* are:{{{color:#14892c} List>
vertices{color}}}
and edges:{{ {color:#14892c}Set> edges{color}}}
It is worth noting that I am intentionally creating a vertices with distinct
Long ids to create List> vertices. After running the code
I am getting the below *nullPointerException*:
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.NullPointerException
at
org.apache.flink.graph.library.CommunityDetection$VertexLabelUpdater.updateVertex(CommunityDetection.java:158)
at
org.apache.flink.graph.spargel.ScatterGatherIteration$GatherUdfSimpleVV.coGroup(ScatterGatherIteration.java:389)
at
org.apache.flink.runtime.operators.CoGroupWithSolutionSetSecondDriver.run(CoGroupWithSolutionSetSecondDriver.java:218)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:490)
at
org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run(AbstractIterativeTask.java:146)
at
org.apache.flink.runtime.iterative.task.IterationTailTask.run(IterationTailTask.java:107)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
was (Author: hadjinik):
Same issue here. I am running:
{{{color:red}return {color:#205081}graph{color}.run(new
CommunityDetection<>(MAX_ITERATIONS_FOR_SEGMENTATION, DELTA)){color}}}
where my *graph *is of type:{{{color:#205081}Graph{color}}}.
The *graph *is generated with:
{{{color:#14892c}Graph.fromDataSet(env.fromCollection(vertices),
env.fromCollection(edges), env);{color}}}
where *vertices* are:{{{color:#14892c} List>
vertices{color}}}
and edges:{{ {color:#14892c}Set> edges{color}}}
It is worth noting that I am intentionally creating a vertices with distinct
Long ids to create List> vertices. After running the code
I am getting the below *nullPointerException*:
{{org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThrea
[jira] [Comment Edited] (FLINK-5506) Java 8 - CommunityDetection.java:158 - java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271183#comment-16271183
]
Christos Hadjinikolis edited comment on FLINK-5506 at 11/29/17 5:38 PM:
Same issue here. I am running:
{{{color:red}return {color:#205081}graph{color}.run(new
CommunityDetection<>(MAX_ITERATIONS_FOR_SEGMENTATION, DELTA)){color}}}
where my *graph *is of type:{{{color:#205081}Graph{color}}}.
The *graph *is generated with:
{{{color:#14892c}Graph.fromDataSet(env.fromCollection(vertices),
env.fromCollection(edges), env);{color}}}
where *vertices* are:{{{color:#14892c} List>
vertices{color}}}
and edges:{{ {color:#14892c}Set> edges{color}}}
It is worth noting that I am intentionally creating a vertices with distinct
Long ids to create List> vertices. After running the code
I am getting the below *nullPointerException*:
{{org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.NullPointerException
at
org.apache.flink.graph.library.CommunityDetection$VertexLabelUpdater.updateVertex(CommunityDetection.java:158)
at
org.apache.flink.graph.spargel.ScatterGatherIteration$GatherUdfSimpleVV.coGroup(ScatterGatherIteration.java:389)
at
org.apache.flink.runtime.operators.CoGroupWithSolutionSetSecondDriver.run(CoGroupWithSolutionSetSecondDriver.java:218)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:490)
at
org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run(AbstractIterativeTask.java:146)
at
org.apache.flink.runtime.iterative.task.IterationTailTask.run(IterationTailTask.java:107)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)}}
was (Author: hadjinik):
Same issue here. I am running:
{{{color:red}return {color:#205081}graph{color}.run(new
CommunityDetection<>(MAX_ITERATIONS_FOR_SEGMENTATION, DELTA)){color}}}
where my *graph *is of type:{{{color:#205081}Graph{color}}}.
The *graph *is generated with:
{{{color:#14892c}Graph.fromDataSet(env.fromCollection(vertices),
env.fromCollection(edges), env);{color}}}
where *vertices* are:{{{color:#14892c} List>
vertices{color}}}
and edges:{{ {color:#14892c}Set> edges{color}}}
It is worth noting that I am intentionally creating a vertices with distinct
Long ids to create List> vertices. After running the code
I am getting the below *nullPointerException*:
{{
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThre
[jira] [Commented] (FLINK-5506) Java 8 - CommunityDetection.java:158 - java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271183#comment-16271183
]
Christos Hadjinikolis commented on FLINK-5506:
--
Same issue here. I am running:
{{{color:red}return {color:#205081}graph{color}.run(new
CommunityDetection<>(MAX_ITERATIONS_FOR_SEGMENTATION, DELTA)){color}}}
where my *graph *is of type:{{{color:#205081}Graph{color}}}.
The *graph *is generated with:
{{{color:#14892c}Graph.fromDataSet(env.fromCollection(vertices),
env.fromCollection(edges), env);{color}}}
where *vertices* are:{{{color:#14892c} List>
vertices{color}}}
and edges:{{ {color:#14892c}Set> edges{color}}}
It is worth noting that I am intentionally creating a vertices with distinct
Long ids to create List> vertices. After running the code
I am getting the below *nullPointerException*:
{{
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:933)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:876)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at
scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.NullPointerException
at
org.apache.flink.graph.library.CommunityDetection$VertexLabelUpdater.updateVertex(CommunityDetection.java:158)
at
org.apache.flink.graph.spargel.ScatterGatherIteration$GatherUdfSimpleVV.coGroup(ScatterGatherIteration.java:389)
at
org.apache.flink.runtime.operators.CoGroupWithSolutionSetSecondDriver.run(CoGroupWithSolutionSetSecondDriver.java:218)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:490)
at
org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run(AbstractIterativeTask.java:146)
at
org.apache.flink.runtime.iterative.task.IterationTailTask.run(IterationTailTask.java:107)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)}}
> Java 8 - CommunityDetection.java:158 - java.lang.NullPointerException
> -
>
> Key: FLINK-5506
> URL: https://issues.apache.org/jira/browse/FLINK-5506
> Project: Flink
> Issue Type: Bug
> Components: Gelly
>Affects Versions: 1.1.4
>Reporter: Miguel E. Coimbra
> Labels: easyfix, newbie
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Reporting this here as per Vasia's advice.
> I am having the following problem while trying out the
> org.apache.flink.graph.library.CommunityDetection algorithm of the Gelly API
> (Java).
> Specs: JDK 1.8.0_102 x64
> Apache Flink: 1.1.4
> Suppose I have a very small (I tried an example with 38 vertices as well)
> dataset stored in a tab-separated file 3-vertex.tsv:
> #id1 id2 score
> 010
> 020
> 030
> This is just a central vertex with 3 neighbors (disconnected between
> themselves).
> I am loading the dataset and executing the algorithm with the following code:
> ---
> // Load the data from the .tsv file.
> final DataSet> edgeTuples =
> env.readCsvFile(inputPath)
> .fieldDelimiter("\t") // node IDs are separated by spaces
> .ignoreComments("#") // comments start with "%"
> .types(Long.class, Long.class, Double.class);
> // Generate a graph and add reverse edges (undirected).
> final Graph graph = Graph.fromTupleDataSet(edgeTuples,
> new MapFunction() {
> private static final long serialVersionUID = 8713516577419451509L;
> public Long map(Long value) {
> return value;
> }
> },
> env).getUndirected();
> // CommunityDetection parameters.
> final double hopAttenuationDelta = 0.5d;
> final int iterationCount = 10;
> // Prepare and
[jira] [Commented] (FLINK-7574) Remove unused dependencies from flink-clients
[ https://issues.apache.org/jira/browse/FLINK-7574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271167#comment-16271167 ] ASF GitHub Bot commented on FLINK-7574: --- Github user yew1eb commented on a diff in the pull request: https://github.com/apache/flink/pull/5076#discussion_r153858786 --- Diff: pom.xml --- @@ -891,6 +905,41 @@ under the License. + + org.apache.maven.plugins + maven-dependency-plugin + 3.0.2 --- End diff -- will remove. > Remove unused dependencies from flink-clients > - > > Key: FLINK-7574 > URL: https://issues.apache.org/jira/browse/FLINK-7574 > Project: Flink > Issue Type: Sub-task > Components: Build System >Affects Versions: 1.3.2 > Environment: Apache Maven 3.3.9, Java version: 1.8.0_144 >Reporter: Hai Zhou UTC+8 >Assignee: Hai Zhou UTC+8 > > [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ > flink-clients_2.11 --- > [WARNING] Used undeclared dependencies found: > [WARNING]org.scala-lang:scala-library:jar:2.11.11:compile > [WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile > [WARNING] Unused declared dependencies found: > [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test > [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile > [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test > [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile > [WARNING]log4j:log4j:jar:1.2.17:test > [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test > [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink pull request #5076: [FLINK-7574][build] POM Cleanup flink-clients
Github user yew1eb commented on a diff in the pull request: https://github.com/apache/flink/pull/5076#discussion_r153858786 --- Diff: pom.xml --- @@ -891,6 +905,41 @@ under the License. + + org.apache.maven.plugins + maven-dependency-plugin + 3.0.2 --- End diff -- will remove. ---
[jira] [Commented] (FLINK-7762) Make WikipediaEditsSourceTest a proper test
[
https://issues.apache.org/jira/browse/FLINK-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271151#comment-16271151
]
ASF GitHub Bot commented on FLINK-7762:
---
Github user uce commented on the issue:
https://github.com/apache/flink/pull/5102
I would be in favour of removing this or moving it to Bahir, but it is
currently used in a [doc
example](https://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/run_example_quickstart.html).
Besides that I doubt that this is of much value to users.
If you think it's OK with it, let's merge this for now and think about the
example thing really needs it.
> Make WikipediaEditsSourceTest a proper test
> ---
>
> Key: FLINK-7762
> URL: https://issues.apache.org/jira/browse/FLINK-7762
> Project: Flink
> Issue Type: Improvement
> Components: Streaming Connectors
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Minor
>
> {{WikipediaEditsSourceTest}} is currently an ITCase even though it's called
> test. Making it a test reduces runtime and also makes it more stable because
> we don't run a whole Flink job.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5102: [FLINK-7762, FLINK-8167] Clean up and harden WikipediaEdi...
Github user uce commented on the issue: https://github.com/apache/flink/pull/5102 I would be in favour of removing this or moving it to Bahir, but it is currently used in a [doc example](https://ci.apache.org/projects/flink/flink-docs-release-1.4/quickstart/run_example_quickstart.html). Besides that I doubt that this is of much value to users. If you think it's OK with it, let's merge this for now and think about the example thing really needs it. ---
[jira] [Commented] (FLINK-7907) Flink Metrics documentation missing Scala examples
[ https://issues.apache.org/jira/browse/FLINK-7907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271136#comment-16271136 ] ASF GitHub Bot commented on FLINK-7907: --- Github user yew1eb commented on the issue: https://github.com/apache/flink/pull/5071 @zentol , thanks for your review. I have updated the PR according to your comments. > Flink Metrics documentation missing Scala examples > -- > > Key: FLINK-7907 > URL: https://issues.apache.org/jira/browse/FLINK-7907 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Colin Williams >Assignee: Hai Zhou UTC+8 >Priority: Minor > > The Flink metrics documentation is missing Scala examples for many of the > metrics types. To be consistent there should be Scala examples for all the > types. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #5071: [FLINK-7907][docs] The metrics documentation missing scal...
Github user yew1eb commented on the issue: https://github.com/apache/flink/pull/5071 @zentol , thanks for your review. I have updated the PR according to your comments. ---
[jira] [Commented] (FLINK-7762) Make WikipediaEditsSourceTest a proper test
[
https://issues.apache.org/jira/browse/FLINK-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271127#comment-16271127
]
ASF GitHub Bot commented on FLINK-7762:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5102
The changes look good! But you comment about needing that IRC channel seem
valid. If we don't have the test, however, we would have no way of knowing that
the code at least works. (We could also drop the source completely)
> Make WikipediaEditsSourceTest a proper test
> ---
>
> Key: FLINK-7762
> URL: https://issues.apache.org/jira/browse/FLINK-7762
> Project: Flink
> Issue Type: Improvement
> Components: Streaming Connectors
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Minor
>
> {{WikipediaEditsSourceTest}} is currently an ITCase even though it's called
> test. Making it a test reduces runtime and also makes it more stable because
> we don't run a whole Flink job.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8158) Rowtime window inner join emits late data
[
https://issues.apache.org/jira/browse/FLINK-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271126#comment-16271126
]
ASF GitHub Bot commented on FLINK-8158:
---
Github user fhueske commented on the issue:
https://github.com/apache/flink/pull/5094
I had a look at `testRowTimeJoinWithCommonBounds2`.
The test processes starts by processing two watermarks for `6000` for the
first and second input. After that three records are added for `6000` (first),
`6000` (second), and `1000` (first). All three records are late because their
timestamp is smaller or equal to the current watermark. Therefore, the join
result should also be late. If all three records would be aligned with the
watermarks, the result should not be late.
> Rowtime window inner join emits late data
> -
>
> Key: FLINK-8158
> URL: https://issues.apache.org/jira/browse/FLINK-8158
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
> Attachments: screenshot-1xxx.png
>
>
> When executing the join, the join operator needs to make sure that no late
> data is emitted. Currently, this achieved by holding back watermarks.
> However, the window border is not handled correctly. For the sql bellow:
> {quote}
> val sqlQuery =
> """
> SELECT t2.key, t2.id, t1.id
> FROM T1 as t1 join T2 as t2 ON
> t1.key = t2.key AND
> t1.rt BETWEEN t2.rt - INTERVAL '5' SECOND AND
> t2.rt + INTERVAL '1' SECOND
> """.stripMargin
> val data1 = new mutable.MutableList[(String, String, Long)]
> // for boundary test
> data1.+=(("A", "LEFT1", 6000L))
> val data2 = new mutable.MutableList[(String, String, Long)]
> data2.+=(("A", "RIGHT1", 6000L))
> {quote}
> Join will output a watermark with timestamp 1000, but if left comes with
> another data ("A", "LEFT1", 1000L), join will output a record with timestamp
> 1000 which equals previous watermark.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (FLINK-8172) Remove unnecessary synchronisation in RecordSerializer
Piotr Nowojski created FLINK-8172: - Summary: Remove unnecessary synchronisation in RecordSerializer Key: FLINK-8172 URL: https://issues.apache.org/jira/browse/FLINK-8172 Project: Flink Issue Type: Improvement Components: Network Affects Versions: 1.3.2, 1.4.0 Reporter: Piotr Nowojski Assignee: Piotr Nowojski Fix For: 1.5.0 While writing the records, RecordSerializer is the only owner of the `Buffer` into which data are written. Yet we are synchronisation twice per record while accessing MemorySegment. Removing this synchronisation speeds up the Network throughput in point to point benchmark by a factor of two (from ~12500records/ms up to 23000 records/ms). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #5102: [FLINK-7762, FLINK-8167] Clean up and harden WikipediaEdi...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5102 The changes look good! But you comment about needing that IRC channel seem valid. If we don't have the test, however, we would have no way of knowing that the code at least works. (We could also drop the source completely) ---
[GitHub] flink issue #5094: [FLINK-8158] [table] Fix rowtime window inner join emits ...
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/5094 I had a look at `testRowTimeJoinWithCommonBounds2`. The test processes starts by processing two watermarks for `6000` for the first and second input. After that three records are added for `6000` (first), `6000` (second), and `1000` (first). All three records are late because their timestamp is smaller or equal to the current watermark. Therefore, the join result should also be late. If all three records would be aligned with the watermarks, the result should not be late. ---
[GitHub] flink issue #4985: [FLINK-8027] Generalize existing rest handlers to work wi...
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/4985 Thanks for the review @shuai-xu. Will rebase it and once Travis gives green light, I'll merge it. ---
[jira] [Commented] (FLINK-8027) Generalize existing rest handlers to work with arbitrary RestfulGateway
[
https://issues.apache.org/jira/browse/FLINK-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271117#comment-16271117
]
ASF GitHub Bot commented on FLINK-8027:
---
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4985#discussion_r153852356
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TestingTaskExecutorGateway.java
---
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.taskexecutor;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.clusterframework.types.SlotID;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.executiongraph.ExecutionAttemptID;
+import org.apache.flink.runtime.executiongraph.PartitionInfo;
+import org.apache.flink.runtime.jobmaster.JobMasterId;
+import org.apache.flink.runtime.messages.Acknowledge;
+import org.apache.flink.runtime.resourcemanager.ResourceManagerId;
+import org.apache.flink.util.Preconditions;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Simple {@link TaskExecutorGateway} implementation for testing purposes.
+ */
+public class TestingTaskExecutorGateway implements TaskExecutorGateway {
+
+ private final String address;
+
+ private final String hostname;
+
+ public TestingTaskExecutorGateway() {
+ this("foobar:1234", "foobar");
+ }
+
+ public TestingTaskExecutorGateway(String address, String hostname) {
+ this.address = Preconditions.checkNotNull(address);
+ this.hostname = Preconditions.checkNotNull(hostname);
+ }
+
+ @Override
+ public CompletableFuture requestSlot(SlotID slotId, JobID
jobId, AllocationID allocationId, String targetAddress, ResourceManagerId
resourceManagerId, Time timeout) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture
submitTask(TaskDeploymentDescriptor tdd, JobMasterId jobMasterId, Time timeout)
{
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture
updatePartitions(ExecutionAttemptID executionAttemptID, Iterable
partitionInfos, Time timeout) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public void failPartition(ExecutionAttemptID executionAttemptID) {
+ // noop
+ }
+
+ @Override
+ public CompletableFuture
triggerCheckpoint(ExecutionAttemptID executionAttemptID, long checkpointID,
long checkpointTimestamp, CheckpointOptions checkpointOptions) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture
confirmCheckpoint(ExecutionAttemptID executionAttemptID, long checkpointId,
long checkpointTimestamp) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture stopTask(ExecutionAttemptID
executionAttemptID, Time timeout) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture cancelTask(ExecutionAttemptID
executionAttemptID, Time timeout) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public void heartbeatFromJobManager(ResourceID heartbeatOrigin) {
+ // noop
+ }
+
+ @Override
[jira] [Commented] (FLINK-8027) Generalize existing rest handlers to work with arbitrary RestfulGateway
[
https://issues.apache.org/jira/browse/FLINK-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271118#comment-16271118
]
ASF GitHub Bot commented on FLINK-8027:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4985
Thanks for the review @shuai-xu. Will rebase it and once Travis gives green
light, I'll merge it.
> Generalize existing rest handlers to work with arbitrary RestfulGateway
> ---
>
> Key: FLINK-8027
> URL: https://issues.apache.org/jira/browse/FLINK-8027
> Project: Flink
> Issue Type: Sub-task
> Components: REST
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Labels: flip-6
> Fix For: 1.5.0
>
>
> In order to reuse the existing {{AbstractRestHandler}} we should refactor
> them such that they work with arbitrary {{RestfulGateway}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4985: [FLINK-8027] Generalize existing rest handlers to ...
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/4985#discussion_r153852356
--- Diff:
flink-runtime/src/test/java/org/apache/flink/runtime/taskexecutor/TestingTaskExecutorGateway.java
---
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.taskexecutor;
+
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceID;
+import org.apache.flink.runtime.clusterframework.types.SlotID;
+import org.apache.flink.runtime.deployment.TaskDeploymentDescriptor;
+import org.apache.flink.runtime.executiongraph.ExecutionAttemptID;
+import org.apache.flink.runtime.executiongraph.PartitionInfo;
+import org.apache.flink.runtime.jobmaster.JobMasterId;
+import org.apache.flink.runtime.messages.Acknowledge;
+import org.apache.flink.runtime.resourcemanager.ResourceManagerId;
+import org.apache.flink.util.Preconditions;
+
+import java.util.concurrent.CompletableFuture;
+
+/**
+ * Simple {@link TaskExecutorGateway} implementation for testing purposes.
+ */
+public class TestingTaskExecutorGateway implements TaskExecutorGateway {
+
+ private final String address;
+
+ private final String hostname;
+
+ public TestingTaskExecutorGateway() {
+ this("foobar:1234", "foobar");
+ }
+
+ public TestingTaskExecutorGateway(String address, String hostname) {
+ this.address = Preconditions.checkNotNull(address);
+ this.hostname = Preconditions.checkNotNull(hostname);
+ }
+
+ @Override
+ public CompletableFuture requestSlot(SlotID slotId, JobID
jobId, AllocationID allocationId, String targetAddress, ResourceManagerId
resourceManagerId, Time timeout) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture
submitTask(TaskDeploymentDescriptor tdd, JobMasterId jobMasterId, Time timeout)
{
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture
updatePartitions(ExecutionAttemptID executionAttemptID, Iterable
partitionInfos, Time timeout) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public void failPartition(ExecutionAttemptID executionAttemptID) {
+ // noop
+ }
+
+ @Override
+ public CompletableFuture
triggerCheckpoint(ExecutionAttemptID executionAttemptID, long checkpointID,
long checkpointTimestamp, CheckpointOptions checkpointOptions) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture
confirmCheckpoint(ExecutionAttemptID executionAttemptID, long checkpointId,
long checkpointTimestamp) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture stopTask(ExecutionAttemptID
executionAttemptID, Time timeout) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public CompletableFuture cancelTask(ExecutionAttemptID
executionAttemptID, Time timeout) {
+ return CompletableFuture.completedFuture(Acknowledge.get());
+ }
+
+ @Override
+ public void heartbeatFromJobManager(ResourceID heartbeatOrigin) {
+ // noop
+ }
+
+ @Override
+ public void heartbeatFromResourceManager(ResourceID heartbeatOrigin) {
+ // noop
+ }
+
+ @Override
+ public void disconnectJobManager(JobID jobId, Exception cause) {
+ // nooop
--- End diff --
[jira] [Commented] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint
[
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271116#comment-16271116
]
ASF GitHub Bot commented on FLINK-7652:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4734
> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
> Issue Type: Sub-task
> Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Labels: flip-6
> Fix For: 1.5.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #4734: [FLINK-7652] [flip6] Port CurrentJobIdsHandler to ...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/4734 ---
[jira] [Resolved] (FLINK-7652) Port CurrentJobIdsHandler to new REST endpoint
[
https://issues.apache.org/jira/browse/FLINK-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann resolved FLINK-7652.
--
Resolution: Done
Added via 67aad88ee025ce02053ab560f2504762f53b87d9
> Port CurrentJobIdsHandler to new REST endpoint
> --
>
> Key: FLINK-7652
> URL: https://issues.apache.org/jira/browse/FLINK-7652
> Project: Flink
> Issue Type: Sub-task
> Components: REST, Webfrontend
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
> Labels: flip-6
> Fix For: 1.5.0
>
>
> Port existing {{CurrentJobIdsHandler}} to new REST endpoint
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5100: [FLINK-4812][metrics] Expose currentLowWatermark for all ...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5100 I think the fact the the operator needs this in `setup()` is already problematic, the latency metrics implementation shouldn't have been merged like this. Wouldn't we need the output gauge only in `OperatorChain` because that creates the `Output`s that operators use to emit watermarks? ---
[jira] [Commented] (FLINK-4812) Report Watermark metrics in all operators
[ https://issues.apache.org/jira/browse/FLINK-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271100#comment-16271100 ] ASF GitHub Bot commented on FLINK-4812: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5100 I think the fact the the operator needs this in `setup()` is already problematic, the latency metrics implementation shouldn't have been merged like this. Wouldn't we need the output gauge only in `OperatorChain` because that creates the `Output`s that operators use to emit watermarks? > Report Watermark metrics in all operators > - > > Key: FLINK-4812 > URL: https://issues.apache.org/jira/browse/FLINK-4812 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.5.0 > > > As reported by a user, Flink does currently not export the current low > watermark for sources > (http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/currentLowWatermark-metric-not-reported-for-all-tasks-td13770.html). > This JIRA is for adding such a metric for the sources as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8171) Remove work arounds in Flip6LocalStreamEnvironment
[
https://issues.apache.org/jira/browse/FLINK-8171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271097#comment-16271097
]
ASF GitHub Bot commented on FLINK-8171:
---
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/5101
I think we can remove the comment but not enabling the queued scheduling
mode which is mandatory for Flip-6 to work. I'll remove the comment.
> Remove work arounds in Flip6LocalStreamEnvironment
> --
>
> Key: FLINK-8171
> URL: https://issues.apache.org/jira/browse/FLINK-8171
> Project: Flink
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Labels: flip-6
> Fix For: 1.5.0
>
>
> After adding FLINK-7956, it is no longer necessary that the
> {{Flip6LocalStreamEnvironment}} waits for the registration of TaskManagers
> before submitting a job. Moreover, it is also possible to use slot sharing
> when submitting jobs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5101: [FLINK-8171] [flip6] Remove work arounds from Flip6LocalS...
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/5101 I think we can remove the comment but not enabling the queued scheduling mode which is mandatory for Flip-6 to work. I'll remove the comment. ---
[GitHub] flink pull request #5102: [FLINK-7762, FLINK-8167] Clean up and harden Wikip...
GitHub user uce opened a pull request: https://github.com/apache/flink/pull/5102 [FLINK-7762, FLINK-8167] Clean up and harden WikipediaEditsSource ## What is the purpose of the change This pull requests addresses two related issues with the WikipediaEditsSource. It makes the WikipediaEditsSourceTest a proper test instead of unnecessarily starting a FlinkMiniCluster and addresses a potential test instability. In general, the WikipediaEditsSource is not in good shape and could benefit from further refactoring. One potential area for improvement is integration with the asynchronous channel listener that reports events like errors or being kicked out of a channel, etc. I did not do this due to time constraints and the fact that this is not a production source. In general, it is questionable whether we should keep the test as is or remove it since it depends on connectivity to an IRC channel. ## Brief change log - Harden WikipediaEditsSource with eager sanity checks - Make WikipediaEditsSourceTest proper test ## Verifying this change This change is a rework/code cleanup without any new test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): yes, but only to `flink-test-utils-junit` - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable You can merge this pull request into a Git repository by running: $ git pull https://github.com/uce/flink 7762-8167-wikiedits Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5102.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5102 commit b2ab66f05ce545214a8132dc2d46b3143939b015 Author: Ufuk Celebi Date: 2017-11-29T15:28:18Z [FLINK-8167] [connector-wikiedits] Harden WikipediaEditsSource - Minor eager sanity checks - Use UUID suffix for nickname. As reported in FLINK-8167, the current nickname suffix can result in nickname clashes which lead to test failures. commit 06ec1542963bbe2afaf1ad1fd55a54d13f855304 Author: Ufuk Celebi Date: 2017-11-29T15:36:29Z [FLINK-7762] [connector-wikiedits] Make WikipediaEditsSourceTest proper test The WikipediaEditsSourceTest unnecessarily implements an integration test that starts a FlinkMiniCluster and executes a small Flink program. This simply creates a source and executes run in a separate thread until a single WikipediaEditEvent is received. ---
[jira] [Commented] (FLINK-7762) Make WikipediaEditsSourceTest a proper test
[
https://issues.apache.org/jira/browse/FLINK-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271083#comment-16271083
]
ASF GitHub Bot commented on FLINK-7762:
---
GitHub user uce opened a pull request:
https://github.com/apache/flink/pull/5102
[FLINK-7762, FLINK-8167] Clean up and harden WikipediaEditsSource
## What is the purpose of the change
This pull requests addresses two related issues with the
WikipediaEditsSource. It makes the WikipediaEditsSourceTest a proper test
instead of unnecessarily starting a FlinkMiniCluster and addresses a potential
test instability.
In general, the WikipediaEditsSource is not in good shape and could benefit
from further refactoring. One potential area for improvement is integration
with the asynchronous channel listener that reports events like errors or being
kicked out of a channel, etc.
I did not do this due to time constraints and the fact that this is not a
production source. In general, it is questionable whether we should keep the
test as is or remove it since it depends on connectivity to an IRC channel.
## Brief change log
- Harden WikipediaEditsSource with eager sanity checks
- Make WikipediaEditsSourceTest proper test
## Verifying this change
This change is a rework/code cleanup without any new test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): yes, but only to
`flink-test-utils-junit`
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uce/flink 7762-8167-wikiedits
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5102.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5102
commit b2ab66f05ce545214a8132dc2d46b3143939b015
Author: Ufuk Celebi
Date: 2017-11-29T15:28:18Z
[FLINK-8167] [connector-wikiedits] Harden WikipediaEditsSource
- Minor eager sanity checks
- Use UUID suffix for nickname. As reported in FLINK-8167, the current
nickname suffix can result in nickname clashes which lead to test
failures.
commit 06ec1542963bbe2afaf1ad1fd55a54d13f855304
Author: Ufuk Celebi
Date: 2017-11-29T15:36:29Z
[FLINK-7762] [connector-wikiedits] Make WikipediaEditsSourceTest proper test
The WikipediaEditsSourceTest unnecessarily implements an integration
test that starts a FlinkMiniCluster and executes a small Flink program.
This simply creates a source and executes run in a separate thread until
a single WikipediaEditEvent is received.
> Make WikipediaEditsSourceTest a proper test
> ---
>
> Key: FLINK-7762
> URL: https://issues.apache.org/jira/browse/FLINK-7762
> Project: Flink
> Issue Type: Improvement
> Components: Streaming Connectors
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Minor
>
> {{WikipediaEditsSourceTest}} is currently an ITCase even though it's called
> test. Making it a test reduces runtime and also makes it more stable because
> we don't run a whole Flink job.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8148) Test instability in YarnFileStageTest
[
https://issues.apache.org/jira/browse/FLINK-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271074#comment-16271074
]
ASF GitHub Bot commented on FLINK-8148:
---
Github user NicoK commented on the issue:
https://github.com/apache/flink/pull/5066
oh, sorry for that
> Test instability in YarnFileStageTest
> -
>
> Key: FLINK-8148
> URL: https://issues.apache.org/jira/browse/FLINK-8148
> Project: Flink
> Issue Type: Bug
> Components: Tests, YARN
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Critical
> Labels: test-stability
>
> {code}
> Running org.apache.flink.yarn.YarnFileStageTestS3ITCase
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 13.152 sec
> <<< FAILURE! - in org.apache.flink.yarn.YarnFileStageTestS3ITCase
> testRecursiveUploadForYarnS3(org.apache.flink.yarn.YarnFileStageTestS3ITCase)
> Time elapsed: 8.515 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarn(YarnFileStageTestS3ITCase.java:171)
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3(YarnFileStageTestS3ITCase.java:192)
> {code}
> from https://travis-ci.org/apache/flink/jobs/305861539
> {{YarnFileStageTestS3ITCase.testRecursiveUploadForYarn}} verifies that the
> test directory used is cleaned up by
> {{YarnFileStageTest.testCopyFromLocalRecursive}} which should clean up the
> directory (in a {{finally}} block). However, for S3, we may not always see
> our own deletes.
> Quoting from https://aws.amazon.com/s3/faqs/ here:
> {quote}Q: What data consistency model does Amazon S3 employ?
> Amazon S3 buckets in all Regions provide read-after-write consistency for
> PUTS of new objects and eventual consistency for overwrite PUTS and
> DELETES.{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5066: [FLINK-8148][yarn/s3] fix test instability in YarnFileSta...
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5066 oh, sorry for that ---
[jira] [Commented] (FLINK-8149) Replace usages of deprecated SerializationSchema
[
https://issues.apache.org/jira/browse/FLINK-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271050#comment-16271050
]
ASF GitHub Bot commented on FLINK-8149:
---
Github user yew1eb commented on the issue:
https://github.com/apache/flink/pull/5069
@tzulitai, thanks for your review. I have updated the PR according to your
comments.
> Replace usages of deprecated SerializationSchema
>
>
> Key: FLINK-8149
> URL: https://issues.apache.org/jira/browse/FLINK-8149
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Affects Versions: 1.4.0
>Reporter: Hai Zhou UTC+8
>Assignee: Hai Zhou UTC+8
> Fix For: 1.5.0
>
>
> The deprecated {{SerializationSchema}} in {{flink-streaming-java}}, has been
> moved to {{flink-core}}.
> But, the deprecate {{SerializationSchema}} is still used in
> {{flink-connector-kinesis}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5069: [FLINK-8149][kinesis] Replace usages of deprecated Serial...
Github user yew1eb commented on the issue: https://github.com/apache/flink/pull/5069 @tzulitai, thanks for your review. I have updated the PR according to your comments. ---
[jira] [Commented] (FLINK-8149) Replace usages of deprecated SerializationSchema
[
https://issues.apache.org/jira/browse/FLINK-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271046#comment-16271046
]
ASF GitHub Bot commented on FLINK-8149:
---
Github user yew1eb commented on a diff in the pull request:
https://github.com/apache/flink/pull/5069#discussion_r153841399
--- Diff:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java
---
@@ -41,7 +42,6 @@
import
org.apache.flink.streaming.connectors.kinesis.serialization.KinesisDeserializationSchema;
import
org.apache.flink.streaming.connectors.kinesis.serialization.KinesisDeserializationSchemaWrapper;
import
org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil;
-import org.apache.flink.streaming.util.serialization.DeserializationSchema;
--- End diff --
make sense.
> Replace usages of deprecated SerializationSchema
>
>
> Key: FLINK-8149
> URL: https://issues.apache.org/jira/browse/FLINK-8149
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Affects Versions: 1.4.0
>Reporter: Hai Zhou UTC+8
>Assignee: Hai Zhou UTC+8
> Fix For: 1.5.0
>
>
> The deprecated {{SerializationSchema}} in {{flink-streaming-java}}, has been
> moved to {{flink-core}}.
> But, the deprecate {{SerializationSchema}} is still used in
> {{flink-connector-kinesis}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #5069: [FLINK-8149][kinesis] Replace usages of deprecated...
Github user yew1eb commented on a diff in the pull request: https://github.com/apache/flink/pull/5069#discussion_r153841399 --- Diff: flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java --- @@ -41,7 +42,6 @@ import org.apache.flink.streaming.connectors.kinesis.serialization.KinesisDeserializationSchema; import org.apache.flink.streaming.connectors.kinesis.serialization.KinesisDeserializationSchemaWrapper; import org.apache.flink.streaming.connectors.kinesis.util.KinesisConfigUtil; -import org.apache.flink.streaming.util.serialization.DeserializationSchema; --- End diff -- make sense. ---
[jira] [Commented] (FLINK-7907) Flink Metrics documentation missing Scala examples
[
https://issues.apache.org/jira/browse/FLINK-7907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271043#comment-16271043
]
ASF GitHub Bot commented on FLINK-7907:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5071#discussion_r153839062
--- Diff: docs/monitoring/metrics.md ---
@@ -61,9 +63,35 @@ public class MyMapper extends RichMapFunction {
}
{% endhighlight %}
+
+
+
+{% highlight scala %}
+
+public class MyMapper extends RichMapFunction[String,String] {
+ var counter: Counter
--- End diff --
reasonable sure this must be lazy (also applies to remaining examples)
> Flink Metrics documentation missing Scala examples
> --
>
> Key: FLINK-7907
> URL: https://issues.apache.org/jira/browse/FLINK-7907
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Colin Williams
>Assignee: Hai Zhou UTC+8
>Priority: Minor
>
> The Flink metrics documentation is missing Scala examples for many of the
> metrics types. To be consistent there should be Scala examples for all the
> types.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7907) Flink Metrics documentation missing Scala examples
[
https://issues.apache.org/jira/browse/FLINK-7907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271042#comment-16271042
]
ASF GitHub Bot commented on FLINK-7907:
---
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5071#discussion_r153838811
--- Diff: docs/monitoring/metrics.md ---
@@ -61,9 +63,35 @@ public class MyMapper extends RichMapFunction {
}
{% endhighlight %}
+
+
+
+{% highlight scala %}
+
+public class MyMapper extends RichMapFunction[String,String] {
--- End diff --
remove public keyword (also applies to remaining examples)
> Flink Metrics documentation missing Scala examples
> --
>
> Key: FLINK-7907
> URL: https://issues.apache.org/jira/browse/FLINK-7907
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Colin Williams
>Assignee: Hai Zhou UTC+8
>Priority: Minor
>
> The Flink metrics documentation is missing Scala examples for many of the
> metrics types. To be consistent there should be Scala examples for all the
> types.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink pull request #5071: [FLINK-7907][docs] The metrics documentation missi...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5071#discussion_r153839062
--- Diff: docs/monitoring/metrics.md ---
@@ -61,9 +63,35 @@ public class MyMapper extends RichMapFunction {
}
{% endhighlight %}
+
+
+
+{% highlight scala %}
+
+public class MyMapper extends RichMapFunction[String,String] {
+ var counter: Counter
--- End diff --
reasonable sure this must be lazy (also applies to remaining examples)
---
[GitHub] flink pull request #5071: [FLINK-7907][docs] The metrics documentation missi...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/5071#discussion_r153838811
--- Diff: docs/monitoring/metrics.md ---
@@ -61,9 +63,35 @@ public class MyMapper extends RichMapFunction {
}
{% endhighlight %}
+
+
+
+{% highlight scala %}
+
+public class MyMapper extends RichMapFunction[String,String] {
--- End diff --
remove public keyword (also applies to remaining examples)
---
[jira] [Commented] (FLINK-8158) Rowtime window inner join emits late data
[
https://issues.apache.org/jira/browse/FLINK-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271030#comment-16271030
]
ASF GitHub Bot commented on FLINK-8158:
---
Github user xccui commented on the issue:
https://github.com/apache/flink/pull/5094
Hi @hequn8128, as @fhueske explained, watermarks should not define the
semantics of the join. We just emit join results, in spite of whether they are
late and the join function should not care about the lateness. You can take
"holding back watermarks" as a special watermark assigner, based on which,
there could be out-of-ordered data (join results) being taken as late.
Currently, we just generates watermarks with a static delay that equals to the
larger relative offset. In the future, we should dynamically decide whether a
join result is late according to the real cache size.
> Rowtime window inner join emits late data
> -
>
> Key: FLINK-8158
> URL: https://issues.apache.org/jira/browse/FLINK-8158
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
> Attachments: screenshot-1xxx.png
>
>
> When executing the join, the join operator needs to make sure that no late
> data is emitted. Currently, this achieved by holding back watermarks.
> However, the window border is not handled correctly. For the sql bellow:
> {quote}
> val sqlQuery =
> """
> SELECT t2.key, t2.id, t1.id
> FROM T1 as t1 join T2 as t2 ON
> t1.key = t2.key AND
> t1.rt BETWEEN t2.rt - INTERVAL '5' SECOND AND
> t2.rt + INTERVAL '1' SECOND
> """.stripMargin
> val data1 = new mutable.MutableList[(String, String, Long)]
> // for boundary test
> data1.+=(("A", "LEFT1", 6000L))
> val data2 = new mutable.MutableList[(String, String, Long)]
> data2.+=(("A", "RIGHT1", 6000L))
> {quote}
> Join will output a watermark with timestamp 1000, but if left comes with
> another data ("A", "LEFT1", 1000L), join will output a record with timestamp
> 1000 which equals previous watermark.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5094: [FLINK-8158] [table] Fix rowtime window inner join emits ...
Github user xccui commented on the issue: https://github.com/apache/flink/pull/5094 Hi @hequn8128, as @fhueske explained, watermarks should not define the semantics of the join. We just emit join results, in spite of whether they are late and the join function should not care about the lateness. You can take "holding back watermarks" as a special watermark assigner, based on which, there could be out-of-ordered data (join results) being taken as late. Currently, we just generates watermarks with a static delay that equals to the larger relative offset. In the future, we should dynamically decide whether a join result is late according to the real cache size. ---
[jira] [Commented] (FLINK-8148) Test instability in YarnFileStageTest
[
https://issues.apache.org/jira/browse/FLINK-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271028#comment-16271028
]
ASF GitHub Bot commented on FLINK-8148:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5066
will fix the checkstyle violation on the way.
> Test instability in YarnFileStageTest
> -
>
> Key: FLINK-8148
> URL: https://issues.apache.org/jira/browse/FLINK-8148
> Project: Flink
> Issue Type: Bug
> Components: Tests, YARN
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Critical
> Labels: test-stability
>
> {code}
> Running org.apache.flink.yarn.YarnFileStageTestS3ITCase
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 13.152 sec
> <<< FAILURE! - in org.apache.flink.yarn.YarnFileStageTestS3ITCase
> testRecursiveUploadForYarnS3(org.apache.flink.yarn.YarnFileStageTestS3ITCase)
> Time elapsed: 8.515 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarn(YarnFileStageTestS3ITCase.java:171)
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3(YarnFileStageTestS3ITCase.java:192)
> {code}
> from https://travis-ci.org/apache/flink/jobs/305861539
> {{YarnFileStageTestS3ITCase.testRecursiveUploadForYarn}} verifies that the
> test directory used is cleaned up by
> {{YarnFileStageTest.testCopyFromLocalRecursive}} which should clean up the
> directory (in a {{finally}} block). However, for S3, we may not always see
> our own deletes.
> Quoting from https://aws.amazon.com/s3/faqs/ here:
> {quote}Q: What data consistency model does Amazon S3 employ?
> Amazon S3 buckets in all Regions provide read-after-write consistency for
> PUTS of new objects and eventual consistency for overwrite PUTS and
> DELETES.{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5066: [FLINK-8148][yarn/s3] fix test instability in YarnFileSta...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5066 will fix the checkstyle violation on the way. ---
[jira] [Commented] (FLINK-8148) Test instability in YarnFileStageTest
[
https://issues.apache.org/jira/browse/FLINK-8148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271025#comment-16271025
]
ASF GitHub Bot commented on FLINK-8148:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5066
merging.
> Test instability in YarnFileStageTest
> -
>
> Key: FLINK-8148
> URL: https://issues.apache.org/jira/browse/FLINK-8148
> Project: Flink
> Issue Type: Bug
> Components: Tests, YARN
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Critical
> Labels: test-stability
>
> {code}
> Running org.apache.flink.yarn.YarnFileStageTestS3ITCase
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 13.152 sec
> <<< FAILURE! - in org.apache.flink.yarn.YarnFileStageTestS3ITCase
> testRecursiveUploadForYarnS3(org.apache.flink.yarn.YarnFileStageTestS3ITCase)
> Time elapsed: 8.515 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarn(YarnFileStageTestS3ITCase.java:171)
> at
> org.apache.flink.yarn.YarnFileStageTestS3ITCase.testRecursiveUploadForYarnS3(YarnFileStageTestS3ITCase.java:192)
> {code}
> from https://travis-ci.org/apache/flink/jobs/305861539
> {{YarnFileStageTestS3ITCase.testRecursiveUploadForYarn}} verifies that the
> test directory used is cleaned up by
> {{YarnFileStageTest.testCopyFromLocalRecursive}} which should clean up the
> directory (in a {{finally}} block). However, for S3, we may not always see
> our own deletes.
> Quoting from https://aws.amazon.com/s3/faqs/ here:
> {quote}Q: What data consistency model does Amazon S3 employ?
> Amazon S3 buckets in all Regions provide read-after-write consistency for
> PUTS of new objects and eventual consistency for overwrite PUTS and
> DELETES.{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5066: [FLINK-8148][yarn/s3] fix test instability in YarnFileSta...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5066 merging. ---
[jira] [Commented] (FLINK-8171) Remove work arounds in Flip6LocalStreamEnvironment
[
https://issues.apache.org/jira/browse/FLINK-8171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271021#comment-16271021
]
ASF GitHub Bot commented on FLINK-8171:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5101
Can we also remove this bit
```
// temporary hack for FLIP-6 since slot-sharing isn't implemented yet
jobGraph.setAllowQueuedScheduling(true);
```
from the `RestClusterClient`?
> Remove work arounds in Flip6LocalStreamEnvironment
> --
>
> Key: FLINK-8171
> URL: https://issues.apache.org/jira/browse/FLINK-8171
> Project: Flink
> Issue Type: Improvement
> Components: Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Labels: flip-6
> Fix For: 1.5.0
>
>
> After adding FLINK-7956, it is no longer necessary that the
> {{Flip6LocalStreamEnvironment}} waits for the registration of TaskManagers
> before submitting a job. Moreover, it is also possible to use slot sharing
> when submitting jobs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5101: [FLINK-8171] [flip6] Remove work arounds from Flip6LocalS...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5101 Can we also remove this bit ``` // temporary hack for FLIP-6 since slot-sharing isn't implemented yet jobGraph.setAllowQueuedScheduling(true); ``` from the `RestClusterClient`? ---
[jira] [Closed] (FLINK-7989) flink-dist jar is deployed twice onto YARN
[
https://issues.apache.org/jira/browse/FLINK-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-7989.
---
Resolution: Fixed
Fix Version/s: (was: 1.4.1)
1.4.0
1.4: 3d568f52070a7719803a2bea5346d0e6f2a72615
> flink-dist jar is deployed twice onto YARN
> --
>
> Key: FLINK-7989
> URL: https://issues.apache.org/jira/browse/FLINK-7989
> Project: Flink
> Issue Type: Bug
> Components: YARN
>Affects Versions: 1.4.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
> Fix For: 1.4.0, 1.5.0
>
>
> We always add the {{flink-dist*.jar}} ourselves, but it could also be inside
> a shipped folder such as the "lib/" folder and is then deployed again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-8171) Remove work arounds in Flip6LocalStreamEnvironment
[ https://issues.apache.org/jira/browse/FLINK-8171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270996#comment-16270996 ] ASF GitHub Bot commented on FLINK-8171: --- GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/5101 [FLINK-8171] [flip6] Remove work arounds from Flip6LocalStreamEnvironment ## What is the purpose of the change It is no longer needed to wait for the registration of task managers and to not use slot sharing when submitting jobs to the Flip-6 MiniCluster. Therefore, we can remove these work arounds from the Flip6LocalStreamEnvironment. This PR is based on #5091. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink fixLocalStreamEnvironmentITCase Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5101.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5101 commit d30dde83548dbeff4249f3b57b67cdb6247af510 Author: Till Rohrmann Date: 2017-11-14T22:50:52Z [FLINK-8078] Introduce LogicalSlot interface The LogicalSlot interface decouples the task deployment from the actual slot implementation which at the moment is Slot, SimpleSlot and SharedSlot. This is a helpful step to introduce a different slot implementation for Flip-6. commit e5da9566a6fc8a36ac8b06bae911c0dff5554e5d Author: Till Rohrmann Date: 2017-11-15T13:20:27Z [FLINK-8085] Thin out LogicalSlot interface Remove isCanceled, isReleased method and decouple logical slot from Execution by introducing a Payload interface which is set for a LogicalSlot. The Payload interface is implemented by the Execution and allows to fail an implementation and obtaining a termination future. Introduce proper Execution#releaseFuture which is completed once the Execution's assigned resource has been released. commit 84d86bebe2f9f8395430e7c71dd2393ba117b44f Author: Till Rohrmann Date: 2017-11-24T17:03:49Z [FLINK-8087] Decouple Slot from AllocatedSlot This commit introduces the SlotContext which is an abstraction for the SimpleSlot to obtain the relevant slot information to do the communication with the TaskManager without relying on the AllocatedSlot which is now only used by the SlotPool. commit 80a3cc848a0c724a2bc09b1b967cc9e6ccec5942 Author: Till Rohrmann Date: 2017-11-24T17:06:10Z [FLINK-8088] Associate logical slots with the slot request id Before logical slots like the SimpleSlot and SharedSlot where associated to the actually allocated slot via the AllocationID. This, however, was sub-optimal because allocated slots can be re-used to fulfill also other slot requests (logical slots). Therefore, we should bind the logical slots to the right id with the right lifecycle which is the slot request id. commit 3e4550c0607744b20893dc90c587b63e68e4de1e Author: Till Rohrmann Date: 2017-11-13T14:42:07Z [FLINK-8089] Also check for other pending slot requests in offerSlot Not only check for a slot request with the right allocation id but also check whether we can fulfill other pending slot requests with an unclaimed offered slot before adding it to the list of available slots. commit b04dda46aaf298d921929910574662970d9c5093 Author: Till Rohrmann Date: 2017-11-24T22:29:53Z [hotfix] Speed up RecoveryITCase commit e512558917f9bb5005024630b8a015cd624164b4 Author: Till Rohrmann Date: 2017-11-24T17:08:38Z [FLINK-7956] [flip6] Add support for queued scheduling with slot sharing to SlotPool This commit adds support for queued scheduling with slot sharing to the SlotPool. The idea of slot sharing is that multiple tasks can run in the same slot. Moreover, queued scheduling means that a slot request must not be completed right away but at a later point in time. This allows to start new TaskExecutors in case that there are no more slots left. The main component responsible for the management o
[jira] [Commented] (FLINK-7574) Remove unused dependencies from flink-clients
[ https://issues.apache.org/jira/browse/FLINK-7574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270997#comment-16270997 ] ASF GitHub Bot commented on FLINK-7574: --- Github user yew1eb commented on the issue: https://github.com/apache/flink/pull/5076 Hi @greghogan, You're quite right, Thanks for your advice. For a discussion of this improvement, you can see #4712. > Remove unused dependencies from flink-clients > - > > Key: FLINK-7574 > URL: https://issues.apache.org/jira/browse/FLINK-7574 > Project: Flink > Issue Type: Sub-task > Components: Build System >Affects Versions: 1.3.2 > Environment: Apache Maven 3.3.9, Java version: 1.8.0_144 >Reporter: Hai Zhou UTC+8 >Assignee: Hai Zhou UTC+8 > > [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ > flink-clients_2.11 --- > [WARNING] Used undeclared dependencies found: > [WARNING]org.scala-lang:scala-library:jar:2.11.11:compile > [WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile > [WARNING] Unused declared dependencies found: > [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test > [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile > [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test > [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile > [WARNING]log4j:log4j:jar:1.2.17:test > [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test > [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #5076: [FLINK-7574][build] POM Cleanup flink-clients
Github user yew1eb commented on the issue: https://github.com/apache/flink/pull/5076 Hi @greghogan, You're quite right, Thanks for your advice. For a discussion of this improvement, you can see #4712. ---
[GitHub] flink pull request #5101: [FLINK-8171] [flip6] Remove work arounds from Flip...
GitHub user tillrohrmann opened a pull request: https://github.com/apache/flink/pull/5101 [FLINK-8171] [flip6] Remove work arounds from Flip6LocalStreamEnvironment ## What is the purpose of the change It is no longer needed to wait for the registration of task managers and to not use slot sharing when submitting jobs to the Flip-6 MiniCluster. Therefore, we can remove these work arounds from the Flip6LocalStreamEnvironment. This PR is based on #5091. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/tillrohrmann/flink fixLocalStreamEnvironmentITCase Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/5101.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #5101 commit d30dde83548dbeff4249f3b57b67cdb6247af510 Author: Till Rohrmann Date: 2017-11-14T22:50:52Z [FLINK-8078] Introduce LogicalSlot interface The LogicalSlot interface decouples the task deployment from the actual slot implementation which at the moment is Slot, SimpleSlot and SharedSlot. This is a helpful step to introduce a different slot implementation for Flip-6. commit e5da9566a6fc8a36ac8b06bae911c0dff5554e5d Author: Till Rohrmann Date: 2017-11-15T13:20:27Z [FLINK-8085] Thin out LogicalSlot interface Remove isCanceled, isReleased method and decouple logical slot from Execution by introducing a Payload interface which is set for a LogicalSlot. The Payload interface is implemented by the Execution and allows to fail an implementation and obtaining a termination future. Introduce proper Execution#releaseFuture which is completed once the Execution's assigned resource has been released. commit 84d86bebe2f9f8395430e7c71dd2393ba117b44f Author: Till Rohrmann Date: 2017-11-24T17:03:49Z [FLINK-8087] Decouple Slot from AllocatedSlot This commit introduces the SlotContext which is an abstraction for the SimpleSlot to obtain the relevant slot information to do the communication with the TaskManager without relying on the AllocatedSlot which is now only used by the SlotPool. commit 80a3cc848a0c724a2bc09b1b967cc9e6ccec5942 Author: Till Rohrmann Date: 2017-11-24T17:06:10Z [FLINK-8088] Associate logical slots with the slot request id Before logical slots like the SimpleSlot and SharedSlot where associated to the actually allocated slot via the AllocationID. This, however, was sub-optimal because allocated slots can be re-used to fulfill also other slot requests (logical slots). Therefore, we should bind the logical slots to the right id with the right lifecycle which is the slot request id. commit 3e4550c0607744b20893dc90c587b63e68e4de1e Author: Till Rohrmann Date: 2017-11-13T14:42:07Z [FLINK-8089] Also check for other pending slot requests in offerSlot Not only check for a slot request with the right allocation id but also check whether we can fulfill other pending slot requests with an unclaimed offered slot before adding it to the list of available slots. commit b04dda46aaf298d921929910574662970d9c5093 Author: Till Rohrmann Date: 2017-11-24T22:29:53Z [hotfix] Speed up RecoveryITCase commit e512558917f9bb5005024630b8a015cd624164b4 Author: Till Rohrmann Date: 2017-11-24T17:08:38Z [FLINK-7956] [flip6] Add support for queued scheduling with slot sharing to SlotPool This commit adds support for queued scheduling with slot sharing to the SlotPool. The idea of slot sharing is that multiple tasks can run in the same slot. Moreover, queued scheduling means that a slot request must not be completed right away but at a later point in time. This allows to start new TaskExecutors in case that there are no more slots left. The main component responsible for the management of shared slots is the SlotSharingManager. The SlotSharingManager maintains internally a tree-like structure which stores the SlotContext future of the underlying AllocatedSlot. Whenever this future is completed potentially pending LogicalSl
[jira] [Created] (FLINK-8171) Remove work arounds in Flip6LocalStreamEnvironment
Till Rohrmann created FLINK-8171:
Summary: Remove work arounds in Flip6LocalStreamEnvironment
Key: FLINK-8171
URL: https://issues.apache.org/jira/browse/FLINK-8171
Project: Flink
Issue Type: Improvement
Components: Tests
Affects Versions: 1.5.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Fix For: 1.5.0
After adding FLINK-7956, it is no longer necessary that the
{{Flip6LocalStreamEnvironment}} waits for the registration of TaskManagers
before submitting a job. Moreover, it is also possible to use slot sharing when
submitting jobs.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7573) Introduce Http protocol connector for Elasticsearch2
[
https://issues.apache.org/jira/browse/FLINK-7573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270979#comment-16270979
]
Dylan Meissner commented on FLINK-7573:
---
Worth noting, as of Elasticsearch 5.6 there is an official [Java High Level
REST
Client](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high.html).
It _accepts the same request arguments as the TransportClient and returns the
same response objects_.
> Introduce Http protocol connector for Elasticsearch2
>
>
> Key: FLINK-7573
> URL: https://issues.apache.org/jira/browse/FLINK-7573
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector
>Reporter: mingleizhang
>
> Currently, all connectors as far as I have known that merely support the TCP
> transport protocol of Elasticsearch, but some of company's ES cluster just
> relies on the HTTP protocol, and close the TCP port on production
> environment. So, I suggest add a new implemention for creating a HTTP
> protocol by using {{JestClient}}, which is a Java HTTP Rest client for
> ElasticSearch.
> FYI
> I used 9300 port to access ES cluster. It is really awful and scared. Because
> program can not run on a production environment based on a
> {{TransportClient}}. So, I have to access the ES cluster by 9200 port
> instead. 9300 is for TCP. 9200 is for HTTP for accessing I guess here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (FLINK-7573) Introduce Http protocol connector for Elasticsearch2
[
https://issues.apache.org/jira/browse/FLINK-7573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270979#comment-16270979
]
Dylan Meissner edited comment on FLINK-7573 at 11/29/17 3:52 PM:
-
Worth noting, as of Elasticsearch 5.6 there is an official "Java High Level
REST Client". It _accepts the same request arguments as the TransportClient and
returns the same response objects_.
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high.html
was (Author: dylanmei):
Worth noting, as of Elasticsearch 5.6 there is an official [Java High Level
REST
Client](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high.html).
It _accepts the same request arguments as the TransportClient and returns the
same response objects_.
> Introduce Http protocol connector for Elasticsearch2
>
>
> Key: FLINK-7573
> URL: https://issues.apache.org/jira/browse/FLINK-7573
> Project: Flink
> Issue Type: Improvement
> Components: ElasticSearch Connector
>Reporter: mingleizhang
>
> Currently, all connectors as far as I have known that merely support the TCP
> transport protocol of Elasticsearch, but some of company's ES cluster just
> relies on the HTTP protocol, and close the TCP port on production
> environment. So, I suggest add a new implemention for creating a HTTP
> protocol by using {{JestClient}}, which is a Java HTTP Rest client for
> ElasticSearch.
> FYI
> I used 9300 port to access ES cluster. It is really awful and scared. Because
> program can not run on a production environment based on a
> {{TransportClient}}. So, I have to access the ES cluster by 9200 port
> instead. 9300 is for TCP. 9200 is for HTTP for accessing I guess here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-4812) Report Watermark metrics in all operators
[ https://issues.apache.org/jira/browse/FLINK-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270976#comment-16270976 ] ASF GitHub Bot commented on FLINK-4812: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5100 the coupling of input to output gauge becomes rather tricky. The input gauge must be reachable in `Stream[Two]InputProcessor`, `OperatorChain#createChainedOperator`, and the output gauge in `AbstractStreamOperator#setup`. For sources, sinks and watermark assigners we must refer to the same gauge in both places. I don't see how we can do this in a nice way. The rather nice thing the current approach has is that everything is contained in `AbstractStreamOperator` and isn't scattered all over the place. `Given that we allow people to completely re-define how operators handle watermarks, wouldn't it make sense to give them the same freedom when it comes to the input watermark metrics? > Report Watermark metrics in all operators > - > > Key: FLINK-4812 > URL: https://issues.apache.org/jira/browse/FLINK-4812 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.5.0 > > > As reported by a user, Flink does currently not export the current low > watermark for sources > (http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/currentLowWatermark-metric-not-reported-for-all-tasks-td13770.html). > This JIRA is for adding such a metric for the sources as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #5100: [FLINK-4812][metrics] Expose currentLowWatermark for all ...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5100 the coupling of input to output gauge becomes rather tricky. The input gauge must be reachable in `Stream[Two]InputProcessor`, `OperatorChain#createChainedOperator`, and the output gauge in `AbstractStreamOperator#setup`. For sources, sinks and watermark assigners we must refer to the same gauge in both places. I don't see how we can do this in a nice way. The rather nice thing the current approach has is that everything is contained in `AbstractStreamOperator` and isn't scattered all over the place. `Given that we allow people to completely re-define how operators handle watermarks, wouldn't it make sense to give them the same freedom when it comes to the input watermark metrics? ---
[jira] [Commented] (FLINK-7762) Make WikipediaEditsSourceTest a proper test
[
https://issues.apache.org/jira/browse/FLINK-7762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270974#comment-16270974
]
Hai Zhou UTC+8 commented on FLINK-7762:
---
Hi [~uce], yes. If you want, then you can work on it.
> Make WikipediaEditsSourceTest a proper test
> ---
>
> Key: FLINK-7762
> URL: https://issues.apache.org/jira/browse/FLINK-7762
> Project: Flink
> Issue Type: Improvement
> Components: Streaming Connectors
>Reporter: Aljoscha Krettek
>Assignee: Ufuk Celebi
>Priority: Minor
>
> {{WikipediaEditsSourceTest}} is currently an ITCase even though it's called
> test. Making it a test reduces runtime and also makes it more stable because
> we don't run a whole Flink job.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-4812) Report Watermark metrics in all operators
[ https://issues.apache.org/jira/browse/FLINK-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270954#comment-16270954 ] ASF GitHub Bot commented on FLINK-4812: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5100 ah, weÄ#re doing the same already for `numRecordsIn` maybe we can do this after all. > Report Watermark metrics in all operators > - > > Key: FLINK-4812 > URL: https://issues.apache.org/jira/browse/FLINK-4812 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.5.0 > > > As reported by a user, Flink does currently not export the current low > watermark for sources > (http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/currentLowWatermark-metric-not-reported-for-all-tasks-td13770.html). > This JIRA is for adding such a metric for the sources as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #5100: [FLINK-4812][metrics] Expose currentLowWatermark for all ...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5100 ah, weÃ#re doing the same already for `numRecordsIn` maybe we can do this after all. ---
[jira] [Commented] (FLINK-7595) Removing stateless task from task chain breaks savepoint restore
[
https://issues.apache.org/jira/browse/FLINK-7595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270939#comment-16270939
]
ASF GitHub Bot commented on FLINK-7595:
---
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/4651
Rebased on top of master, let's see what travis says. Didn't get any
conflicts so 🤞 .
> Removing stateless task from task chain breaks savepoint restore
>
>
> Key: FLINK-7595
> URL: https://issues.apache.org/jira/browse/FLINK-7595
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Ufuk Celebi
>Assignee: Chesnay Schepler
> Attachments: ChainedTaskRemoveTest.java
>
>
> When removing a stateless operator from a 2-task chain where the head
> operator is stateful breaks savepoint restore with
> {code}
> Caused by: java.lang.IllegalStateException: Failed to rollback to savepoint
> /var/folders/py/s_1l8vln6f19ygc77m8c4zhrgn/T/junit1167397515334838028/junit8006766303945373008/savepoint-cb0bcf-3cfa67865ac0.
> Cannot map savepoint state...
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4651: [FLINK-7595] [Savepoints] Allow removing stateless operat...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/4651 Rebased on top of master, let's see what travis says. Didn't get any conflicts so ð¤ . ---
[jira] [Commented] (FLINK-4812) Report Watermark metrics in all operators
[ https://issues.apache.org/jira/browse/FLINK-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270933#comment-16270933 ] ASF GitHub Bot commented on FLINK-4812: --- Github user zentol commented on the issue: https://github.com/apache/flink/pull/5100 You can't measure input metrics in the ChainingOutput (nicely) since for a chain A->B does B at no point have access to the output of A. We would need to have the task setup all watermark metrics and either pass them on to operators for registration or have the task register them for operators, which is just backwards. This also clashes heavily with the idea of operators deciding whether the same metric is used for input&output. > Report Watermark metrics in all operators > - > > Key: FLINK-4812 > URL: https://issues.apache.org/jira/browse/FLINK-4812 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.5.0 > > > As reported by a user, Flink does currently not export the current low > watermark for sources > (http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/currentLowWatermark-metric-not-reported-for-all-tasks-td13770.html). > This JIRA is for adding such a metric for the sources as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (FLINK-8167) Failing WikipediaEditsSourceTest on Travis
[
https://issues.apache.org/jira/browse/FLINK-8167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270934#comment-16270934
]
Ufuk Celebi commented on FLINK-8167:
I reproduced this locally. A nickname clash as indicated by the Travis run log
results in a connection timeout which subsequently results in the test to
timeout.
> Failing WikipediaEditsSourceTest on Travis
> --
>
> Key: FLINK-8167
> URL: https://issues.apache.org/jira/browse/FLINK-8167
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Ufuk Celebi
>Priority: Critical
> Labels: test-stability
> Fix For: 1.5.0
>
>
> I've observed the {{WikipediaEditsSourceTest}} failing on Travis.
> https://travis-ci.org/tillrohrmann/flink/jobs/308351298
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5100: [FLINK-4812][metrics] Expose currentLowWatermark for all ...
Github user zentol commented on the issue: https://github.com/apache/flink/pull/5100 You can't measure input metrics in the ChainingOutput (nicely) since for a chain A->B does B at no point have access to the output of A. We would need to have the task setup all watermark metrics and either pass them on to operators for registration or have the task register them for operators, which is just backwards. This also clashes heavily with the idea of operators deciding whether the same metric is used for input&output. ---
[jira] [Commented] (FLINK-8158) Rowtime window inner join emits late data
[
https://issues.apache.org/jira/browse/FLINK-8158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270924#comment-16270924
]
ASF GitHub Bot commented on FLINK-8158:
---
Github user hequn8128 commented on the issue:
https://github.com/apache/flink/pull/5094
Thanks for your replies.
@fhueske : The watermark must be aligned with timestamps, and it is the
main reason why watermarks are hold back(right?). Current window join may
output a record with timestamp equals to the earlier **output** watermark(see
`testRowTimeJoinWithCommonBounds2 ` in `JoinHarnessTest `).
@xccui : I am considering that if we can cache more late records to have a
more complete join result, and this can be achieved by caching both left and
right data that is later than the holding back watermark.
> Rowtime window inner join emits late data
> -
>
> Key: FLINK-8158
> URL: https://issues.apache.org/jira/browse/FLINK-8158
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Reporter: Hequn Cheng
>Assignee: Hequn Cheng
> Attachments: screenshot-1xxx.png
>
>
> When executing the join, the join operator needs to make sure that no late
> data is emitted. Currently, this achieved by holding back watermarks.
> However, the window border is not handled correctly. For the sql bellow:
> {quote}
> val sqlQuery =
> """
> SELECT t2.key, t2.id, t1.id
> FROM T1 as t1 join T2 as t2 ON
> t1.key = t2.key AND
> t1.rt BETWEEN t2.rt - INTERVAL '5' SECOND AND
> t2.rt + INTERVAL '1' SECOND
> """.stripMargin
> val data1 = new mutable.MutableList[(String, String, Long)]
> // for boundary test
> data1.+=(("A", "LEFT1", 6000L))
> val data2 = new mutable.MutableList[(String, String, Long)]
> data2.+=(("A", "RIGHT1", 6000L))
> {quote}
> Join will output a watermark with timestamp 1000, but if left comes with
> another data ("A", "LEFT1", 1000L), join will output a record with timestamp
> 1000 which equals previous watermark.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #5094: [FLINK-8158] [table] Fix rowtime window inner join emits ...
Github user hequn8128 commented on the issue: https://github.com/apache/flink/pull/5094 Thanks for your replies. @fhueske : The watermark must be aligned with timestamps, and it is the main reason why watermarks are hold back(right?). Current window join may output a record with timestamp equals to the earlier **output** watermark(see `testRowTimeJoinWithCommonBounds2 ` in `JoinHarnessTest `). @xccui : I am considering that if we can cache more late records to have a more complete join result, and this can be achieved by caching both left and right data that is later than the holding back watermark. ---
[jira] [Commented] (FLINK-7595) Removing stateless task from task chain breaks savepoint restore
[
https://issues.apache.org/jira/browse/FLINK-7595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270923#comment-16270923
]
ASF GitHub Bot commented on FLINK-7595:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/4651
This change looks good! Does it still build when rebased on
master/release-1.4?
> Removing stateless task from task chain breaks savepoint restore
>
>
> Key: FLINK-7595
> URL: https://issues.apache.org/jira/browse/FLINK-7595
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Reporter: Ufuk Celebi
>Assignee: Chesnay Schepler
> Attachments: ChainedTaskRemoveTest.java
>
>
> When removing a stateless operator from a 2-task chain where the head
> operator is stateful breaks savepoint restore with
> {code}
> Caused by: java.lang.IllegalStateException: Failed to rollback to savepoint
> /var/folders/py/s_1l8vln6f19ygc77m8c4zhrgn/T/junit1167397515334838028/junit8006766303945373008/savepoint-cb0bcf-3cfa67865ac0.
> Cannot map savepoint state...
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] flink issue #4651: [FLINK-7595] [Savepoints] Allow removing stateless operat...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/4651 This change looks good! Does it still build when rebased on master/release-1.4? ---
[jira] [Commented] (FLINK-4812) Report Watermark metrics in all operators
[ https://issues.apache.org/jira/browse/FLINK-4812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270898#comment-16270898 ] ASF GitHub Bot commented on FLINK-4812: --- Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5100 Ah stupid, so we have to do it in the chain and in the input processors. Because the problem of operators overriding `processWatermark()` is real. 😅 > Report Watermark metrics in all operators > - > > Key: FLINK-4812 > URL: https://issues.apache.org/jira/browse/FLINK-4812 > Project: Flink > Issue Type: Improvement > Components: Metrics >Reporter: Robert Metzger >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.5.0 > > > As reported by a user, Flink does currently not export the current low > watermark for sources > (http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/currentLowWatermark-metric-not-reported-for-all-tasks-td13770.html). > This JIRA is for adding such a metric for the sources as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] flink issue #5100: [FLINK-4812][metrics] Expose currentLowWatermark for all ...
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/5100 Ah stupid, so we have to do it in the chain and in the input processors. Because the problem of operators overriding `processWatermark()` is real. ð ---
