[jira] [Created] (FLINK-5703) ExecutionGraph recovery based on reconciliation with TaskManager reports

[Zhijiang Wang (JIRA)]

Zhijiang Wang created FLINK-5703:


 Summary: ExecutionGraph recovery based on reconciliation with 
TaskManager reports
 Key: FLINK-5703
 URL: https://issues.apache.org/jira/browse/FLINK-5703
 Project: Flink
  Issue Type: Sub-task
Reporter: Zhijiang Wang
Assignee: Zhijiang Wang






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (FLINK-3398) Flink Kafka consumer should support auto-commit opt-outs

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849733#comment-15849733
 ] 

Tzu-Li (Gordon) Tai edited comment on FLINK-3398 at 2/3/17 6:39 AM:


I think we should keep this JIRA instead of merging it into FLINK-4280.
They try to address different new features: FLINK-4280 is for more flexible 
start position configuration independent from Kafka offsets.
This JIRA is for opt-out from offset committing on Flink's checkpoints.

I'll move the respective proposed API for commit opt-out here:

{code}
Properties props = new Properties();
...
FlinkKafkaConsumer09 kafka = new FlinkKafkaConsumer09("topic", schema, props);
kafka.setEnableOffsetCommittingOnCheckpoints(boolean); // if true 
(checkpointing should be enabled), overrides periodic checkpointing if 
"enable.auto.commit" is set in props.
{code}

So the scenarios breaks down into:

1. {{props.put("auto.commit.enable", "true")}} & 
{{setEnableOffsetCommittingOnCheckpoints(false)}}:
Perform auto periodic committing with the internal client (or in 0.8 case our 
own periodic committer), regardless of whether or not Flink checkpointing is 
on. Usage documentation should state clearly that if using Kafka's auto 
periodic committing, the offsets in Kafka will not be in-sync with the 
checkpoint offsets.

2. {{props.put("auto.commit.enable", "true")}} & 
{{setEnableOffsetCommittingOnCheckpoints(true)}}:
Overrides whatever value in props for "auto.commit.enable", only commit on 
checkpoints. If checkpointing isn't enabled, the consumer fails due to 
conflicting configuration.

3. {{props.put("auto.commit.enable", "false")}} & 
{{setEnableOffsetCommittingOnCheckpoints(false)}}: No offset committing at all.



was (Author: tzulitai):
I think we should keep this JIRA instead of merging it into FLINK-4280.
They try to address different new features: FLINK-4280 is for more flexible 
start position configuration independent from Kafka offsets.
This JIRA is for opt-out from offset committing on Flink's checkpoints.

I'll move the respective proposed API for commit opt-out here:

{code}
Properties props = new Properties();
...
FlinkKafkaConsumer09 kafka = new FlinkKafkaConsumer09("topic", schema, props);
kafka.setEnableOffsetCommittingOnCheckpoints(boolean); // if true 
(checkpointing should be enabled), overrides periodic checkpointing if 
"enable.auto.commit" is set in props.
{code}

So the scenarios breaks down into:

1. {{props.put("auto.commit.enable", "true")}} & 
{{setEnableOffsetCommittingOnCheckpoints(false)}}:
Perform auto periodic committing with the internal client (or in 0.8 case our 
own periodic committer), regardless of whether or not checkpointing is on. 
Usage documentation should state clearly that if using Kafka's auto periodic 
committing, the offsets in Kafka will not be in-sync with the checkpoint 
offsets.

2. {{props.put("auto.commit.enable", "true")}} & 
{{setEnableOffsetCommittingOnCheckpoints(true)}}:
Overrides whatever value in props for "auto.commit.enable", only commit on 
checkpoints. If checkpointing isn't enabled, the consumer fails due to 
conflicting configuration.

3. {{props.put("auto.commit.enable", "false")}} & 
{{setEnableOffsetCommittingOnCheckpoints(false)}}: No offset committing at all.


> Flink Kafka consumer should support auto-commit opt-outs
> 
>
> Key: FLINK-3398
> URL: https://issues.apache.org/jira/browse/FLINK-3398
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Shikhar Bhushan
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Critical
> Fix For: 1.3.0
>
>
> Currently the Kafka source will commit consumer offsets to Zookeeper, either 
> upon a checkpoint if checkpointing is enabled, otherwise periodically based 
> on {{auto.commit.interval.ms}}
> It should be possible to opt-out of committing consumer offsets to Zookeeper. 
> Kafka has this config as {{auto.commit.enable}} (0.8) and 
> {{enable.auto.commit}} (0.9).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (FLINK-3398) Flink Kafka consumer should support auto-commit opt-outs

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849733#comment-15849733
 ] 

Tzu-Li (Gordon) Tai edited comment on FLINK-3398 at 2/3/17 6:39 AM:


I think we should keep this JIRA instead of merging it into FLINK-4280.
They try to address different new features: FLINK-4280 is for more flexible 
start position configuration independent from Kafka offsets.
This JIRA is for opt-out from offset committing on Flink's checkpoints.

I'll move the respective proposed API for commit opt-out here:

{code}
Properties props = new Properties();
...
FlinkKafkaConsumer09 kafka = new FlinkKafkaConsumer09("topic", schema, props);
kafka.setEnableOffsetCommittingOnCheckpoints(boolean); // if true 
(checkpointing should be enabled), overrides periodic checkpointing if 
"enable.auto.commit" is set in props.
{code}

So the scenarios breaks down into:

1. {{props.put("auto.commit.enable", "true")}} & 
{{setEnableOffsetCommittingOnCheckpoints(false)}}:
Perform auto periodic committing with the internal client (or in 0.8 case our 
own periodic committer), regardless of whether or not checkpointing is on. 
Usage documentation should state clearly that if using Kafka's auto periodic 
committing, the offsets in Kafka will not be in-sync with the checkpoint 
offsets.

2. {{props.put("auto.commit.enable", "true")}} & 
{{setEnableOffsetCommittingOnCheckpoints(true)}}:
Overrides whatever value in props for "auto.commit.enable", only commit on 
checkpoints. If checkpointing isn't enabled, the consumer fails due to 
conflicting configuration.

3. {{props.put("auto.commit.enable", "false")}} & 
{{setEnableOffsetCommittingOnCheckpoints(false)}}: No offset committing at all.



was (Author: tzulitai):
I think we should keep this JIRA instead of merging it into FLINK-4280.
They try to address different new features: FLINK-4280 is for more flexible 
start position configuration independent from Kafka offsets.
This JIRA is for opt-out from offset committing on Flink's checkpoints.

I'll move the respective proposed API for commit opt-out here:

{code}
Properties props = new Properties();
...
FlinkKafkaConsumer09 kafka = new FlinkKafkaConsumer09("topic", schema, props);
kafka.setEnableOffsetCommittingOnCheckpoints(boolean); // if true 
(checkpointing should be enabled), overrides periodic checkpointing if 
"enable.auto.commit" is set in props.
{code}

So the scenarios breaks down into:

1. {{props.put("auto.commit.enable", "true")}} & 
{{setEnableOffsetCommittingOnCheckpoints(false)}}:
Perform auto committing with the internal client (or in 0.8 case our own 
periodic committer).

2. {{props.put("auto.commit.enable", "true")}} & 
{{setEnableOffsetCommittingOnCheckpoints(true)}}:
Overrides whatever value in props for "auto.commit.enable", only commit on 
checkpoints. If checkpointing isn't enabled, the consumer fails due to 
conflicting configuration.

3. {{props.put("auto.commit.enable", "false")}} & 
{{setEnableOffsetCommittingOnCheckpoints(false)}}: No offset committing at all.


> Flink Kafka consumer should support auto-commit opt-outs
> 
>
> Key: FLINK-3398
> URL: https://issues.apache.org/jira/browse/FLINK-3398
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Shikhar Bhushan
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Critical
> Fix For: 1.3.0
>
>
> Currently the Kafka source will commit consumer offsets to Zookeeper, either 
> upon a checkpoint if checkpointing is enabled, otherwise periodically based 
> on {{auto.commit.interval.ms}}
> It should be possible to opt-out of committing consumer offsets to Zookeeper. 
> Kafka has this config as {{auto.commit.enable}} (0.8) and 
> {{enable.auto.commit}} (0.9).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-5701) FlinkKafkaProducer should check asyncException on checkpoints

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-5701:
---
Summary: FlinkKafkaProducer should check asyncException on checkpoints  
(was: FlinkKafkaPrdocuer should check asyncException on checkpoints)

> FlinkKafkaProducer should check asyncException on checkpoints
> -
>
> Key: FLINK-5701
> URL: https://issues.apache.org/jira/browse/FLINK-5701
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Reported in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html
> The problem:
> The producer holds a {{pendingRecords}} value that is incremented on each 
> invoke() and decremented on each callback, used to check if the producer 
> needs to sync on pending callbacks on checkpoints.
> On each checkpoint, we should only consider the checkpoint succeeded iff 
> after flushing the {{pendingRecords == 0}} and {{asyncException == null}} 
> (currently, we’re only checking {{pendingRecords}}).
> A quick fix for this is to check and rethrow async exceptions in the 
> {{snapshotState}} method both before and after flushing and 
> {{pendingRecords}} becomes 0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-5702) Kafka Producer docs should warn if using setLogFailuresOnly, at-least-once is compromised

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-5702:
--

 Summary: Kafka Producer docs should warn if using 
setLogFailuresOnly, at-least-once is compromised
 Key: FLINK-5702
 URL: https://issues.apache.org/jira/browse/FLINK-5702
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Kafka Connector
Reporter: Tzu-Li (Gordon) Tai


The documentation for FlinkKafkaProducer does not have any information about 
the {{setLogFailuresOnly}}. It should emphasize that if users choose to only 
log failures instead of failing the sink, at-least-once can not be guaranteed .



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-5701) FlinkKafkaPrdocuer should check asyncException on checkpoints

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-5701:
---
Description: 
Reported in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html

The problem:

The producer holds a {{pendingRecords}} value that is incremented on each 
invoke() and decremented on each callback, used to check if the producer needs 
to sync on pending callbacks on checkpoints.
On each checkpoint, we should only consider the checkpoint succeeded iff after 
flushing the {{pendingRecords == 0}} and {{asyncException == null}} (currently, 
we’re only checking {{pendingRecords}}).

A quick fix for this is to check and rethrow async exceptions in the 
{{snapshotState}} method both before and after flushing and {{pendingRecords}} 
becomes 0.

  was:
Reported in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html

The problem:

The producer holds a {{pendingRecords}} value that is incremented on each 
invoke() and decremented on each callback, used to check if the producer needs 
to sync on pending callbacks on checkpoints.
On each checkpoint, we should only consider the checkpoint succeeded iff after 
flushing the {{pendingRecords == 0}} and {{asyncException == null}} (currently, 
we’re only checking {{pendingRecords}}).

A quick fix for this is to check and rethrow async exceptions in the 
`snapshotState` method both before and after flushing and `pendingRecords` 
becomes 0.


> FlinkKafkaPrdocuer should check asyncException on checkpoints
> -
>
> Key: FLINK-5701
> URL: https://issues.apache.org/jira/browse/FLINK-5701
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>
> Reported in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html
> The problem:
> The producer holds a {{pendingRecords}} value that is incremented on each 
> invoke() and decremented on each callback, used to check if the producer 
> needs to sync on pending callbacks on checkpoints.
> On each checkpoint, we should only consider the checkpoint succeeded iff 
> after flushing the {{pendingRecords == 0}} and {{asyncException == null}} 
> (currently, we’re only checking {{pendingRecords}}).
> A quick fix for this is to check and rethrow async exceptions in the 
> {{snapshotState}} method both before and after flushing and 
> {{pendingRecords}} becomes 0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-5701) FlinkKafkaPrdocuer should check asyncException on checkpoints

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-5701:
---
Description: 
Reported in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html

The problem:

The producer holds a {{pendingRecords}} value that is incremented on each 
invoke() and decremented on each callback, used to check if the producer needs 
to sync on pending callbacks on checkpoints.
On each checkpoint, we should only consider the checkpoint succeeded iff after 
flushing the {{pendingRecords == 0}} and {{asyncException == null}} (currently, 
we’re only checking {{pendingRecords}}).

A quick fix for this is to check and rethrow async exceptions in the 
`snapshotState` method both before and after flushing and `pendingRecords` 
becomes 0.

  was:
Reported in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html

The problem:

The producer holds a {{pendingRecords}} value that is incremented on each 
invoke() and decremented on each callback, used to check if the producer needs 
to sync on pending callbacks on checkpoints.
On each checkpoint, we should only consider the checkpoint succeeded iff after 
flushing the {{pendingRecords == 0}} and {{asyncException == null}} (currently, 
we’re only checking {{pendingRecords}}).

Generally, to fix this, we need to handle exceptions in the callback and re-add 
the original record back into the producer. I think the {{onComplete}} method 
is called after the KafkaProducer internally finishes all retry attempts and is 
removed from the buffer, so if we don’t do anything with the exception other 
than just logging it, the message will be lost.

Two additional things we need to address in order to solve this:
1. {{FlinkKafkaProducer}} needs to keep a map of callback to their 
corresponding original record.

2. We need to determine what async exceptions to actually re-add to the 
FlinkKafkaProducer. We simply cannot re-add for every exception, otherwise  
errors that simply cannot be resolved by retrying will hang the checkpoint 
flush process forever, and it'll be unclear to the user why the checkpoint is 
taking so long. The ElasticsearchSink has similar issues (FLINK-5353 and 
FLINK-5122). The proposed approach for this, instead of determining which async 
exceptions to retry case by case, is to let the user provide async failure 
handlers and let them implement logic on which exceptions to handle / re-add.


> FlinkKafkaPrdocuer should check asyncException on checkpoints
> -
>
> Key: FLINK-5701
> URL: https://issues.apache.org/jira/browse/FLINK-5701
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>
> Reported in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html
> The problem:
> The producer holds a {{pendingRecords}} value that is incremented on each 
> invoke() and decremented on each callback, used to check if the producer 
> needs to sync on pending callbacks on checkpoints.
> On each checkpoint, we should only consider the checkpoint succeeded iff 
> after flushing the {{pendingRecords == 0}} and {{asyncException == null}} 
> (currently, we’re only checking {{pendingRecords}}).
> A quick fix for this is to check and rethrow async exceptions in the 
> `snapshotState` method both before and after flushing and `pendingRecords` 
> becomes 0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-5701) FlinkKafkaPrdocuer should check asyncException on checkpoints

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-5701:
---
Priority: Critical  (was: Major)

> FlinkKafkaPrdocuer should check asyncException on checkpoints
> -
>
> Key: FLINK-5701
> URL: https://issues.apache.org/jira/browse/FLINK-5701
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Critical
>
> Reported in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html
> The problem:
> The producer holds a {{pendingRecords}} value that is incremented on each 
> invoke() and decremented on each callback, used to check if the producer 
> needs to sync on pending callbacks on checkpoints.
> On each checkpoint, we should only consider the checkpoint succeeded iff 
> after flushing the {{pendingRecords == 0}} and {{asyncException == null}} 
> (currently, we’re only checking {{pendingRecords}}).
> A quick fix for this is to check and rethrow async exceptions in the 
> {{snapshotState}} method both before and after flushing and 
> {{pendingRecords}} becomes 0.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-5701) FlinkKafkaPrdocuer should check asyncException on checkpoints

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-5701:
---
Summary: FlinkKafkaPrdocuer should check asyncException on checkpoints  
(was: FlinkKafkaPrdocuer violates at-least-once by not handling failed records)

> FlinkKafkaPrdocuer should check asyncException on checkpoints
> -
>
> Key: FLINK-5701
> URL: https://issues.apache.org/jira/browse/FLINK-5701
> Project: Flink
>  Issue Type: Bug
>  Components: Kafka Connector, Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>
> Reported in ML: 
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html
> The problem:
> The producer holds a {{pendingRecords}} value that is incremented on each 
> invoke() and decremented on each callback, used to check if the producer 
> needs to sync on pending callbacks on checkpoints.
> On each checkpoint, we should only consider the checkpoint succeeded iff 
> after flushing the {{pendingRecords == 0}} and {{asyncException == null}} 
> (currently, we’re only checking {{pendingRecords}}).
> Generally, to fix this, we need to handle exceptions in the callback and 
> re-add the original record back into the producer. I think the {{onComplete}} 
> method is called after the KafkaProducer internally finishes all retry 
> attempts and is removed from the buffer, so if we don’t do anything with the 
> exception other than just logging it, the message will be lost.
> Two additional things we need to address in order to solve this:
> 1. {{FlinkKafkaProducer}} needs to keep a map of callback to their 
> corresponding original record.
> 2. We need to determine what async exceptions to actually re-add to the 
> FlinkKafkaProducer. We simply cannot re-add for every exception, otherwise  
> errors that simply cannot be resolved by retrying will hang the checkpoint 
> flush process forever, and it'll be unclear to the user why the checkpoint is 
> taking so long. The ElasticsearchSink has similar issues (FLINK-5353 and 
> FLINK-5122). The proposed approach for this, instead of determining which 
> async exceptions to retry case by case, is to let the user provide async 
> failure handlers and let them implement logic on which exceptions to handle / 
> re-add.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-5701) FlinkKafkaPrdocuer violates at-least-once by not handling failed records

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-5701:
--

 Summary: FlinkKafkaPrdocuer violates at-least-once by not handling 
failed records
 Key: FLINK-5701
 URL: https://issues.apache.org/jira/browse/FLINK-5701
 Project: Flink
  Issue Type: Bug
  Components: Kafka Connector, Streaming Connectors
Reporter: Tzu-Li (Gordon) Tai


Reported in ML: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-td11413.html

The problem:

The producer holds a {{pendingRecords}} value that is incremented on each 
invoke() and decremented on each callback, used to check if the producer needs 
to sync on pending callbacks on checkpoints.
On each checkpoint, we should only consider the checkpoint succeeded iff after 
flushing the {{pendingRecords == 0}} and {{asyncException == null}} (currently, 
we’re only checking {{pendingRecords}}).

Generally, to fix this, we need to handle exceptions in the callback and re-add 
the original record back into the producer. I think the {{onComplete}} method 
is called after the KafkaProducer internally finishes all retry attempts and is 
removed from the buffer, so if we don’t do anything with the exception other 
than just logging it, the message will be lost.

Two additional things we need to address in order to solve this:
1. {{FlinkKafkaProducer}} needs to keep a map of callback to their 
corresponding original record.

2. We need to determine what async exceptions to actually re-add to the 
FlinkKafkaProducer. We simply cannot re-add for every exception, otherwise  
errors that simply cannot be resolved by retrying will hang the checkpoint 
flush process forever, and it'll be unclear to the user why the checkpoint is 
taking so long. The ElasticsearchSink has similar issues (FLINK-5353 and 
FLINK-5122). The proposed approach for this, instead of determining which async 
exceptions to retry case by case, is to let the user provide async failure 
handlers and let them implement logic on which exceptions to handle / re-add.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5499) Try to reuse the resource location of prior execution attempt in allocating slot

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851009#comment-15851009
 ] 

ASF GitHub Bot commented on FLINK-5499:
---

Github user wangzhijiang999 commented on the issue:

https://github.com/apache/flink/pull/3125
  
@StephanEwen 
Yes, the current concern is only focusing on state restore performance. 
This PR does not consider all the scenarios and it may be only the first step 
for the slot location implementation.

If the location do not exist,  it can add other strategies to decide the 
locations, such as co-loated by input for batch job as you mentioned. And it 
can be the second step for the implementation.

Wish your further comments!


> Try to reuse the resource location of prior execution attempt in allocating 
> slot
> 
>
> Key: FLINK-5499
> URL: https://issues.apache.org/jira/browse/FLINK-5499
> Project: Flink
>  Issue Type: Improvement
>  Components: JobManager
>Reporter: Zhijiang Wang
>Assignee: Zhijiang Wang
>
> Currently when schedule execution to request to allocate slot from 
> {{SlotPool}}, the {{TaskManagerLocation}} parameter is empty collection. So 
> for task fail over scenario, the new execution attempt may be deployed to 
> different task managers. If setting rockDB as state backend, the performance 
> is better if the data can be restored from local machine. So we try to reuse 
> the {{TaskManagerLocation}} of prior execution attempt when allocating slot 
> from {{SlotPool}}. If the {{TaskManagerLocation}} is empty from prior 
> executions, the behavior is the same with current status.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3125: [FLINK-5499][JobManager]Reuse the resource location of pr...

[wangzhijiang999]

Github user wangzhijiang999 commented on the issue:

https://github.com/apache/flink/pull/3125
  
@StephanEwen 
Yes, the current concern is only focusing on state restore performance. 
This PR does not consider all the scenarios and it may be only the first step 
for the slot location implementation.

If the location do not exist,  it can add other strategies to decide the 
locations, such as co-loated by input for batch job as you mentioned. And it 
can be the second step for the implementation.

Wish your further comments!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-5166) TextInputFormatTest.testNestedFileRead

[shijinkui (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

shijinkui closed FLINK-5166.

Resolution: Won't Fix

> TextInputFormatTest.testNestedFileRead
> --
>
> Key: FLINK-5166
> URL: https://issues.apache.org/jira/browse/FLINK-5166
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats, Tests
>Reporter: shijinkui
>
> `mvn clean package -P \!scala-2.11,scala-2.11  -U`
> Failed tests:
>   TextInputFormatTest.testNestedFileRead:140 Test erroneous
> Tests run: 846, Failures: 1, Errors: 0, Skipped: 0



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3113: [FLINK-4912] Introduce RECONCILIATING state in ExecutionG...

[wangzhijiang999]

Github user wangzhijiang999 commented on the issue:

https://github.com/apache/flink/pull/3113
  
@StephanEwen , thank you for the concrete suggestions. Sorry for delay 
response because of Chinese Spring Festival Holiday.

I have considered and added some tests to validate the state transitions of 
the state machine related with the later processes which would be submitted in 
the following PRs together.

I totally agree with the consideration of the above possible state 
transitions. And I plan to give a detail explanation of my implementation in 
another jira soon. It is actually a bit complex to do that ,so I try to break 
them down into small ones in order to review and merge quickly.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4912) Introduce RECONCILING state in ExecutionGraph

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851002#comment-15851002
 ] 

ASF GitHub Bot commented on FLINK-4912:
---

Github user wangzhijiang999 commented on the issue:

https://github.com/apache/flink/pull/3113
  
@StephanEwen , thank you for the concrete suggestions. Sorry for delay 
response because of Chinese Spring Festival Holiday.

I have considered and added some tests to validate the state transitions of 
the state machine related with the later processes which would be submitted in 
the following PRs together.

I totally agree with the consideration of the above possible state 
transitions. And I plan to give a detail explanation of my implementation in 
another jira soon. It is actually a bit complex to do that ,so I try to break 
them down into small ones in order to review and merge quickly.



> Introduce RECONCILING state in ExecutionGraph
> -
>
> Key: FLINK-4912
> URL: https://issues.apache.org/jira/browse/FLINK-4912
> Project: Flink
>  Issue Type: Sub-task
>  Components: Distributed Coordination
>Reporter: Stephan Ewen
>Assignee: Zhijiang Wang
>
> This is part of the non-disruptive JobManager failure recovery.
> I suggest to add a JobStatus and ExecutionState {{RECONCILING}}.
> If a job is started on a that JobManager for master recovery (tbd how to 
> determine that) the {{ExecutionGraph}} and the {{Execution}}s start in the 
> reconciling state.
> From {{RECONCILING}}, tasks can go to {{RUNNING}} (execution reconciled with 
> TaskManager) or to {{FAILED}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5640) configure the explicit Unit Test file suffix

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850984#comment-15850984
 ] 

ASF GitHub Bot commented on FLINK-5640:
---

Github user shijinkui commented on the issue:

https://github.com/apache/flink/pull/3211
  
cc @rmetzger 


> configure the explicit Unit Test file suffix
> 
>
> Key: FLINK-5640
> URL: https://issues.apache.org/jira/browse/FLINK-5640
> Project: Flink
>  Issue Type: Test
>  Components: Tests
>Affects Versions: 1.2.1
>Reporter: shijinkui
>Assignee: shijinkui
>
> There are four types of Unit Test file: *ITCase.java, *Test.java, 
> *ITSuite.scala, *Suite.scala
> File name ending with "IT.java" is integration test. File name ending with 
> "Test.java"  is unit test.
> It's clear for Surefire plugin of default-test execution to declare that 
> "*Test.*" is Java Unit Test.
> The test file statistics below:
> * Suite  total: 10
> * ITCase  total: 378
> * Test  total: 1008
> * ITSuite  total: 14



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3211: [FLINK-5640][test]onfigure the explicit Unit Test file su...

[shijinkui]

Github user shijinkui commented on the issue:

https://github.com/apache/flink/pull/3211
  
cc @rmetzger 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-5695) Optimize table type systems based on database semantics

[sunjincheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850965#comment-15850965
 ] 

sunjincheng edited comment on FLINK-5695 at 2/3/17 2:48 AM:


Hi, [~fhueske] , Yes, I would like to explain my purpose:
   There are only one "select" operation for WindowGroupedTable. 
GroupedTable also has only one "select" operation, and all of them are 
generated by "groupBy" operation (WindowedTable.groupBy | Table.groupBy). The 
only difference is groupBy's keys, The keys of "WindowedTable" "GroupBy" 
operation contain the "window", for the user is just add window-related fields 
in the table, such as windowStrat and windowEnd. "GroupedTable.select" and 
"WindowGroupedTable.select" semantics is no different, GroupedTable type easy 
to understand by users, and  "GroupedTable" should have all of the 
characteristics which "WindowGroupedTable" have, so I personally feel should be 
removed "WindowGroupedTable" type.
 Likewise, I think we can write a document that accept by the user if we 
present the explicit table type to the user. What do you think?


was (Author: sunjincheng121):
Hi, [~fhueske] , Yes, I would like to explain my purpose:
   There are only one "select" operation for WindowGroupedTable. 
GroupedTable also has only one "select" operation, and all of them are 
generated by "groupBy" operation (WindowedTable.groupBy | Table.groupBy). The 
only difference is groupBy's keys, The keys of "WindowedTable" "GroupBy" 
operation contain the "window", for the user is just add window-related fields 
in the table, such as windowStrat and windowEnd. "GroupedTable.select" and 
"WindowGroupedTable.select" semantics is no different, GroupedTable type easy 
to understand by users, and  "GroupedTable" should have all of the 
characteristics which "WindowGroupedTable" have, so I personally feel should be 
removed "WindowGroupedTable" type.
   Likewise, I think we can write a document that accepts the user if we 
present the explicit table type to the user. What do you think?

> Optimize table type systems based on database semantics
> ---
>
> Key: FLINK-5695
> URL: https://issues.apache.org/jira/browse/FLINK-5695
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Optimize table type systems based on database semantics.As follows:
> {code}
>  groupBy
>  >
> Table GroupedTable
>   ∧   <   ∧
>| select|
>|   |
>| where|
>| select| groupBy
>| agg   |
>| ...  |
>| window |
>   ∨ -> 
>  TableWindowedTable
>   <-
>   select
> {code}
> What do you think? [~fhueske] 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5166) TextInputFormatTest.testNestedFileRead

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850972#comment-15850972
 ] 

ASF GitHub Bot commented on FLINK-5166:
---

Github user shijinkui closed the pull request at:

https://github.com/apache/flink/pull/2876


> TextInputFormatTest.testNestedFileRead
> --
>
> Key: FLINK-5166
> URL: https://issues.apache.org/jira/browse/FLINK-5166
> Project: Flink
>  Issue Type: Bug
>  Components: Batch Connectors and Input/Output Formats, Tests
>Reporter: shijinkui
>
> `mvn clean package -P \!scala-2.11,scala-2.11  -U`
> Failed tests:
>   TextInputFormatTest.testNestedFileRead:140 Test erroneous
> Tests run: 846, Failures: 1, Errors: 0, Skipped: 0



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #2876: [FLINK-5166] TextInputFormatTest.testNestedFileRea...

[shijinkui]

Github user shijinkui closed the pull request at:

https://github.com/apache/flink/pull/2876


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (FLINK-5695) Optimize table type systems based on database semantics

[sunjincheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850965#comment-15850965
 ] 

sunjincheng edited comment on FLINK-5695 at 2/3/17 2:47 AM:


Hi, [~fhueske] , Yes, I would like to explain my purpose:
   There are only one "select" operation for WindowGroupedTable. 
GroupedTable also has only one "select" operation, and all of them are 
generated by "groupBy" operation (WindowedTable.groupBy | Table.groupBy). The 
only difference is groupBy's keys, The keys of "WindowedTable" "GroupBy" 
operation contain the "window", for the user is just add window-related fields 
in the table, such as windowStrat and windowEnd. "GroupedTable.select" and 
"WindowGroupedTable.select" semantics is no different, GroupedTable type easy 
to understand by users, and  "GroupedTable" should have all of the 
characteristics which "WindowGroupedTable" have, so I personally feel should be 
removed "WindowGroupedTable" type.
   Likewise, I think we can write a document that accepts the user if we 
present the explicit table type to the user. What do you think?


was (Author: sunjincheng121):
Hi, [~fhueske] , Yes, I would like to explain my purpose:
   There are only one "select" operation for WindowGroupedTable. 
GroupedTable also has only one "select" operation, and all of them are 
generated by "groupBy" operation (WindowedTable.groupBy | Table.groupBy). The 
only difference is groupBy's keys, The keys of "WindowedTable" "GroupBy" 
operation contain the "window", for the user is just add window-related fields 
in the table, such as windowStrat and windowEnd. "GroupedTable.select" and 
"WindowGroupedTable.select" semantics is no different, GroupedTable type easy 
to understand by users, and  "GroupedTable" should have all of the 
characteristics which "WindowGroupedTable" have, so I personally feel should be 
removed "WindowGroupedTable" type.

> Optimize table type systems based on database semantics
> ---
>
> Key: FLINK-5695
> URL: https://issues.apache.org/jira/browse/FLINK-5695
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Optimize table type systems based on database semantics.As follows:
> {code}
>  groupBy
>  >
> Table GroupedTable
>   ∧   <   ∧
>| select|
>|   |
>| where|
>| select| groupBy
>| agg   |
>| ...  |
>| window |
>   ∨ -> 
>  TableWindowedTable
>   <-
>   select
> {code}
> What do you think? [~fhueske] 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5695) Optimize table type systems based on database semantics

[sunjincheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850965#comment-15850965
 ] 

sunjincheng commented on FLINK-5695:


Hi, [~fhueske] , Yes, I would like to explain my purpose:
   There are only one "select" operation for WindowGroupedTable. 
GroupedTable also has only one "select" operation, and all of them are 
generated by "groupBy" operation (WindowedTable.groupBy | Table.groupBy). The 
only difference is groupBy's keys, The keys of "WindowedTable" "GroupBy" 
operation contain the "window", for the user is just add window-related fields 
in the table, such as windowStrat and windowEnd. "GroupedTable.select" and 
"WindowGroupedTable.select" semantics is no different, GroupedTable type easy 
to understand by users, and  "GroupedTable" should have all of the 
characteristics which "WindowGroupedTable" have, so I personally feel should be 
removed "WindowGroupedTable" type.

> Optimize table type systems based on database semantics
> ---
>
> Key: FLINK-5695
> URL: https://issues.apache.org/jira/browse/FLINK-5695
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Optimize table type systems based on database semantics.As follows:
> {code}
>  groupBy
>  >
> Table GroupedTable
>   ∧   <   ∧
>| select|
>|   |
>| where|
>| select| groupBy
>| agg   |
>| ...  |
>| window |
>   ∨ -> 
>  TableWindowedTable
>   <-
>   select
> {code}
> What do you think? [~fhueske] 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5624) Support tumbling window on streaming tables in the SQL API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850641#comment-15850641
 ] 

ASF GitHub Bot commented on FLINK-5624:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/3252
  
Hi @haohui, thanks for your contribution!

The referenced JIRA is about adding support for group windows to SQL, not 
OVER (or row) windows. It should enable queries such as:

```
SELECT a, sum(b) as sumB, TUMBLE_END(rowtime(), INTERVAL '1' HOUR) AS t,
  FROM myT
GROUP BY TUMBLE(rowtime(), INTERVAL '1' HOUR), a;
```

I saw that you contributed `TUMBLE` just very recently to Calcite, so this 
feature is not yet available in a Calcite release that we could link against. 
Until then, we could add support for the more manual version of SQL tumbling 
windows:

```
SELECT a, SUM(b) AS sumB, CEIL(rowtime() TO HOUR) AS t,
  FROM myT
GROUP BY CEIL(rowtime() TO HOUR), a
```

We would also need to find a way to reference the `rowtime`. We do not want 
to expose this as an actual attribute in Flink's SQL (internally, Flink treats 
record timestamps as metadata which may not be modified by a query). The 
current approach would be to implement a built-in function which serves as a 
marker and is replaced during the translation.

Best, Fabian


> Support tumbling window on streaming tables in the SQL API
> --
>
> Key: FLINK-5624
> URL: https://issues.apache.org/jira/browse/FLINK-5624
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>
> This is a follow up of FLINK-4691.
> FLINK-4691 adds supports for group-windows for streaming tables. This jira 
> proposes to expose the functionality in the SQL layer via the {{GROUP BY}} 
> clauses, as described in 
> http://calcite.apache.org/docs/stream.html#tumbling-windows.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3252: [FLINK-5624] Support tumbling window on streaming tables ...

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/3252
  
Hi @haohui, thanks for your contribution!

The referenced JIRA is about adding support for group windows to SQL, not 
OVER (or row) windows. It should enable queries such as:

```
SELECT a, sum(b) as sumB, TUMBLE_END(rowtime(), INTERVAL '1' HOUR) AS t,
  FROM myT
GROUP BY TUMBLE(rowtime(), INTERVAL '1' HOUR), a;
```

I saw that you contributed `TUMBLE` just very recently to Calcite, so this 
feature is not yet available in a Calcite release that we could link against. 
Until then, we could add support for the more manual version of SQL tumbling 
windows:

```
SELECT a, SUM(b) AS sumB, CEIL(rowtime() TO HOUR) AS t,
  FROM myT
GROUP BY CEIL(rowtime() TO HOUR), a
```

We would also need to find a way to reference the `rowtime`. We do not want 
to expose this as an actual attribute in Flink's SQL (internally, Flink treats 
record timestamps as metadata which may not be modified by a query). The 
current approach would be to implement a built-in function which serves as a 
marker and is replaced during the translation.

Best, Fabian


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2168) Add HBaseTableSource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850613#comment-15850613
 ] 

ASF GitHub Bot commented on FLINK-2168:
---

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/3149
  
Thanks for the update @ramkrish86 and your patience!
I skimmed over the PR and it looks good.
I'll do a final pass early next week before merging it.

Thanks, Fabian


> Add HBaseTableSource
> 
>
> Key: FLINK-2168
> URL: https://issues.apache.org/jira/browse/FLINK-2168
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Affects Versions: 0.9
>Reporter: Fabian Hueske
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Add a {{HBaseTableSource}} to read data from a HBase table. The 
> {{HBaseTableSource}} should implement the {{ProjectableTableSource}} 
> (FLINK-3848) and {{FilterableTableSource}} (FLINK-3849) interfaces.
> The implementation can be based on Flink's {{TableInputFormat}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3149: FLINK-2168 Add HBaseTableSource

[fhueske]

Github user fhueske commented on the issue:

https://github.com/apache/flink/pull/3149
  
Thanks for the update @ramkrish86 and your patience!
I skimmed over the PR and it looks good.
I'll do a final pass early next week before merging it.

Thanks, Fabian


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5654) Add processing time OVER RANGE BETWEEN x PRECEDING aggregation to SQL

[Fabian Hueske (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850584#comment-15850584
 ] 

Fabian Hueske commented on FLINK-5654:
--

I agree [~rtudoran], the second approach looks much easier to handle at 
optimization time.
It is (usually) not a problem to add rules as needed. 

Thanks, Fabian

> Add processing time OVER RANGE BETWEEN x PRECEDING aggregation to SQL
> -
>
> Key: FLINK-5654
> URL: https://issues.apache.org/jira/browse/FLINK-5654
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: radu
>
> The goal of this issue is to add support for OVER RANGE aggregations on 
> processing time streams to the SQL interface.
> Queries similar to the following should be supported:
> {code}
> SELECT 
>   a, 
>   SUM(b) OVER (PARTITION BY c ORDER BY procTime() RANGE BETWEEN INTERVAL '1' 
> HOUR PRECEDING AND CURRENT ROW) AS sumB,
>   MIN(b) OVER (PARTITION BY c ORDER BY procTime() RANGE BETWEEN INTERVAL '1' 
> HOUR PRECEDING AND CURRENT ROW) AS minB
> FROM myStream
> {code}
> The following restrictions should initially apply:
> - All OVER clauses in the same SELECT clause must be exactly the same.
> - The PARTITION BY clause is optional (no partitioning results in single 
> threaded execution).
> - The ORDER BY clause may only have procTime() as parameter. procTime() is a 
> parameterless scalar function that just indicates processing time mode.
> - UNBOUNDED PRECEDING is not supported (see FLINK-5657)
> - FOLLOWING is not supported.
> The restrictions will be resolved in follow up issues. If we find that some 
> of the restrictions are trivial to address, we can add the functionality in 
> this issue as well.
> This issue includes:
> - Design of the DataStream operator to compute OVER ROW aggregates
> - Translation from Calcite's RelNode representation (LogicalProject with 
> RexOver expression).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-3016) Update Flink roadmap

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-3016.
-
Resolution: Resolved

This info is now available in 
[FLIPs|https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals]
 and in the [Release and Feature 
Plan|https://cwiki.apache.org/confluence/display/FLINK/Flink+Release+and+Feature+Plan].

> Update Flink roadmap
> 
>
> Key: FLINK-3016
> URL: https://issues.apache.org/jira/browse/FLINK-3016
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Reporter: Slim Baltagi
>Priority: Minor
>
> The current Flink roadmap on this wiki page: : 
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Roadmap
> is not up-to-date. 
> It needs to be updated. Remove completed tasks and list new major ones 
> especially that 0.10 is supposed to be the last major release before 1.0



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-5700) Document akka.client.timeout

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-5700.

Resolution: Fixed

Fixed via 56d3611f3138a5605f61b4d1e0f59238f04296c6

> Document akka.client.timeout
> 
>
> Key: FLINK-5700
> URL: https://issues.apache.org/jira/browse/FLINK-5700
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 1.3.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> The configuration parameter {{akka.client.timeout}} has not been properly 
> documented.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-5494) Improve Mesos documentation

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-5494.

   Resolution: Fixed
Fix Version/s: 1.3.0

1.3.0: f8f9cae191a5504450cfc1b141b114641c2216b3
1.2.0: 8d81bfdaa2cd18ca9ef84c1618fc8c94e0f0e199

> Improve Mesos documentation
> ---
>
> Key: FLINK-5494
> URL: https://issues.apache.org/jira/browse/FLINK-5494
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.3.0, 1.2.0
>
>
> Flink's Mesos documentation could benefit from more details how to set things 
> up and which parameters to use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3236: [FLINK-5494] [docs] Add more details to the Mesos ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3236


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5494) Improve Mesos documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850540#comment-15850540
 ] 

ASF GitHub Bot commented on FLINK-5494:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3236


> Improve Mesos documentation
> ---
>
> Key: FLINK-5494
> URL: https://issues.apache.org/jira/browse/FLINK-5494
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.2.0, 1.3.0
>
>
> Flink's Mesos documentation could benefit from more details how to set things 
> up and which parameters to use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5494) Improve Mesos documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850536#comment-15850536
 ] 

ASF GitHub Bot commented on FLINK-5494:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/3236
  
Thanks for your review @tzulitai. I've addressed your comments. Merging 
this PR now.


> Improve Mesos documentation
> ---
>
> Key: FLINK-5494
> URL: https://issues.apache.org/jira/browse/FLINK-5494
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.2.0
>
>
> Flink's Mesos documentation could benefit from more details how to set things 
> up and which parameters to use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3236: [FLINK-5494] [docs] Add more details to the Mesos ...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99223588
  
--- Diff: docs/setup/mesos.md ---
@@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
 In the `/bin` directory of the Flink distribution, you find two startup 
scripts
 which manage the Flink processes in a Mesos cluster:
 
-1. mesos-appmaster.sh
-   This starts the Mesos application master which will register the Mesos
-   scheduler. It is also responsible for starting up the worker nodes.
+1. `mesos-appmaster.sh`
+   This starts the Mesos application master which will register the Mesos 
scheduler.
+   It is also responsible for starting up the worker nodes.

-2. mesos-taskmanager.sh
-   The entry point for the Mesos worker processes. You don't need to 
explicitly
-   execute this script. It is automatically launched by the Mesos worker 
node to
-   bring up a new TaskManager.
+2. `mesos-taskmanager.sh`
+   The entry point for the Mesos worker processes.
+   You don't need to explicitly execute this script.
+   It is automatically launched by the Mesos worker node to bring up a new 
TaskManager.
+
+In order to run the `mesos-appmaster.sh` script you have to define 
`mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to 
the Java process.
+Additionally, you should define the number of task managers which are 
started by Mesos via `mesos.initial-tasks`.
+This value can also be defined in the `flink-conf.yaml` or passed as a 
Java property.
+
+When executing `mesos-appmaster.sh`, it will create a job manager on the 
machine where you executed the script.
+In contrast to that, the task managers will be run as Mesos tasks in the 
Mesos cluster.
+
+ General configuration
+
+It is possible to completely parameterize a Mesos application through Java 
properties passed to the Mesos application master.
+This also allows to specify general Flink configuration parameters.
+For example:
+
+bin/mesos-appmaster.sh \
+-Dmesos.master=master.foobar.org:5050
+-Djobmanager.heap.mb=1024 \
+-Djobmanager.rpc.port=6123 \
+-Djobmanager.web.port=8081 \
+-Dmesos.initial-tasks=10 \
+-Dmesos.resourcemanager.tasks.mem=4096 \
+-Dtaskmanager.heap.mb=3500 \
+-Dtaskmanager.numberOfTaskSlots=2 \
+-Dparallelism.default=10
 
 
 ### High Availability
 
-You will need to run a service like Marathon or Apache Aurora which takes 
care
-of restarting the Flink master process in case of node or process 
failures. In
-addition, Zookeeper needs to be configured like described in the
-[High Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
+You will need to run a service like Marathon or Apache Aurora which takes 
care of restarting the Flink master process in case of node or process failures.
+In addition, Zookeeper needs to be configured like described in the [High 
Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
 
-For the reconciliation of tasks to work correctly, please also set
-`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
+For the reconciliation of tasks to work correctly, please also set 
`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
 
  Marathon
 
-Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. 
In
-particular, it should also adjust any configuration parameters for the 
Flink
-cluster. 
+Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
+In particular, it should also adjust any configuration parameters for the 
Flink cluster.
 
 Here is an example configuration for Marathon:
 
 {
-"id": "basic-0", 
-"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue 
-DanotherEntry=anotherValue ...",
+"id": "flink",
+"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 
-Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 
-Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 
-Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 
-Dmesos.resourcemanager.tasks.cpus=1",
 "cpus": 1.0,
-"mem": 2048,
+"mem": 1024
 }
 
+When running Flink with Marathon, the whole Flink cluster including the 
job manager will be run as Mesos tasks in the Mesos cluster.
+
 ### Configuration parameters
 
  Mesos configuration entries
 
+`mesos.initial-tasks`: The initial workers to bring up when the master 
starts (**DEFAULT**: The number of workers specified at cluster 

[GitHub] flink issue #3236: [FLINK-5494] [docs] Add more details to the Mesos documen...

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/3236
  
Thanks for your review @tzulitai. I've addressed your comments. Merging 
this PR now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5494) Improve Mesos documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850535#comment-15850535
 ] 

ASF GitHub Bot commented on FLINK-5494:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99223588
  
--- Diff: docs/setup/mesos.md ---
@@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
 In the `/bin` directory of the Flink distribution, you find two startup 
scripts
 which manage the Flink processes in a Mesos cluster:
 
-1. mesos-appmaster.sh
-   This starts the Mesos application master which will register the Mesos
-   scheduler. It is also responsible for starting up the worker nodes.
+1. `mesos-appmaster.sh`
+   This starts the Mesos application master which will register the Mesos 
scheduler.
+   It is also responsible for starting up the worker nodes.

-2. mesos-taskmanager.sh
-   The entry point for the Mesos worker processes. You don't need to 
explicitly
-   execute this script. It is automatically launched by the Mesos worker 
node to
-   bring up a new TaskManager.
+2. `mesos-taskmanager.sh`
+   The entry point for the Mesos worker processes.
+   You don't need to explicitly execute this script.
+   It is automatically launched by the Mesos worker node to bring up a new 
TaskManager.
+
+In order to run the `mesos-appmaster.sh` script you have to define 
`mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to 
the Java process.
+Additionally, you should define the number of task managers which are 
started by Mesos via `mesos.initial-tasks`.
+This value can also be defined in the `flink-conf.yaml` or passed as a 
Java property.
+
+When executing `mesos-appmaster.sh`, it will create a job manager on the 
machine where you executed the script.
+In contrast to that, the task managers will be run as Mesos tasks in the 
Mesos cluster.
+
+ General configuration
+
+It is possible to completely parameterize a Mesos application through Java 
properties passed to the Mesos application master.
+This also allows to specify general Flink configuration parameters.
+For example:
+
+bin/mesos-appmaster.sh \
+-Dmesos.master=master.foobar.org:5050
+-Djobmanager.heap.mb=1024 \
+-Djobmanager.rpc.port=6123 \
+-Djobmanager.web.port=8081 \
+-Dmesos.initial-tasks=10 \
+-Dmesos.resourcemanager.tasks.mem=4096 \
+-Dtaskmanager.heap.mb=3500 \
+-Dtaskmanager.numberOfTaskSlots=2 \
+-Dparallelism.default=10
 
 
 ### High Availability
 
-You will need to run a service like Marathon or Apache Aurora which takes 
care
-of restarting the Flink master process in case of node or process 
failures. In
-addition, Zookeeper needs to be configured like described in the
-[High Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
+You will need to run a service like Marathon or Apache Aurora which takes 
care of restarting the Flink master process in case of node or process failures.
+In addition, Zookeeper needs to be configured like described in the [High 
Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
 
-For the reconciliation of tasks to work correctly, please also set
-`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
+For the reconciliation of tasks to work correctly, please also set 
`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
 
  Marathon
 
-Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. 
In
-particular, it should also adjust any configuration parameters for the 
Flink
-cluster. 
+Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
+In particular, it should also adjust any configuration parameters for the 
Flink cluster.
 
 Here is an example configuration for Marathon:
 
 {
-"id": "basic-0", 
-"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue 
-DanotherEntry=anotherValue ...",
+"id": "flink",
+"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 
-Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 
-Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 
-Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 
-Dmesos.resourcemanager.tasks.cpus=1",
 "cpus": 1.0,
-"mem": 2048,
+"mem": 1024
 }
 
+When running Flink with Marathon, the whole Flink cluster including the 
job manager will be run as Mesos tasks in the Mesos 

[jira] [Closed] (FLINK-2896) Right-hand side menu items of website are missing in narrow browsers

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-2896.
-
Resolution: Fixed

The new web design places menu items vertically on the left side of the page.

> Right-hand side menu items of website are missing in narrow browsers
> 
>
> Key: FLINK-2896
> URL: https://issues.apache.org/jira/browse/FLINK-2896
> Project: Flink
>  Issue Type: Bug
>  Components: Project Website
>Reporter: Fabian Hueske
>
> The right menu items (Blog, Community, Project) are missing if the browser 
> window is too narrow.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-2883) Add documentation to forbid key-modifying ReduceFunction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850521#comment-15850521
 ] 

ASF GitHub Bot commented on FLINK-2883:
---

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/3256

[FLINK-2883] [docs] Add documentation to forbid key-modifying ReduceFunction



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
2883_add_documentation_to_forbid_keymodifying_reducefunction

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3256.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3256


commit 2a858819e2163a4c935521670d766ebaaba5b99d
Author: Greg Hogan 
Date:   2017-02-02T21:15:52Z

[FLINK-2883] [docs] Add documentation to forbid key-modifying ReduceFunction




> Add documentation to forbid key-modifying ReduceFunction
> 
>
> Key: FLINK-2883
> URL: https://issues.apache.org/jira/browse/FLINK-2883
> Project: Flink
>  Issue Type: Task
>  Components: DataStream API, Documentation
>Affects Versions: 0.10.0
>Reporter: Till Rohrmann
>Assignee: Greg Hogan
>
> If one uses a combinable reduce operation which also changes the key value of 
> the underlying data element, then the results of the reduce operation can 
> become wrong. The reason is that after the combine phase, another reduce 
> operator is executed which will then reduce the elements based on the new key 
> values. This might be not so surprising if one explicitly defined ones 
> {{GroupReduceOperation}} as combinable. However, the {{ReduceFunction}} 
> conceals the fact that a combiner is used implicitly. Furthermore, the API 
> does not prevent the user from changing the key fields which could solve the 
> problem.
> The following example program illustrates the problem
> {code}
> val env = ExecutionEnvironment.getExecutionEnvironment
> env.setParallelism(1)
> val input = env.fromElements((1,2), (1,3), (2,3), (3,3), (3,4))
> val result = input.groupBy(0).reduce{
>   (left, right) =>
> (left._1 + right._1, left._2 + right._2)
> }
> result.output(new PrintingOutputFormat[Int]())
> env.execute()
> {code}
> The expected output is 
> {code}
> (2, 5)
> (2, 3)
> (6, 7)
> {code}
> However, the actual output is
> {code}
> (4, 8)
> (6, 7)
> {code}
> I think that the underlying problem is that associativity and commutativity 
> is not sufficient for a combinable reduce operation. Additionally we also 
> need to make sure that the key stays the same.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3256: [FLINK-2883] [docs] Add documentation to forbid ke...

[greghogan]

GitHub user greghogan opened a pull request:

https://github.com/apache/flink/pull/3256

[FLINK-2883] [docs] Add documentation to forbid key-modifying ReduceFunction



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/greghogan/flink 
2883_add_documentation_to_forbid_keymodifying_reducefunction

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3256.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3256


commit 2a858819e2163a4c935521670d766ebaaba5b99d
Author: Greg Hogan 
Date:   2017-02-02T21:15:52Z

[FLINK-2883] [docs] Add documentation to forbid key-modifying ReduceFunction




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-2883) Add documentation to forbid key-modifying ReduceFunction

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan reassigned FLINK-2883:
-

Assignee: Greg Hogan

> Add documentation to forbid key-modifying ReduceFunction
> 
>
> Key: FLINK-2883
> URL: https://issues.apache.org/jira/browse/FLINK-2883
> Project: Flink
>  Issue Type: Task
>  Components: DataStream API, Documentation
>Affects Versions: 0.10.0
>Reporter: Till Rohrmann
>Assignee: Greg Hogan
>
> If one uses a combinable reduce operation which also changes the key value of 
> the underlying data element, then the results of the reduce operation can 
> become wrong. The reason is that after the combine phase, another reduce 
> operator is executed which will then reduce the elements based on the new key 
> values. This might be not so surprising if one explicitly defined ones 
> {{GroupReduceOperation}} as combinable. However, the {{ReduceFunction}} 
> conceals the fact that a combiner is used implicitly. Furthermore, the API 
> does not prevent the user from changing the key fields which could solve the 
> problem.
> The following example program illustrates the problem
> {code}
> val env = ExecutionEnvironment.getExecutionEnvironment
> env.setParallelism(1)
> val input = env.fromElements((1,2), (1,3), (2,3), (3,3), (3,4))
> val result = input.groupBy(0).reduce{
>   (left, right) =>
> (left._1 + right._1, left._2 + right._2)
> }
> result.output(new PrintingOutputFormat[Int]())
> env.execute()
> {code}
> The expected output is 
> {code}
> (2, 5)
> (2, 3)
> (6, 7)
> {code}
> However, the actual output is
> {code}
> (4, 8)
> (6, 7)
> {code}
> I think that the underlying problem is that associativity and commutativity 
> is not sufficient for a combinable reduce operation. Additionally we also 
> need to make sure that the key stays the same.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5494) Improve Mesos documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850516#comment-15850516
 ] 

ASF GitHub Bot commented on FLINK-5494:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99221601
  
--- Diff: docs/setup/mesos.md ---
@@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
 In the `/bin` directory of the Flink distribution, you find two startup 
scripts
 which manage the Flink processes in a Mesos cluster:
 
-1. mesos-appmaster.sh
-   This starts the Mesos application master which will register the Mesos
-   scheduler. It is also responsible for starting up the worker nodes.
+1. `mesos-appmaster.sh`
+   This starts the Mesos application master which will register the Mesos 
scheduler.
+   It is also responsible for starting up the worker nodes.

-2. mesos-taskmanager.sh
-   The entry point for the Mesos worker processes. You don't need to 
explicitly
-   execute this script. It is automatically launched by the Mesos worker 
node to
-   bring up a new TaskManager.
+2. `mesos-taskmanager.sh`
+   The entry point for the Mesos worker processes.
+   You don't need to explicitly execute this script.
+   It is automatically launched by the Mesos worker node to bring up a new 
TaskManager.
+
+In order to run the `mesos-appmaster.sh` script you have to define 
`mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to 
the Java process.
+Additionally, you should define the number of task managers which are 
started by Mesos via `mesos.initial-tasks`.
+This value can also be defined in the `flink-conf.yaml` or passed as a 
Java property.
+
+When executing `mesos-appmaster.sh`, it will create a job manager on the 
machine where you executed the script.
+In contrast to that, the task managers will be run as Mesos tasks in the 
Mesos cluster.
+
+ General configuration
+
+It is possible to completely parameterize a Mesos application through Java 
properties passed to the Mesos application master.
+This also allows to specify general Flink configuration parameters.
+For example:
+
+bin/mesos-appmaster.sh \
+-Dmesos.master=master.foobar.org:5050
+-Djobmanager.heap.mb=1024 \
+-Djobmanager.rpc.port=6123 \
+-Djobmanager.web.port=8081 \
+-Dmesos.initial-tasks=10 \
+-Dmesos.resourcemanager.tasks.mem=4096 \
+-Dtaskmanager.heap.mb=3500 \
+-Dtaskmanager.numberOfTaskSlots=2 \
+-Dparallelism.default=10
 
 
 ### High Availability
 
-You will need to run a service like Marathon or Apache Aurora which takes 
care
-of restarting the Flink master process in case of node or process 
failures. In
-addition, Zookeeper needs to be configured like described in the
-[High Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
+You will need to run a service like Marathon or Apache Aurora which takes 
care of restarting the Flink master process in case of node or process failures.
+In addition, Zookeeper needs to be configured like described in the [High 
Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
 
-For the reconciliation of tasks to work correctly, please also set
-`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
+For the reconciliation of tasks to work correctly, please also set 
`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
 
  Marathon
 
-Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. 
In
-particular, it should also adjust any configuration parameters for the 
Flink
-cluster. 
+Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
+In particular, it should also adjust any configuration parameters for the 
Flink cluster.
 
 Here is an example configuration for Marathon:
 
 {
-"id": "basic-0", 
-"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue 
-DanotherEntry=anotherValue ...",
+"id": "flink",
+"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 
-Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 
-Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 
-Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 
-Dmesos.resourcemanager.tasks.cpus=1",
 "cpus": 1.0,
-"mem": 2048,
+"mem": 1024
 }
 
+When running Flink with Marathon, the whole Flink cluster including the 
job manager will be run as Mesos tasks in the Mesos 

[GitHub] flink pull request #3236: [FLINK-5494] [docs] Add more details to the Mesos ...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99221601
  
--- Diff: docs/setup/mesos.md ---
@@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
 In the `/bin` directory of the Flink distribution, you find two startup 
scripts
 which manage the Flink processes in a Mesos cluster:
 
-1. mesos-appmaster.sh
-   This starts the Mesos application master which will register the Mesos
-   scheduler. It is also responsible for starting up the worker nodes.
+1. `mesos-appmaster.sh`
+   This starts the Mesos application master which will register the Mesos 
scheduler.
+   It is also responsible for starting up the worker nodes.

-2. mesos-taskmanager.sh
-   The entry point for the Mesos worker processes. You don't need to 
explicitly
-   execute this script. It is automatically launched by the Mesos worker 
node to
-   bring up a new TaskManager.
+2. `mesos-taskmanager.sh`
+   The entry point for the Mesos worker processes.
+   You don't need to explicitly execute this script.
+   It is automatically launched by the Mesos worker node to bring up a new 
TaskManager.
+
+In order to run the `mesos-appmaster.sh` script you have to define 
`mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to 
the Java process.
+Additionally, you should define the number of task managers which are 
started by Mesos via `mesos.initial-tasks`.
+This value can also be defined in the `flink-conf.yaml` or passed as a 
Java property.
+
+When executing `mesos-appmaster.sh`, it will create a job manager on the 
machine where you executed the script.
+In contrast to that, the task managers will be run as Mesos tasks in the 
Mesos cluster.
+
+ General configuration
+
+It is possible to completely parameterize a Mesos application through Java 
properties passed to the Mesos application master.
+This also allows to specify general Flink configuration parameters.
+For example:
+
+bin/mesos-appmaster.sh \
+-Dmesos.master=master.foobar.org:5050
+-Djobmanager.heap.mb=1024 \
+-Djobmanager.rpc.port=6123 \
+-Djobmanager.web.port=8081 \
+-Dmesos.initial-tasks=10 \
+-Dmesos.resourcemanager.tasks.mem=4096 \
+-Dtaskmanager.heap.mb=3500 \
+-Dtaskmanager.numberOfTaskSlots=2 \
+-Dparallelism.default=10
 
 
 ### High Availability
 
-You will need to run a service like Marathon or Apache Aurora which takes 
care
-of restarting the Flink master process in case of node or process 
failures. In
-addition, Zookeeper needs to be configured like described in the
-[High Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
+You will need to run a service like Marathon or Apache Aurora which takes 
care of restarting the Flink master process in case of node or process failures.
+In addition, Zookeeper needs to be configured like described in the [High 
Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
 
-For the reconciliation of tasks to work correctly, please also set
-`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
+For the reconciliation of tasks to work correctly, please also set 
`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
 
  Marathon
 
-Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. 
In
-particular, it should also adjust any configuration parameters for the 
Flink
-cluster. 
+Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
+In particular, it should also adjust any configuration parameters for the 
Flink cluster.
 
 Here is an example configuration for Marathon:
 
 {
-"id": "basic-0", 
-"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue 
-DanotherEntry=anotherValue ...",
+"id": "flink",
+"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 
-Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 
-Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 
-Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 
-Dmesos.resourcemanager.tasks.cpus=1",
 "cpus": 1.0,
-"mem": 2048,
+"mem": 1024
 }
 
+When running Flink with Marathon, the whole Flink cluster including the 
job manager will be run as Mesos tasks in the Mesos cluster.
+
 ### Configuration parameters
 
  Mesos configuration entries
--- End diff --

Yes they should also go there.


---
If your project is set up for it, you can reply to this email and have your
reply 

[jira] [Commented] (FLINK-5494) Improve Mesos documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850505#comment-15850505
 ] 

ASF GitHub Bot commented on FLINK-5494:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99220661
  
--- Diff: docs/setup/mesos.md ---
@@ -145,60 +174,75 @@ If set to 'docker', specify the image name:
 In the `/bin` directory of the Flink distribution, you find two startup 
scripts
 which manage the Flink processes in a Mesos cluster:
 
-1. mesos-appmaster.sh
-   This starts the Mesos application master which will register the Mesos
-   scheduler. It is also responsible for starting up the worker nodes.
+1. `mesos-appmaster.sh`
+   This starts the Mesos application master which will register the Mesos 
scheduler.
+   It is also responsible for starting up the worker nodes.

-2. mesos-taskmanager.sh
-   The entry point for the Mesos worker processes. You don't need to 
explicitly
-   execute this script. It is automatically launched by the Mesos worker 
node to
-   bring up a new TaskManager.
+2. `mesos-taskmanager.sh`
+   The entry point for the Mesos worker processes.
+   You don't need to explicitly execute this script.
+   It is automatically launched by the Mesos worker node to bring up a new 
TaskManager.
+
+In order to run the `mesos-appmaster.sh` script you have to define 
`mesos.master` in the `flink-conf.yaml` or pass it via `-Dmesos.master=...` to 
the Java process.
+Additionally, you should define the number of task managers which are 
started by Mesos via `mesos.initial-tasks`.
+This value can also be defined in the `flink-conf.yaml` or passed as a 
Java property.
+
+When executing `mesos-appmaster.sh`, it will create a job manager on the 
machine where you executed the script.
+In contrast to that, the task managers will be run as Mesos tasks in the 
Mesos cluster.
+
+ General configuration
+
+It is possible to completely parameterize a Mesos application through Java 
properties passed to the Mesos application master.
+This also allows to specify general Flink configuration parameters.
+For example:
+
+bin/mesos-appmaster.sh \
+-Dmesos.master=master.foobar.org:5050
+-Djobmanager.heap.mb=1024 \
+-Djobmanager.rpc.port=6123 \
+-Djobmanager.web.port=8081 \
+-Dmesos.initial-tasks=10 \
+-Dmesos.resourcemanager.tasks.mem=4096 \
+-Dtaskmanager.heap.mb=3500 \
+-Dtaskmanager.numberOfTaskSlots=2 \
+-Dparallelism.default=10
 
 
 ### High Availability
 
-You will need to run a service like Marathon or Apache Aurora which takes 
care
-of restarting the Flink master process in case of node or process 
failures. In
-addition, Zookeeper needs to be configured like described in the
-[High Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
+You will need to run a service like Marathon or Apache Aurora which takes 
care of restarting the Flink master process in case of node or process failures.
+In addition, Zookeeper needs to be configured like described in the [High 
Availability section of the Flink docs]({{ site.baseurl 
}}/setup/jobmanager_high_availability.html)
 
-For the reconciliation of tasks to work correctly, please also set
-`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
+For the reconciliation of tasks to work correctly, please also set 
`recovery.zookeeper.path.mesos-workers` to a valid Zookeeper path.
 
  Marathon
 
-Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script. 
In
-particular, it should also adjust any configuration parameters for the 
Flink
-cluster. 
+Marathon needs to be set up to launch the `bin/mesos-appmaster.sh` script.
+In particular, it should also adjust any configuration parameters for the 
Flink cluster.
 
 Here is an example configuration for Marathon:
 
 {
-"id": "basic-0", 
-"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -DconfigEntry=configValue 
-DanotherEntry=anotherValue ...",
+"id": "flink",
+"cmd": "$FLINK_HOME/bin/mesos-appmaster.sh -Djobmanager.heap.mb=1024 
-Djobmanager.rpc.port=6123 -Djobmanager.web.port=8081 -Dmesos.initial-tasks=1 
-Dmesos.resourcemanager.tasks.mem=1024 -Dtaskmanager.heap.mb=1024 
-Dtaskmanager.numberOfTaskSlots=2 -Dparallelism.default=2 
-Dmesos.resourcemanager.tasks.cpus=1",
 "cpus": 1.0,
-"mem": 2048,
+"mem": 1024
--- End diff --

Added indentation.


> Improve Mesos documentation
> ---
>
> Key: FLINK-5494
>

[jira] [Commented] (FLINK-5494) Improve Mesos documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850498#comment-15850498
 ] 

ASF GitHub Bot commented on FLINK-5494:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99220223
  
--- Diff: docs/setup/mesos.md ---
@@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the 
meantime:
 
 ## Mesos without DC/OS
 
-Let's take a look at how to setup Flink on Mesos without DC/OS. 
+You can also run Mesos without DC/OS.
 
-### Prerequisites
+### Installing Mesos
 
-Please follow the
-[instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/). 
+Please follow the [instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/getting-started/).
+
+After installation you have to configure the set of master and agent nodes 
by creating the files `MESOS_HOME/etc/mesos/masters` and 
`MESOS_HOME/etc/mesos/slaves`.
+These files contain in each row a single hostname on which the respective 
component will be started (assuming SSH access to these nodes).
+
+Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use 
the template found in the same directory.
+In this file, you have to define
+
+export MESOS_work_dir=WORK_DIRECTORY
+
+and it is recommended to uncommment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+
+
+In order to configure the Mesos agents, you have to create 
`MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same 
directory.
+You have to configure
+
+export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
+
+and uncomment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+export MESOS_work_dir=WORK_DIRECTORY
+
+ Mesos Library
+
+In order to run Java applications with Mesos you have to export 
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
+
+ Deploying Mesos
+
+In order to start your mesos cluster, use the deployment script 
`MESOS_HOME/sbin/mesos-start-cluster.sh`.
+In order to stop your mesos cluster, use the deployment script 
`MESOS_HOME/sbin/mesos-stop-cluster.sh`.
+More information about the deployment scripts can be found 
[here](http://mesos.apache.org/documentation/latest/deploy-scripts/).
+
+### Installing Marathon
+
+Optionally, you may also [install 
Marathon](https://mesosphere.github.io/marathon/docs/) which will be necessary 
to run Flink in high availability (HA) mode.
 
 ### Optional dependencies
 
-Optionally,
-you may also install [Marathon](https://mesosphere.github.io/marathon/) 
which
-will be necessary if you want your Flink cluster to be highly available in 
the
-presence of master node failures. Additionally, you probably want to 
install a
-distributed file system to share data across nodes and make use of Flink's
-checkpointing mechanism.
+Additionally, you probably want to install a distributed file system to 
share data across nodes and make use of Flink's checkpointing mechanism.
--- End diff --

Have removed the sentence.


> Improve Mesos documentation
> ---
>
> Key: FLINK-5494
> URL: https://issues.apache.org/jira/browse/FLINK-5494
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.2.0
>
>
> Flink's Mesos documentation could benefit from more details how to set things 
> up and which parameters to use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5494) Improve Mesos documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850494#comment-15850494
 ] 

ASF GitHub Bot commented on FLINK-5494:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99220004
  
--- Diff: docs/setup/mesos.md ---
@@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the 
meantime:
 
 ## Mesos without DC/OS
 
-Let's take a look at how to setup Flink on Mesos without DC/OS. 
+You can also run Mesos without DC/OS.
 
-### Prerequisites
+### Installing Mesos
 
-Please follow the
-[instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/). 
+Please follow the [instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/getting-started/).
+
+After installation you have to configure the set of master and agent nodes 
by creating the files `MESOS_HOME/etc/mesos/masters` and 
`MESOS_HOME/etc/mesos/slaves`.
+These files contain in each row a single hostname on which the respective 
component will be started (assuming SSH access to these nodes).
+
+Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use 
the template found in the same directory.
+In this file, you have to define
+
+export MESOS_work_dir=WORK_DIRECTORY
+
+and it is recommended to uncommment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+
+
+In order to configure the Mesos agents, you have to create 
`MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same 
directory.
+You have to configure
+
+export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
+
+and uncomment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+export MESOS_work_dir=WORK_DIRECTORY
+
+ Mesos Library
+
+In order to run Java applications with Mesos you have to export 
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
+
+ Deploying Mesos
+
+In order to start your mesos cluster, use the deployment script 
`MESOS_HOME/sbin/mesos-start-cluster.sh`.
+In order to stop your mesos cluster, use the deployment script 
`MESOS_HOME/sbin/mesos-stop-cluster.sh`.
+More information about the deployment scripts can be found 
[here](http://mesos.apache.org/documentation/latest/deploy-scripts/).
+
+### Installing Marathon
+
+Optionally, you may also [install 
Marathon](https://mesosphere.github.io/marathon/docs/) which will be necessary 
to run Flink in high availability (HA) mode.
 
 ### Optional dependencies
--- End diff --

Will change it that way.


> Improve Mesos documentation
> ---
>
> Key: FLINK-5494
> URL: https://issues.apache.org/jira/browse/FLINK-5494
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.2.0
>
>
> Flink's Mesos documentation could benefit from more details how to set things 
> up and which parameters to use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3236: [FLINK-5494] [docs] Add more details to the Mesos ...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99220223
  
--- Diff: docs/setup/mesos.md ---
@@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the 
meantime:
 
 ## Mesos without DC/OS
 
-Let's take a look at how to setup Flink on Mesos without DC/OS. 
+You can also run Mesos without DC/OS.
 
-### Prerequisites
+### Installing Mesos
 
-Please follow the
-[instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/). 
+Please follow the [instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/getting-started/).
+
+After installation you have to configure the set of master and agent nodes 
by creating the files `MESOS_HOME/etc/mesos/masters` and 
`MESOS_HOME/etc/mesos/slaves`.
+These files contain in each row a single hostname on which the respective 
component will be started (assuming SSH access to these nodes).
+
+Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use 
the template found in the same directory.
+In this file, you have to define
+
+export MESOS_work_dir=WORK_DIRECTORY
+
+and it is recommended to uncommment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+
+
+In order to configure the Mesos agents, you have to create 
`MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same 
directory.
+You have to configure
+
+export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
+
+and uncomment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+export MESOS_work_dir=WORK_DIRECTORY
+
+ Mesos Library
+
+In order to run Java applications with Mesos you have to export 
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
+
+ Deploying Mesos
+
+In order to start your mesos cluster, use the deployment script 
`MESOS_HOME/sbin/mesos-start-cluster.sh`.
+In order to stop your mesos cluster, use the deployment script 
`MESOS_HOME/sbin/mesos-stop-cluster.sh`.
+More information about the deployment scripts can be found 
[here](http://mesos.apache.org/documentation/latest/deploy-scripts/).
+
+### Installing Marathon
+
+Optionally, you may also [install 
Marathon](https://mesosphere.github.io/marathon/docs/) which will be necessary 
to run Flink in high availability (HA) mode.
 
 ### Optional dependencies
 
-Optionally,
-you may also install [Marathon](https://mesosphere.github.io/marathon/) 
which
-will be necessary if you want your Flink cluster to be highly available in 
the
-presence of master node failures. Additionally, you probably want to 
install a
-distributed file system to share data across nodes and make use of Flink's
-checkpointing mechanism.
+Additionally, you probably want to install a distributed file system to 
share data across nodes and make use of Flink's checkpointing mechanism.
--- End diff --

Have removed the sentence.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3236: [FLINK-5494] [docs] Add more details to the Mesos ...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99220004
  
--- Diff: docs/setup/mesos.md ---
@@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the 
meantime:
 
 ## Mesos without DC/OS
 
-Let's take a look at how to setup Flink on Mesos without DC/OS. 
+You can also run Mesos without DC/OS.
 
-### Prerequisites
+### Installing Mesos
 
-Please follow the
-[instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/). 
+Please follow the [instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/getting-started/).
+
+After installation you have to configure the set of master and agent nodes 
by creating the files `MESOS_HOME/etc/mesos/masters` and 
`MESOS_HOME/etc/mesos/slaves`.
+These files contain in each row a single hostname on which the respective 
component will be started (assuming SSH access to these nodes).
+
+Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use 
the template found in the same directory.
+In this file, you have to define
+
+export MESOS_work_dir=WORK_DIRECTORY
+
+and it is recommended to uncommment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+
+
+In order to configure the Mesos agents, you have to create 
`MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same 
directory.
+You have to configure
+
+export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
+
+and uncomment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+export MESOS_work_dir=WORK_DIRECTORY
+
+ Mesos Library
+
+In order to run Java applications with Mesos you have to export 
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
+
+ Deploying Mesos
+
+In order to start your mesos cluster, use the deployment script 
`MESOS_HOME/sbin/mesos-start-cluster.sh`.
+In order to stop your mesos cluster, use the deployment script 
`MESOS_HOME/sbin/mesos-stop-cluster.sh`.
+More information about the deployment scripts can be found 
[here](http://mesos.apache.org/documentation/latest/deploy-scripts/).
+
+### Installing Marathon
+
+Optionally, you may also [install 
Marathon](https://mesosphere.github.io/marathon/docs/) which will be necessary 
to run Flink in high availability (HA) mode.
 
 ### Optional dependencies
--- End diff --

Will change it that way.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3236: [FLINK-5494] [docs] Add more details to the Mesos ...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99219038
  
--- Diff: docs/setup/mesos.md ---
@@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the 
meantime:
 
 ## Mesos without DC/OS
 
-Let's take a look at how to setup Flink on Mesos without DC/OS. 
+You can also run Mesos without DC/OS.
 
-### Prerequisites
+### Installing Mesos
 
-Please follow the
-[instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/). 
+Please follow the [instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/getting-started/).
+
+After installation you have to configure the set of master and agent nodes 
by creating the files `MESOS_HOME/etc/mesos/masters` and 
`MESOS_HOME/etc/mesos/slaves`.
+These files contain in each row a single hostname on which the respective 
component will be started (assuming SSH access to these nodes).
+
+Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use 
the template found in the same directory.
+In this file, you have to define
+
+export MESOS_work_dir=WORK_DIRECTORY
+
+and it is recommended to uncommment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+
+
+In order to configure the Mesos agents, you have to create 
`MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same 
directory.
+You have to configure
+
+export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
+
+and uncomment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+export MESOS_work_dir=WORK_DIRECTORY
+
+ Mesos Library
+
+In order to run Java applications with Mesos you have to export 
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
--- End diff --

Good idea. Will add this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5494) Improve Mesos documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850488#comment-15850488
 ] 

ASF GitHub Bot commented on FLINK-5494:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3236#discussion_r99219038
  
--- Diff: docs/setup/mesos.md ---
@@ -107,29 +107,58 @@ Unvierse. Please use the following workaround in the 
meantime:
 
 ## Mesos without DC/OS
 
-Let's take a look at how to setup Flink on Mesos without DC/OS. 
+You can also run Mesos without DC/OS.
 
-### Prerequisites
+### Installing Mesos
 
-Please follow the
-[instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/). 
+Please follow the [instructions on how to setup Mesos on the official 
website](http://mesos.apache.org/documentation/latest/getting-started/).
+
+After installation you have to configure the set of master and agent nodes 
by creating the files `MESOS_HOME/etc/mesos/masters` and 
`MESOS_HOME/etc/mesos/slaves`.
+These files contain in each row a single hostname on which the respective 
component will be started (assuming SSH access to these nodes).
+
+Next you have to create `MESOS_HOME/etc/mesos/mesos-master-env.sh` or use 
the template found in the same directory.
+In this file, you have to define
+
+export MESOS_work_dir=WORK_DIRECTORY
+
+and it is recommended to uncommment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+
+
+In order to configure the Mesos agents, you have to create 
`MESOS_HOME/etc/mesos/mesos-agent-env.sh` or use the template found in the same 
directory.
+You have to configure
+
+export MESOS_master=MASTER_HOSTNAME:MASTER_PORT
+
+and uncomment
+
+export MESOS_log_dir=LOGGING_DIRECTORY
+export MESOS_work_dir=WORK_DIRECTORY
+
+ Mesos Library
+
+In order to run Java applications with Mesos you have to export 
`MESOS_NATIVE_JAVA_LIBRARY=MESOS_HOME/lib/libmesos.so`.
--- End diff --

Good idea. Will add this.


> Improve Mesos documentation
> ---
>
> Key: FLINK-5494
> URL: https://issues.apache.org/jira/browse/FLINK-5494
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
> Fix For: 1.2.0
>
>
> Flink's Mesos documentation could benefit from more details how to set things 
> up and which parameters to use.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5474) Extend DC/OS documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850485#comment-15850485
 ] 

ASF GitHub Bot commented on FLINK-5474:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3237


> Extend DC/OS documentation
> --
>
> Key: FLINK-5474
> URL: https://issues.apache.org/jira/browse/FLINK-5474
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.3.0
>
>
> We could extend the DC/OS documentation a little bit to include information 
> about how to submit a job (where to find the connection information) and that 
> one has to install the DC/OS cli in order to add the development universe.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-5474) Extend DC/OS documentation

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-5474.

Resolution: Fixed

1.3.0: 817e3c257d7b58f9de46ddb657d3d2707757d1c9
1.2.0: b0a95a9e5c4b6f4b1f81943327e611e95f40135e

> Extend DC/OS documentation
> --
>
> Key: FLINK-5474
> URL: https://issues.apache.org/jira/browse/FLINK-5474
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.3.0, 1.2.0
>
>
> We could extend the DC/OS documentation a little bit to include information 
> about how to submit a job (where to find the connection information) and that 
> one has to install the DC/OS cli in order to add the development universe.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3237: [FLINK-5474] [docs] Extend DC/OS documentation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/3237


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5474) Extend DC/OS documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850478#comment-15850478
 ] 

ASF GitHub Bot commented on FLINK-5474:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/3237
  
Thanks for the review @tzulitai. I'll address your comments and will then 
merge this PR.


> Extend DC/OS documentation
> --
>
> Key: FLINK-5474
> URL: https://issues.apache.org/jira/browse/FLINK-5474
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.3.0
>
>
> We could extend the DC/OS documentation a little bit to include information 
> about how to submit a job (where to find the connection information) and that 
> one has to install the DC/OS cli in order to add the development universe.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5474) Extend DC/OS documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850474#comment-15850474
 ] 

ASF GitHub Bot commented on FLINK-5474:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3237#discussion_r99217632
  
--- Diff: docs/setup/mesos.md ---
@@ -92,18 +92,25 @@ Universe. In the search prompt, just search for Flink.
 **Note**: At the time of this writing, Flink was not yet available in the
 Unvierse. Please use the following workaround in the meantime:
 
-1. Add the Development Universe
+1. [Install the DC/OS CLI](https://dcos.io/docs/1.8/usage/cli/install/)
 
-`dcos marathon app add 
https://raw.githubusercontent.com/mesosphere/dcos-flink-service/Makman2/quickstart/universe-server.json`
+2. Add the Development Universe
+
+`./dcos marathon app add 
https://raw.githubusercontent.com/mesosphere/dcos-flink-service/Makman2/quickstart/universe-server.json`
 
-2. Add the local Universe repository:
+3. Add the local Universe repository:
 
-   `dcos package repo add --index=0 dev-universe 
http://universe.marathon.mesos:8085/repo`
+   `./dcos package repo add --index=0 dev-universe 
http://universe.marathon.mesos:8085/repo`
 
-3. Install Flink through the Universe page or using the `dcos` command:
+4. Install Flink through the Universe page or using the `dcos` command:

-   `dcos package install flink`
+   `./dcos package install flink`
+
+In order to execute a Flink job on a DC/OS hosted Flink cluster, you first 
have to find out the address of the launched job manager.
--- End diff --

I'll change it.


> Extend DC/OS documentation
> --
>
> Key: FLINK-5474
> URL: https://issues.apache.org/jira/browse/FLINK-5474
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.3.0
>
>
> We could extend the DC/OS documentation a little bit to include information 
> about how to submit a job (where to find the connection information) and that 
> one has to install the DC/OS cli in order to add the development universe.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5474) Extend DC/OS documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850475#comment-15850475
 ] 

ASF GitHub Bot commented on FLINK-5474:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3237#discussion_r99217650
  
--- Diff: docs/setup/mesos.md ---
@@ -92,18 +92,25 @@ Universe. In the search prompt, just search for Flink.
 **Note**: At the time of this writing, Flink was not yet available in the
 Unvierse. Please use the following workaround in the meantime:
 
-1. Add the Development Universe
+1. [Install the DC/OS CLI](https://dcos.io/docs/1.8/usage/cli/install/)
 
-`dcos marathon app add 
https://raw.githubusercontent.com/mesosphere/dcos-flink-service/Makman2/quickstart/universe-server.json`
+2. Add the Development Universe
+
+`./dcos marathon app add 
https://raw.githubusercontent.com/mesosphere/dcos-flink-service/Makman2/quickstart/universe-server.json`
 
-2. Add the local Universe repository:
+3. Add the local Universe repository:
 
-   `dcos package repo add --index=0 dev-universe 
http://universe.marathon.mesos:8085/repo`
+   `./dcos package repo add --index=0 dev-universe 
http://universe.marathon.mesos:8085/repo`
 
-3. Install Flink through the Universe page or using the `dcos` command:
+4. Install Flink through the Universe page or using the `dcos` command:

-   `dcos package install flink`
+   `./dcos package install flink`
+
+In order to execute a Flink job on a DC/OS hosted Flink cluster, you first 
have to find out the address of the launched job manager.
+The job manager address can be found out by opening the Flink service, 
going to *Job Manager* and then using the address specified under 
`jobmanager.rpc.address` and `jobmanager.rpc.port`.
--- End diff --

Same here.


> Extend DC/OS documentation
> --
>
> Key: FLINK-5474
> URL: https://issues.apache.org/jira/browse/FLINK-5474
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.3.0
>
>
> We could extend the DC/OS documentation a little bit to include information 
> about how to submit a job (where to find the connection information) and that 
> one has to install the DC/OS cli in order to add the development universe.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3237: [FLINK-5474] [docs] Extend DC/OS documentation

[tillrohrmann]

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/3237
  
Thanks for the review @tzulitai. I'll address your comments and will then 
merge this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3237: [FLINK-5474] [docs] Extend DC/OS documentation

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3237#discussion_r99217592
  
--- Diff: docs/setup/mesos.md ---
@@ -92,18 +92,25 @@ Universe. In the search prompt, just search for Flink.
 **Note**: At the time of this writing, Flink was not yet available in the
 Unvierse. Please use the following workaround in the meantime:
--- End diff --

Will correct it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #3237: [FLINK-5474] [docs] Extend DC/OS documentation

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3237#discussion_r99217650
  
--- Diff: docs/setup/mesos.md ---
@@ -92,18 +92,25 @@ Universe. In the search prompt, just search for Flink.
 **Note**: At the time of this writing, Flink was not yet available in the
 Unvierse. Please use the following workaround in the meantime:
 
-1. Add the Development Universe
+1. [Install the DC/OS CLI](https://dcos.io/docs/1.8/usage/cli/install/)
 
-`dcos marathon app add 
https://raw.githubusercontent.com/mesosphere/dcos-flink-service/Makman2/quickstart/universe-server.json`
+2. Add the Development Universe
+
+`./dcos marathon app add 
https://raw.githubusercontent.com/mesosphere/dcos-flink-service/Makman2/quickstart/universe-server.json`
 
-2. Add the local Universe repository:
+3. Add the local Universe repository:
 
-   `dcos package repo add --index=0 dev-universe 
http://universe.marathon.mesos:8085/repo`
+   `./dcos package repo add --index=0 dev-universe 
http://universe.marathon.mesos:8085/repo`
 
-3. Install Flink through the Universe page or using the `dcos` command:
+4. Install Flink through the Universe page or using the `dcos` command:

-   `dcos package install flink`
+   `./dcos package install flink`
+
+In order to execute a Flink job on a DC/OS hosted Flink cluster, you first 
have to find out the address of the launched job manager.
+The job manager address can be found out by opening the Flink service, 
going to *Job Manager* and then using the address specified under 
`jobmanager.rpc.address` and `jobmanager.rpc.port`.
--- End diff --

Same here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5474) Extend DC/OS documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850473#comment-15850473
 ] 

ASF GitHub Bot commented on FLINK-5474:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3237#discussion_r99217592
  
--- Diff: docs/setup/mesos.md ---
@@ -92,18 +92,25 @@ Universe. In the search prompt, just search for Flink.
 **Note**: At the time of this writing, Flink was not yet available in the
 Unvierse. Please use the following workaround in the meantime:
--- End diff --

Will correct it.


> Extend DC/OS documentation
> --
>
> Key: FLINK-5474
> URL: https://issues.apache.org/jira/browse/FLINK-5474
> Project: Flink
>  Issue Type: Sub-task
>  Components: Mesos
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.3.0
>
>
> We could extend the DC/OS documentation a little bit to include information 
> about how to submit a job (where to find the connection information) and that 
> one has to install the DC/OS cli in order to add the development universe.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3237: [FLINK-5474] [docs] Extend DC/OS documentation

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/3237#discussion_r99217632
  
--- Diff: docs/setup/mesos.md ---
@@ -92,18 +92,25 @@ Universe. In the search prompt, just search for Flink.
 **Note**: At the time of this writing, Flink was not yet available in the
 Unvierse. Please use the following workaround in the meantime:
 
-1. Add the Development Universe
+1. [Install the DC/OS CLI](https://dcos.io/docs/1.8/usage/cli/install/)
 
-`dcos marathon app add 
https://raw.githubusercontent.com/mesosphere/dcos-flink-service/Makman2/quickstart/universe-server.json`
+2. Add the Development Universe
+
+`./dcos marathon app add 
https://raw.githubusercontent.com/mesosphere/dcos-flink-service/Makman2/quickstart/universe-server.json`
 
-2. Add the local Universe repository:
+3. Add the local Universe repository:
 
-   `dcos package repo add --index=0 dev-universe 
http://universe.marathon.mesos:8085/repo`
+   `./dcos package repo add --index=0 dev-universe 
http://universe.marathon.mesos:8085/repo`
 
-3. Install Flink through the Universe page or using the `dcos` command:
+4. Install Flink through the Universe page or using the `dcos` command:

-   `dcos package install flink`
+   `./dcos package install flink`
+
+In order to execute a Flink job on a DC/OS hosted Flink cluster, you first 
have to find out the address of the launched job manager.
--- End diff --

I'll change it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2772) Introduce dedicated garbage collector config parameter and set default values for batch and streaming

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850469#comment-15850469
 ] 

Greg Hogan commented on FLINK-2772:
---

Is this still apropos? I'm not seeing a distinction between streaming and batch 
in the current version of {{taskmanager.sh}}.

> Introduce dedicated garbage collector config parameter and set default values 
> for batch and streaming
> -
>
> Key: FLINK-2772
> URL: https://issues.apache.org/jira/browse/FLINK-2772
> Project: Flink
>  Issue Type: Improvement
>  Components: Build System
>Reporter: Till Rohrmann
>Priority: Minor
>
> The {{taskmanager.sh}} start up script sets a default garbage collector if 
> the execution mode is {{streaming}} and if no {{env.java.opts}} configuration 
> value has been set. In the batch case, nothing is done. This seems to be 
> inconsistent. Therefore, I propose to add a dedicated configuration parameter 
> for the garbage collector. If the parameter is not set, then a default value 
> is set (e.g. G1 or CMS) for batch and streaming.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-2171) Add instruction to build Flink with Scala 2.11

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850299#comment-15850299
 ] 

Greg Hogan commented on FLINK-2171:
---

Is this now covered? 
https://ci.apache.org/projects/flink/flink-docs-master/setup/building.html#scala-versions

> Add instruction to build Flink with Scala 2.11
> --
>
> Key: FLINK-2171
> URL: https://issues.apache.org/jira/browse/FLINK-2171
> Project: Flink
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 0.9
>Reporter: Fabian Hueske
>Priority: Minor
>
> Flink can be built for Scala 2.11. However, the build documentation does not 
> cover include instructions for that.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-2042) Increase font size for stack figure on website home

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850293#comment-15850293
 ] 

Greg Hogan commented on FLINK-2042:
---

This looks to be resolved with the new website design.

> Increase font size for stack figure on website home
> ---
>
> Key: FLINK-2042
> URL: https://issues.apache.org/jira/browse/FLINK-2042
> Project: Flink
>  Issue Type: Improvement
>  Components: Project Website
>Reporter: Fabian Hueske
>Priority: Minor
>  Labels: starter
>
> The font of the stack figure on the website home is quite small and could be 
> increased.
> The image is also a bit blurred.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-1759) Execution statistics for vertex-centric iterations

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850264#comment-15850264
 ] 

Greg Hogan commented on FLINK-1759:
---

Might look at FLINK-3030 for the UI changes.

> Execution statistics for vertex-centric iterations
> --
>
> Key: FLINK-1759
> URL: https://issues.apache.org/jira/browse/FLINK-1759
> Project: Flink
>  Issue Type: Improvement
>  Components: Gelly
>Affects Versions: 0.9
>Reporter: Vasia Kalavri
>Priority: Minor
>
> It would be nice to add an option for gathering execution statistics from 
> VertexCentricIteration.
> In particular, the following metrics could be useful:
> - total number of supersteps
> - number of messages sent (total / per superstep)
> - bytes of messages exchanged (total / per superstep)
> - execution time (total / per superstep)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (FLINK-1703) option to balance load according to CPU available rather than slots available

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-1703.
-
Resolution: Duplicate

> option to balance load according to CPU available rather than slots available
> -
>
> Key: FLINK-1703
> URL: https://issues.apache.org/jira/browse/FLINK-1703
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Emmanuel Leroy
>  Labels: balancer, load
>
> To be able to accommodate for loading and running several different 
> topologies on a cluster, one may want to assign more slots per node than 
> there are CPUs available (many slots per CPU).
>  
> For example, with 3 nodes with 3 CPUs each, defining 3 slots per node (9 
> slots total) limits the number of task that can be loaded at any one time. 
> => Increasing the number of slots to 3 per CPU, gives 27 slots, allowing to 
> load more topologies.
> In this configuration (3 nodes, 3 CPUs, 27 slots), if a job specifies a 
> parallelism of 9, 9 slots out of 27 are used, but the distribution strategy 
> gives preference to slots on the same host as the stream source to limit 
> network IOs: as a consequence, the first job using 9 slots has the 9 slots 
> allocated on a single host (the one on the same host as the stream source), 
> even though this host only has 3 CPUs available. 
> These 9 slots are therefore sharing the same 3 CPUs, while the other 2 nodes 
> (6 CPUs) stay idle.
> The distribution strategy should take into account the number of CPUs 
> available along with the number of slots rather than rely only on the slot 
> availability.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-1337) Create an Amazon EMR Bootstrap Action

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850229#comment-15850229
 ] 

Greg Hogan commented on FLINK-1337:
---

Amazon EMR includes Flink as of December 2016 (version 
[5.2.1|http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-components.html]).
 Is there additional work under this ticket?

> Create an Amazon EMR Bootstrap Action
> -
>
> Key: FLINK-1337
> URL: https://issues.apache.org/jira/browse/FLINK-1337
> Project: Flink
>  Issue Type: New Feature
>  Components: Build System
>Reporter: Stephan Ewen
>Assignee: Timur Fayruzov
>Priority: Minor
>
> EMR offers bootstrap actions that prepare the cluster by installing 
> additional components, etc..
> We can offer a Flink bootstrap action that downloads, unpacks, and configures 
> Flink. It may optionally install libraries that we like to use (such as 
> Python, BLAS/JBLAS, ...)
> http://blogs.aws.amazon.com/bigdata/post/TxO6EHTHQALSIB/Getting-Started-with-Amazon-EMR-Bootstrap-Actions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-2168) Add HBaseTableSource

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850207#comment-15850207
 ] 

ASF GitHub Bot commented on FLINK-2168:
---

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/3149
  
The `NestedFieldsProjectableTableSource` makes sense to me. +1 to implement 
it in a separate JIRA.


> Add HBaseTableSource
> 
>
> Key: FLINK-2168
> URL: https://issues.apache.org/jira/browse/FLINK-2168
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Affects Versions: 0.9
>Reporter: Fabian Hueske
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> Add a {{HBaseTableSource}} to read data from a HBase table. The 
> {{HBaseTableSource}} should implement the {{ProjectableTableSource}} 
> (FLINK-3848) and {{FilterableTableSource}} (FLINK-3849) interfaces.
> The implementation can be based on Flink's {{TableInputFormat}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3149: FLINK-2168 Add HBaseTableSource

[wuchong]

Github user wuchong commented on the issue:

https://github.com/apache/flink/pull/3149
  
The `NestedFieldsProjectableTableSource` makes sense to me. +1 to implement 
it in a separate JIRA.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-3164) Spread out scheduling strategy

[Greg Hogan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Hogan closed FLINK-3164.
-
Resolution: Duplicate

> Spread out scheduling strategy
> --
>
> Key: FLINK-3164
> URL: https://issues.apache.org/jira/browse/FLINK-3164
> Project: Flink
>  Issue Type: Improvement
>  Components: DataSet API, Distributed Coordination
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>
> The size of a Flink cluster is bounded by the amount of memory allocated for 
> network buffers. The all-to-all distribution of data during a network shuffle 
> means that doubling the number of TaskManager slots quadruples the required 
> number of network buffers.
> A Flink job can be configured to execute operators with lower parallelism 
> which reduces the number of network buffers used across the cluster. Since 
> the Flink scheduler clusters tasks the number of network buffers to be 
> configured cannot be reduced.
> For example, if each TaskManager has 32 slots and the cluster has 32 
> TaskManagers the maximum parallelism can be set to 1024. If the preceding 
> operator has a parallelism of 32 then the TaskManager fan-out is between 
> 1*1024 (tasks evenly distributed) and 32*1024 (executed on a single 
> TaskManager).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5654) Add processing time OVER RANGE BETWEEN x PRECEDING aggregation to SQL

[radu (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850152#comment-15850152
 ] 

radu commented on FLINK-5654:
-

There is another decision point to be made. When we do the parsing of the SQL 
query two options are possible:

1) the entire expression is translated to a LogicalCalc
An example is
rel#8:LogicalCalc.NONE(input=rel#2:Subset#0.NONE,expr#0..5={inputs},expr#6=17:34:58,expr#7=360,expr#8=COUNT($t5)
 OVER (PARTITION BY $t2 ORDER BY $t6 RANGE BETWEEN $t7 PRECEDING AND CURRENT 
ROW),expr#9=0,expr#10=>($t8, $t9),expr#11=$SUM0($t5) OVER (PARTITION BY $t2 
ORDER BY $t6 RANGE BETWEEN $t7 PRECEDING AND CURRENT 
ROW),expr#12=CAST($t11):DOUBLE,expr#13=null,expr#14=CASE($t10, $t12, 
$t13),timeevent=$t0,sumB=$t14)

Notice that the logic related to window (boundaries, aggregates,partition over 
clause...) are Rex objects in the LogicalCalc (e.g. RexOver, RexWindow...)

2) the other option is that the whole logic of the over clause to be mapped 
directly to a LogicalWindow operator:
rel#14:LogicalWindow.NONE(input=rel#13:Subset#2.NONE,window#0=window(partition 
{1} order by [] range between $4 PRECEDING and CURRENT ROW aggs [COUNT($2), 
$SUM0($2)]))

Notice that all the information specific to the window processing is now in 
available in the LogicalWindow object, while the other information related to 
projection or calc will have a dedicated LogicalCalc node/operator.

>From my point of view it is more elegant and the code is more robust with the 
>second option. This is what i propose to be used (and the option that i 
>selected).
However, based on default rules adopted from Calcite in Flink, the 2 rules that 
enable to translate a query to the second option are not enabled.
That is why i added the 2 rules to the FlinkRuleSet

ProjectWindowTransposeRule.INSTANCE,
ProjectToWindowRule.INSTANCE,

I will push these modifications together with the rest of the code. However, if 
anyone has a strong preference for one option versus the other, or there is a 
specific reason that i am missing for which the two rules were not enabled in 
the first place, please let me know

> Add processing time OVER RANGE BETWEEN x PRECEDING aggregation to SQL
> -
>
> Key: FLINK-5654
> URL: https://issues.apache.org/jira/browse/FLINK-5654
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API & SQL
>Reporter: Fabian Hueske
>Assignee: radu
>
> The goal of this issue is to add support for OVER RANGE aggregations on 
> processing time streams to the SQL interface.
> Queries similar to the following should be supported:
> {code}
> SELECT 
>   a, 
>   SUM(b) OVER (PARTITION BY c ORDER BY procTime() RANGE BETWEEN INTERVAL '1' 
> HOUR PRECEDING AND CURRENT ROW) AS sumB,
>   MIN(b) OVER (PARTITION BY c ORDER BY procTime() RANGE BETWEEN INTERVAL '1' 
> HOUR PRECEDING AND CURRENT ROW) AS minB
> FROM myStream
> {code}
> The following restrictions should initially apply:
> - All OVER clauses in the same SELECT clause must be exactly the same.
> - The PARTITION BY clause is optional (no partitioning results in single 
> threaded execution).
> - The ORDER BY clause may only have procTime() as parameter. procTime() is a 
> parameterless scalar function that just indicates processing time mode.
> - UNBOUNDED PRECEDING is not supported (see FLINK-5657)
> - FOLLOWING is not supported.
> The restrictions will be resolved in follow up issues. If we find that some 
> of the restrictions are trivial to address, we can add the functionality in 
> this issue as well.
> This issue includes:
> - Design of the DataStream operator to compute OVER ROW aggregates
> - Translation from Calcite's RelNode representation (LogicalProject with 
> RexOver expression).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-867) Webinterface JobHistory does not work offline

[Greg Hogan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850127#comment-15850127
 ] 

Greg Hogan commented on FLINK-867:
--

[~StephanEwen] is this still an issue with the new web frontend?

> Webinterface JobHistory does not work offline
> -
>
> Key: FLINK-867
> URL: https://issues.apache.org/jira/browse/FLINK-867
> Project: Flink
>  Issue Type: Bug
>  Components: Webfrontend
>Reporter: Jonathan Hasenburg
>  Labels: github-import
> Fix For: pre-apache
>
>
> @StephanEwen found out that there is a problem with the JobHistory of the 
> Webinterface while there is no internet connection. It is because the 
> visualization API of google that you are only allowed to use online:
> ![bildschirmfoto 2014-05-23 um 12 06 
> 02|https://cloud.githubusercontent.com/assets/5738978/3081862/a7bc7332-e4c7-11e3-8c3e-6c68381d0a44.png)
> https://developers.google.com/chart/interactive/faq (the hole faq]
> So if we decide that there needs to be an offline function it must be 
> replaced by someone.
>  Imported from GitHub 
> Url: https://github.com/stratosphere/stratosphere/issues/867
> Created by: [JonathanH5|https://github.com/JonathanH5]
> Labels: 
> Created at: Mon May 26 13:19:57 CEST 2014
> State: open



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-2944) Collect, expose and display operator-specific stats

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-2944.
---
Resolution: Fixed

> Collect, expose and display operator-specific stats
> ---
>
> Key: FLINK-2944
> URL: https://issues.apache.org/jira/browse/FLINK-2944
> Project: Flink
>  Issue Type: Sub-task
>  Components: Local Runtime, Webfrontend
>Reporter: Fabian Hueske
>  Labels: requires-design-doc
>
> It would be nice to collect operator-specific stats such as:
> - HashJoin: bytes spilled build side, bytes spilled probe side, num keys 
> (build / probe)
> - GroupBy: bytes spilled, num keys, max group size, avg. group size
> - Combiner: combine rate (in/out ratio)
> - Streaming Ops: avg. throughput
> - etc.
> and display these stats in the webUI. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (FLINK-3766) Add a new tab for monitoring metrics in the web interface

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-3766.
---
   Resolution: Fixed
Fix Version/s: 1.2.0

This has been merged for Flink 1.2.0

> Add a new tab for monitoring metrics in the web interface
> -
>
> Key: FLINK-3766
> URL: https://issues.apache.org/jira/browse/FLINK-3766
> Project: Flink
>  Issue Type: New Feature
>  Components: Webfrontend
>Reporter: Robert Metzger
> Fix For: 1.2.0
>
> Attachments: metricsMock-selected.png, metricsMock-unselected.png
>
>
> Add a new tab for showing operator/task specific metrics in the Flink web ui.
> I propose to add a drop down into the tab that allows selecting the metrics.
> For each metric, we show a box in a grid.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (FLINK-4830) Show latency statistics in web interface

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger reassigned FLINK-4830:
-

Assignee: (was: Robert Metzger)

> Show latency statistics in web interface
> 
>
> Key: FLINK-4830
> URL: https://issues.apache.org/jira/browse/FLINK-4830
> Project: Flink
>  Issue Type: Sub-task
>  Components: Metrics, Webfrontend
>Reporter: Robert Metzger
> Fix For: pre-apache
>
>
> With FLINK-3660, we added a metric that measures the latency of records 
> flowing through the system.
> With this JIRA, I would like to expose the latency also in the web frontend.
> Therefore, we'll probably need to change the format the latency is reported 
> in the metric.
> I think we should show the latencies in two different views in the interface:
> - A global end-to-end latency in the overview of running jobs. This number is 
> probably pretty inaccurate, but should give users a first impression of the 
> latency characteristics currently present in the job
> - A detailed latency drill-down view, that allows to see how much latency is 
> added on average at each operator, for the sources.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5552) Make the JMX port available through RESTful API

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850107#comment-15850107
 ] 

Robert Metzger commented on FLINK-5552:
---

[~Zentol] any thoughts on this issue?
I wonder if we can create some infrastructure for a metric reporter to 
advertise its ports / hostnames etc?

Getting the JMX port for the TaskManagers is probably pretty hard because the 
reporter instances would need to report the port back to the TaskManager.

> Make the JMX port available through RESTful API
> ---
>
> Key: FLINK-5552
> URL: https://issues.apache.org/jira/browse/FLINK-5552
> Project: Flink
>  Issue Type: Improvement
>  Components: Metrics, Webfrontend
>Reporter: david.wang
>
> Currently, JMXReporter will create a server for JMX viewer retrieving JMX 
> stat. The port can be configured through configuration options, but for large 
> cluster with many machines running many Flink instances and other processes, 
> we can set a fixed port to JMX server, making it difficult to get the JMX 
> port.
> This JIRA is to suggest adding an api at web frontend so that it is very easy 
> to get the JMX port for JM and TM.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3250: Eratosthenes Sieve, a helpful and interesting example of ...

[greghogan]

Github user greghogan commented on the issue:

https://github.com/apache/flink/pull/3250
  
The code would need to be cleaned up. I would look to the other examples, 
add comments, remove unused code, change anonymous inner classes to be static 
inner classes, etc. There is also the question, what makes a good Flink 
example? Single-threaded programs 
[1](http://primesieve.org/segmented_sieve.html) and 
[2](http://sweet.ua.pt/tos/software/prime_sieve.html) can sieve 10^9 numbers in 
under 1 second. I think we are better off with batch examples that demonstrate 
Flink's performance (such as processing sparse datasets).

If you are interested in adding to the existing examples that would be a 
good discussion for the 
[dev](https://flink.apache.org/community.html#mailing-lists) mailing list. I 
also expect that more users are reading the online documentation than 
inspecting the examples so that is another opportunity for contributions.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-5502) Add documentation about migrating functions from 1.1 to 1.2

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek closed FLINK-5502.
---
Resolution: Fixed

Implemented on release-1.2:
da6ac7b5f7caf14bb2a8581db2e405e0585f7872

on master:
35460d24fcd1c736eb60033840aa5303c710ff6e

> Add documentation about migrating functions from 1.1 to 1.2
> ---
>
> Key: FLINK-5502
> URL: https://issues.apache.org/jira/browse/FLINK-5502
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Affects Versions: 1.2.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>  Labels: documentation
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5502) Add documentation about migrating functions from 1.1 to 1.2

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15850053#comment-15850053
 ] 

ASF GitHub Bot commented on FLINK-5502:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/3130
  
Thanks!  I merged your changes, could you please close this PR?


> Add documentation about migrating functions from 1.1 to 1.2
> ---
>
> Key: FLINK-5502
> URL: https://issues.apache.org/jira/browse/FLINK-5502
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Affects Versions: 1.2.0
>Reporter: Kostas Kloudas
>Assignee: Kostas Kloudas
>  Labels: documentation
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3130: [FLINK-5502] Add the function migration guide in docs.

[aljoscha]

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/3130
  
Thanks! 😄 I merged your changes, could you please close this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-5700) Document akka.client.timeout

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-5700:


 Summary: Document akka.client.timeout
 Key: FLINK-5700
 URL: https://issues.apache.org/jira/browse/FLINK-5700
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.3.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
Priority: Minor


The configuration parameter {{akka.client.timeout}} has not been properly 
documented.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-5699) Cancel with savepoint fails with a NPE if savepoint target directory not set

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-5699:


 Summary: Cancel with savepoint fails with a NPE if savepoint 
target directory not set
 Key: FLINK-5699
 URL: https://issues.apache.org/jira/browse/FLINK-5699
 Project: Flink
  Issue Type: Bug
  Components: State Backends, Checkpointing
Affects Versions: 1.3.0
Reporter: Till Rohrmann
Priority: Minor


When canceling a job with savepoint where one has not configured a savepoint 
directory, then the command fails with the following exception

{code}
java.lang.Exception: Canceling the job with ID 663f9769f0f3565b8ebc2acf0091431a 
failed.
at org.apache.flink.client.CliFrontend.cancel(CliFrontend.java:633)
at 
org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1082)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1123)
at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
at 
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at 
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1120)
Caused by: java.lang.Exception: Failed to cancel job 
663f9769f0f3565b8ebc2acf0091431a with savepoint.
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:634)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at 
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:36)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33)
at 
scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
at 
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118)
at 
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at 
org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:118)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.NullPointerException: Savepoint target directory
at 
org.apache.flink.util.Preconditions.checkNotNull(Preconditions.java:75)
at 
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.triggerSavepoint(CheckpointCoordinator.java:296)
at 
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:598)
... 22 more
{code}

I think we could return a more meaningful exception then the NPE to the user.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3255: [FLINK-5509] Replace QueryableStateClient keyHashC...

[dawidwys]

GitHub user dawidwys opened a pull request:

https://github.com/apache/flink/pull/3255

[FLINK-5509] Replace QueryableStateClient keyHashCode argument

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dawidwys/flink keyHashCode

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3255.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3255


commit e143e7d01af83c19169becfba6b7a25958f0f8b8
Author: Dawid Wysakowicz 
Date:   2017-02-02T14:21:47Z

[FLINK-5509] Replace QueryableStateClient keyHashCode argument




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-5509) Replace QueryableStateClient keyHashCode argument

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849963#comment-15849963
 ] 

ASF GitHub Bot commented on FLINK-5509:
---

GitHub user dawidwys opened a pull request:

https://github.com/apache/flink/pull/3255

[FLINK-5509] Replace QueryableStateClient keyHashCode argument

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dawidwys/flink keyHashCode

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3255.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3255


commit e143e7d01af83c19169becfba6b7a25958f0f8b8
Author: Dawid Wysakowicz 
Date:   2017-02-02T14:21:47Z

[FLINK-5509] Replace QueryableStateClient keyHashCode argument




> Replace QueryableStateClient keyHashCode argument
> -
>
> Key: FLINK-5509
> URL: https://issues.apache.org/jira/browse/FLINK-5509
> Project: Flink
>  Issue Type: Sub-task
>  Components: Queryable State
>Reporter: Ufuk Celebi
>Assignee: Dawid Wysakowicz
>Priority: Minor
>
> When going over the low level QueryableStateClient with [~NicoK] we noticed 
> that the key hashCode argument can be confusing to users:
> {code}
> Future getKvState(
>   JobID jobId,
>   String name,
>   int keyHashCode,
>   byte[] serializedKeyAndNamespace)
> {code}
> The {{keyHashCode}} argument is the result of calling {{hashCode()}} on the 
> key to look up. This is what is send to the JobManager in order to look up 
> the location of the key. While pretty straight forward, it is repetitive and 
> possibly confusing.
> As an alternative we suggest to make the method generic and simply call 
> hashCode on the object ourselves. This way the user just provides the key 
> object.
> Since there are some early users of the queryable state API already, we would 
> suggest to rename the method in order to provoke a compilation error after 
> upgrading to the actually released 1.2 version.
> (This would also work without renaming since the hashCode of Integer (what 
> users currently provide) is the same number, but it would be confusing why it 
> acutally works.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #2925: [FLINK-4574] [kinesis] Strengthen fetch interval implemen...

[tony810430]

Github user tony810430 commented on the issue:

https://github.com/apache/flink/pull/2925
  
Hi, @tzulitai 

I think there is no better way if trying to use `Timer` to deal with `0` 
fetch interval. Even though I implement my own `Timer` scheduler, it just move 
the `runForever` inside from `ShardConsumerTask`. Because fetch interval is 
`0`, there is no way to get the next expected execution time and the solution 
would be running the function forever.

I think a better solution is defined non-positive fetch interval differs 
from positive fetch interval and would be executed in another way in the 
document. The implementation will be two parts: the original way remains the 
same without `runForever` in `ShardConsumerFetcherTask` and the other is for 
non-positive fetch interval implemented by using a Thread running 
`ShardConsumerFetcherTask::run()` forever.

It is more reasonable for me to distinguish these two configuration and 
implement them by using two ways in `ShardConsumer::run()`. What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-4574) Strengthen fetch interval implementation in Kinesis consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849919#comment-15849919
 ] 

ASF GitHub Bot commented on FLINK-4574:
---

Github user tony810430 commented on the issue:

https://github.com/apache/flink/pull/2925
  
Hi, @tzulitai 

I think there is no better way if trying to use `Timer` to deal with `0` 
fetch interval. Even though I implement my own `Timer` scheduler, it just move 
the `runForever` inside from `ShardConsumerTask`. Because fetch interval is 
`0`, there is no way to get the next expected execution time and the solution 
would be running the function forever.

I think a better solution is defined non-positive fetch interval differs 
from positive fetch interval and would be executed in another way in the 
document. The implementation will be two parts: the original way remains the 
same without `runForever` in `ShardConsumerFetcherTask` and the other is for 
non-positive fetch interval implemented by using a Thread running 
`ShardConsumerFetcherTask::run()` forever.

It is more reasonable for me to distinguish these two configuration and 
implement them by using two ways in `ShardConsumer::run()`. What do you think?


> Strengthen fetch interval implementation in Kinesis consumer
> 
>
> Key: FLINK-4574
> URL: https://issues.apache.org/jira/browse/FLINK-4574
> Project: Flink
>  Issue Type: Improvement
>  Components: Kinesis Connector
>Affects Versions: 1.1.0
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Wei-Che Wei
>
> As pointed out by [~rmetzger], right now the fetch interval implementation in 
> the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer 
> interval times than specified by the user, ex. say the specified fetch 
> interval is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and 
> {{y}} to complete processing the fetched records for emitting, than the 
> actual interval between each fetch is actually {{f+x+y}}.
> The main problem with this is that we can never guarantee how much time has 
> past since the last {{getRecords}} call, thus can not guarantee that returned 
> shard iterators will not have expired the next time we use them, even if we 
> limit the user-given value for {{f}} to not be longer than the iterator 
> expire time.
> I propose to improve this by, per {{ShardConsumer}}, use a 
> {{ScheduledExecutorService}} / {{Timer}} to do the fixed-interval fetching, 
> and a separate blocking queue that collects the fetched records for emitting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-4574) Strengthen fetch interval implementation in Kinesis consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849898#comment-15849898
 ] 

ASF GitHub Bot commented on FLINK-4574:
---

Github user tony810430 commented on a diff in the pull request:

https://github.com/apache/flink/pull/2925#discussion_r99117155
  
--- Diff: 
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
 ---
@@ -154,42 +166,115 @@ public void run() {
}
}
 
-   // set the nextShardItr so we can 
continue iterating in the next while loop
-   nextShardItr = 
getRecordsResult.getNextShardIterator();
+   // set the startShardItr so we can 
continue iterating in the next while loop
+   startShardItr = 
getRecordsResult.getNextShardIterator();
} else {
// the last record was non-aggregated, 
so we can simply start from the next record
-   nextShardItr = 
kinesis.getShardIterator(subscribedShard, 
ShardIteratorType.AFTER_SEQUENCE_NUMBER.toString(), 
lastSequenceNum.getSequenceNumber());
+   startShardItr = 
kinesis.getShardIterator(subscribedShard, 
ShardIteratorType.AFTER_SEQUENCE_NUMBER.toString(), 
lastSequenceNum.getSequenceNumber());
}
}
 
-   while(isRunning()) {
-   if (nextShardItr == null) {
-   
fetcherRef.updateState(subscribedShardStateIndex, 
SentinelSequenceNumber.SENTINEL_SHARD_ENDING_SEQUENCE_NUM.get());
-
-   // we can close this consumer thread 
once we've reached the end of the subscribed shard
-   break;
-   } else {
-   if (fetchIntervalMillis != 0) {
-   
Thread.sleep(fetchIntervalMillis);
-   }
+   ArrayBlockingQueue queue = new 
ArrayBlockingQueue<>(maxNumberOfRecordsPerFetch);
+   ShardConsumerFetcher shardConsumerFetcher;
 
-   GetRecordsResult getRecordsResult = 
getRecords(nextShardItr, maxNumberOfRecordsPerFetch);
+   if (fetchIntervalMillis > 0L) {
+   shardConsumerFetcher = new 
ShardConsumerFetcher(this, startShardItr, queue, false);
+   timer.scheduleAtFixedRate(shardConsumerFetcher, 
0L, fetchIntervalMillis);
+   } else {
+   // if fetchIntervalMillis is 0, make the task 
run forever and schedule it once only.
+   shardConsumerFetcher = new 
ShardConsumerFetcher(this, startShardItr, queue, true);
+   timer.schedule(shardConsumerFetcher, 0L);
+   }
 
-   // each of the Kinesis records may be 
aggregated, so we must deaggregate them before proceeding
-   List fetchedRecords = 
deaggregateRecords(
-   getRecordsResult.getRecords(),
-   
subscribedShard.getShard().getHashKeyRange().getStartingHashKey(),
-   
subscribedShard.getShard().getHashKeyRange().getEndingHashKey());
+   while(isRunning()) {
+   UserRecord record = queue.poll();
+   if (record != null) {
+   
deserializeRecordForCollectionAndUpdateState(record);
+   } else {
+   if (shardConsumerFetcher.nextShardItr 
== null) {
+   
fetcherRef.updateState(subscribedShardStateIndex, 
SentinelSequenceNumber.SENTINEL_SHARD_ENDING_SEQUENCE_NUM.get());
 
-   for (UserRecord record : 
fetchedRecords) {
-   
deserializeRecordForCollectionAndUpdateState(record);
+   // we can close this consumer 
thread once we've reached the end of the subscribed shard
+   break;
}
+   }
 
-   nextShardItr = 

[GitHub] flink pull request #2925: [FLINK-4574] [kinesis] Strengthen fetch interval i...

[tony810430]

Github user tony810430 commented on a diff in the pull request:

https://github.com/apache/flink/pull/2925#discussion_r99117155
  
--- Diff: 
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
 ---
@@ -154,42 +166,115 @@ public void run() {
}
}
 
-   // set the nextShardItr so we can 
continue iterating in the next while loop
-   nextShardItr = 
getRecordsResult.getNextShardIterator();
+   // set the startShardItr so we can 
continue iterating in the next while loop
+   startShardItr = 
getRecordsResult.getNextShardIterator();
} else {
// the last record was non-aggregated, 
so we can simply start from the next record
-   nextShardItr = 
kinesis.getShardIterator(subscribedShard, 
ShardIteratorType.AFTER_SEQUENCE_NUMBER.toString(), 
lastSequenceNum.getSequenceNumber());
+   startShardItr = 
kinesis.getShardIterator(subscribedShard, 
ShardIteratorType.AFTER_SEQUENCE_NUMBER.toString(), 
lastSequenceNum.getSequenceNumber());
}
}
 
-   while(isRunning()) {
-   if (nextShardItr == null) {
-   
fetcherRef.updateState(subscribedShardStateIndex, 
SentinelSequenceNumber.SENTINEL_SHARD_ENDING_SEQUENCE_NUM.get());
-
-   // we can close this consumer thread 
once we've reached the end of the subscribed shard
-   break;
-   } else {
-   if (fetchIntervalMillis != 0) {
-   
Thread.sleep(fetchIntervalMillis);
-   }
+   ArrayBlockingQueue queue = new 
ArrayBlockingQueue<>(maxNumberOfRecordsPerFetch);
+   ShardConsumerFetcher shardConsumerFetcher;
 
-   GetRecordsResult getRecordsResult = 
getRecords(nextShardItr, maxNumberOfRecordsPerFetch);
+   if (fetchIntervalMillis > 0L) {
+   shardConsumerFetcher = new 
ShardConsumerFetcher(this, startShardItr, queue, false);
+   timer.scheduleAtFixedRate(shardConsumerFetcher, 
0L, fetchIntervalMillis);
+   } else {
+   // if fetchIntervalMillis is 0, make the task 
run forever and schedule it once only.
+   shardConsumerFetcher = new 
ShardConsumerFetcher(this, startShardItr, queue, true);
+   timer.schedule(shardConsumerFetcher, 0L);
+   }
 
-   // each of the Kinesis records may be 
aggregated, so we must deaggregate them before proceeding
-   List fetchedRecords = 
deaggregateRecords(
-   getRecordsResult.getRecords(),
-   
subscribedShard.getShard().getHashKeyRange().getStartingHashKey(),
-   
subscribedShard.getShard().getHashKeyRange().getEndingHashKey());
+   while(isRunning()) {
+   UserRecord record = queue.poll();
+   if (record != null) {
+   
deserializeRecordForCollectionAndUpdateState(record);
+   } else {
+   if (shardConsumerFetcher.nextShardItr 
== null) {
+   
fetcherRef.updateState(subscribedShardStateIndex, 
SentinelSequenceNumber.SENTINEL_SHARD_ENDING_SEQUENCE_NUM.get());
 
-   for (UserRecord record : 
fetchedRecords) {
-   
deserializeRecordForCollectionAndUpdateState(record);
+   // we can close this consumer 
thread once we've reached the end of the subscribed shard
+   break;
}
+   }
 
-   nextShardItr = 
getRecordsResult.getNextShardIterator();
+   Throwable throwable = this.error.get();
+   if (throwable != null) {
+   throw throwable;
}
 

[GitHub] flink issue #3254: Fix typo in code snippet

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3254
  
+1 to merge


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-5509) Replace QueryableStateClient keyHashCode argument

[Dawid Wysakowicz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Wysakowicz reassigned FLINK-5509:
---

Assignee: Dawid Wysakowicz

> Replace QueryableStateClient keyHashCode argument
> -
>
> Key: FLINK-5509
> URL: https://issues.apache.org/jira/browse/FLINK-5509
> Project: Flink
>  Issue Type: Sub-task
>  Components: Queryable State
>Reporter: Ufuk Celebi
>Assignee: Dawid Wysakowicz
>Priority: Minor
>
> When going over the low level QueryableStateClient with [~NicoK] we noticed 
> that the key hashCode argument can be confusing to users:
> {code}
> Future getKvState(
>   JobID jobId,
>   String name,
>   int keyHashCode,
>   byte[] serializedKeyAndNamespace)
> {code}
> The {{keyHashCode}} argument is the result of calling {{hashCode()}} on the 
> key to look up. This is what is send to the JobManager in order to look up 
> the location of the key. While pretty straight forward, it is repetitive and 
> possibly confusing.
> As an alternative we suggest to make the method generic and simply call 
> hashCode on the object ourselves. This way the user just provides the key 
> object.
> Since there are some early users of the queryable state API already, we would 
> suggest to rename the method in order to provoke a compilation error after 
> upgrading to the actually released 1.2 version.
> (This would also work without renaming since the hashCode of Integer (what 
> users currently provide) is the same number, but it would be confusing why it 
> acutally works.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3250: Eratosthenes Sieve, a helpful and interesting example of ...

[dsnz]

Github user dsnz commented on the issue:

https://github.com/apache/flink/pull/3250
  
Hello,
I can describe the algorithm but I cannot say anything about 
appropriateness as an example running on a distributed stream processor. I 
don't use Flink, I just tried to learn quickly its API and get a handle of it 
and certainly I cannot run/try distributed instances.
Is this okay ? How can I best help with this example ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-5698) Add NestedFieldsProjectableTableSource interface

[Anton Solovev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Solovev updated FLINK-5698:
-
Issue Type: New Feature  (was: Improvement)

> Add NestedFieldsProjectableTableSource interface
> 
>
> Key: FLINK-5698
> URL: https://issues.apache.org/jira/browse/FLINK-5698
> Project: Flink
>  Issue Type: New Feature
>  Components: Table API & SQL
>Reporter: Anton Solovev
>
> Add a NestedFieldsProjectableTableSource interface for some TableSource 
> implementation that support nesting projection push-down.
> The interface could look as follows
> {code}
> def trait ProjectableTableSource {
>   def projectNestedFields(fields: Array[String]): 
> NestedFieldsProjectableTableSource[T]
> }
> {code}
> This interface works together with ProjectableTableSource 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (FLINK-5698) Add NestedFieldsProjectableTableSource interface

[Anton Solovev (JIRA)]

Anton Solovev created FLINK-5698:


 Summary: Add NestedFieldsProjectableTableSource interface
 Key: FLINK-5698
 URL: https://issues.apache.org/jira/browse/FLINK-5698
 Project: Flink
  Issue Type: Improvement
  Components: Table API & SQL
Reporter: Anton Solovev


Add a NestedFieldsProjectableTableSource interface for some TableSource 
implementation that support nesting projection push-down.
The interface could look as follows
{code}
def trait ProjectableTableSource {
  def projectNestedFields(fields: Array[String]): 
NestedFieldsProjectableTableSource[T]
}
{code}
This interface works together with ProjectableTableSource 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5481) Cannot create Collection of Row

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849803#comment-15849803
 ] 

ASF GitHub Bot commented on FLINK-5481:
---

Github user tonycox commented on the issue:

https://github.com/apache/flink/pull/3127
  
```scala
import Types._
  ROW(Array("person", "additional"),
ROW(Array("name", "age"), STRING, STRING),
ROW(Array("more_info", "and_so_on"), STRING, STRING))
```


> Cannot create Collection of Row 
> 
>
> Key: FLINK-5481
> URL: https://issues.apache.org/jira/browse/FLINK-5481
> Project: Flink
>  Issue Type: Bug
>  Components: DataSet API, Table API & SQL
>Affects Versions: 1.2.0
>Reporter: Anton Solovev
>Assignee: Anton Solovev
>Priority: Trivial
>
> When we use {{ExecutionEnvironment#fromElements(X... data)}} it takes first 
> element of {{data}} to define a type. If first Row in collection has wrong 
> number of fields (there are nulls) TypeExtractor returns not RowTypeInfo, but 
> GenericType



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink issue #3127: [FLINK-5481] Add type extraction from collection

[tonycox]

Github user tonycox commented on the issue:

https://github.com/apache/flink/pull/3127
  
```scala
import Types._
  ROW(Array("person", "additional"),
ROW(Array("name", "age"), STRING, STRING),
ROW(Array("more_info", "and_so_on"), STRING, STRING))
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (FLINK-5635) Improve Docker tooling to make it easier to build images and launch Flink via Docker tools

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-5635:
--
Fix Version/s: (was: 1.2.0)

> Improve Docker tooling to make it easier to build images and launch Flink via 
> Docker tools
> --
>
> Key: FLINK-5635
> URL: https://issues.apache.org/jira/browse/FLINK-5635
> Project: Flink
>  Issue Type: Improvement
>  Components: Docker
>Affects Versions: 1.2.0
>Reporter: Jamie Grier
>Assignee: Jamie Grier
>
> This is a bit of a catch-all ticket for general improvements to the Flink on 
> Docker experience.
> Things to improve:
>   - Make it possible to build a Docker image from your own flink-dist 
> directory as well as official releases.
>   - Make it possible to override the image name so a user can more easily 
> publish these images to their Docker repository
>   - Provide scripts that show how to properly run on Docker Swarm or similar 
> environments with overlay networking (Kubernetes) without using host 
> networking.
>   - Log to stdout rather than to files.
>   - Work properly with docker-compose for local deployment as well as 
> production deployments (Swarm/k8s)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (FLINK-5634) Flink should not always redirect stdout to a file.

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger updated FLINK-5634:
--
Fix Version/s: (was: 1.2.0)

> Flink should not always redirect stdout to a file.
> --
>
> Key: FLINK-5634
> URL: https://issues.apache.org/jira/browse/FLINK-5634
> Project: Flink
>  Issue Type: Bug
>  Components: Docker
>Affects Versions: 1.2.0
>Reporter: Jamie Grier
>Assignee: Jamie Grier
>
> Flink always redirects stdout to a file.  While often convenient this isn't 
> always what people want.  The most obvious case of this is a Docker 
> deployment.
> It should be possible to have Flink log to stdout.
> Here is a PR for this:  https://github.com/apache/flink/pull/3204



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3254: Fix typo in code snippet

[katsharp]

GitHub user katsharp opened a pull request:

https://github.com/apache/flink/pull/3254

Fix typo in code snippet

* fix typo in code snippet CheckpoingConfig -> CheckpointConfig
* no jira as this is a typo fix, as per 
http://flink.apache.org/contribute-documentation.html
* built docs locally and checked the page loaded
* mvn clean verify passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/katsharp/flink typoInCheckpointDocs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3254.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3254


commit 00b12d42a9c22f50aa0fba944109b791eb3e29bf
Author: Kathleen Sharp 
Date:   2017-02-02T09:08:48Z

Fix typo in code snippet

* fix typo in code snippet CheckpoingConfig -> CheckpointConfig




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (FLINK-5606) Remove magic number in key and namespace serialization

[Dawid Wysakowicz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dawid Wysakowicz reassigned FLINK-5606:
---

Assignee: Dawid Wysakowicz

> Remove magic number in key and namespace serialization
> --
>
> Key: FLINK-5606
> URL: https://issues.apache.org/jira/browse/FLINK-5606
> Project: Flink
>  Issue Type: Sub-task
>  Components: Queryable State
>Reporter: Ufuk Celebi
>Assignee: Dawid Wysakowicz
>Priority: Minor
>
> The serialized key and namespace for state queries contains a magic number 
> between the key and namespace: {{key|42|namespace}}. This was for historical 
> reasons in order to skip deserialization of the key and namespace for our old 
> {{RocksDBStateBackend}} which used the same format. This has now been 
> superseded by the keygroup aware state backends and there is no point in 
> doing this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (FLINK-5606) Remove magic number in key and namespace serialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849771#comment-15849771
 ] 

ASF GitHub Bot commented on FLINK-5606:
---

GitHub user dawidwys opened a pull request:

https://github.com/apache/flink/pull/3253

[FLINK-5606] Remove magic number in key and namespace serialization

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dawidwys/flink magicNumber

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3253.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3253


commit 19b9c9c03f10bbeaef7ab7ed008b263842c18d40
Author: Dawid Wysakowicz 
Date:   2017-02-02T10:30:03Z

[FLINK-5606] Remove magic number in key and namespace serialization




> Remove magic number in key and namespace serialization
> --
>
> Key: FLINK-5606
> URL: https://issues.apache.org/jira/browse/FLINK-5606
> Project: Flink
>  Issue Type: Sub-task
>  Components: Queryable State
>Reporter: Ufuk Celebi
>Priority: Minor
>
> The serialized key and namespace for state queries contains a magic number 
> between the key and namespace: {{key|42|namespace}}. This was for historical 
> reasons in order to skip deserialization of the key and namespace for our old 
> {{RocksDBStateBackend}} which used the same format. This has now been 
> superseded by the keygroup aware state backends and there is no point in 
> doing this.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] flink pull request #3253: [FLINK-5606] Remove magic number in key and namesp...

[dawidwys]

GitHub user dawidwys opened a pull request:

https://github.com/apache/flink/pull/3253

[FLINK-5606] Remove magic number in key and namespace serialization

Thanks for contributing to Apache Flink. Before you open your pull request, 
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your 
pull request. For more information and/or questions please refer to the [How To 
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful 
description of your changes.

- [x] General
  - The pull request references the related JIRA issue ("[FLINK-XXX] Jira 
title text")
  - The pull request addresses only one issue
  - Each commit in the PR has a meaningful commit message (including the 
JIRA id)

- [x] Documentation
  - Documentation has been added for new functionality
  - Old documentation affected by the pull request has been updated
  - JavaDoc for public methods has been added

- [x] Tests & Build
  - Functionality added by the pull request is covered by tests
  - `mvn clean verify` has been executed successfully locally or a Travis 
build has passed


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dawidwys/flink magicNumber

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/3253.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3253


commit 19b9c9c03f10bbeaef7ab7ed008b263842c18d40
Author: Dawid Wysakowicz 
Date:   2017-02-02T10:30:03Z

[FLINK-5606] Remove magic number in key and namespace serialization




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---