[jira] [Commented] (FLINK-3179) Combiner is not injected if Reduce or GroupReduce input is explicitly partitioned

[ramkrishna.s.vasudevan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148072#comment-15148072
 ] 

ramkrishna.s.vasudevan commented on FLINK-3179:
---

I have updated the PR [~fhueske]. Things seems to work but am not sure on the 
way things are done. May need some refinement. Feedback and comments welcome.

> Combiner is not injected if Reduce or GroupReduce input is explicitly 
> partitioned
> -
>
> Key: FLINK-3179
> URL: https://issues.apache.org/jira/browse/FLINK-3179
> Project: Flink
>  Issue Type: Bug
>  Components: Optimizer
>Affects Versions: 0.10.1
>Reporter: Fabian Hueske
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 1.0.0, 0.10.2
>
>
> The optimizer does not inject a combiner if the input of a Reducer or 
> GroupReducer is explicitly partitioned as in the following example
> {code}
> DataSet> words = ...
> DataSet> counts = words
>   .partitionByHash(0)
>   .groupBy(0)
>   .sum(1);
> {code}
> Explicit partitioning can be useful to enforce partitioning on a subset of 
> keys or to use a different partitioning method (custom or range partitioning).
> This issue should be fixed by changing the {{instantiate()}} methods of the 
> {{ReduceProperties}} and {{GroupReduceWithCombineProperties}} classes such 
> that a combine is injected in front of a {{PartitionPlanNode}} if it is the 
> input of a Reduce or GroupReduce operator. This should only happen, if the 
> Reducer is the only successor of the Partition operator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147831#comment-15147831
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user twalthr commented on the pull request:

https://github.com/apache/flink/pull/1634#issuecomment-184412261
  
@fhueske you were totally right. I reworked the casting again.


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Casting support for arithmetic op...

[twalthr]

Github user twalthr commented on the pull request:

https://github.com/apache/flink/pull/1634#issuecomment-184412261
  
@fhueske you were totally right. I reworked the casting again.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3410) setting setNumberOfExecutionRetries to 0 still leads to RESTARTs.

[Robert Metzger (JIRA)]

Robert Metzger created FLINK-3410:
-

 Summary: setting setNumberOfExecutionRetries to 0 still leads to 
RESTARTs.
 Key: FLINK-3410
 URL: https://issues.apache.org/jira/browse/FLINK-3410
 Project: Flink
  Issue Type: Bug
  Components: JobManager
Affects Versions: 1.0.0
Reporter: Robert Metzger


While testing the RC0 for 1.0.0 I found the following issue:

Setting the number of retries to 0 still leads to the job being restarted:

{code}
final StreamExecutionEnvironment see = 
StreamExecutionEnvironment.getExecutionEnvironment();
see.setNumberOfExecutionRetries(0);
{code}

{code}
21:19:50,677 INFO  org.apache.flink.runtime.jobmanager.JobManager   
 - Status of job 0e78d0825da485167aabee7e63c8e913 (Data Generator) changed to 
RESTARTING.
21:19:50,678 INFO  
org.apache.flink.runtime.executiongraph.restart.FixedDelayRestartStrategy  - 
Delaying retry of job execution for 1 ms ...
{code}

While looking through the code, it seems that the execution config is returning 
{{null}} when the number of retries is set to 0. With {{null}} the jobManager 
picks the default restart strategy.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3409) Integrate STOP with Savepoints

[Matthias J. Sax (JIRA)]

Matthias J. Sax created FLINK-3409:
--

 Summary: Integrate STOP with Savepoints
 Key: FLINK-3409
 URL: https://issues.apache.org/jira/browse/FLINK-3409
 Project: Flink
  Issue Type: Improvement
  Components: Command-line client, JobManager
Reporter: Matthias J. Sax


Right now, if a savepoint is triggered the job keeps running. At the same time, 
on STOP a last checkpoint should be collected for a clean shutdown (WIP: 
https://issues.apache.org/jira/browse/FLINK-3408).

This work should enable stopping a job and getting a final savepoint at the 
same time. Thus, a job can get stopped gracefully and resumed later on using 
the gathered checkpoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3408) Make STOP signal reliable

[Matthias J. Sax (JIRA)]

Matthias J. Sax created FLINK-3408:
--

 Summary: Make STOP signal reliable
 Key: FLINK-3408
 URL: https://issues.apache.org/jira/browse/FLINK-3408
 Project: Flink
  Issue Type: Improvement
  Components: Distributed Runtime, JobManager, TaskManager
Reporter: Matthias J. Sax


Right now, STOP signal does not give any consistency guarantees. Thus, the 
signal might get (partial or completely) lost.

In this work, it should be ensured, that STOP is never lost and a job is 
stopped gracefully if possible, including job restart after failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3407) Extend TwitterSource with interface StoppableFunction

[Matthias J. Sax (JIRA)]

Matthias J. Sax created FLINK-3407:
--

 Summary: Extend TwitterSource with interface StoppableFunction
 Key: FLINK-3407
 URL: https://issues.apache.org/jira/browse/FLINK-3407
 Project: Flink
  Issue Type: Improvement
  Components: Streaming Connectors
Reporter: Matthias J. Sax


{{TwitterSource}} is not stoppable right now. To make it stoppable, is must 
implement {{StoppableFunction}}. Implementing method {{stop()}} must ensure, 
that the source stops receiving new Tweets and issues a final checkpoint. 
Afterwards, {{run()}} must return.

When implementing this, keep in mind, that the gathered checkpoint might later 
be used as a savepoint.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3406) Extend RabbitMQ source with interface StoppableFunction

[Matthias J. Sax (JIRA)]

Matthias J. Sax created FLINK-3406:
--

 Summary: Extend RabbitMQ source with interface StoppableFunction
 Key: FLINK-3406
 URL: https://issues.apache.org/jira/browse/FLINK-3406
 Project: Flink
  Issue Type: Improvement
  Components: Streaming Connectors
Reporter: Matthias J. Sax


{{RMQSource}} is not stoppable right now. To make it stoppable, is must 
implement {{StoppableFunction}}. Implementing method {{stop()}} must ensure, 
that the source stops receiving new messages from RabbitMQ and issues a final 
checkpoint. Afterwards, {{run()}} must return.

When implementing this, keep in mind, that the gathered checkpoint might later 
be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3405) Extend NiFiSource with interface StoppableFunction

[Matthias J. Sax (JIRA)]

Matthias J. Sax created FLINK-3405:
--

 Summary: Extend NiFiSource with interface StoppableFunction
 Key: FLINK-3405
 URL: https://issues.apache.org/jira/browse/FLINK-3405
 Project: Flink
  Issue Type: Improvement
  Components: Streaming Connectors
Reporter: Matthias J. Sax


Nifi source is not stoppable right now. To make it stoppable, is must implement 
{{StoppableFunction}}. Implementing method {{stop()}} must ensure, that the 
source stops receiving new messages from Nifi and issues a final checkpoint. 
Afterwards, {{run()}} must return.

When implementing this, keep in mind, that the gathered checkpoint might later 
be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3404) Extend Kafka consumers with interface StoppableFunction

[Matthias J. Sax (JIRA)]

Matthias J. Sax created FLINK-3404:
--

 Summary: Extend Kafka consumers with interface StoppableFunction
 Key: FLINK-3404
 URL: https://issues.apache.org/jira/browse/FLINK-3404
 Project: Flink
  Issue Type: Improvement
  Components: Kafka Connector
Reporter: Matthias J. Sax


Kafka consumers are not stoppable right now. To make them stoppable, they must 
implement {{StoppableFunction}}. Implementing method {{stop()}} must ensure, 
that the consumer stops pulling new messages from Kafka and issues a final 
checkpoint with the last offset. Afterwards, {{run()}} must return.

When implementing this, keep in mind, that the gathered checkpoint might later 
be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-3158) Shading does not remove google guava from flink-dist fat jar

[Robert Metzger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Metzger resolved FLINK-3158.
---
Resolution: Fixed

As of http://git-wip-us.apache.org/repos/asf/flink/commit/65d08058, we use the 
maven-enforcer-plugin to ensure a maven version < 3.3 is used for creating the 
release.

> Shading does not remove google guava from flink-dist fat jar
> 
>
> Key: FLINK-3158
> URL: https://issues.apache.org/jira/browse/FLINK-3158
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 1.0.0, 0.10.1
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Blocker
>
> It seems that guava somehow slipped our checks and made it into the 
> flink-dist fat jar again.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3403) Create Section "Working with Time" in Streaming Guide

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-3403:
---

 Summary: Create Section "Working with Time" in Streaming Guide
 Key: FLINK-3403
 URL: https://issues.apache.org/jira/browse/FLINK-3403
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek


We should add a proper page for this. Where we explain the notions of time and 
how to setup programs.

Also, we would explain how to work with watermarks and the different timestamp 
extractors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-3402) Refactor Common Parts of Stream/Batch Documentation

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-3402:
---

 Summary: Refactor Common Parts of Stream/Batch Documentation
 Key: FLINK-3402
 URL: https://issues.apache.org/jira/browse/FLINK-3402
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek


I want to build on the work of [~uce] in refactoring the streaming guide. With 
the release on the horizon I think it is important to have good structure in 
the documentation.

I propose to move the following sections from the Streaming Doc to a new 
Section "Basic Concepts" (name up for discussion):

 - Linking With Flink
 - DataStream abstraction (remove, this can be covered by an extended "Lazy 
Evaluation")
 - Lazy Evaluation
 - Specifying Keys
 - Passing Functions to Flink
 - Data Types
 - Debugging
 - Program Packaging and Distributed Execution
 - Parallel Execution
 - Execution Plans



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147650#comment-15147650
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1639#issuecomment-184326419
  
Looks good. :-) 
+1 to merge.


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Translation of explicit casting

[fhueske]

Github user fhueske commented on the pull request:

https://github.com/apache/flink/pull/1639#issuecomment-184326419
  
Looks good. :-) 
+1 to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-3401) AscendingTimestampExtractor should not fail on order violation

[Stephan Ewen (JIRA)]

Stephan Ewen created FLINK-3401:
---

 Summary: AscendingTimestampExtractor should not fail on order 
violation
 Key: FLINK-3401
 URL: https://issues.apache.org/jira/browse/FLINK-3401
 Project: Flink
  Issue Type: Sub-task
  Components: Streaming
Affects Versions: 0.10.1
Reporter: Stephan Ewen
Assignee: Stephan Ewen
 Fix For: 1.0.0


Currently, the {{AscendingTimestampExtractor}} fails hard when the order of 
timestamps is violated.

In the spirit of "streaming jobs never fail", I propose to change the default 
behavior such that violations are only logged (later collected in metrics) and 
that hard failures have to be explicitly activated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147622#comment-15147622
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1634#discussion_r52925323
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/OperatorCodeGen.scala
 ---
@@ -18,20 +18,56 @@
 package org.apache.flink.api.table.codegen
 
 import org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO
-import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.common.typeinfo.{NumericTypeInfo, 
TypeInformation}
 import org.apache.flink.api.table.codegen.CodeGenUtils._
 
 object OperatorCodeGen {
 
-   def generateArithmeticOperator(
+  def generateArithmeticOperator(
   operator: String,
   nullCheck: Boolean,
   resultType: TypeInformation[_],
   left: GeneratedExpression,
   right: GeneratedExpression)
 : GeneratedExpression = {
-generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+// String arithmetic // TODO rework
+if (isString(left)) {
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
   (leftTerm, rightTerm) => s"$leftTerm $operator $rightTerm"
+  }
+}
+// Numeric arithmetic
+else if (isNumeric(left) && isNumeric(right)) {
+  val leftType = left.resultType.asInstanceOf[NumericTypeInfo[_]]
+  val rightType = right.resultType.asInstanceOf[NumericTypeInfo[_]]
+
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+  (leftTerm, rightTerm) =>
+// insert auto casting for "narrowing primitive conversions"
+if (leftType != rightType) {
+  // leftType can not be casted to rightType automatically -> 
narrow
+  if (!leftType.shouldAutocastTo(rightType)) {
--- End diff --

Just checked, double is actually downcasted.
The following test fails:
```
val t = env.fromElements((10.0d: Double, 1: Byte)).toTable.select('_1 + 
'_2)
val expected = "11.0"
```


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Casting support for arithmetic op...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1634#discussion_r52925323
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/OperatorCodeGen.scala
 ---
@@ -18,20 +18,56 @@
 package org.apache.flink.api.table.codegen
 
 import org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO
-import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.common.typeinfo.{NumericTypeInfo, 
TypeInformation}
 import org.apache.flink.api.table.codegen.CodeGenUtils._
 
 object OperatorCodeGen {
 
-   def generateArithmeticOperator(
+  def generateArithmeticOperator(
   operator: String,
   nullCheck: Boolean,
   resultType: TypeInformation[_],
   left: GeneratedExpression,
   right: GeneratedExpression)
 : GeneratedExpression = {
-generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+// String arithmetic // TODO rework
+if (isString(left)) {
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
   (leftTerm, rightTerm) => s"$leftTerm $operator $rightTerm"
+  }
+}
+// Numeric arithmetic
+else if (isNumeric(left) && isNumeric(right)) {
+  val leftType = left.resultType.asInstanceOf[NumericTypeInfo[_]]
+  val rightType = right.resultType.asInstanceOf[NumericTypeInfo[_]]
+
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+  (leftTerm, rightTerm) =>
+// insert auto casting for "narrowing primitive conversions"
+if (leftType != rightType) {
+  // leftType can not be casted to rightType automatically -> 
narrow
+  if (!leftType.shouldAutocastTo(rightType)) {
--- End diff --

Just checked, double is actually downcasted.
The following test fails:
```
val t = env.fromElements((10.0d: Double, 1: Byte)).toTable.select('_1 + 
'_2)
val expected = "11.0"
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3396) Job submission Savepoint restore logic flawed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147615#comment-15147615
 ] 

ASF GitHub Bot commented on FLINK-3396:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1633#issuecomment-184313039
  
Changes look good to me @uce. I had only one inline question concerning a 
semantic change.

Apart from that +1 for merging.


> Job submission Savepoint restore logic flawed
> -
>
> Key: FLINK-3396
> URL: https://issues.apache.org/jira/browse/FLINK-3396
> Project: Flink
>  Issue Type: Bug
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
> Fix For: 1.0.0
>
>
> When savepoint restoring fails, the thrown Exception fails the execution 
> graph, but the client is not informed about the failure.
> The expected behaviour is that the submission should be acked with success or 
> failure in any case. With savepoint restore failures, the ack message will be 
> skipped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3396] [runtime] Fail job submission aft...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1633#issuecomment-184313039
  
Changes look good to me @uce. I had only one inline question concerning a 
semantic change.

Apart from that +1 for merging.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3396) Job submission Savepoint restore logic flawed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147613#comment-15147613
 ] 

ASF GitHub Bot commented on FLINK-3396:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1633#discussion_r52924511
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -1073,57 +1073,73 @@ class JobManager(
   // execute the recovery/writing the jobGraph into the 
SubmittedJobGraphStore asynchronously
   // because it is a blocking operation
   future {
-try {
-  if (isRecovery) {
-executionGraph.restoreLatestCheckpointedState()
-  }
-  else {
-val snapshotSettings = jobGraph.getSnapshotSettings
-if (snapshotSettings != null) {
-  val savepointPath = snapshotSettings.getSavepointPath()
+val restoreStateSuccess =
+  try {
+if (isRecovery) {
+  executionGraph.restoreLatestCheckpointedState()
--- End diff --

Is it intended that now failures in the `restoreLatestCheckpointedState` 
are non recoverable as well? This seems to be different from the former 
implementation.


> Job submission Savepoint restore logic flawed
> -
>
> Key: FLINK-3396
> URL: https://issues.apache.org/jira/browse/FLINK-3396
> Project: Flink
>  Issue Type: Bug
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
> Fix For: 1.0.0
>
>
> When savepoint restoring fails, the thrown Exception fails the execution 
> graph, but the client is not informed about the failure.
> The expected behaviour is that the submission should be acked with success or 
> failure in any case. With savepoint restore failures, the ack message will be 
> skipped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3396] [runtime] Fail job submission aft...

[tillrohrmann]

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/1633#discussion_r52924511
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -1073,57 +1073,73 @@ class JobManager(
   // execute the recovery/writing the jobGraph into the 
SubmittedJobGraphStore asynchronously
   // because it is a blocking operation
   future {
-try {
-  if (isRecovery) {
-executionGraph.restoreLatestCheckpointedState()
-  }
-  else {
-val snapshotSettings = jobGraph.getSnapshotSettings
-if (snapshotSettings != null) {
-  val savepointPath = snapshotSettings.getSavepointPath()
+val restoreStateSuccess =
+  try {
+if (isRecovery) {
+  executionGraph.restoreLatestCheckpointedState()
--- End diff --

Is it intended that now failures in the `restoreLatestCheckpointedState` 
are non recoverable as well? This seems to be different from the former 
implementation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147608#comment-15147608
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1634#discussion_r52924026
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/OperatorCodeGen.scala
 ---
@@ -18,20 +18,56 @@
 package org.apache.flink.api.table.codegen
 
 import org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO
-import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.common.typeinfo.{NumericTypeInfo, 
TypeInformation}
 import org.apache.flink.api.table.codegen.CodeGenUtils._
 
 object OperatorCodeGen {
 
-   def generateArithmeticOperator(
+  def generateArithmeticOperator(
   operator: String,
   nullCheck: Boolean,
   resultType: TypeInformation[_],
   left: GeneratedExpression,
   right: GeneratedExpression)
 : GeneratedExpression = {
-generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+// String arithmetic // TODO rework
+if (isString(left)) {
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
   (leftTerm, rightTerm) => s"$leftTerm $operator $rightTerm"
+  }
+}
+// Numeric arithmetic
+else if (isNumeric(left) && isNumeric(right)) {
+  val leftType = left.resultType.asInstanceOf[NumericTypeInfo[_]]
+  val rightType = right.resultType.asInstanceOf[NumericTypeInfo[_]]
+
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+  (leftTerm, rightTerm) =>
+// insert auto casting for "narrowing primitive conversions"
+if (leftType != rightType) {
+  // leftType can not be casted to rightType automatically -> 
narrow
+  if (!leftType.shouldAutocastTo(rightType)) {
--- End diff --

Wouldn't this downcast `double` to `short` given an expression: `double + 
short`?


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Casting support for arithmetic op...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1634#discussion_r52924026
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/OperatorCodeGen.scala
 ---
@@ -18,20 +18,56 @@
 package org.apache.flink.api.table.codegen
 
 import org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO
-import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.common.typeinfo.{NumericTypeInfo, 
TypeInformation}
 import org.apache.flink.api.table.codegen.CodeGenUtils._
 
 object OperatorCodeGen {
 
-   def generateArithmeticOperator(
+  def generateArithmeticOperator(
   operator: String,
   nullCheck: Boolean,
   resultType: TypeInformation[_],
   left: GeneratedExpression,
   right: GeneratedExpression)
 : GeneratedExpression = {
-generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+// String arithmetic // TODO rework
+if (isString(left)) {
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
   (leftTerm, rightTerm) => s"$leftTerm $operator $rightTerm"
+  }
+}
+// Numeric arithmetic
+else if (isNumeric(left) && isNumeric(right)) {
+  val leftType = left.resultType.asInstanceOf[NumericTypeInfo[_]]
+  val rightType = right.resultType.asInstanceOf[NumericTypeInfo[_]]
+
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+  (leftTerm, rightTerm) =>
+// insert auto casting for "narrowing primitive conversions"
+if (leftType != rightType) {
+  // leftType can not be casted to rightType automatically -> 
narrow
+  if (!leftType.shouldAutocastTo(rightType)) {
--- End diff --

Wouldn't this downcast `double` to `short` given an expression: `double + 
short`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3387) Replace all ByteArrayOutputStream and DataOutputStream use in State Backends by DataOutputSerializer

[Subhobrata Dey (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147597#comment-15147597
 ] 

Subhobrata Dey commented on FLINK-3387:
---

Hello [~StephanEwen],

if nobody is working on this issue, I would like to work on it.

> Replace all ByteArrayOutputStream and DataOutputStream use in State Backends 
> by DataOutputSerializer
> 
>
> Key: FLINK-3387
> URL: https://issues.apache.org/jira/browse/FLINK-3387
> Project: Flink
>  Issue Type: Bug
>  Components: state backends
>Affects Versions: 1.0.0
>Reporter: Stephan Ewen
>
> Currently, the {{byte[]}} to object conversion and vice versa is always done 
> by creating new byte array streams and Data Input/Output streams. That is 
> pretty wasteful, we can optimize this with Flink's own classes:
>   - {{DataInputSerializer}}
>   - {{DataOutputSerializer}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Casting support for arithmetic op...

[fhueske]

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1634#discussion_r52921997
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/OperatorCodeGen.scala
 ---
@@ -18,20 +18,56 @@
 package org.apache.flink.api.table.codegen
 
 import org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO
-import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.common.typeinfo.{NumericTypeInfo, 
TypeInformation}
 import org.apache.flink.api.table.codegen.CodeGenUtils._
 
 object OperatorCodeGen {
 
-   def generateArithmeticOperator(
+  def generateArithmeticOperator(
   operator: String,
   nullCheck: Boolean,
   resultType: TypeInformation[_],
   left: GeneratedExpression,
   right: GeneratedExpression)
 : GeneratedExpression = {
-generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+// String arithmetic // TODO rework
+if (isString(left)) {
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
   (leftTerm, rightTerm) => s"$leftTerm $operator $rightTerm"
+  }
+}
+// Numeric arithmetic
+else if (isNumeric(left) && isNumeric(right)) {
+  val leftType = left.resultType.asInstanceOf[NumericTypeInfo[_]]
+  val rightType = right.resultType.asInstanceOf[NumericTypeInfo[_]]
+
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+  (leftTerm, rightTerm) =>
+// insert auto casting for "narrowing primitive conversions"
+if (leftType != rightType) {
--- End diff --

Shouldn't we compare and cast to the result type?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147585#comment-15147585
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user fhueske commented on a diff in the pull request:

https://github.com/apache/flink/pull/1634#discussion_r52921997
  
--- Diff: 
flink-libraries/flink-table/src/main/scala/org/apache/flink/api/table/codegen/OperatorCodeGen.scala
 ---
@@ -18,20 +18,56 @@
 package org.apache.flink.api.table.codegen
 
 import org.apache.flink.api.common.typeinfo.BasicTypeInfo.BOOLEAN_TYPE_INFO
-import org.apache.flink.api.common.typeinfo.TypeInformation
+import org.apache.flink.api.common.typeinfo.{NumericTypeInfo, 
TypeInformation}
 import org.apache.flink.api.table.codegen.CodeGenUtils._
 
 object OperatorCodeGen {
 
-   def generateArithmeticOperator(
+  def generateArithmeticOperator(
   operator: String,
   nullCheck: Boolean,
   resultType: TypeInformation[_],
   left: GeneratedExpression,
   right: GeneratedExpression)
 : GeneratedExpression = {
-generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+// String arithmetic // TODO rework
+if (isString(left)) {
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
   (leftTerm, rightTerm) => s"$leftTerm $operator $rightTerm"
+  }
+}
+// Numeric arithmetic
+else if (isNumeric(left) && isNumeric(right)) {
+  val leftType = left.resultType.asInstanceOf[NumericTypeInfo[_]]
+  val rightType = right.resultType.asInstanceOf[NumericTypeInfo[_]]
+
+  generateOperatorIfNotNull(nullCheck, resultType, left, right) {
+  (leftTerm, rightTerm) =>
+// insert auto casting for "narrowing primitive conversions"
+if (leftType != rightType) {
--- End diff --

Shouldn't we compare and cast to the result type?


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3332) Provide an exactly-once Cassandra connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147584#comment-15147584
 ] 

ASF GitHub Bot commented on FLINK-3332:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1640#issuecomment-184299327
  
I agree with @uce on this one. It seems to me a bit too much like marketing 
since, technically speaking, it gives you at least once guarantees.


> Provide an exactly-once Cassandra connector
> ---
>
> Key: FLINK-3332
> URL: https://issues.apache.org/jira/browse/FLINK-3332
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>
> With FLINK-3311, we are adding a Cassandra connector to Flink.
> It would be good to also provide an "exactly-once" C* connector.
> I would like to first discuss how we are going to implement this in Flink.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3332] Add Exactly-Once Cassandra connec...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1640#issuecomment-184299327
  
I agree with @uce on this one. It seems to me a bit too much like marketing 
since, technically speaking, it gives you at least once guarantees.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3332) Provide an exactly-once Cassandra connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147549#comment-15147549
 ] 

ASF GitHub Bot commented on FLINK-3332:
---

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1640#issuecomment-184284513
  
Great addition. But in my personal opinion it's not a good idea to name the 
sink exactly once when you say: *Note that a job failure while the data is 
being committed will cause duplicate data to be committed, but the chance of 
this happening is much smaller than for a naive At-Least-once implementation.*

What do others sink (I'll leave this typo... :smile:)?


> Provide an exactly-once Cassandra connector
> ---
>
> Key: FLINK-3332
> URL: https://issues.apache.org/jira/browse/FLINK-3332
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>
> With FLINK-3311, we are adding a Cassandra connector to Flink.
> It would be good to also provide an "exactly-once" C* connector.
> I would like to first discuss how we are going to implement this in Flink.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3332] Add Exactly-Once Cassandra connec...

[uce]

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1640#issuecomment-184284513
  
Great addition. But in my personal opinion it's not a good idea to name the 
sink exactly once when you say: *Note that a job failure while the data is 
being committed will cause duplicate data to be committed, but the chance of 
this happening is much smaller than for a naive At-Least-once implementation.*

What do others sink (I'll leave this typo... :smile:)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147548#comment-15147548
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user twalthr closed the pull request at:

https://github.com/apache/flink/pull/1624


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Translation from and to POJOs for...

[twalthr]

Github user twalthr closed the pull request at:

https://github.com/apache/flink/pull/1624


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3332) Provide an exactly-once Cassandra connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147547#comment-15147547
 ] 

ASF GitHub Bot commented on FLINK-3332:
---

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/1640

[FLINK-3332] Add Exactly-Once Cassandra connector

2nd attempt to add an Exactly-Once Cassandra Sink. I've addressed all 
issues brought up in the last PR bar one: This sink only works with Tuples. A 
simpler Cassandra Sink is in the works (see FLINK-3311) that will feature POJO 
support and i intend to copy the code-paste that code once it's done.

The Exactly-once guarantee is made by saving incoming records in the 
OperatorState, and only committing them into Cassandra when a checkpoint 
completes. Whether a operator committed data is saved using a new 
CheckpointCommitter object, that saves this information in an external and 
retry-persistent resource. Note that a job failure while the data is being 
committed will cause duplicate data to be committed, but the chance of this 
happening is much smaller than for a naive At-Least-once implementation.

The CassandraExactlyOnceSink is implemented as a custom operator to get 
access to the Statebackend. Values are committed with single inserts using a 
PreparedStatement that is supplied by the user, similiar to the Batch 
JDBC-Outputformat.

The Exactly-Once logic is completely contained in a GenericExactlyOnceSink 
class that can be used by virtually every sink, requiring no knowledge about 
the checkpointing mechamism.

The CassandraExactlyOnceSink and GenericExactlyOnceSink are covered by 
tests that use the OneInputStreamTaskHarness to generate a task environment, 
verifying that stored data is discarded when a state is restored; all data is 
being committed when a notify is missed; and of course that everything works 
when nothing goes wrong.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3332_cassandra

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1640.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1640


commit 4b91168dd60680502bf50f00e49b94d190f2d601
Author: zentol 
Date:   2016-02-10T13:14:18Z

[FLINK-3332] Add Exactly-Once Cassandra connector




> Provide an exactly-once Cassandra connector
> ---
>
> Key: FLINK-3332
> URL: https://issues.apache.org/jira/browse/FLINK-3332
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming Connectors
>Reporter: Robert Metzger
>Assignee: Chesnay Schepler
>
> With FLINK-3311, we are adding a Cassandra connector to Flink.
> It would be good to also provide an "exactly-once" C* connector.
> I would like to first discuss how we are going to implement this in Flink.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3332] Add Exactly-Once Cassandra connec...

[zentol]

GitHub user zentol opened a pull request:

https://github.com/apache/flink/pull/1640

[FLINK-3332] Add Exactly-Once Cassandra connector

2nd attempt to add an Exactly-Once Cassandra Sink. I've addressed all 
issues brought up in the last PR bar one: This sink only works with Tuples. A 
simpler Cassandra Sink is in the works (see FLINK-3311) that will feature POJO 
support and i intend to copy the code-paste that code once it's done.

The Exactly-once guarantee is made by saving incoming records in the 
OperatorState, and only committing them into Cassandra when a checkpoint 
completes. Whether a operator committed data is saved using a new 
CheckpointCommitter object, that saves this information in an external and 
retry-persistent resource. Note that a job failure while the data is being 
committed will cause duplicate data to be committed, but the chance of this 
happening is much smaller than for a naive At-Least-once implementation.

The CassandraExactlyOnceSink is implemented as a custom operator to get 
access to the Statebackend. Values are committed with single inserts using a 
PreparedStatement that is supplied by the user, similiar to the Batch 
JDBC-Outputformat.

The Exactly-Once logic is completely contained in a GenericExactlyOnceSink 
class that can be used by virtually every sink, requiring no knowledge about 
the checkpointing mechamism.

The CassandraExactlyOnceSink and GenericExactlyOnceSink are covered by 
tests that use the OneInputStreamTaskHarness to generate a task environment, 
verifying that stored data is discarded when a state is restored; all data is 
being committed when a notify is missed; and of course that everything works 
when nothing goes wrong.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zentol/flink 3332_cassandra

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1640.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1640


commit 4b91168dd60680502bf50f00e49b94d190f2d601
Author: zentol 
Date:   2016-02-10T13:14:18Z

[FLINK-3332] Add Exactly-Once Cassandra connector




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3226] Translation from and to POJOs for...

[vasia]

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1624#issuecomment-184278657
  
You can close the PR @twalthr. It's merged (but doesn't automatically 
close).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147541#comment-15147541
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user vasia commented on the pull request:

https://github.com/apache/flink/pull/1624#issuecomment-184278657
  
You can close the PR @twalthr. It's merged (but doesn't automatically 
close).


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3377) Remove final flag from ResultPartitionWriter class

[Chesnay Schepler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler closed FLINK-3377.
---
Resolution: Won't Fix

> Remove final flag from ResultPartitionWriter class
> --
>
> Key: FLINK-3377
> URL: https://issues.apache.org/jira/browse/FLINK-3377
> Project: Flink
>  Issue Type: Wish
>  Components: Distributed Runtime
>Affects Versions: 0.10.1
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Trivial
> Fix For: 1.0.0
>
>
> The final flag on the 
> org.apache.flink.runtime.io.network.api.writer.ResultPartitionWriter class is 
> causing issues for me.
> The flag requires me to run a test I'm working on with a 
> @RunWith(PowerMockRunner.class) annotation so that i can use 
> @PrepareForTest({ResultPartitionWriter.class}).
> But it breaks my TemporaryFolder annotated with @ClassRule. (apart from that 
> there also was a classloader issue, but i could resolve that)
> To me these seem like unnecessary problems, as such i propose removing the 
> final flag.
> The 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147533#comment-15147533
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/1639

[FLINK-3226] Translation of explicit casting

This PR implements explicit casting from and to all supported types so far.

@fhueske @vasia can someone review?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink ExplicitCasting

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1639






> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Translation of explicit casting

[twalthr]

GitHub user twalthr opened a pull request:

https://github.com/apache/flink/pull/1639

[FLINK-3226] Translation of explicit casting

This PR implements explicit casting from and to all supported types so far.

@fhueske @vasia can someone review?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/twalthr/flink ExplicitCasting

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1639.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1639






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2111) Add "stop" signal to cleanly shutdown streaming jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147525#comment-15147525
 ] 

ASF GitHub Bot commented on FLINK-2111:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/750#issuecomment-184273250
  
Thanks for your great work and patience @mjsax. I know this PR lingered 
around for far too long but I think we merged now a really good first version 
:-)

I also wanted to complete the 300 interactions on this PR thread ;-)


> Add "stop" signal to cleanly shutdown streaming jobs
> 
>
> Key: FLINK-2111
> URL: https://issues.apache.org/jira/browse/FLINK-2111
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime, JobManager, Local Runtime, 
> Streaming, TaskManager, Webfrontend
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
>
> Currently, streaming jobs can only be stopped using "cancel" command, what is 
> a "hard" stop with no clean shutdown.
> The new introduced "stop" signal, will only affect streaming source tasks 
> such that the sources can stop emitting data and shutdown cleanly, resulting 
> in a clean shutdown of the whole streaming job.
> This feature is a pre-requirment for 
> https://issues.apache.org/jira/browse/FLINK-1929



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147504#comment-15147504
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user zentol commented on the pull request:

https://github.com/apache/flink/pull/1583#issuecomment-184259383
  
Made a few remarks, but didn't find any issue that isn't a minor one. Good 
job!


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[zentol]

Github user zentol commented on the pull request:

https://github.com/apache/flink/pull/1583#issuecomment-184259383
  
Made a few remarks, but didn't find any issue that isn't a minor one. Good 
job!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147502#comment-15147502
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52912765
  
--- Diff: 
flink-yarn-tests/src/main/java/org/apache/flink/yarn/YarnTestBase.java ---
@@ -112,7 +112,7 @@
 * lib/ folder of the flink distribution.
 */
protected static File flinkLibFolder;
-
+   
--- End diff --

only change in this file is formatting


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52912765
  
--- Diff: 
flink-yarn-tests/src/main/java/org/apache/flink/yarn/YarnTestBase.java ---
@@ -112,7 +112,7 @@
 * lib/ folder of the flink distribution.
 */
protected static File flinkLibFolder;
-
+   
--- End diff --

only change in this file is formatting


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147499#comment-15147499
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52912585
  
--- Diff: 
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGeneratorTest.java
 ---
@@ -75,16 +75,22 @@ public void testExecutionConfigSerialization() throws 
IOException, ClassNotFound
config.disableSysoutLogging();
}
config.setParallelism(dop);
-   
+
JobGraph jobGraph = compiler.createJobGraph("test");
-   
+
+   final String exec_config_key = "runtime.config";
--- End diff --

variable doesn't follow naming conventions


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52912585
  
--- Diff: 
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGeneratorTest.java
 ---
@@ -75,16 +75,22 @@ public void testExecutionConfigSerialization() throws 
IOException, ClassNotFound
config.disableSysoutLogging();
}
config.setParallelism(dop);
-   
+
JobGraph jobGraph = compiler.createJobGraph("test");
-   
+
+   final String exec_config_key = "runtime.config";
--- End diff --

variable doesn't follow naming conventions


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147500#comment-15147500
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52912601
  
--- Diff: 
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGeneratorTest.java
 ---
@@ -75,16 +75,22 @@ public void testExecutionConfigSerialization() throws 
IOException, ClassNotFound
config.disableSysoutLogging();
}
config.setParallelism(dop);
-   
+
JobGraph jobGraph = compiler.createJobGraph("test");
-   
+
+   final String exec_config_key = "runtime.config";
+
+   
InstantiationUtil.writeObjectToConfig(jobGraph.getExecutionConfig(),
+   jobGraph.getJobConfiguration(),
+   exec_config_key);
+
ExecutionConfig executionConfig = 
InstantiationUtil.readObjectFromConfig(
jobGraph.getJobConfiguration(),
-   ExecutionConfig.CONFIG_KEY,
+   exec_config_key,
Thread.currentThread().getContextClassLoader());
-   
+
assertNotNull(executionConfig);
-   
--- End diff --

this class contains a relatively high number of pure formatting changes.


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52912601
  
--- Diff: 
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGeneratorTest.java
 ---
@@ -75,16 +75,22 @@ public void testExecutionConfigSerialization() throws 
IOException, ClassNotFound
config.disableSysoutLogging();
}
config.setParallelism(dop);
-   
+
JobGraph jobGraph = compiler.createJobGraph("test");
-   
+
+   final String exec_config_key = "runtime.config";
+
+   
InstantiationUtil.writeObjectToConfig(jobGraph.getExecutionConfig(),
+   jobGraph.getJobConfiguration(),
+   exec_config_key);
+
ExecutionConfig executionConfig = 
InstantiationUtil.readObjectFromConfig(
jobGraph.getJobConfiguration(),
-   ExecutionConfig.CONFIG_KEY,
+   exec_config_key,
Thread.currentThread().getContextClassLoader());
-   
+
assertNotNull(executionConfig);
-   
--- End diff --

this class contains a relatively high number of pure formatting changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2111) Add "stop" signal to cleanly shutdown streaming jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147487#comment-15147487
 ] 

ASF GitHub Bot commented on FLINK-2111:
---

Github user mjsax commented on the pull request:

https://github.com/apache/flink/pull/750#issuecomment-184254082
  
Thanks for merging! Very happy about it!


> Add "stop" signal to cleanly shutdown streaming jobs
> 
>
> Key: FLINK-2111
> URL: https://issues.apache.org/jira/browse/FLINK-2111
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime, JobManager, Local Runtime, 
> Streaming, TaskManager, Webfrontend
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
>
> Currently, streaming jobs can only be stopped using "cancel" command, what is 
> a "hard" stop with no clean shutdown.
> The new introduced "stop" signal, will only affect streaming source tasks 
> such that the sources can stop emitting data and shutdown cleanly, resulting 
> in a clean shutdown of the whole streaming job.
> This feature is a pre-requirment for 
> https://issues.apache.org/jira/browse/FLINK-1929



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147494#comment-15147494
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52912014
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -1051,7 +1052,8 @@ class JobManager(
   }
   catch {
 case t: Throwable =>
-  log.error(s"Failed to submit job $jobId ($jobName)", t)
+  val message = t.getMessage
+  log.error(s"Failed to submit job $jobId ($jobName): $message", t)
--- End diff --

does the exception's message not show up in the logs without this change?


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52912014
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/JobManager.scala
 ---
@@ -1051,7 +1052,8 @@ class JobManager(
   }
   catch {
 case t: Throwable =>
-  log.error(s"Failed to submit job $jobId ($jobName)", t)
+  val message = t.getMessage
+  log.error(s"Failed to submit job $jobId ($jobName): $message", t)
--- End diff --

does the exception's message not show up in the logs without this change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...

[mjsax]

Github user mjsax commented on the pull request:

https://github.com/apache/flink/pull/750#issuecomment-184254082
  
Thanks for merging! Very happy about it!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147486#comment-15147486
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52911198
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobGraph.java ---
@@ -225,22 +233,12 @@ public int getNumberOfExecutionRetries() {
 * @return The delay of time in milliseconds the system will try to 
re-execute failed tasks.
 */
public long getExecutionRetryDelay() {
-   return executionRetryDelay;
-   }
-
-   /**
-* Sets the delay that failed tasks are re-executed. A value of zero
-* effectively disables fault tolerance. A value of {@code -1} 
indicates that the system
-* default value (as defined in the configuration) should be used.
-*
-* @param executionRetryDelay The delay of time the system will wait to 
re-execute failed tasks.
-*/
-   public void setExecutionRetryDelay(long executionRetryDelay){
-   if (executionRetryDelay < -1) {
+   long retryDelay = executionConfig.getExecutionRetryDelay();
+   if (retryDelay < -1) {
--- End diff --

same as above


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52911198
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobGraph.java ---
@@ -225,22 +233,12 @@ public int getNumberOfExecutionRetries() {
 * @return The delay of time in milliseconds the system will try to 
re-execute failed tasks.
 */
public long getExecutionRetryDelay() {
-   return executionRetryDelay;
-   }
-
-   /**
-* Sets the delay that failed tasks are re-executed. A value of zero
-* effectively disables fault tolerance. A value of {@code -1} 
indicates that the system
-* default value (as defined in the configuration) should be used.
-*
-* @param executionRetryDelay The delay of time the system will wait to 
re-execute failed tasks.
-*/
-   public void setExecutionRetryDelay(long executionRetryDelay){
-   if (executionRetryDelay < -1) {
+   long retryDelay = executionConfig.getExecutionRetryDelay();
+   if (retryDelay < -1) {
--- End diff --

same as above


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147482#comment-15147482
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52911077
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobGraph.java ---
@@ -215,7 +218,12 @@ public void setNumberOfExecutionRetries(int 
numberOfExecutionRetries) {
 * @return The number of times the system will try to re-execute failed 
tasks.
 */
public int getNumberOfExecutionRetries() {
-   return numExecutionRetries;
+   int retries = executionConfig.getNumberOfExecutionRetries();
+   if (retries < -1) {
--- End diff --

this check shouldn't be necessary, since setNumberOfExecutionRetries 
already checks for it.


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (FLINK-2111) Add "stop" signal to cleanly shutdown streaming jobs

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann resolved FLINK-2111.
--
Resolution: Fixed

Added via f60f8fbc69cd1c36a3a99b08b0e76100b9032fae

> Add "stop" signal to cleanly shutdown streaming jobs
> 
>
> Key: FLINK-2111
> URL: https://issues.apache.org/jira/browse/FLINK-2111
> Project: Flink
>  Issue Type: Improvement
>  Components: Distributed Runtime, JobManager, Local Runtime, 
> Streaming, TaskManager, Webfrontend
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Minor
>
> Currently, streaming jobs can only be stopped using "cancel" command, what is 
> a "hard" stop with no clean shutdown.
> The new introduced "stop" signal, will only affect streaming source tasks 
> such that the sources can stop emitting data and shutdown cleanly, resulting 
> in a clean shutdown of the whole streaming job.
> This feature is a pre-requirment for 
> https://issues.apache.org/jira/browse/FLINK-1929



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-2111] Add "stop" signal to cleanly shut...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/750


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52911077
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobGraph.java ---
@@ -215,7 +218,12 @@ public void setNumberOfExecutionRetries(int 
numberOfExecutionRetries) {
 * @return The number of times the system will try to re-execute failed 
tasks.
 */
public int getNumberOfExecutionRetries() {
-   return numExecutionRetries;
+   int retries = executionConfig.getNumberOfExecutionRetries();
+   if (retries < -1) {
--- End diff --

this check shouldn't be necessary, since setNumberOfExecutionRetries 
already checks for it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147472#comment-15147472
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user zentol commented on the pull request:

https://github.com/apache/flink/pull/1583#issuecomment-184252247
  
we could probably heavily reduce the diff by providing constructors 
with/without the executionconfig argument. (if none is given, we just 
instantiate a new one)


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2922) Add Queryable Window Operator

[Gyula Fora (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147471#comment-15147471
 ] 

Gyula Fora commented on FLINK-2922:
---

I think it would be also nice to extend the same general idea to all stateful 
operators, so that users would have a straightforward way to query them without 
having to hack that into the user functions.

An idea would be that for each state (statedescriptor) the user would also 
define a query function that tells the system what should be the query output. 
So if your state is a giant list you might only one to query the last element 
anyways.

> Add Queryable Window Operator
> -
>
> Key: FLINK-2922
> URL: https://issues.apache.org/jira/browse/FLINK-2922
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>  Labels: requires-design-doc
> Attachments: FLINK-2922.pdf
>
>
> The idea is to provide a window operator that allows to query the current 
> window result at any time without discarding the current result.
> For example, a user might have an aggregation window operation with tumbling 
> windows of 1 hour. Now, at any time they might be interested in the current 
> aggregated value for the currently in-flight hour window.
> The idea is to make the operator a two input operator where normal elements 
> arrive on input one while queries arrive on input two. The query stream must 
> be keyed by the same key as the input stream. If an input arrives for a key 
> the current value for that key is emitted along with the query element so 
> that the user can map the result to the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[zentol]

Github user zentol commented on the pull request:

https://github.com/apache/flink/pull/1583#issuecomment-184252247
  
we could probably heavily reduce the diff by providing constructors 
with/without the executionconfig argument. (if none is given, we just 
instantiate a new one)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3399) Count with timeout trigger

[Shikhar Bhushan (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147467#comment-15147467
 ] 

Shikhar Bhushan commented on FLINK-3399:


Compound triggers would be pretty useful, this could probably be composed out 
of the existing {{CountTrigger}} + {{ContinuousProcessingTimeTrigger}} (though 
slight difference in semantics with the latter using rounded time intervals vs 
this impl using a static delta from current time when registering the timer). 
I'm curious if you have thoughts already about the internal changes required.

> Count with timeout trigger
> --
>
> Key: FLINK-3399
> URL: https://issues.apache.org/jira/browse/FLINK-3399
> Project: Flink
>  Issue Type: Improvement
>Reporter: Shikhar Bhushan
>Priority: Minor
>
> I created an implementation of a trigger that I'd like to contribute, 
> https://gist.github.com/shikhar/2cb9f1b792be31b7c16e
> An example application - if a sink function operates more efficiently if it 
> is writing in a batched fashion, then the windowing mechanism + this trigger 
> can be used. Count to have an upper bound on batch size & better control on 
> memory usage, and timeout to ensure timeliness of the outputs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3396) Job submission Savepoint restore logic flawed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147465#comment-15147465
 ] 

ASF GitHub Bot commented on FLINK-3396:
---

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1633#issuecomment-184250973
  
@tillrohrmann, could you have a look at this change?


> Job submission Savepoint restore logic flawed
> -
>
> Key: FLINK-3396
> URL: https://issues.apache.org/jira/browse/FLINK-3396
> Project: Flink
>  Issue Type: Bug
>Reporter: Ufuk Celebi
>Assignee: Ufuk Celebi
> Fix For: 1.0.0
>
>
> When savepoint restoring fails, the thrown Exception fails the execution 
> graph, but the client is not informed about the failure.
> The expected behaviour is that the submission should be acked with success or 
> failure in any case. With savepoint restore failures, the ack message will be 
> skipped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3396] [runtime] Fail job submission aft...

[uce]

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1633#issuecomment-184250973
  
@tillrohrmann, could you have a look at this change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147464#comment-15147464
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52910257
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/FlinkYarnSessionCli.java ---
@@ -393,7 +393,7 @@ public int run(String[] args) {
printUsage();
return 1;
}
-
+   
--- End diff --

only change made in this file is formatting.


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3390) Savepoint resume is not retried

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147463#comment-15147463
 ] 

ASF GitHub Bot commented on FLINK-3390:
---

Github user uce closed the pull request at:

https://github.com/apache/flink/pull/1637


> Savepoint resume is not retried
> ---
>
> Key: FLINK-3390
> URL: https://issues.apache.org/jira/browse/FLINK-3390
> Project: Flink
>  Issue Type: Bug
>Reporter: Sebastian Klemke
> Attachments: savepoint_restore_violates_exactly_once.patch
>
>
> When during resuming from a savepoint, restoreState fails for a task node, 
> job is retried but without retrying resume from savepoint state. This leads 
> to the job being restarted with empty state. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[zentol]

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/1583#discussion_r52910257
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/FlinkYarnSessionCli.java ---
@@ -393,7 +393,7 @@ public int run(String[] args) {
printUsage();
return 1;
}
-
+   
--- End diff --

only change made in this file is formatting.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3390] [runtime, tests] Reset Execution ...

[uce]

Github user uce closed the pull request at:

https://github.com/apache/flink/pull/1637


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3390) Savepoint resume is not retried

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147462#comment-15147462
 ] 

ASF GitHub Bot commented on FLINK-3390:
---

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1637#issuecomment-184250578
  
I think there is a better way to do this, which should also cover 
[FLINK-3397](https://issues.apache.org/jira/browse/FLINK-3397).


> Savepoint resume is not retried
> ---
>
> Key: FLINK-3390
> URL: https://issues.apache.org/jira/browse/FLINK-3390
> Project: Flink
>  Issue Type: Bug
>Reporter: Sebastian Klemke
> Attachments: savepoint_restore_violates_exactly_once.patch
>
>
> When during resuming from a savepoint, restoreState fails for a task node, 
> job is retried but without retrying resume from savepoint state. This leads 
> to the job being restarted with empty state. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3390] [runtime, tests] Reset Execution ...

[uce]

Github user uce commented on the pull request:

https://github.com/apache/flink/pull/1637#issuecomment-184250578
  
I think there is a better way to do this, which should also cover 
[FLINK-3397](https://issues.apache.org/jira/browse/FLINK-3397).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3187) Decouple restart strategy from ExecutionGraph

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147444#comment-15147444
 ] 

ASF GitHub Bot commented on FLINK-3187:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1470


> Decouple restart strategy from ExecutionGraph
> -
>
> Key: FLINK-3187
> URL: https://issues.apache.org/jira/browse/FLINK-3187
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> Currently, the {{ExecutionGraph}} supports the following restart logic: 
> Whenever a failure occurs and the number of restart attempts aren't depleted, 
> wait for a fixed amount of time and then try to restart. This behaviour can 
> be controlled by the configuration parameters {{execution-retries.default}} 
> and {{execution-retries.delay}}.
> I propose to decouple the restart logic from the {{ExecutionGraph}} a bit by 
> introducing a strategy pattern. That way it would not only allow us to define 
> a job specific restart behaviour but also to implement different restart 
> strategies. Conceivable strategies could be: Fixed timeout restart, 
> exponential backoff restart, partial topology restarts, etc.
> This change is a preliminary step towards having a restart strategy which 
> will scale the parallelism of a job down in case that not enough slots are 
> available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (FLINK-3187) Decouple restart strategy from ExecutionGraph

[Till Rohrmann (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Till Rohrmann closed FLINK-3187.

Resolution: Fixed

Fixed via 5eae47f5d381cd55729660f62c714b52d28413ed

> Decouple restart strategy from ExecutionGraph
> -
>
> Key: FLINK-3187
> URL: https://issues.apache.org/jira/browse/FLINK-3187
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> Currently, the {{ExecutionGraph}} supports the following restart logic: 
> Whenever a failure occurs and the number of restart attempts aren't depleted, 
> wait for a fixed amount of time and then try to restart. This behaviour can 
> be controlled by the configuration parameters {{execution-retries.default}} 
> and {{execution-retries.delay}}.
> I propose to decouple the restart logic from the {{ExecutionGraph}} a bit by 
> introducing a strategy pattern. That way it would not only allow us to define 
> a job specific restart behaviour but also to implement different restart 
> strategies. Conceivable strategies could be: Fixed timeout restart, 
> exponential backoff restart, partial topology restarts, etc.
> This change is a preliminary step towards having a restart strategy which 
> will scale the parallelism of a job down in case that not enough slots are 
> available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3187] Introduce RestartStrategy to deco...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1470


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3187] Introduce RestartStrategy to deco...

[tillrohrmann]

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1470#issuecomment-184239679
  
Merging this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3187) Decouple restart strategy from ExecutionGraph

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147442#comment-15147442
 ] 

ASF GitHub Bot commented on FLINK-3187:
---

Github user tillrohrmann commented on the pull request:

https://github.com/apache/flink/pull/1470#issuecomment-184239679
  
Merging this.


> Decouple restart strategy from ExecutionGraph
> -
>
> Key: FLINK-3187
> URL: https://issues.apache.org/jira/browse/FLINK-3187
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.0.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
>
> Currently, the {{ExecutionGraph}} supports the following restart logic: 
> Whenever a failure occurs and the number of restart attempts aren't depleted, 
> wait for a fixed amount of time and then try to restart. This behaviour can 
> be controlled by the configuration parameters {{execution-retries.default}} 
> and {{execution-retries.delay}}.
> I propose to decouple the restart logic from the {{ExecutionGraph}} a bit by 
> introducing a strategy pattern. That way it would not only allow us to define 
> a job specific restart behaviour but also to implement different restart 
> strategies. Conceivable strategies could be: Fixed timeout restart, 
> exponential backoff restart, partial topology restarts, etc.
> This change is a preliminary step towards having a restart strategy which 
> will scale the parallelism of a job down in case that not enough slots are 
> available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3327) Attach the ExecutionConfig to the JobGraph and make it accessible to the AbstractInvocable.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147431#comment-15147431
 ] 

ASF GitHub Bot commented on FLINK-3327:
---

Github user kl0u commented on the pull request:

https://github.com/apache/flink/pull/1583#issuecomment-184236773
  
Hello! Just rebased to the new master. Please review.


> Attach the ExecutionConfig to the JobGraph and make it accessible to the 
> AbstractInvocable.
> ---
>
> Key: FLINK-3327
> URL: https://issues.apache.org/jira/browse/FLINK-3327
> Project: Flink
>  Issue Type: Sub-task
>  Components: TaskManager
>Reporter: Klou
>Assignee: Klou
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-2523) Make task canceling interrupt interval configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147433#comment-15147433
 ] 

ASF GitHub Bot commented on FLINK-2523:
---

Github user kl0u commented on the pull request:

https://github.com/apache/flink/pull/1612#issuecomment-184236866
  
Hello! Just rebased to the new master. Please review.


> Make task canceling interrupt interval configurable
> ---
>
> Key: FLINK-2523
> URL: https://issues.apache.org/jira/browse/FLINK-2523
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Affects Versions: 0.10.0
>Reporter: Stephan Ewen
>Assignee: Klou
> Fix For: 1.0.0
>
>
> When a task is canceled, the cancellation calls periodically "interrupt()" on 
> the task thread, if the task thread does not cancel with a certain time.
> Currently, this value is hard coded to 10 seconds. We should make that time 
> configurable.
> Until then, I would like to increase the value to 30 seconds, as many tasks 
> (here I am observing it for Kafka consumers) can take longer then 10 seconds 
> for proper cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: FLINK-2523: Makes the task cancellation interv...

[kl0u]

Github user kl0u commented on the pull request:

https://github.com/apache/flink/pull/1612#issuecomment-184236866
  
Hello! Just rebased to the new master. Please review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: FLINK-3327: ExecutionConfig to JobGraph.

[kl0u]

Github user kl0u commented on the pull request:

https://github.com/apache/flink/pull/1583#issuecomment-184236773
  
Hello! Just rebased to the new master. Please review.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3354] Determine correct size for RocksD...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1638


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3354) RocksDB should compute checkpoint size based on backup file size

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147422#comment-15147422
 ] 

ASF GitHub Bot commented on FLINK-3354:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1638


> RocksDB should compute checkpoint size based on backup file size
> 
>
> Key: FLINK-3354
> URL: https://issues.apache.org/jira/browse/FLINK-3354
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Gyula Fora
>Priority: Critical
>
> Currently the RocksDB backend returns 0 for state size, the actual state size 
> could be computed using:
> fs.getContentSummary(path).getLength();



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3034][Streaming Connectors] Redis Sink ...

[mjsax]

Github user mjsax commented on the pull request:

https://github.com/apache/flink/pull/1580#issuecomment-184216894
  
Any progress here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3034) Redis SInk Connector

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147375#comment-15147375
 ] 

ASF GitHub Bot commented on FLINK-3034:
---

Github user mjsax commented on the pull request:

https://github.com/apache/flink/pull/1580#issuecomment-184216894
  
Any progress here?


> Redis SInk Connector
> 
>
> Key: FLINK-3034
> URL: https://issues.apache.org/jira/browse/FLINK-3034
> Project: Flink
>  Issue Type: New Feature
>  Components: Streaming Connectors
>Reporter: Matthias J. Sax
>Assignee: Subhankar Biswas
>Priority: Minor
>
> Flink does not provide a sink connector for Redis.
> See FLINK-3033



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3226) Translate optimized logical Table API plans into physical plans representing DataSet programs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147343#comment-15147343
 ] 

ASF GitHub Bot commented on FLINK-3226:
---

Github user vasia closed the pull request at:

https://github.com/apache/flink/pull/1632


> Translate optimized logical Table API plans into physical plans representing 
> DataSet programs
> -
>
> Key: FLINK-3226
> URL: https://issues.apache.org/jira/browse/FLINK-3226
> Project: Flink
>  Issue Type: Sub-task
>  Components: Table API
>Reporter: Fabian Hueske
>Assignee: Chengxiang Li
>
> This issue is about translating an (optimized) logical Table API (see 
> FLINK-3225) query plan into a physical plan. The physical plan is a 1-to-1 
> representation of the DataSet program that will be executed. This means:
> - Each Flink RelNode refers to exactly one Flink DataSet or DataStream 
> operator.
> - All (join and grouping) keys of Flink operators are correctly specified.
> - The expressions which are to be executed in user-code are identified.
> - All fields are referenced with their physical execution-time index.
> - Flink type information is available.
> - Optional: Add physical execution hints for joins
> The translation should be the final part of Calcite's optimization process.
> For this task we need to:
> - implement a set of Flink DataSet RelNodes. Each RelNode corresponds to one 
> Flink DataSet operator (Map, Reduce, Join, ...). The RelNodes must hold all 
> relevant operator information (keys, user-code expression, strategy hints, 
> parallelism).
> - implement rules to translate optimized Calcite RelNodes into Flink 
> RelNodes. We start with a straight-forward mapping and later add rules that 
> merge several relational operators into a single Flink operator, e.g., merge 
> a join followed by a filter. Timo implemented some rules for the first SQL 
> implementation which can be used as a starting point.
> - Integrate the translation rules into the Calcite optimization process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3226] Translate logical joins to physic...

[vasia]

Github user vasia closed the pull request at:

https://github.com/apache/flink/pull/1632


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-2719) ProcessFailureStreamingRecoveryITCase>AbstractProcessFailureRecoveryTest.testTaskManagerProcessFailure failed on Travis

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147338#comment-15147338
 ] 

Till Rohrmann commented on FLINK-2719:
--

This is not another instance of the reported problem here. The test case which 
is failing for you is {{TaskManagerProcessFailureBatchRecoveryITCase}} and not 
the {{ProcessFailureStreamingRecoveryITCase}}.

> ProcessFailureStreamingRecoveryITCase>AbstractProcessFailureRecoveryTest.testTaskManagerProcessFailure
>  failed on Travis
> ---
>
> Key: FLINK-2719
> URL: https://issues.apache.org/jira/browse/FLINK-2719
> Project: Flink
>  Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Till Rohrmann
>Priority: Critical
>  Labels: test-stability
> Fix For: 1.0.0
>
>
> The test case 
> {{ProcessFailureStreamingRecoveryITCase>AbstractProcessFailureRecoveryTest.testTaskManagerProcessFailure}}
>  failed on travis with the following exception
> {code}
> Failed tests: 
>   
> ProcessFailureStreamingRecoveryITCase>AbstractProcessFailureRecoveryTest.testTaskManagerProcessFailure:211
>  The program encountered a FileNotFoundException : File does not exist: 
> /tmp/cbe4a9aa-3b9a-455d-b7b4-a9abf7c2d9d5/03801d139e79e850249e386ffd89c13ca727bcd8
> {code}
> Most likely, this is a problem of the Travis infrastructure that we could not 
> create the temp file. Maybe we should harden this.
> https://s3.amazonaws.com/archive.travis-ci.org/jobs/81028955/log.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3400) RocksDB Backend does not work when not in Flink lib folder

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147323#comment-15147323
 ] 

Aljoscha Krettek commented on FLINK-3400:
-

I see two possibilities for solving this:

 - Move the utilities to a package that is always in the classpath anyways. 
Candidates would be {{flink-core}}, {{flink-runtime}} or 
{{flink-streaming-java}}. I would gravitate towards {{flink-streaming-java}} 
here.
 - Figure out how to get the path to the user-code jar/jars and add that to the 
classpath of the started external process. Are the user-code jars always 
available in a directory on the machine where a Task is running.

I think I'll go with option 1) move to {{flink-streaming-java}}. But what do 
others think?

> RocksDB Backend does not work when not in Flink lib folder
> --
>
> Key: FLINK-3400
> URL: https://issues.apache.org/jira/browse/FLINK-3400
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.0.0
>
>
> The problem is that the external process runner cannot load the 
> {{HDFSCopyFromLocal}} and {{HDFSCopyToLocal}} classes when the rocksdb 
> backend jar is not in the Flink lib folder. If rocksdb is only in the 
> user-code jar the code cannot be loaded when trying to start the external 
> copy runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3243) Fix Interplay of TimeCharacteristic and Time Windows

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147322#comment-15147322
 ] 

ASF GitHub Bot commented on FLINK-3243:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1513#issuecomment-184196790
  
I rebased on top of current master.

Btw, still still has the issue that the time characteristic only applies to 
window operations that are created after setting the time characteristic. 
Changing the time characteristic does not affect already created window 
operations.


> Fix Interplay of TimeCharacteristic and Time Windows
> 
>
> Key: FLINK-3243
> URL: https://issues.apache.org/jira/browse/FLINK-3243
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
>
> As per the discussion on the Dev ML: 
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Time-Behavior-in-Streaming-Jobs-Event-time-processing-time-td9616.html.
> The discussion seems to have converged on option 2):
> - Add dedicated WindowAssigners for processing time and event time
> - {{timeWindow()}} and {{timeWindowAll()}} respect the set 
> {{TimeCharacteristic}}. 
> This will make the easy stuff easy, i.e. using time windows and quickly 
> switching the time characteristic. Users will then have the flexibility to 
> mix different kinds of window assigners in their job.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3243] Fix Interplay of TimeCharacterist...

[aljoscha]

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1513#issuecomment-184196790
  
I rebased on top of current master.

Btw, still still has the issue that the time characteristic only applies to 
window operations that are created after setting the time characteristic. 
Changing the time characteristic does not affect already created window 
operations.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3354) RocksDB should compute checkpoint size based on backup file size

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147321#comment-15147321
 ] 

ASF GitHub Bot commented on FLINK-3354:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1638#issuecomment-184196345
  
thanks


> RocksDB should compute checkpoint size based on backup file size
> 
>
> Key: FLINK-3354
> URL: https://issues.apache.org/jira/browse/FLINK-3354
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Gyula Fora
>Priority: Critical
>
> Currently the RocksDB backend returns 0 for state size, the actual state size 
> could be computed using:
> fs.getContentSummary(path).getLength();



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3354] Determine correct size for RocksD...

[aljoscha]

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1638#issuecomment-184196345
  
thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (FLINK-3160) Aggregate operator statistics by TaskManager

[Ufuk Celebi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ufuk Celebi closed FLINK-3160.
--
   Resolution: Implemented
Fix Version/s: 1.0.0

Implemented in 73bc35f.

> Aggregate operator statistics by TaskManager
> 
>
> Key: FLINK-3160
> URL: https://issues.apache.org/jira/browse/FLINK-3160
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.0.0
>
>
> The web client job info page presents a table of the following per task 
> statistics: start time, end time, duration, bytes received, records received, 
> bytes sent, records sent, attempt, host, status.
> Flink supports clusters with thousands of slots and a job setting a high 
> parallelism renders this job info page unwieldy and difficult to analyze in 
> real-time.
> It would be helpful to optionally or automatically aggregate statistics by 
> TaskManager. These rows could then be expanded to reveal the current per task 
> statistics.
> Start time, end time, duration, and attempt are not applicable to a 
> TaskManager since new tasks for repeated attempts may be started. Bytes 
> received, records received, bytes sent, and records sent are summed. Any 
> throughput metrics can be averaged over the total task time or time window. 
> Status could reference the number of running tasks on the TaskManager or an 
> idle state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3160) Aggregate operator statistics by TaskManager

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147320#comment-15147320
 ] 

ASF GitHub Bot commented on FLINK-3160:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1564


> Aggregate operator statistics by TaskManager
> 
>
> Key: FLINK-3160
> URL: https://issues.apache.org/jira/browse/FLINK-3160
> Project: Flink
>  Issue Type: Improvement
>  Components: Webfrontend
>Affects Versions: 1.0.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
> Fix For: 1.0.0
>
>
> The web client job info page presents a table of the following per task 
> statistics: start time, end time, duration, bytes received, records received, 
> bytes sent, records sent, attempt, host, status.
> Flink supports clusters with thousands of slots and a job setting a high 
> parallelism renders this job info page unwieldy and difficult to analyze in 
> real-time.
> It would be helpful to optionally or automatically aggregate statistics by 
> TaskManager. These rows could then be expanded to reveal the current per task 
> statistics.
> Start time, end time, duration, and attempt are not applicable to a 
> TaskManager since new tasks for repeated attempts may be started. Bytes 
> received, records received, bytes sent, and records sent are summed. Any 
> throughput metrics can be averaged over the total task time or time window. 
> Status could reference the number of running tasks on the TaskManager or an 
> idle state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3160] [web-dashboard] Aggregate operato...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/1564


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request: [FLINK-3354] Determine correct size for RocksD...

[gyfora]

Github user gyfora commented on the pull request:

https://github.com/apache/flink/pull/1638#issuecomment-184196183
  
Sure


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3354) RocksDB should compute checkpoint size based on backup file size

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147315#comment-15147315
 ] 

ASF GitHub Bot commented on FLINK-3354:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1638#issuecomment-184195688
  
Ok, could you please go ahead and merge it if no one else objects. :+1: 


> RocksDB should compute checkpoint size based on backup file size
> 
>
> Key: FLINK-3354
> URL: https://issues.apache.org/jira/browse/FLINK-3354
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Gyula Fora
>Priority: Critical
>
> Currently the RocksDB backend returns 0 for state size, the actual state size 
> could be computed using:
> fs.getContentSummary(path).getLength();



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3354] Determine correct size for RocksD...

[aljoscha]

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1638#issuecomment-184195688
  
Ok, could you please go ahead and merge it if no one else objects. :+1: 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3354) RocksDB should compute checkpoint size based on backup file size

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147314#comment-15147314
 ] 

ASF GitHub Bot commented on FLINK-3354:
---

Github user gyfora commented on the pull request:

https://github.com/apache/flink/pull/1638#issuecomment-184195205
  
It counts the size without HDFS replication I think that is better.


> RocksDB should compute checkpoint size based on backup file size
> 
>
> Key: FLINK-3354
> URL: https://issues.apache.org/jira/browse/FLINK-3354
> Project: Flink
>  Issue Type: Improvement
>  Components: Streaming
>Reporter: Gyula Fora
>Priority: Critical
>
> Currently the RocksDB backend returns 0 for state size, the actual state size 
> could be computed using:
> fs.getContentSummary(path).getLength();



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request: [FLINK-3354] Determine correct size for RocksD...

[gyfora]

Github user gyfora commented on the pull request:

https://github.com/apache/flink/pull/1638#issuecomment-184195205
  
It counts the size without HDFS replication I think that is better.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---