[GitHub] flink issue #2060: [FLINK-3921] StringParser encoding

[rekhajoshm]

Github user rekhajoshm commented on the issue:

https://github.com/apache/flink/pull/2060
  
Thanks @greghogan for your inputs. updated. Please have a look. 
Please note though the configurable Charset will have no impact on 
StringValueParser(or other parsers) except StringParser.In SVParser we 
explicitly call setValueAscii(), and hence works with ascii only. @StephanEwen 
@fhueske 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (FLINK-3921) StringParser not specifying encoding to use

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315704#comment-15315704
 ] 

ASF GitHub Bot commented on FLINK-3921:
---

Github user rekhajoshm commented on the issue:

https://github.com/apache/flink/pull/2060
  
Thanks @greghogan for your inputs. updated. Please have a look. 
Please note though the configurable Charset will have no impact on 
StringValueParser(or other parsers) except StringParser.In SVParser we 
explicitly call setValueAscii(), and hence works with ascii only. @StephanEwen 
@fhueske 



> StringParser not specifying encoding to use
> ---
>
> Key: FLINK-3921
> URL: https://issues.apache.org/jira/browse/FLINK-3921
> Project: Flink
>  Issue Type: Improvement
>  Components: Core
>Affects Versions: 1.0.3
>Reporter: Tatu Saloranta
>Assignee: Rekha Joshi
>Priority: Trivial
>
> Class `flink.types.parser.StringParser` has javadocs indicating that contents 
> are expected to be Ascii, similar to `StringValueParser`. That makes sense, 
> but when constructing actual instance, no encoding is specified; on line 66 
> f.ex:
>this.result = new String(bytes, startPos+1, i - startPos - 2);
> which leads to using whatever default platform encoding is. If contents 
> really are always Ascii (would not count on that as parser is used from CSV 
> reader), not a big deal, but it can lead to the usual Latin-1-VS-UTF-8 issues.
> So I think that encoding should be explicitly specified, whatever is to be 
> used: javadocs claim ascii, so could be "us-ascii", but could well be UTF-8 
> or even ISO-8859-1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3650) Add maxBy/minBy to Scala DataSet API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315569#comment-15315569
 ] 

ASF GitHub Bot commented on FLINK-3650:
---

Github user ramkrish86 commented on the issue:

https://github.com/apache/flink/pull/1856
  
This time the build seems to pass except one and there is no long lines 
now. 


> Add maxBy/minBy to Scala DataSet API
> 
>
> Key: FLINK-3650
> URL: https://issues.apache.org/jira/browse/FLINK-3650
> Project: Flink
>  Issue Type: Improvement
>  Components: Java API, Scala API
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Assignee: ramkrishna.s.vasudevan
>
> The stable Java DataSet API contains the API calls {{maxBy}} and {{minBy}}. 
> These methods are not supported by the Scala DataSet API. These methods 
> should be added in order to have a consistent API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink issue #1856: FLINK-3650 Add maxBy/minBy to Scala DataSet API

[ramkrish86]

Github user ramkrish86 commented on the issue:

https://github.com/apache/flink/pull/1856
  
This time the build seems to pass except one and there is no long lines 
now. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4021) Problem of setting autoread for netty channel when more tasks sharing the same Tcp connection

[Zhijiang Wang (JIRA)]

Zhijiang Wang created FLINK-4021:


 Summary: Problem of setting autoread for netty channel when more 
tasks sharing the same Tcp connection
 Key: FLINK-4021
 URL: https://issues.apache.org/jira/browse/FLINK-4021
 Project: Flink
  Issue Type: Bug
  Components: Distributed Runtime
Affects Versions: 1.0.2
Reporter: Zhijiang Wang
Assignee: Zhijiang Wang
 Fix For: 1.1.0


More than one task sharing the same Tcp connection for shuffling data.
If the downstream task said as "A" has no available memory segment to read 
netty buffer from network, it will set autoread as false for the channel.
When the task A is failed or has available segments again, the netty handler 
will be notified to process the staging buffers first, then reset autoread as 
true. But in some scenarios, the autoread will not be set as true any more.
That is when processing staging buffers, first find the corresponding input 
channel for the buffer, if the task for that input channel is failed, the 
decodeMsg method in PartitionRequestClientHandler will return false, that means 
setting autoread as true will not be done anymore.
In summary,  if one task "A" sets the autoread as false because of no available 
segments, and resulting in some staging buffers. If another task "B" is failed 
by accident corresponding to one staging buffer. When task A trys to reset 
autoread as true, the process can not work because of task B failed.
I have fixed this problem in our application by adding one boolean parameter in 
decodeBufferOrEvent method to distinguish whether this method is invoke by 
netty IO thread channel read or staged message handler task in 
PartitionRequestClientHandler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (FLINK-3231) Handle Kinesis-side resharding in Kinesis streaming consumer

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315494#comment-15315494
 ] 

Tzu-Li (Gordon) Tai edited comment on FLINK-3231 at 6/4/16 1:04 PM:


Hi [~StephanEwen],

Finally reached the point where we could start working on this. I'm currently 
in the designing phase for this feature.

Merged state on restore sounds interesting (is there currently any doc / thread 
/ JIRA that's issuing this merged states feature?).

However, I don't think it will be able to fully solve the described problem for 
this JIRA, unless we are expecting the streaming job to fail and restore every 
time resharding happens. We still need coordination between the subtasks to 
gracefully handle the resharding. 

On the other hand, if the merged state feature also provides access (including 
incomplete checkpoints) from subtasks during job execution, then it might be 
possible to figure out an implementation.


was (Author: tzulitai):
Hi Stephan,

Finally reached the point where we could start working on this. I'm currently 
in the designing phase for this feature.

Merged state on restore sounds interesting (is there currently any doc / thread 
/ JIRA that's issuing this merged states feature?).

However, I don't think it will be able to fully solve the described problem for 
this JIRA, unless we are expecting the streaming job to fail and restore every 
time resharding happens. We still need coordination between the subtasks to 
gracefully handle the resharding. 

On the other hand, if the merged state feature also provides access (including 
incomplete checkpoints) from subtasks during job execution, then it might be 
possible to figure out an implementation.

> Handle Kinesis-side resharding in Kinesis streaming consumer
> 
>
> Key: FLINK-3231
> URL: https://issues.apache.org/jira/browse/FLINK-3231
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Tzu-Li (Gordon) Tai
>
> A big difference between Kinesis shards and Kafka partitions is that Kinesis 
> users can choose to "merge" and "split" shards at any time for adjustable 
> stream throughput capacity. This article explains this quite clearly: 
> https://brandur.org/kinesis-by-example.
> This will break the static shard-to-task mapping implemented in the basic 
> version of the Kinesis consumer 
> (https://issues.apache.org/jira/browse/FLINK-3229). The static shard-to-task 
> mapping is done in a simple round-robin-like distribution which can be 
> locally determined at each Flink consumer task (Flink Kafka consumer does 
> this too).
> To handle Kinesis resharding, we will need some way to let the Flink consumer 
> tasks coordinate which shards they are currently handling, and allow the 
> tasks to ask the coordinator for a shards reassignment when the task finds 
> out it has found a closed shard at runtime (shards will be closed by Kinesis 
> when it is merged and split).
> We need a centralized coordinator state store which is visible to all Flink 
> consumer tasks. Tasks can use this state store to locally determine what 
> shards it can be reassigned. Amazon KCL uses a DynamoDB table for the 
> coordination, but as described in 
> https://issues.apache.org/jira/browse/FLINK-3211, we unfortunately can't use 
> KCL for the implementation of the consumer if we want to leverage Flink's 
> checkpointing mechanics. For our own implementation, Zookeeper can be used 
> for this state store, but that means it would require the user to set up ZK 
> to work.
> Since this feature introduces extensive work, it is opened as a separate 
> sub-task from the basic implementation 
> https://issues.apache.org/jira/browse/FLINK-3229.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3231) Handle Kinesis-side resharding in Kinesis streaming consumer

[Tzu-Li (Gordon) Tai (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315494#comment-15315494
 ] 

Tzu-Li (Gordon) Tai commented on FLINK-3231:


Hi Stephan,

Finally reached the point where we could start working on this. I'm currently 
in the designing phase for this feature.

Merged state on restore sounds interesting (is there currently any doc / thread 
/ JIRA that's issuing this merged states feature?).

However, I don't think it will be able to fully solve the described problem for 
this JIRA, unless we are expecting the streaming job to fail and restore every 
time resharding happens. We still need coordination between the subtasks to 
gracefully handle the resharding. 

On the other hand, if the merged state feature also provides access (including 
incomplete checkpoints) from subtasks during job execution, then it might be 
possible to figure out an implementation.

> Handle Kinesis-side resharding in Kinesis streaming consumer
> 
>
> Key: FLINK-3231
> URL: https://issues.apache.org/jira/browse/FLINK-3231
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Tzu-Li (Gordon) Tai
>
> A big difference between Kinesis shards and Kafka partitions is that Kinesis 
> users can choose to "merge" and "split" shards at any time for adjustable 
> stream throughput capacity. This article explains this quite clearly: 
> https://brandur.org/kinesis-by-example.
> This will break the static shard-to-task mapping implemented in the basic 
> version of the Kinesis consumer 
> (https://issues.apache.org/jira/browse/FLINK-3229). The static shard-to-task 
> mapping is done in a simple round-robin-like distribution which can be 
> locally determined at each Flink consumer task (Flink Kafka consumer does 
> this too).
> To handle Kinesis resharding, we will need some way to let the Flink consumer 
> tasks coordinate which shards they are currently handling, and allow the 
> tasks to ask the coordinator for a shards reassignment when the task finds 
> out it has found a closed shard at runtime (shards will be closed by Kinesis 
> when it is merged and split).
> We need a centralized coordinator state store which is visible to all Flink 
> consumer tasks. Tasks can use this state store to locally determine what 
> shards it can be reassigned. Amazon KCL uses a DynamoDB table for the 
> coordination, but as described in 
> https://issues.apache.org/jira/browse/FLINK-3211, we unfortunately can't use 
> KCL for the implementation of the consumer if we want to leverage Flink's 
> checkpointing mechanics. For our own implementation, Zookeeper can be used 
> for this state store, but that means it would require the user to set up ZK 
> to work.
> Since this feature introduces extensive work, it is opened as a separate 
> sub-task from the basic implementation 
> https://issues.apache.org/jira/browse/FLINK-3229.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-4020) Remove shard list querying from Kinesis consumer constructor

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-4020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-4020:
---
Description: 
Currently FlinkKinesisConsumer is querying for the whole list of shards in the 
constructor, forcing the client to be able to access Kinesis as well. This is 
also a drawback for handling Kinesis-side resharding, since we'd want all shard 
listing / shard-to-task assigning / shard end (result of resharding) handling 
logic to be capable of being independently done within task life cycle methods, 
with defined and definite results.

Main thing to overcome is coordination between parallel subtasks. All subtasks 
will need to retry (due to Amazon's operation rate limits) until all subtasks 
have succeeded. We could probably use either ZK or Amazon DynamoDB (user 
configurable) for coordinating subtask status.

  was:Currently FlinkKinesisConsumer is querying for the whole list of shards 
in the constructor, forcing the client to be able to access Kinesis as well. 
This is also a drawback for handling Kinesis-side resharding, since we'd want 
all shard listing / shard-to-task assigning / shard end (result of resharding) 
handling logic to be capable of being independently done within task life cycle 
methods, with defined and definite results.


> Remove shard list querying from Kinesis consumer constructor
> 
>
> Key: FLINK-4020
> URL: https://issues.apache.org/jira/browse/FLINK-4020
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming Connectors
>Reporter: Tzu-Li (Gordon) Tai
>
> Currently FlinkKinesisConsumer is querying for the whole list of shards in 
> the constructor, forcing the client to be able to access Kinesis as well. 
> This is also a drawback for handling Kinesis-side resharding, since we'd want 
> all shard listing / shard-to-task assigning / shard end (result of 
> resharding) handling logic to be capable of being independently done within 
> task life cycle methods, with defined and definite results.
> Main thing to overcome is coordination between parallel subtasks. All 
> subtasks will need to retry (due to Amazon's operation rate limits) until all 
> subtasks have succeeded. We could probably use either ZK or Amazon DynamoDB 
> (user configurable) for coordinating subtask status.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (FLINK-3231) Handle Kinesis-side resharding in Kinesis streaming consumer

[Tzu-Li (Gordon) Tai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/FLINK-3231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tzu-Li (Gordon) Tai updated FLINK-3231:
---
Affects Version/s: (was: 1.0.0)
   1.1.0

> Handle Kinesis-side resharding in Kinesis streaming consumer
> 
>
> Key: FLINK-3231
> URL: https://issues.apache.org/jira/browse/FLINK-3231
> Project: Flink
>  Issue Type: Sub-task
>  Components: Streaming Connectors
>Affects Versions: 1.1.0
>Reporter: Tzu-Li (Gordon) Tai
>
> A big difference between Kinesis shards and Kafka partitions is that Kinesis 
> users can choose to "merge" and "split" shards at any time for adjustable 
> stream throughput capacity. This article explains this quite clearly: 
> https://brandur.org/kinesis-by-example.
> This will break the static shard-to-task mapping implemented in the basic 
> version of the Kinesis consumer 
> (https://issues.apache.org/jira/browse/FLINK-3229). The static shard-to-task 
> mapping is done in a simple round-robin-like distribution which can be 
> locally determined at each Flink consumer task (Flink Kafka consumer does 
> this too).
> To handle Kinesis resharding, we will need some way to let the Flink consumer 
> tasks coordinate which shards they are currently handling, and allow the 
> tasks to ask the coordinator for a shards reassignment when the task finds 
> out it has found a closed shard at runtime (shards will be closed by Kinesis 
> when it is merged and split).
> We need a centralized coordinator state store which is visible to all Flink 
> consumer tasks. Tasks can use this state store to locally determine what 
> shards it can be reassigned. Amazon KCL uses a DynamoDB table for the 
> coordination, but as described in 
> https://issues.apache.org/jira/browse/FLINK-3211, we unfortunately can't use 
> KCL for the implementation of the consumer if we want to leverage Flink's 
> checkpointing mechanics. For our own implementation, Zookeeper can be used 
> for this state store, but that means it would require the user to set up ZK 
> to work.
> Since this feature introduces extensive work, it is opened as a separate 
> sub-task from the basic implementation 
> https://issues.apache.org/jira/browse/FLINK-3229.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3948) EventTimeWindowCheckpointingITCase Fails with Core Dump

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15315477#comment-15315477
 ] 

ASF GitHub Bot commented on FLINK-3948:
---

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/2072

[FLINK-3948] Protect RocksDB cleanup by cleanup lock

Before, it could happen that an asynchronous checkpoint was going on
when trying to do cleanup. Now we protect cleanup and asynchronous
checkpointing by a lock.

This was what caused `EventTimeWindowCheckpointingITCase` to fail. I now 
ran it more than a 100 times on travis and haven't observed a build failure 
related to this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink rocksdb/fix-core-dump

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2072.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2072


commit c8456b45c47e67cc316d5bb979de36a6225eebd4
Author: Aljoscha Krettek 
Date:   2016-06-04T05:59:48Z

Revert "[FLINK-3960] ignore EventTimeWindowCheckpointingITCase for now"

This reverts commit 98a939552e12fc699ff39111bbe877e112460ceb.

commit 13c8593ec9074aa086caf4329b21e331a1c54d58
Author: Aljoscha Krettek 
Date:   2016-05-20T20:37:14Z

[FLINK-3948] Protect RocksDB cleanup by cleanup lock

Before, it could happen that an asynchronous checkpoint was going on
when trying to do cleanup. Now we protect cleanup and asynchronous
checkpointing by a lock.




> EventTimeWindowCheckpointingITCase Fails with Core Dump
> ---
>
> Key: FLINK-3948
> URL: https://issues.apache.org/jira/browse/FLINK-3948
> Project: Flink
>  Issue Type: Bug
>  Components: state backends
>Affects Versions: 1.1.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Critical
>
> It fails because of a core dump in RocksDB. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] flink pull request #2072: [FLINK-3948] Protect RocksDB cleanup by cleanup lo...

[aljoscha]

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/2072

[FLINK-3948] Protect RocksDB cleanup by cleanup lock

Before, it could happen that an asynchronous checkpoint was going on
when trying to do cleanup. Now we protect cleanup and asynchronous
checkpointing by a lock.

This was what caused `EventTimeWindowCheckpointingITCase` to fail. I now 
ran it more than a 100 times on travis and haven't observed a build failure 
related to this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink rocksdb/fix-core-dump

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2072.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2072


commit c8456b45c47e67cc316d5bb979de36a6225eebd4
Author: Aljoscha Krettek 
Date:   2016-06-04T05:59:48Z

Revert "[FLINK-3960] ignore EventTimeWindowCheckpointingITCase for now"

This reverts commit 98a939552e12fc699ff39111bbe877e112460ceb.

commit 13c8593ec9074aa086caf4329b21e331a1c54d58
Author: Aljoscha Krettek 
Date:   2016-05-20T20:37:14Z

[FLINK-3948] Protect RocksDB cleanup by cleanup lock

Before, it could happen that an asynchronous checkpoint was going on
when trying to do cleanup. Now we protect cleanup and asynchronous
checkpointing by a lock.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (FLINK-4020) Remove shard list querying from Kinesis consumer constructor

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-4020:
--

 Summary: Remove shard list querying from Kinesis consumer 
constructor
 Key: FLINK-4020
 URL: https://issues.apache.org/jira/browse/FLINK-4020
 Project: Flink
  Issue Type: Sub-task
  Components: Streaming Connectors
Reporter: Tzu-Li (Gordon) Tai


Currently FlinkKinesisConsumer is querying for the whole list of shards in the 
constructor, forcing the client to be able to access Kinesis as well. This is 
also a drawback for handling Kinesis-side resharding, since we'd want all shard 
listing / shard-to-task assigning / shard end (result of resharding) handling 
logic to be capable of being independently done within task life cycle methods, 
with defined and definite results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (FLINK-4019) Expose approximateArrivalTimestamp through the KinesisDeserializationSchema interface

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-4019:
--

 Summary: Expose approximateArrivalTimestamp through the 
KinesisDeserializationSchema interface
 Key: FLINK-4019
 URL: https://issues.apache.org/jira/browse/FLINK-4019
 Project: Flink
  Issue Type: Sub-task
  Components: Streaming Connectors
Affects Versions: 1.1.0
Reporter: Tzu-Li (Gordon) Tai


Amazon's Record class also gives information about the timestamp of when 
Kinesis successfully receives the record: 
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/Record.html#getApproximateArrivalTimestamp().

This should be useful info for users and should be exposed through the 
deserialization schema.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)