[jira] [Updated] (FLINK-4574) Strengthen fetch interval implementation in Kinesis consumer
[
https://issues.apache.org/jira/browse/FLINK-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-4574:
---
Description:
As pointed out by [~rmetzger], right now the fetch interval implementation in
the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
interval times than specified by the user, ex. say the specified fetch interval
is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and {{y}} to
complete processing the fetched records for emitting, than the actual interval
between each fetch is actually {{f+x+y}}.
The main problem with this is that we can never guarantee how much time has
past since the last {{getRecords}} call, thus can not guarantee that returned
shard iterators will not have expired the next time we use them, even if we
limit the user-given value for {{f}} to not be longer than the iterator expire
time.
I propose to improve this by, per {{ShardConsumer}}, use a
{{ScheduledExecutorService}} / {{Timer}} to do the fixed-interval fetching, and
a separate blocking queue that collects the fetched records for emitting.
was:
As pointed out by [~rmetzger], right now the fetch interval implementation in
the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
interval times than specified by the user, ex. say the specified fetch interval
is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and {{y}} to
complete processing the fetched records for emitting, than the actual interval
between each fetch is actually {{f+x+y}}.
The main problem with this is that we can never guarantee how much time has
past since the last {{getRecords}} call, thus can not guarantee that returned
shard iterators will not have expired the next time we use them, even if we
limit the user-given value for {{f}} to not be longer than the iterator expire
time.
I propose to improve this by, per {{ShardConsumer}}, use a
{{ScheduledExecutorService}} to do the fixed-interval fetching, and a separate
blocking queue that collects the fetched records for emitting.
> Strengthen fetch interval implementation in Kinesis consumer
>
>
> Key: FLINK-4574
> URL: https://issues.apache.org/jira/browse/FLINK-4574
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Affects Versions: 1.1.0
>Reporter: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0, 1.1.3
>
>
> As pointed out by [~rmetzger], right now the fetch interval implementation in
> the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
> interval times than specified by the user, ex. say the specified fetch
> interval is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and
> {{y}} to complete processing the fetched records for emitting, than the
> actual interval between each fetch is actually {{f+x+y}}.
> The main problem with this is that we can never guarantee how much time has
> past since the last {{getRecords}} call, thus can not guarantee that returned
> shard iterators will not have expired the next time we use them, even if we
> limit the user-given value for {{f}} to not be longer than the iterator
> expire time.
> I propose to improve this by, per {{ShardConsumer}}, use a
> {{ScheduledExecutorService}} / {{Timer}} to do the fixed-interval fetching,
> and a separate blocking queue that collects the fetched records for emitting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4559) Kinesis Producer not setting credentials provider properly when AWS_CREDENTIALS_PROVIDER is "AUTO"
[
https://issues.apache.org/jira/browse/FLINK-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15462090#comment-15462090
]
ASF GitHub Bot commented on FLINK-4559:
---
Github user tzulitai commented on the issue:
https://github.com/apache/flink/pull/2470
Thanks for the review @greghogan.
I think it makes sense to let `AUTO` be the default, as the usual use case
for most AWS users is to auto-detect the credentials delivered to instances by
AWS's metadata service. Changing this should not break user code either. Would
you like to open the ticket for that?
> Kinesis Producer not setting credentials provider properly when
> AWS_CREDENTIALS_PROVIDER is "AUTO"
> --
>
> Key: FLINK-4559
> URL: https://issues.apache.org/jira/browse/FLINK-4559
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
>Affects Versions: 1.1.0, 1.1.1, 1.1.2
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Minor
> Fix For: 1.1.3
>
>
> If the {{AWS_CREDENTIAL_PROVIDER}} option is set to {{AUTO}},
> {{AWSUtils.getCredentialsProvider}} will return {{null}}, so
> {{setCredentialsProvider}} should not be explicitly called on the internally
> built {{KinesisProducerConfiguration}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2470: [FLINK-4559][kinesis-connector] Fix AWSUtil.getCredential...
Github user tzulitai commented on the issue: https://github.com/apache/flink/pull/2470 Thanks for the review @greghogan. I think it makes sense to let `AUTO` be the default, as the usual use case for most AWS users is to auto-detect the credentials delivered to instances by AWS's metadata service. Changing this should not break user code either. Would you like to open the ticket for that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4513) Kafka connector documentation refers to Flink 1.1-SNAPSHOT
[ https://issues.apache.org/jira/browse/FLINK-4513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461833#comment-15461833 ] Neelesh Srinivas Salian commented on FLINK-4513: Shall I work on this? Need to change all the connector pages to appropriate this. Will check if this is true for other versions too. > Kafka connector documentation refers to Flink 1.1-SNAPSHOT > -- > > Key: FLINK-4513 > URL: https://issues.apache.org/jira/browse/FLINK-4513 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.1.1 >Reporter: Fabian Hueske >Priority: Trivial > Fix For: 1.1.2 > > > The Kafka connector documentation: > https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/connectors/kafka.html > of Flink 1.1 refers to a Flink 1.1-SNAPSHOT Maven version. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request #2471: Broken links on website
GitHub user apivovarov opened a pull request:
https://github.com/apache/flink/pull/2471
Broken links on website
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [ ] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [ ] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [ ] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
The website has the following broken links
DataSet API for static data embedded in Java, Scala, and Python,
http://ci.apache.org/projects/flink/flink-docs-master/apis/batch/index.html
Table API with a SQL-like expression language embedded in Java and Scala.
http://ci.apache.org/projects/flink/flink-docs-master/apis/batch/libs/table.html
Gelly, a graph processing API and library.
http://ci.apache.org/projects/flink/flink-docs-master/apis/batch/libs/gelly.html
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apivovarov/flink patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2471.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2471
commit b6d56682a55ae46e70cba33326ec58eb753fa73a
Author: Alexander Pivovarov
Date: 2016-09-03T21:14:58Z
Broken links on website
The website has the following broken links
DataSet API for static data embedded in Java, Scala, and Python,
http://ci.apache.org/projects/flink/flink-docs-master/apis/batch/index.html
Table API with a SQL-like expression language embedded in Java and Scala.
http://ci.apache.org/projects/flink/flink-docs-master/apis/batch/libs/table.html
Gelly, a graph processing API and library.
http://ci.apache.org/projects/flink/flink-docs-master/apis/batch/libs/gelly.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Commented] (FLINK-4559) Kinesis Producer not setting credentials provider properly when AWS_CREDENTIALS_PROVIDER is "AUTO"
[
https://issues.apache.org/jira/browse/FLINK-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461654#comment-15461654
]
ASF GitHub Bot commented on FLINK-4559:
---
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/2470
Looks good to me.
Should we also create a ticket to make AUTO the default as this checks for
credentials in multiple places?
http://docs.aws.amazon.com/java-sdk/latest/developer-guide/credentials.html
> Kinesis Producer not setting credentials provider properly when
> AWS_CREDENTIALS_PROVIDER is "AUTO"
> --
>
> Key: FLINK-4559
> URL: https://issues.apache.org/jira/browse/FLINK-4559
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
>Affects Versions: 1.1.0, 1.1.1, 1.1.2
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Minor
> Fix For: 1.1.3
>
>
> If the {{AWS_CREDENTIAL_PROVIDER}} option is set to {{AUTO}},
> {{AWSUtils.getCredentialsProvider}} will return {{null}}, so
> {{setCredentialsProvider}} should not be explicitly called on the internally
> built {{KinesisProducerConfiguration}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink issue #2470: [FLINK-4559][kinesis-connector] Fix AWSUtil.getCredential...
Github user greghogan commented on the issue: https://github.com/apache/flink/pull/2470 Looks good to me. Should we also create a ticket to make AUTO the default as this checks for credentials in multiple places? http://docs.aws.amazon.com/java-sdk/latest/developer-guide/credentials.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4559) Kinesis Producer not setting credentials provider properly when AWS_CREDENTIALS_PROVIDER is "AUTO"
[
https://issues.apache.org/jira/browse/FLINK-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461528#comment-15461528
]
ASF GitHub Bot commented on FLINK-4559:
---
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/2470
[FLINK-4559][kinesis-connector] Fix AWSUtil.getCredentialsProvider() to not
return null
Let `AWSUtil#getCredentialsProvider` return a
`DefaultAWSCredentialsProviderChain` instead of `null` when
`AWS_CREDENTIALS_PROVIDER` config is set to `AUTO`.
This mainly fixes the Kinesis producer to correctly get credentials when
using the `AUTO` option.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-4559
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2470.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2470
commit 7b86137672a4bdec7d67108bf423ef10269ddfa7
Author: Tzu-Li (Gordon) Tai
Date: 2016-09-03T17:37:39Z
[FLINK-4559][kinesis-connector] Fix AWSUtil.getCredentialsProvider() to not
return null
Return a DefaultAWSCredentialsProviderChain instead of null when
AWS_CREDENTIALS_PROVIDER config is set to "AUTO"
> Kinesis Producer not setting credentials provider properly when
> AWS_CREDENTIALS_PROVIDER is "AUTO"
> --
>
> Key: FLINK-4559
> URL: https://issues.apache.org/jira/browse/FLINK-4559
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
>Affects Versions: 1.1.0, 1.1.1, 1.1.2
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Minor
> Fix For: 1.1.3
>
>
> If the {{AWS_CREDENTIAL_PROVIDER}} option is set to {{AUTO}},
> {{AWSUtils.getCredentialsProvider}} will return {{null}}, so
> {{setCredentialsProvider}} should not be explicitly called on the internally
> built {{KinesisProducerConfiguration}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2470: [FLINK-4559][kinesis-connector] Fix AWSUtil.getCre...
GitHub user tzulitai opened a pull request: https://github.com/apache/flink/pull/2470 [FLINK-4559][kinesis-connector] Fix AWSUtil.getCredentialsProvider() to not return null Let `AWSUtil#getCredentialsProvider` return a `DefaultAWSCredentialsProviderChain` instead of `null` when `AWS_CREDENTIALS_PROVIDER` config is set to `AUTO`. This mainly fixes the Kinesis producer to correctly get credentials when using the `AUTO` option. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tzulitai/flink FLINK-4559 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2470.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2470 commit 7b86137672a4bdec7d67108bf423ef10269ddfa7 Author: Tzu-Li (Gordon) Tai Date: 2016-09-03T17:37:39Z [FLINK-4559][kinesis-connector] Fix AWSUtil.getCredentialsProvider() to not return null Return a DefaultAWSCredentialsProviderChain instead of null when AWS_CREDENTIALS_PROVIDER config is set to "AUTO" --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Assigned] (FLINK-4559) Kinesis Producer not setting credentials provider properly when AWS_CREDENTIALS_PROVIDER is "AUTO"
[
https://issues.apache.org/jira/browse/FLINK-4559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai reassigned FLINK-4559:
--
Assignee: Tzu-Li (Gordon) Tai
> Kinesis Producer not setting credentials provider properly when
> AWS_CREDENTIALS_PROVIDER is "AUTO"
> --
>
> Key: FLINK-4559
> URL: https://issues.apache.org/jira/browse/FLINK-4559
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
>Affects Versions: 1.1.0, 1.1.1, 1.1.2
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Minor
> Fix For: 1.1.3
>
>
> If the {{AWS_CREDENTIAL_PROVIDER}} option is set to {{AUTO}},
> {{AWSUtils.getCredentialsProvider}} will return {{null}}, so
> {{setCredentialsProvider}} should not be explicitly called on the internally
> built {{KinesisProducerConfiguration}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4194) Implement isEndOfStream() for KinesisDeserializationSchema
[
https://issues.apache.org/jira/browse/FLINK-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461269#comment-15461269
]
Tzu-Li (Gordon) Tai commented on FLINK-4194:
On second thought, I think we ought to discuss whether or not adding the
function back to {{KinesisDeserializationSchema}} and breaking user code is a
good idea now, since the Kinesis connector is now in a released version and
would need a migration plan. Its not a highly requested feature at the moment,
so I would say it is currently not worth the effort. Perhaps a good time to add
this back is when it is actually possible to "stop" parallel consumption across
the subtasks with an atomic checkpoint-and-stop, so an {{isEndOfStream()}}
behaviour can be truly defined.
What do you think, [~rmetzger]?
> Implement isEndOfStream() for KinesisDeserializationSchema
> --
>
> Key: FLINK-4194
> URL: https://issues.apache.org/jira/browse/FLINK-4194
> Project: Flink
> Issue Type: New Feature
> Components: Kinesis Connector
>Reporter: Robert Metzger
>Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0
>
>
> **Original JIRA title: KinesisDeserializationSchema.isEndOfStream() is never
> called. The corresponding part of the code has been commented out with
> reference to this JIRA.**
> The Kinesis connector does not respect the
> {{KinesisDeserializationSchema.isEndOfStream()}} method.
> The purpose of this method is to stop consuming from a source, based on input
> data.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-4574) Strengthen fetch interval implementation in Kinesis consumer
[
https://issues.apache.org/jira/browse/FLINK-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-4574:
---
Description:
As pointed out by [~rmetzger], right now the fetch interval implementation in
the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
interval times than specified by the user, ex. say the specified fetch interval
is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and {{y}} to
complete processing the fetched records for emitting, than the actual interval
between each fetch is actually {{f+x+y}}.
The main problem with this is that we can never guarantee how much time has
past since the last {{getRecords}} call, thus can not guarantee that returned
shard iterators will not have expired the next time we use them, even if we
limit the user-given value for {{f}} to not be longer than the iterator expire
time.
I propose to improve this by, per {{ShardConsumer}}, use a
{{ScheduledExecutorService}} to do the fixed-interval fetching, and a separate
blocking queue that collects the fetched records for emitting.
was:
As pointed out by [~rmetzger], right now the fetch interval implementation in
the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
interval times than specified by the user, ex. say the specified fetch interval
is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and {{y}} to
complete processing the fetched records for emitting, than the actual interval
between each fetch is actually {{f+x+y}}.
The main problem with this is that we can never guarantee how much time has
past since the last {{getRecords}} call, thus can not guarantee that returned
shard iterators will not have expired the next time we use them, even if we
limit the user-given value for {{f}} to not be longer than the iterator expire
time.
I propose to change to, per {{ShardConsumer}}, use a
{{ScheduledExecutorService}} to do the fixed-interval fetching, and a separate
blocking queue that collects the fetched records for emitting.
> Strengthen fetch interval implementation in Kinesis consumer
>
>
> Key: FLINK-4574
> URL: https://issues.apache.org/jira/browse/FLINK-4574
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Affects Versions: 1.1.0
>Reporter: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0, 1.1.3
>
>
> As pointed out by [~rmetzger], right now the fetch interval implementation in
> the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
> interval times than specified by the user, ex. say the specified fetch
> interval is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and
> {{y}} to complete processing the fetched records for emitting, than the
> actual interval between each fetch is actually {{f+x+y}}.
> The main problem with this is that we can never guarantee how much time has
> past since the last {{getRecords}} call, thus can not guarantee that returned
> shard iterators will not have expired the next time we use them, even if we
> limit the user-given value for {{f}} to not be longer than the iterator
> expire time.
> I propose to improve this by, per {{ShardConsumer}}, use a
> {{ScheduledExecutorService}} to do the fixed-interval fetching, and a
> separate blocking queue that collects the fetched records for emitting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (FLINK-4574) Strengthen fetch interval implementation in Kinesis consumer
[
https://issues.apache.org/jira/browse/FLINK-4574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai updated FLINK-4574:
---
Description:
As pointed out by [~rmetzger], right now the fetch interval implementation in
the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
interval times than specified by the user, ex. say the specified fetch interval
is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and {{y}} to
complete processing the fetched records for emitting, than the actual interval
between each fetch is actually {{f+x+y}}.
The main problem with this is that we can never guarantee how much time has
past since the last {{getRecords}} call, thus can not guarantee that returned
shard iterators will not have expired the next time we use them, even if we
limit the user-given value for {{f}} to not be longer than the iterator expire
time.
I propose to change to, per {{ShardConsumer}}, use a
{{ScheduledExecutorService}} to do the fixed-interval fetching, and a separate
blocking queue that collects the fetched records for emitting.
was:
As pointed out by [~rmetzger], right now the fetch interval implementation in
the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
interval times than specified by the user, ex. say the specified fetch interval
is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and {{y}} to
complete processing the fetched records for emitting, than the actual interval
between each fetch is actually {{f+x+y}}.
The main problem with this is that we can never guarantee how much time has
past since the last {{getRecords}} call, thus can not guarantee that returned
shard iterators will not have expired the next time we use them.
I propose to change to, per {{ShardConsumer}}, use a
{{ScheduledExecutorService}} to do the fixed-interval fetching, and a separate
blocking queue that collects the fetched records for emitting.
> Strengthen fetch interval implementation in Kinesis consumer
>
>
> Key: FLINK-4574
> URL: https://issues.apache.org/jira/browse/FLINK-4574
> Project: Flink
> Issue Type: Improvement
> Components: Kinesis Connector
>Affects Versions: 1.1.0
>Reporter: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0, 1.1.3
>
>
> As pointed out by [~rmetzger], right now the fetch interval implementation in
> the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
> interval times than specified by the user, ex. say the specified fetch
> interval is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and
> {{y}} to complete processing the fetched records for emitting, than the
> actual interval between each fetch is actually {{f+x+y}}.
> The main problem with this is that we can never guarantee how much time has
> past since the last {{getRecords}} call, thus can not guarantee that returned
> shard iterators will not have expired the next time we use them, even if we
> limit the user-given value for {{f}} to not be longer than the iterator
> expire time.
> I propose to change to, per {{ShardConsumer}}, use a
> {{ScheduledExecutorService}} to do the fixed-interval fetching, and a
> separate blocking queue that collects the fetched records for emitting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (FLINK-4574) Strengthen fetch interval implementation in Kinesis consumer
Tzu-Li (Gordon) Tai created FLINK-4574:
--
Summary: Strengthen fetch interval implementation in Kinesis
consumer
Key: FLINK-4574
URL: https://issues.apache.org/jira/browse/FLINK-4574
Project: Flink
Issue Type: Improvement
Components: Kinesis Connector
Affects Versions: 1.1.0
Reporter: Tzu-Li (Gordon) Tai
Fix For: 1.2.0, 1.1.3
As pointed out by [~rmetzger], right now the fetch interval implementation in
the {{ShardConsumer}} class of the Kinesis consumer can lead to much longer
interval times than specified by the user, ex. say the specified fetch interval
is {{f}}, it takes {{x}} to complete a {{getRecords()}} call, and {{y}} to
complete processing the fetched records for emitting, than the actual interval
between each fetch is actually {{f+x+y}}.
The main problem with this is that we can never guarantee how much time has
past since the last {{getRecords}} call, thus can not guarantee that returned
shard iterators will not have expired the next time we use them.
I propose to change to, per {{ShardConsumer}}, use a
{{ScheduledExecutorService}} to do the fixed-interval fetching, and a separate
blocking queue that collects the fetched records for emitting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-4572) Convert to negative in LongValueToIntValue
[
https://issues.apache.org/jira/browse/FLINK-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15461121#comment-15461121
]
ASF GitHub Bot commented on FLINK-4572:
---
GitHub user greghogan opened a pull request:
https://github.com/apache/flink/pull/2469
[FLINK-4572] [gelly] Convert to negative in LongValueToIntValue
The Gelly drivers expect that scale 32 edges, represented by the lower 32
bits of long values, can be converted to int values. Values between 2^31 and
2^32 - 1 should be converted to negative integers.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/greghogan/flink
4572_convert_to_negative_in_longvaluetointvalue
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2469.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2469
commit 8a9fad5bb72e2f399b2afd0d095b6e29c9e5c9d0
Author: Greg Hogan
Date: 2016-09-02T16:01:29Z
[FLINK-4572] [gelly] Convert to negative in LongValueToIntValue
The Gelly drivers expect that scale 32 edges, represented by the lower
32 bits of long values, can be converted to int values. Values between
2^31 and 2^32 - 1 should be converted to negative integers.
> Convert to negative in LongValueToIntValue
> --
>
> Key: FLINK-4572
> URL: https://issues.apache.org/jira/browse/FLINK-4572
> Project: Flink
> Issue Type: Bug
> Components: Gelly
>Affects Versions: 1.2.0
>Reporter: Greg Hogan
>Assignee: Greg Hogan
>Priority: Minor
>
> The Gelly drivers expect that scale 32 edges, represented by the lower 32
> bits of {{long}} values, can be converted to {{int}} values. Values between
> 2^31 and 2^32 - 1 should be converted to negative integers, which is not
> supported by {{MathUtils.checkedDownCast}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2469: [FLINK-4572] [gelly] Convert to negative in LongVa...
GitHub user greghogan opened a pull request: https://github.com/apache/flink/pull/2469 [FLINK-4572] [gelly] Convert to negative in LongValueToIntValue The Gelly drivers expect that scale 32 edges, represented by the lower 32 bits of long values, can be converted to int values. Values between 2^31 and 2^32 - 1 should be converted to negative integers. You can merge this pull request into a Git repository by running: $ git pull https://github.com/greghogan/flink 4572_convert_to_negative_in_longvaluetointvalue Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2469.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2469 commit 8a9fad5bb72e2f399b2afd0d095b6e29c9e5c9d0 Author: Greg Hogan Date: 2016-09-02T16:01:29Z [FLINK-4572] [gelly] Convert to negative in LongValueToIntValue The Gelly drivers expect that scale 32 edges, represented by the lower 32 bits of long values, can be converted to int values. Values between 2^31 and 2^32 - 1 should be converted to negative integers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-4561) replace all the scala version as a `scala.binary.version` property
[ https://issues.apache.org/jira/browse/FLINK-4561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460619#comment-15460619 ] ASF GitHub Bot commented on FLINK-4561: --- Github user shijinkui commented on the issue: https://github.com/apache/flink/pull/2459 @chiwanpark @StephanEwen thanks for the reply. dependency's version which contain scala 2.10 replace as property? > replace all the scala version as a `scala.binary.version` property > -- > > Key: FLINK-4561 > URL: https://issues.apache.org/jira/browse/FLINK-4561 > Project: Flink > Issue Type: Improvement >Reporter: shijinkui > > replace all the scala version(2.10) as a property `scala.binary.version` > defined in root pom properties. default scala version property is 2.10. > modify: > 1. dependency include scala version > 2. module defining include scala version > 3. scala version upgrade to 2.11.8 from 2.11.7 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink issue #2459: [FLINK-4561] replace all the scala version as a `scala.bi...
Github user shijinkui commented on the issue: https://github.com/apache/flink/pull/2459 @chiwanpark @StephanEwen thanks for the reply. dependency's version which contain scala 2.10 replace as property? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-3580) Reintroduce Date/Time and implement scalar functions for it
[
https://issues.apache.org/jira/browse/FLINK-3580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460594#comment-15460594
]
ASF GitHub Bot commented on FLINK-3580:
---
GitHub user twalthr opened a pull request:
https://github.com/apache/flink/pull/2468
[FLINK-3580] [table] Add OVERLAPS function
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
This PR implements the SQL OVERLAPS function for Table API and SQL. It
allows for checking if two anchored intervals overlap. The SQL documentation is
missing as it will be reworked anyway shortly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/twalthr/flink FLINK-3580_OVERLAPS
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2468.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2468
commit edc61d6f85005cad3904a319289669a1ade9c46d
Author: twalthr
Date: 2016-09-03T06:00:58Z
[FLINK-3580] [table] Add OVERLAPS function
> Reintroduce Date/Time and implement scalar functions for it
> ---
>
> Key: FLINK-3580
> URL: https://issues.apache.org/jira/browse/FLINK-3580
> Project: Flink
> Issue Type: Sub-task
> Components: Table API & SQL
>Reporter: Timo Walther
>Assignee: Timo Walther
>
> This task includes:
> {code}
> DATETIME_PLUS
> EXTRACT_DATE
> FLOOR
> CEIL
> CURRENT_TIME
> CURRENT_TIMESTAMP
> LOCALTIME
> LOCALTIMESTAMP
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request #2468: [FLINK-3580] [table] Add OVERLAPS function
GitHub user twalthr opened a pull request:
https://github.com/apache/flink/pull/2468
[FLINK-3580] [table] Add OVERLAPS function
Thanks for contributing to Apache Flink. Before you open your pull request,
please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your
pull request. For more information and/or questions please refer to the [How To
Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful
description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira
title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the
JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis
build has passed
This PR implements the SQL OVERLAPS function for Table API and SQL. It
allows for checking if two anchored intervals overlap. The SQL documentation is
missing as it will be reworked anyway shortly.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/twalthr/flink FLINK-3580_OVERLAPS
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2468.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2468
commit edc61d6f85005cad3904a319289669a1ade9c46d
Author: twalthr
Date: 2016-09-03T06:00:58Z
[FLINK-3580] [table] Add OVERLAPS function
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
