[jira] [Resolved] (FLINK-8985) End-to-end test: CLI
[ https://issues.apache.org/jira/browse/FLINK-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther resolved FLINK-8985. - Resolution: Fixed Fix Version/s: 1.7.0 Fixed in 1.8: 44f217555f6ca72d1aa38d8626c1673a76c12b22, 8106e7236edc0b8035adc92a53ddba79456bc05a Fixed in 1.7: a73f979ea56d03a80d620a76bfd9078818d8c6d5, 89a19ec21eb0bd2fc0d3f8ef23c1f0388b512905 > End-to-end test: CLI > > > Key: FLINK-8985 > URL: https://issues.apache.org/jira/browse/FLINK-8985 > Project: Flink > Issue Type: Sub-task > Components: Client, Tests >Affects Versions: 1.5.0 >Reporter: Till Rohrmann >Assignee: Rong Rong >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > We should add end-to-end test which verifies that all client commands are > working correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10807) KafkaConsumer still consume removed topic after changing topics list
[ https://issues.apache.org/jira/browse/FLINK-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-10807: --- Description: subscribedPartitionsToStartOffsets in KafkaConsumerBase is getting values from restoredState, which is initialized in initializeState and discovering partitions. However, if we remove a topic in topics list and restore the Flink program, the restoredState still keeps removed topic, and the fetcher still fetches the data of the removed topic. (was: subscribedPartitionsToStartOffsets in KafkaConsumerBase is getting values from restoredState, which is initialized in initializeState and discovering partitions. However, if we remove a topic in topics list and restore the Flink program, the restoredState still keeps removed topic, and the fetcher will still fetch the data of the topic.) > KafkaConsumer still consume removed topic after changing topics list > > > Key: FLINK-10807 > URL: https://issues.apache.org/jira/browse/FLINK-10807 > Project: Flink > Issue Type: Bug > Components: Kafka Connector >Affects Versions: 1.6.2 >Reporter: Jiayi Liao >Assignee: Jiayi Liao >Priority: Major > > subscribedPartitionsToStartOffsets in KafkaConsumerBase is getting values > from restoredState, which is initialized in initializeState and discovering > partitions. However, if we remove a topic in topics list and restore the > Flink program, the restoredState still keeps removed topic, and the fetcher > still fetches the data of the removed topic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10807) KafkaConsumer still consume removed topic after changing topics list
Jiayi Liao created FLINK-10807: -- Summary: KafkaConsumer still consume removed topic after changing topics list Key: FLINK-10807 URL: https://issues.apache.org/jira/browse/FLINK-10807 Project: Flink Issue Type: Bug Components: Kafka Connector Affects Versions: 1.6.2 Reporter: Jiayi Liao Assignee: Jiayi Liao subscribedPartitionsToStartOffsets in KafkaConsumerBase is getting values from restoredState, which is initialized in initializeState and discovering partitions. However, if we remove a topic in topics list and restore the Flink program, the restoredState still keeps removed topic, and the fetcher will still fetch the data of the topic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10790) Refactor all the StreamPartitioner implementations into runtime module
[
https://issues.apache.org/jira/browse/FLINK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1660#comment-1660
]
ASF GitHub Bot commented on FLINK-10790:
zhijiangW commented on issue #7037: [FLINK-10790][network] Refactor
StreamPartitioner implementations into runtime module
URL: https://github.com/apache/flink/pull/7037#issuecomment-436525702
@pnowojski , there are some problems for this issue now. I will update the
codes later. Sorry for interrupting. I will cc you after ready.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refactor all the StreamPartitioner implementations into runtime module
> --
>
> Key: FLINK-10790
> URL: https://issues.apache.org/jira/browse/FLINK-10790
> Project: Flink
> Issue Type: Sub-task
> Components: Network
>Affects Versions: 1.8.0
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
> Labels: pull-request-available
>
> The motivation is for refactoring {{ChannelSelector}} interface mentioned in
> [FLINK-10662|https://issues.apache.org/jira/browse/FLINK-10662].
> The {{ChannelSelector}} implementations are separate for stream and batch
> currently, resulting in many duplicated logics for selecting channels. The
> root cause is that they are locating in different modules to not reuse the
> existing ones.
> In general, the core runtime codes should be unified for both stream and
> batch jobs as much as possible, then we can reduce the work overhead to
> maintain the duplicated codes.
> This jira would migrate the existing partitioner package from
> {{flink-streaming-jave}} to {{flink-runtime}}. Next we want to de-duplicate
> implementations for {{RoundRobinChannelSelector}} and {{OutputEmitter}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zhijiangW commented on issue #7037: [FLINK-10790][network] Refactor StreamPartitioner implementations into runtime module
zhijiangW commented on issue #7037: [FLINK-10790][network] Refactor StreamPartitioner implementations into runtime module URL: https://github.com/apache/flink/pull/7037#issuecomment-436525702 @pnowojski , there are some problems for this issue now. I will update the codes later. Sorry for interrupting. I will cc you after ready. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Issue Comment Deleted] (FLINK-10384) Add Sinh math function supported in Table API and SQL
[ https://issues.apache.org/jira/browse/FLINK-10384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang updated FLINK-10384: - Comment: was deleted (was: [~pnovi...@gmail.com] I think this issue could be closed.) > Add Sinh math function supported in Table API and SQL > - > > Key: FLINK-10384 > URL: https://issues.apache.org/jira/browse/FLINK-10384 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Reporter: vinoyang >Assignee: vinoyang >Priority: Minor > Labels: pull-request-available > Fix For: 1.7.0 > > > like FLINK-10340 for adding Cosh math function -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-10384) Add Sinh math function supported in Table API and SQL
[ https://issues.apache.org/jira/browse/FLINK-10384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] vinoyang closed FLINK-10384. Resolution: Fixed Fix Version/s: 1.7.0 > Add Sinh math function supported in Table API and SQL > - > > Key: FLINK-10384 > URL: https://issues.apache.org/jira/browse/FLINK-10384 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Reporter: vinoyang >Assignee: vinoyang >Priority: Minor > Labels: pull-request-available > Fix For: 1.7.0 > > > like FLINK-10340 for adding Cosh math function -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] tzulitai commented on issue #7028: [FLINK-10789] New serializer snapshots after 1.6 should implement TypeSerializerSnapshot
tzulitai commented on issue #7028: [FLINK-10789] New serializer snapshots after 1.6 should implement TypeSerializerSnapshot URL: https://github.com/apache/flink/pull/7028#issuecomment-436501351 Thanks for the review @StefanRRichter! Will proceed to merge this and also the 1.7 backport. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10789) Some new serializer snapshots added after 1.6 are not implementing the new TypeSerializerSnapshot interface
[
https://issues.apache.org/jira/browse/FLINK-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677657#comment-16677657
]
ASF GitHub Bot commented on FLINK-10789:
tzulitai commented on issue #7028: [FLINK-10789] New serializer snapshots after
1.6 should implement TypeSerializerSnapshot
URL: https://github.com/apache/flink/pull/7028#issuecomment-436501351
Thanks for the review @StefanRRichter!
Will proceed to merge this and also the 1.7 backport.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Some new serializer snapshots added after 1.6 are not implementing the new
> TypeSerializerSnapshot interface
> ---
>
> Key: FLINK-10789
> URL: https://issues.apache.org/jira/browse/FLINK-10789
> Project: Flink
> Issue Type: Improvement
> Components: Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Critical
> Labels: pull-request-available
>
> In commit
> [3787b89105c6a005ddca999b0b8a0356e812ede6|[https://github.com/apache/flink/commit/3787b89105c6a005ddca999b0b8a0356e812ede6],]
> there were a few serializer snapshot implementations that were newly added,
> namely:
> - {{LockableSerializerConfigSnapshot}}
> - {{ListViewSerializerConfigSnapshot}}
> - {{MapViewSerializerConfigSnapshot}}
> - {{ScalaEitherSerializerConfigSnapshot}}
> The problem is that instead of implementing the newer
> {{TypeSerializerSnapshot}}, they are extending the legacy deprecated
> {{TypeSerializerConfigSnapshot}}. This doesn't break things, but we will be
> redundantly writing Java-serialized with the new snapshots, as well as
> causing us to must introduce migration paths code later on after releasing
> them.
> It would be more ideal to implement them properly as
> {{TypeSerializerSnapshot}} for the release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10791) Provide end-to-end test for Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-10791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677655#comment-16677655 ] ASF GitHub Bot commented on FLINK-10791: yanghua opened a new pull request #7038: [FLINK-10791] Provide end-to-end test for Kafka 0.11 connector URL: https://github.com/apache/flink/pull/7038 ## What is the purpose of the change *Provide end-to-end test for Kafka 0.11 connector* ## Brief change log - *Provide end-to-end test for Kafka 0.11 connector* ## Verifying this change This change is already covered by existing tests, such as *(please describe tests)*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (**yes** / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide end-to-end test for Kafka 0.11 connector > > > Key: FLINK-10791 > URL: https://issues.apache.org/jira/browse/FLINK-10791 > Project: Flink > Issue Type: Test > Components: E2E Tests >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10791) Provide end-to-end test for Kafka 0.11 connector
[ https://issues.apache.org/jira/browse/FLINK-10791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-10791: --- Labels: pull-request-available (was: ) > Provide end-to-end test for Kafka 0.11 connector > > > Key: FLINK-10791 > URL: https://issues.apache.org/jira/browse/FLINK-10791 > Project: Flink > Issue Type: Test > Components: E2E Tests >Reporter: vinoyang >Assignee: vinoyang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] yanghua opened a new pull request #7038: [FLINK-10791] Provide end-to-end test for Kafka 0.11 connector
yanghua opened a new pull request #7038: [FLINK-10791] Provide end-to-end test for Kafka 0.11 connector URL: https://github.com/apache/flink/pull/7038 ## What is the purpose of the change *Provide end-to-end test for Kafka 0.11 connector* ## Brief change log - *Provide end-to-end test for Kafka 0.11 connector* ## Verifying this change This change is already covered by existing tests, such as *(please describe tests)*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (**yes** / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10790) Refactor all the StreamPartitioner implementations into runtime module
[
https://issues.apache.org/jira/browse/FLINK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677646#comment-16677646
]
ASF GitHub Bot commented on FLINK-10790:
zhijiangW commented on issue #7037: [FLINK-10790][network] Refactor
StreamPartitioner implementations into runtime module
URL: https://github.com/apache/flink/pull/7037#issuecomment-436496303
@pnowojski , this is the preparation for unifying implementations in both
stream and batch modes. As I described in above purpose, I would submit the
following PRs for de-duplication if you think this module refactor make sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refactor all the StreamPartitioner implementations into runtime module
> --
>
> Key: FLINK-10790
> URL: https://issues.apache.org/jira/browse/FLINK-10790
> Project: Flink
> Issue Type: Sub-task
> Components: Network
>Affects Versions: 1.8.0
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
> Labels: pull-request-available
>
> The motivation is for refactoring {{ChannelSelector}} interface mentioned in
> [FLINK-10662|https://issues.apache.org/jira/browse/FLINK-10662].
> The {{ChannelSelector}} implementations are separate for stream and batch
> currently, resulting in many duplicated logics for selecting channels. The
> root cause is that they are locating in different modules to not reuse the
> existing ones.
> In general, the core runtime codes should be unified for both stream and
> batch jobs as much as possible, then we can reduce the work overhead to
> maintain the duplicated codes.
> This jira would migrate the existing partitioner package from
> {{flink-streaming-jave}} to {{flink-runtime}}. Next we want to de-duplicate
> implementations for {{RoundRobinChannelSelector}} and {{OutputEmitter}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zhijiangW commented on issue #7037: [FLINK-10790][network] Refactor StreamPartitioner implementations into runtime module
zhijiangW commented on issue #7037: [FLINK-10790][network] Refactor StreamPartitioner implementations into runtime module URL: https://github.com/apache/flink/pull/7037#issuecomment-436496303 @pnowojski , this is the preparation for unifying implementations in both stream and batch modes. As I described in above purpose, I would submit the following PRs for de-duplication if you think this module refactor make sense. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10790) Refactor all the StreamPartitioner implementations into runtime module
[
https://issues.apache.org/jira/browse/FLINK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10790:
---
Labels: pull-request-available (was: )
> Refactor all the StreamPartitioner implementations into runtime module
> --
>
> Key: FLINK-10790
> URL: https://issues.apache.org/jira/browse/FLINK-10790
> Project: Flink
> Issue Type: Sub-task
> Components: Network
>Affects Versions: 1.8.0
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
> Labels: pull-request-available
>
> The motivation is for refactoring {{ChannelSelector}} interface mentioned in
> [FLINK-10662|https://issues.apache.org/jira/browse/FLINK-10662].
> The {{ChannelSelector}} implementations are separate for stream and batch
> currently, resulting in many duplicated logics for selecting channels. The
> root cause is that they are locating in different modules to not reuse the
> existing ones.
> In general, the core runtime codes should be unified for both stream and
> batch jobs as much as possible, then we can reduce the work overhead to
> maintain the duplicated codes.
> This jira would migrate the existing partitioner package from
> {{flink-streaming-jave}} to {{flink-runtime}}. Next we want to de-duplicate
> implementations for {{RoundRobinChannelSelector}} and {{OutputEmitter}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10790) Refactor all the StreamPartitioner implementations into runtime module
[
https://issues.apache.org/jira/browse/FLINK-10790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677643#comment-16677643
]
ASF GitHub Bot commented on FLINK-10790:
zhijiangW opened a new pull request #7037: [FLINK-10790][network] Refactor
StreamPartitioner implementations into runtime module
URL: https://github.com/apache/flink/pull/7037
## What is the purpose of the change
*Before refactoring `ChannelSelector` interface mentioned in
[FLINK-10662](https://issues.apache.org/jira/browse/FLINK-10622), we found
there are many duplicated implementations for `ChannelSelector` in stream and
batch modes separately. The root cause is that they are locating in different
modules, so can not reference each other.*
*The core runtime codes should be unified for both stream and batch modes as
much as possible, then we can reduce the work overhead to maintain the
duplicated codes, especially for the following refactor.*
*Next we would remove duplicated implementations for `OutputEmitter` and
`RoundRobinChannelSelector`.*
## Brief change log
- *Refactor the existing partitioner package from `flink-streaming-jave`
to `flink-runtime`.*
- *Remove `StreamRecord` type explicitly from `SerializationDelegate`.*
- *Modify related tests based on refactor changes.*
## Verifying this change
*This change is already covered by existing tests.*
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (yes)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refactor all the StreamPartitioner implementations into runtime module
> --
>
> Key: FLINK-10790
> URL: https://issues.apache.org/jira/browse/FLINK-10790
> Project: Flink
> Issue Type: Sub-task
> Components: Network
>Affects Versions: 1.8.0
>Reporter: zhijiang
>Assignee: zhijiang
>Priority: Minor
> Labels: pull-request-available
>
> The motivation is for refactoring {{ChannelSelector}} interface mentioned in
> [FLINK-10662|https://issues.apache.org/jira/browse/FLINK-10662].
> The {{ChannelSelector}} implementations are separate for stream and batch
> currently, resulting in many duplicated logics for selecting channels. The
> root cause is that they are locating in different modules to not reuse the
> existing ones.
> In general, the core runtime codes should be unified for both stream and
> batch jobs as much as possible, then we can reduce the work overhead to
> maintain the duplicated codes.
> This jira would migrate the existing partitioner package from
> {{flink-streaming-jave}} to {{flink-runtime}}. Next we want to de-duplicate
> implementations for {{RoundRobinChannelSelector}} and {{OutputEmitter}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] zhijiangW opened a new pull request #7037: [FLINK-10790][network] Refactor StreamPartitioner implementations into runtime module
zhijiangW opened a new pull request #7037: [FLINK-10790][network] Refactor StreamPartitioner implementations into runtime module URL: https://github.com/apache/flink/pull/7037 ## What is the purpose of the change *Before refactoring `ChannelSelector` interface mentioned in [FLINK-10662](https://issues.apache.org/jira/browse/FLINK-10622), we found there are many duplicated implementations for `ChannelSelector` in stream and batch modes separately. The root cause is that they are locating in different modules, so can not reference each other.* *The core runtime codes should be unified for both stream and batch modes as much as possible, then we can reduce the work overhead to maintain the duplicated codes, especially for the following refactor.* *Next we would remove duplicated implementations for `OutputEmitter` and `RoundRobinChannelSelector`.* ## Brief change log - *Refactor the existing partitioner package from `flink-streaming-jave` to `flink-runtime`.* - *Remove `StreamRecord` type explicitly from `SerializationDelegate`.* - *Modify related tests based on refactor changes.* ## Verifying this change *This change is already covered by existing tests.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (yes) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10806) Support multiple consuming offsets when discovering a new topic
[ https://issues.apache.org/jira/browse/FLINK-10806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-10806: --- Summary: Support multiple consuming offsets when discovering a new topic (was: Add optional consuming parameters when discovering a new topic) > Support multiple consuming offsets when discovering a new topic > --- > > Key: FLINK-10806 > URL: https://issues.apache.org/jira/browse/FLINK-10806 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.6.2 >Reporter: Jiayi Liao >Assignee: Jiayi Liao >Priority: Major > > In KafkaConsumerBase, we discover the TopicPartitions and compare them with > the restoredState. It's reasonable when a topic's partitions scaled. However, > if we add a new topic which has too much data and restore the Flink program, > the data of the new topic will be consumed from the start, which may not be > what we want. I think this should be an option for developers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10806) Add optional consuming parameters when discovering a new topic
[ https://issues.apache.org/jira/browse/FLINK-10806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Liao updated FLINK-10806: --- Summary: Add optional consuming parameters when discovering a new topic (was: Support consume from earliest offset with a new topic) > Add optional consuming parameters when discovering a new topic > -- > > Key: FLINK-10806 > URL: https://issues.apache.org/jira/browse/FLINK-10806 > Project: Flink > Issue Type: Improvement > Components: Kafka Connector >Affects Versions: 1.6.2 >Reporter: Jiayi Liao >Assignee: Jiayi Liao >Priority: Major > > In KafkaConsumerBase, we discover the TopicPartitions and compare them with > the restoredState. It's reasonable when a topic's partitions scaled. However, > if we add a new topic which has too much data and restore the Flink program, > the data of the new topic will be consumed from the start, which may not be > what we want. I think this should be an option for developers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-10806) Support consume from earliest offset with a new topic
Jiayi Liao created FLINK-10806: -- Summary: Support consume from earliest offset with a new topic Key: FLINK-10806 URL: https://issues.apache.org/jira/browse/FLINK-10806 Project: Flink Issue Type: Improvement Components: Kafka Connector Affects Versions: 1.6.2 Reporter: Jiayi Liao Assignee: Jiayi Liao In KafkaConsumerBase, we discover the TopicPartitions and compare them with the restoredState. It's reasonable when a topic's partitions scaled. However, if we add a new topic which has too much data and restore the Flink program, the data of the new topic will be consumed from the start, which may not be what we want. I think this should be an option for developers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10706) Update the Flink Dashboard and its components
[ https://issues.apache.org/jira/browse/FLINK-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677570#comment-16677570 ] Yadong Xie commented on FLINK-10706: Hi [~fwollert], would you please assign this issue to me? In order to make this change more easier to be reviewed, I plan to add more comments on the codes and submit them step by step according to different functional modules. Thanks a lot for your efforts > Update the Flink Dashboard and its components > - > > Key: FLINK-10706 > URL: https://issues.apache.org/jira/browse/FLINK-10706 > Project: Flink > Issue Type: Sub-task > Components: Webfrontend >Affects Versions: 1.6.2 >Reporter: Fabian Wollert >Assignee: Fabian Wollert >Priority: Major > Fix For: 1.8.0 > > > The Flink Dashboard uses currently Angular 1, which had its successor coming > out two years ago. Its expected that Angular 1 (or Angular.js how it is > called now) will ceise to exist in the future, due to its successor Angular > 2-7 and React being the more actively developed platforms. > We should move to Angular 7 or React. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-9699) Add api to replace registered table
[ https://issues.apache.org/jira/browse/FLINK-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677502#comment-16677502 ] ASF GitHub Bot commented on FLINK-9699: --- zjffdu commented on issue #6236: [FLINK-9699] [table] Add api to replace registered table URL: https://github.com/apache/flink/pull/6236#issuecomment-436471468 Thanks @hequn8128 @yanghua `registerOrReplaceXXX` make sense for me. I will update the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add api to replace registered table > --- > > Key: FLINK-9699 > URL: https://issues.apache.org/jira/browse/FLINK-9699 > Project: Flink > Issue Type: Improvement >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] zjffdu commented on issue #6236: [FLINK-9699] [table] Add api to replace registered table
zjffdu commented on issue #6236: [FLINK-9699] [table] Add api to replace registered table URL: https://github.com/apache/flink/pull/6236#issuecomment-436471468 Thanks @hequn8128 @yanghua `registerOrReplaceXXX` make sense for me. I will update the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10802) Add metrics on vertex in job graph picture show
[ https://issues.apache.org/jira/browse/FLINK-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677495#comment-16677495 ] lining commented on FLINK-10802: Hi [~till.rohrmann], now user see something need click many page. If we can add something import on the vertex will efficiency. The problem about overload, we can control the refresh rate. Or add job dashbord for user can see overview of job. Such as back pressure, checkpoints， failover. > Add metrics on vertex in job graph picture show > --- > > Key: FLINK-10802 > URL: https://issues.apache.org/jira/browse/FLINK-10802 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: lining >Assignee: lining >Priority: Major > Attachments: image-2018-11-06-18-12-16-914.png > > > Hi. Now, if someone want see job metrics, have to click job overview. if we > can add some import metrics on graph, can impore it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10644) Batch Job: Speculative execution
[ https://issues.apache.org/jira/browse/FLINK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JIN SUN updated FLINK-10644: Description: Strugglers/outlier are tasks that run slower than most of the all tasks in a Batch Job, this somehow impact job latency, as pretty much this straggler will be in the critical path of the job and become as the bottleneck. Tasks may be slow for various reasons, including hardware degradation, or software mis-configuration, or noise neighboring. It's hard for JM to predict the runtime. To reduce the overhead of strugglers, other system such as Hadoop/Tez, Spark has *_speculative execution_*. Speculative execution is a health-check procedure that checks for tasks to be speculated, i.e. running slower in a ExecutionJobVertex than the median of all successfully completed tasks in that EJV, Such slow tasks will be re-submitted to another TM. It will not stop the slow tasks, but run a new copy in parallel. And will kill the others if one of them complete. This JIRA is an umbrella to apply this kind of idea in FLINK. Details will be append later. was: Strugglers/outlier are tasks that run slower than most of the all tasks in a Batch Job, this somehow impact job latency, as pretty much this straggler will be in the critical path of the job and become as the bottleneck. Tasks may be slow for various reasons, including hardware degradation, or software mis-configuration, or noise neighboring. It's hard for JM to predict the runtime. To reduce the overhead of strugglers, other system such as Hadoop/Tez, Spark has *_speculative execution_*. Speculative execution is a health-check procedure that checks for tasks to be speculated, i.e. running slower in a ExecutionJobVertex than the median of all successfully completed tasks in that EJV, Such slow tasks will be re-submitted to another TM. It will not stop the slow tasks, but run a new copy in parallel. And will kill the others if one of them complete. This JIRA is an umbrella to apply this kind of idea in FLINK. Details will be append later. the document contribute by is here: [https://docs.google.com/document/d/1X_Pfo4WcO-TEZmmVTTYNn44LQg5gnFeeaeqM7ZNLQ7M/edit] > Batch Job: Speculative execution > > > Key: FLINK-10644 > URL: https://issues.apache.org/jira/browse/FLINK-10644 > Project: Flink > Issue Type: New Feature > Components: JobManager >Reporter: JIN SUN >Assignee: JIN SUN >Priority: Major > Fix For: 1.8.0 > > > Strugglers/outlier are tasks that run slower than most of the all tasks in a > Batch Job, this somehow impact job latency, as pretty much this straggler > will be in the critical path of the job and become as the bottleneck. > Tasks may be slow for various reasons, including hardware degradation, or > software mis-configuration, or noise neighboring. It's hard for JM to predict > the runtime. > To reduce the overhead of strugglers, other system such as Hadoop/Tez, Spark > has *_speculative execution_*. Speculative execution is a health-check > procedure that checks for tasks to be speculated, i.e. running slower in a > ExecutionJobVertex than the median of all successfully completed tasks in > that EJV, Such slow tasks will be re-submitted to another TM. It will not > stop the slow tasks, but run a new copy in parallel. And will kill the others > if one of them complete. > This JIRA is an umbrella to apply this kind of idea in FLINK. Details will be > append later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10644) Batch Job: Speculative execution
[ https://issues.apache.org/jira/browse/FLINK-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JIN SUN updated FLINK-10644: Description: Strugglers/outlier are tasks that run slower than most of the all tasks in a Batch Job, this somehow impact job latency, as pretty much this straggler will be in the critical path of the job and become as the bottleneck. Tasks may be slow for various reasons, including hardware degradation, or software mis-configuration, or noise neighboring. It's hard for JM to predict the runtime. To reduce the overhead of strugglers, other system such as Hadoop/Tez, Spark has *_speculative execution_*. Speculative execution is a health-check procedure that checks for tasks to be speculated, i.e. running slower in a ExecutionJobVertex than the median of all successfully completed tasks in that EJV, Such slow tasks will be re-submitted to another TM. It will not stop the slow tasks, but run a new copy in parallel. And will kill the others if one of them complete. This JIRA is an umbrella to apply this kind of idea in FLINK. Details will be append later. the document contribute by is here: [https://docs.google.com/document/d/1X_Pfo4WcO-TEZmmVTTYNn44LQg5gnFeeaeqM7ZNLQ7M/edit] was: Strugglers/outlier are tasks that run slower than most of the all tasks in a Batch Job, this somehow impact job latency, as pretty much this straggler will be in the critical path of the job and become as the bottleneck. Tasks may be slow for various reasons, including hardware degradation, or software mis-configuration, or noise neighboring. It's hard for JM to predict the runtime. To reduce the overhead of strugglers, other system such as Hadoop/Tez, Spark has *_speculative execution_*. Speculative execution is a health-check procedure that checks for tasks to be speculated, i.e. running slower in a ExecutionJobVertex than the median of all successfully completed tasks in that EJV, Such slow tasks will be re-submitted to another TM. It will not stop the slow tasks, but run a new copy in parallel. And will kill the others if one of them complete. This JIRA is an umbrella to apply this kind of idea in FLINK. Details will be append later. > Batch Job: Speculative execution > > > Key: FLINK-10644 > URL: https://issues.apache.org/jira/browse/FLINK-10644 > Project: Flink > Issue Type: New Feature > Components: JobManager >Reporter: JIN SUN >Assignee: JIN SUN >Priority: Major > Fix For: 1.8.0 > > > Strugglers/outlier are tasks that run slower than most of the all tasks in a > Batch Job, this somehow impact job latency, as pretty much this straggler > will be in the critical path of the job and become as the bottleneck. > Tasks may be slow for various reasons, including hardware degradation, or > software mis-configuration, or noise neighboring. It's hard for JM to predict > the runtime. > To reduce the overhead of strugglers, other system such as Hadoop/Tez, Spark > has *_speculative execution_*. Speculative execution is a health-check > procedure that checks for tasks to be speculated, i.e. running slower in a > ExecutionJobVertex than the median of all successfully completed tasks in > that EJV, Such slow tasks will be re-submitted to another TM. It will not > stop the slow tasks, but run a new copy in parallel. And will kill the others > if one of them complete. > This JIRA is an umbrella to apply this kind of idea in FLINK. Details will be > append later. > > the document contribute by is here: > [https://docs.google.com/document/d/1X_Pfo4WcO-TEZmmVTTYNn44LQg5gnFeeaeqM7ZNLQ7M/edit] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (FLINK-10671) rest monitoring api Savepoint status call fails if akka.ask.timeout < checkpoint duration
[
https://issues.apache.org/jira/browse/FLINK-10671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao closed FLINK-10671.
Resolution: Not A Problem
Please reopen if necessary.
> rest monitoring api Savepoint status call fails if akka.ask.timeout <
> checkpoint duration
> -
>
> Key: FLINK-10671
> URL: https://issues.apache.org/jira/browse/FLINK-10671
> Project: Flink
> Issue Type: Bug
> Components: REST
>Affects Versions: 1.6.1
>Reporter: Cliff Resnick
>Assignee: Gary Yao
>Priority: Minor
>
> Hi,
>
> There seems to be a problem with REST monitoring API:
> |/jobs/:jobid/savepoints/:triggerid|
>
> The problem is that when the Savepoint represented by {{:triggerid}} is
> called with {{cancel=true}} the above status call seems to fail if the
> savepoint duration exceeds {{akka.ask.timeout}} value.
>
> Below is a log in which I invoke "cancel with savepoint" then poll the above
> endpoint for status at 2 second intervals. {{akka.ask.timout}} is set for
> twenty seconds. The error is repeatable at various values of
> {{akka.ask.timeout}}.
>
> {noformat}
> 2018/10/24 19:42:25 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:27 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:29 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:31 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:33 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:35 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:37 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:39 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:41 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:43 savepoint id 925964b35b2d501f4a45b714eca0a2ca is
> IN_PROGRESS
> 2018/10/24 19:42:45 Cancel with Savepoint may have failed:
> java.util.concurrent.CompletionException: akka.pattern.AskTimeoutException:
> Ask timed out on [Actor[akka://flink/user/jobmanager_0#-234856817]] after
> [2 ms]. Sender[null] sent message of type
> "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
> at
> java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:326)
> at
> java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:338)
> at java.util.concurrent.CompletableFuture.uniRelay(CompletableFuture.java:911)
> at
> java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:899)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:770)
> at akka.dispatch.OnComplete.internal(Future.scala:258)
> at akka.dispatch.OnComplete.internal(Future.scala:256)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/jobmanager_0#-234856817]] after [2 ms].
> Sender[null] sent message of type
> "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
> ... 9 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10356) Add sanity checks to SpillingAdaptiveSpanningRecordDeserializer
[
https://issues.apache.org/jira/browse/FLINK-10356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677127#comment-16677127
]
ASF GitHub Bot commented on FLINK-10356:
NicoK commented on issue #6705: [FLINK-10356][network] add sanity checks to
SpillingAdaptiveSpanningRecordDeserializer
URL: https://github.com/apache/flink/pull/6705#issuecomment-436356439
Thanks @zhijiangW and @pnowojski for the reviews. I have addressed the
comments and committed some changes. Please have a look whether these look
better to you.
I'm honestly not sure the additional code in the tests is worth the
de-duplication...but maybe it is better readable now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add sanity checks to SpillingAdaptiveSpanningRecordDeserializer
> ---
>
> Key: FLINK-10356
> URL: https://issues.apache.org/jira/browse/FLINK-10356
> Project: Flink
> Issue Type: Sub-task
> Components: Network
>Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.6.0, 1.6.1, 1.7.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
>
> {{SpillingAdaptiveSpanningRecordDeserializer}} doesn't have any consistency
> checks for usage calls or serializers behaving properly, e.g. to read only as
> many bytes as available/promised for that record. At least these checks
> should be added:
> # Check that buffers have not been read from yet before adding them (this is
> an invariant {{SpillingAdaptiveSpanningRecordDeserializer}} works with and
> from what I can see, it is followed now.
> # Check that after deserialization, we actually consumed {{recordLength}}
> bytes
> ** If not, in the spanning deserializer, we currently simply skip the
> remaining bytes.
> ** But in the non-spanning deserializer, we currently continue from the
> wrong offset.
> # Protect against {{setNextBuffer}} being called before draining all
> available records
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] NicoK commented on issue #6705: [FLINK-10356][network] add sanity checks to SpillingAdaptiveSpanningRecordDeserializer
NicoK commented on issue #6705: [FLINK-10356][network] add sanity checks to SpillingAdaptiveSpanningRecordDeserializer URL: https://github.com/apache/flink/pull/6705#issuecomment-436356439 Thanks @zhijiangW and @pnowojski for the reviews. I have addressed the comments and committed some changes. Please have a look whether these look better to you. I'm honestly not sure the additional code in the tests is worth the de-duplication...but maybe it is better readable now. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10768) Move external catalog related code from TableEnvironment to CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677099#comment-16677099 ] ASF GitHub Bot commented on FLINK-10768: bowenli86 commented on issue #7011: [FLINK-10768][Table & SQL] Move external catalog related code from TableEnvironment to CatalogManager URL: https://github.com/apache/flink/pull/7011#issuecomment-436350350 holding on this PR as there are some related discussion going on at[ the design doc](https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZCqn4f7dz0zW5aa6T_hBZq5_o/edit) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Move external catalog related code from TableEnvironment to CatalogManager > -- > > Key: FLINK-10768 > URL: https://issues.apache.org/jira/browse/FLINK-10768 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Labels: pull-request-available > Fix For: 1.8.0 > > > Add a new CatalogManager class and port existing Calcite-directly-related > code from TableEnvironment into CatalogManager. > Background: there are two parallel efforts going on right now - FLINK-10686, > driven by Timo, includes moving external catalogs APIs from flink-table to > flink-table-common, also from Scala to Java; FLINK-10744 I'm working on right > now to integrate Flink with Hive and enhance external catalog functionality. > As discussed with @twalthr in FLINK-10689, we'd better parallelize these > efforts while introducing minimal overhead for integrating them later. Our > agreed way is to writing new code/feature related to external catalogs/hive > in Java in flink-table first then move to other module like > flink-table-common, this way we can minimize migration efforts. If existing > classes are modified for a feature we can start migrating them to Java in a > separate commit first and then perform the actual feature changes, and > migrated classes can be placed in flink-table/src/main/java until we find a > better module structure. > Thus, this is NOT a feature, but purely refactor, thus no new functions > should be introduced. It acts the pre-requisite for FLINK-10698 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] bowenli86 commented on issue #7011: [FLINK-10768][Table & SQL] Move external catalog related code from TableEnvironment to CatalogManager
bowenli86 commented on issue #7011: [FLINK-10768][Table & SQL] Move external catalog related code from TableEnvironment to CatalogManager URL: https://github.com/apache/flink/pull/7011#issuecomment-436350350 holding on this PR as there are some related discussion going on at[ the design doc](https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZCqn4f7dz0zW5aa6T_hBZq5_o/edit) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10699) Create FlinkHmsCatalog for persistent Flink meta objects using Hive metastore as a registry
[ https://issues.apache.org/jira/browse/FLINK-10699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-10699: - Summary: Create FlinkHmsCatalog for persistent Flink meta objects using Hive metastore as a registry (was: Create a catalog implementation for persistent Flink meta objects using Hive metastore as a registry) > Create FlinkHmsCatalog for persistent Flink meta objects using Hive metastore > as a registry > --- > > Key: FLINK-10699 > URL: https://issues.apache.org/jira/browse/FLINK-10699 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Affects Versions: 1.6.1 >Reporter: Xuefu Zhang >Assignee: Bowen Li >Priority: Major > > Similar to FLINK-10697, but using Hive metastore as persistent storage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10531) State TTL RocksDb backend end-to-end test failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas updated FLINK-10531:
---
Fix Version/s: 1.7.0
> State TTL RocksDb backend end-to-end test failed on Travis
> --
>
> Key: FLINK-10531
> URL: https://issues.apache.org/jira/browse/FLINK-10531
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.6.1
>Reporter: Till Rohrmann
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.7.0
>
>
> The {{State TTL RocksDb backend end-to-end test}} end-to-end test failed on
> Travis.
> https://travis-ci.org/apache/flink/jobs/438226190
> https://api.travis-ci.org/v3/job/438226190/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10531) State TTL RocksDb backend end-to-end test failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677023#comment-16677023
]
ASF GitHub Bot commented on FLINK-10531:
kl0u commented on issue #7036: [FLINK-10531][e2e] Fix unstable TTL end-to-end
test.
URL: https://github.com/apache/flink/pull/7036#issuecomment-436331588
R @azagrebin
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> State TTL RocksDb backend end-to-end test failed on Travis
> --
>
> Key: FLINK-10531
> URL: https://issues.apache.org/jira/browse/FLINK-10531
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.6.1
>Reporter: Till Rohrmann
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> The {{State TTL RocksDb backend end-to-end test}} end-to-end test failed on
> Travis.
> https://travis-ci.org/apache/flink/jobs/438226190
> https://api.travis-ci.org/v3/job/438226190/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] kl0u commented on issue #7036: [FLINK-10531][e2e] Fix unstable TTL end-to-end test.
kl0u commented on issue #7036: [FLINK-10531][e2e] Fix unstable TTL end-to-end test. URL: https://github.com/apache/flink/pull/7036#issuecomment-436331588 R @azagrebin This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10531) State TTL RocksDb backend end-to-end test failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677018#comment-16677018
]
ASF GitHub Bot commented on FLINK-10531:
kl0u opened a new pull request #7036: [FLINK-10531][e2e] Fix unstable TTL
end-to-end test.
URL: https://github.com/apache/flink/pull/7036
As describe on the JIRA, the problem is that the clock on the machine on
Travis seems to have jumped backwards.
This resulted in a mismatch between the elements discarded as expired by
Flink's internal TTL mechanism, and the "user-code" in the test that computes
the expired elements based on the timestamp of the latest update.
I repeat the explanation from the JIRA here for reference:
---
The way the test works is the following. For a given state:
1) it keeps the state with TTL
2) and keeps all the updates to that state irrespective of their timestamp
(e.g. all the elements added in a ListState)
To verify correctness:
1) it fetches the state from Flink, as cleaned up by the internal TTL
mechanism
2) it reconstructs the expected state from the stored updates by taking the
timestamp
of the latest update (ts) discarding elements with timestamp ts-ttl
As you can see from the stacktrace in the error from Travis, the latest
update has timestamp ts=1538918066021
while there are elements in the list with timestamps greater than ts (e.g.
1538918066136). This means that the internal
clock on that machine went backwards, so Flink's TTL may have removed
elements that appear in the expected state
of the test, as it takes as current timestamp the ts=1538918066021.
The fix is simply to assume that (for the test), processing time increases
monotonically and ignore "updates from the past".
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> State TTL RocksDb backend end-to-end test failed on Travis
> --
>
> Key: FLINK-10531
> URL: https://issues.apache.org/jira/browse/FLINK-10531
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.6.1
>Reporter: Till Rohrmann
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> The {{State TTL RocksDb backend end-to-end test}} end-to-end test failed on
> Travis.
> https://travis-ci.org/apache/flink/jobs/438226190
> https://api.travis-ci.org/v3/job/438226190/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10531) State TTL RocksDb backend end-to-end test failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10531:
---
Labels: pull-request-available test-stability (was: test-stability)
> State TTL RocksDb backend end-to-end test failed on Travis
> --
>
> Key: FLINK-10531
> URL: https://issues.apache.org/jira/browse/FLINK-10531
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.6.1
>Reporter: Till Rohrmann
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> The {{State TTL RocksDb backend end-to-end test}} end-to-end test failed on
> Travis.
> https://travis-ci.org/apache/flink/jobs/438226190
> https://api.travis-ci.org/v3/job/438226190/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] kl0u opened a new pull request #7036: [FLINK-10531][e2e] Fix unstable TTL end-to-end test.
kl0u opened a new pull request #7036: [FLINK-10531][e2e] Fix unstable TTL end-to-end test. URL: https://github.com/apache/flink/pull/7036 As describe on the JIRA, the problem is that the clock on the machine on Travis seems to have jumped backwards. This resulted in a mismatch between the elements discarded as expired by Flink's internal TTL mechanism, and the "user-code" in the test that computes the expired elements based on the timestamp of the latest update. I repeat the explanation from the JIRA here for reference: --- The way the test works is the following. For a given state: 1) it keeps the state with TTL 2) and keeps all the updates to that state irrespective of their timestamp (e.g. all the elements added in a ListState) To verify correctness: 1) it fetches the state from Flink, as cleaned up by the internal TTL mechanism 2) it reconstructs the expected state from the stored updates by taking the timestamp of the latest update (ts) discarding elements with timestamp ts-ttl As you can see from the stacktrace in the error from Travis, the latest update has timestamp ts=1538918066021 while there are elements in the list with timestamps greater than ts (e.g. 1538918066136). This means that the internal clock on that machine went backwards, so Flink's TTL may have removed elements that appear in the expected state of the test, as it takes as current timestamp the ts=1538918066021. The fix is simply to assume that (for the test), processing time increases monotonically and ignore "updates from the past". This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10797) "IntelliJ Setup" link is broken in Readme.md
[ https://issues.apache.org/jira/browse/FLINK-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677010#comment-16677010 ] ASF GitHub Bot commented on FLINK-10797: xndai closed pull request #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md URL: https://github.com/apache/flink/pull/7025 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/README.md b/README.md index 7966ecd3d76..b7a96cdee46 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,7 @@ The IntelliJ IDE supports Maven out of the box and offers a plugin for Scala dev * IntelliJ download: [https://www.jetbrains.com/idea/](https://www.jetbrains.com/idea/) * IntelliJ Scala Plugin: [http://plugins.jetbrains.com/plugin/?id=1347](http://plugins.jetbrains.com/plugin/?id=1347) -Check out our [Setting up IntelliJ](https://github.com/apache/flink/blob/master/docs/internals/ide_setup.md#intellij-idea) guide for details. +Check out our [Setting up IntelliJ](https://github.com/apache/flink/blob/master/docs/flinkDev/ide_setup.md#intellij-idea) guide for details. ### Eclipse Scala IDE This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "IntelliJ Setup" link is broken in Readme.md > > > Key: FLINK-10797 > URL: https://issues.apache.org/jira/browse/FLINK-10797 > Project: Flink > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Xiening Dai >Assignee: TisonKun >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > The link points to > https://github.com/apache/flink/blob/master/docs/internals/ide_setup.md#intellij-idea > which is a 404 not found. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] xndai closed pull request #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md
xndai closed pull request #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md URL: https://github.com/apache/flink/pull/7025 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/README.md b/README.md index 7966ecd3d76..b7a96cdee46 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,7 @@ The IntelliJ IDE supports Maven out of the box and offers a plugin for Scala dev * IntelliJ download: [https://www.jetbrains.com/idea/](https://www.jetbrains.com/idea/) * IntelliJ Scala Plugin: [http://plugins.jetbrains.com/plugin/?id=1347](http://plugins.jetbrains.com/plugin/?id=1347) -Check out our [Setting up IntelliJ](https://github.com/apache/flink/blob/master/docs/internals/ide_setup.md#intellij-idea) guide for details. +Check out our [Setting up IntelliJ](https://github.com/apache/flink/blob/master/docs/flinkDev/ide_setup.md#intellij-idea) guide for details. ### Eclipse Scala IDE This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10796) Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676991#comment-16676991 ] Bowen Li commented on FLINK-10796: -- [~xiaow...@gmail.com] That's right. I've added the requirement in description > Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager > -- > > Key: FLINK-10796 > URL: https://issues.apache.org/jira/browse/FLINK-10796 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.8.0 > > > CatalogManager has a default catalog (as FlinkInMemoryCatalog), which will be > selected when a catalog name isn’t given in a request. > Users can also specify an external catalog as their preferred default one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10796) Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-10796: - Description: CatalogManager has a default catalog (as FlinkInMemoryCatalog), which will be selected when a catalog name isn’t given in a request. Users can also specify an external catalog as their preferred default one. was: CatalogManager has a default catalog, which will be selected when a catalog name isn’t given in a request. Users can also specify an external catalog as their preferred default one. > Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager > -- > > Key: FLINK-10796 > URL: https://issues.apache.org/jira/browse/FLINK-10796 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.8.0 > > > CatalogManager has a default catalog (as FlinkInMemoryCatalog), which will be > selected when a catalog name isn’t given in a request. > Users can also specify an external catalog as their preferred default one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10789) Some new serializer snapshots added after 1.6 are not implementing the new TypeSerializerSnapshot interface
[
https://issues.apache.org/jira/browse/FLINK-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676988#comment-16676988
]
ASF GitHub Bot commented on FLINK-10789:
StefanRRichter commented on a change in pull request #7028: [FLINK-10789] New
serializer snapshots after 1.6 should implement TypeSerializerSnapshot
URL: https://github.com/apache/flink/pull/7028#discussion_r231202779
##
File path:
flink-scala/src/main/java/org/apache/flink/api/scala/typeutils/ScalaEitherSerializerSnapshot.java
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.typeutils;
+
+import org.apache.flink.api.common.typeutils.CompositeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+
+import scala.util.Either;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Configuration snapshot for serializers of Scala's {@link Either} type,
+ * containing configuration snapshots of the Left and Right serializers.
+ */
+public class ScalaEitherSerializerSnapshot implements
TypeSerializerSnapshot> {
+
+
Review comment:
2 newlines.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Some new serializer snapshots added after 1.6 are not implementing the new
> TypeSerializerSnapshot interface
> ---
>
> Key: FLINK-10789
> URL: https://issues.apache.org/jira/browse/FLINK-10789
> Project: Flink
> Issue Type: Improvement
> Components: Type Serialization System
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Critical
> Labels: pull-request-available
>
> In commit
> [3787b89105c6a005ddca999b0b8a0356e812ede6|[https://github.com/apache/flink/commit/3787b89105c6a005ddca999b0b8a0356e812ede6],]
> there were a few serializer snapshot implementations that were newly added,
> namely:
> - {{LockableSerializerConfigSnapshot}}
> - {{ListViewSerializerConfigSnapshot}}
> - {{MapViewSerializerConfigSnapshot}}
> - {{ScalaEitherSerializerConfigSnapshot}}
> The problem is that instead of implementing the newer
> {{TypeSerializerSnapshot}}, they are extending the legacy deprecated
> {{TypeSerializerConfigSnapshot}}. This doesn't break things, but we will be
> redundantly writing Java-serialized with the new snapshots, as well as
> causing us to must introduce migration paths code later on after releasing
> them.
> It would be more ideal to implement them properly as
> {{TypeSerializerSnapshot}} for the release.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] StefanRRichter commented on a change in pull request #7028: [FLINK-10789] New serializer snapshots after 1.6 should implement TypeSerializerSnapshot
StefanRRichter commented on a change in pull request #7028: [FLINK-10789] New
serializer snapshots after 1.6 should implement TypeSerializerSnapshot
URL: https://github.com/apache/flink/pull/7028#discussion_r231202779
##
File path:
flink-scala/src/main/java/org/apache/flink/api/scala/typeutils/ScalaEitherSerializerSnapshot.java
##
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.scala.typeutils;
+
+import org.apache.flink.api.common.typeutils.CompositeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.common.typeutils.TypeSerializerSchemaCompatibility;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+
+import scala.util.Either;
+
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * Configuration snapshot for serializers of Scala's {@link Either} type,
+ * containing configuration snapshots of the Left and Right serializers.
+ */
+public class ScalaEitherSerializerSnapshot implements
TypeSerializerSnapshot> {
+
+
Review comment:
2 newlines.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-10796) Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bowen Li updated FLINK-10796: - Description: CatalogManager has a default catalog, which will be selected when a catalog name isn’t given in a request. Users can also specify an external catalog as their preferred default one. was:CatalogManager has a default catalog, which will be selected when a catalog name isn’t given in a request. > Add a default external catalog (as FlinkInMemoryCatalog) to CatalogManager > -- > > Key: FLINK-10796 > URL: https://issues.apache.org/jira/browse/FLINK-10796 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Reporter: Bowen Li >Assignee: Bowen Li >Priority: Major > Fix For: 1.8.0 > > > CatalogManager has a default catalog, which will be selected when a catalog > name isn’t given in a request. > Users can also specify an external catalog as their preferred default one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] asfgit closed pull request #5863: [FLINK-8985][e2etest] initial support for End-to-end CLI test, excluding YARN test
asfgit closed pull request #5863: [FLINK-8985][e2etest] initial support for
End-to-end CLI test, excluding YARN test
URL: https://github.com/apache/flink/pull/5863
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-end-to-end-tests/flink-api-test/pom.xml
b/flink-end-to-end-tests/flink-api-test/pom.xml
new file mode 100644
index 000..24a85e02bef
--- /dev/null
+++ b/flink-end-to-end-tests/flink-api-test/pom.xml
@@ -0,0 +1,105 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.6-SNAPSHOT
+ ..
+
+
+ 4.0.0
+
+ flink-api-test
+ flink-api-test
+ jar
+
+
+
+ org.apache.flink
+ flink-core
+ ${project.version}
+
+
+ org.apache.flink
+
flink-streaming-java_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-connector-filesystem_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-runtime-web_${scala.binary.version}
+ ${project.version}
+ test-jar
+
+
+
+
+
+
+ org.apache.maven.plugins
+ maven-shade-plugin
+ 3.0.0
+
+
+ package
+
+ shade
+
+
+
PeriodicStreamingJob
+
+
+
com.google.code.findbugs:jsr305
+
org.slf4j:*
+
log4j:*
+
+
+
+
+
*:*
+
+
META-INF/*.SF
+
META-INF/*.DSA
+
META-INF/*.RSA
+
+
+
+
+
+
org.apache.flink.runtime.tests.PeriodicStreamingJob
+
+
+
+
+
+
+
+
+
+
+
diff --git
a/flink-end-to-end-tests/flink-api-test/src/main/java/org/apache/flink/runtime/tests/PeriodicStreamingJob.java
b/flink-end-to-end-tests/flink-api-test/src/main/java/org/apache/flink/runtime/tests/PeriodicStreamingJob.java
new file mode 100644
index 000..ee8a49b8583
--- /dev/null
+++
b/flink-end-to-end-tests/flink-api-test/src/main/java/org/apache/flink/runtime/tests/PeriodicStreamingJob.java
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright
[jira] [Commented] (FLINK-8985) End-to-end test: CLI
[
https://issues.apache.org/jira/browse/FLINK-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676976#comment-16676976
]
ASF GitHub Bot commented on FLINK-8985:
---
asfgit closed pull request #5863: [FLINK-8985][e2etest] initial support for
End-to-end CLI test, excluding YARN test
URL: https://github.com/apache/flink/pull/5863
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/flink-end-to-end-tests/flink-api-test/pom.xml
b/flink-end-to-end-tests/flink-api-test/pom.xml
new file mode 100644
index 000..24a85e02bef
--- /dev/null
+++ b/flink-end-to-end-tests/flink-api-test/pom.xml
@@ -0,0 +1,105 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+
+
+ flink-end-to-end-tests
+ org.apache.flink
+ 1.6-SNAPSHOT
+ ..
+
+
+ 4.0.0
+
+ flink-api-test
+ flink-api-test
+ jar
+
+
+
+ org.apache.flink
+ flink-core
+ ${project.version}
+
+
+ org.apache.flink
+
flink-streaming-java_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-connector-filesystem_${scala.binary.version}
+ ${project.version}
+
+
+ org.apache.flink
+
flink-runtime-web_${scala.binary.version}
+ ${project.version}
+ test-jar
+
+
+
+
+
+
+ org.apache.maven.plugins
+ maven-shade-plugin
+ 3.0.0
+
+
+ package
+
+ shade
+
+
+
PeriodicStreamingJob
+
+
+
com.google.code.findbugs:jsr305
+
org.slf4j:*
+
log4j:*
+
+
+
+
+
*:*
+
+
META-INF/*.SF
+
META-INF/*.DSA
+
META-INF/*.RSA
+
+
+
+
+
+
org.apache.flink.runtime.tests.PeriodicStreamingJob
+
+
+
+
+
+
+
+
+
+
+
diff --git
a/flink-end-to-end-tests/flink-api-test/src/main/java/org/apache/flink/runtime/tests/PeriodicStreamingJob.java
b/flink-end-to-end-tests/flink-api-test/src/main/java/org/apache/flink/runtime/tests/PeriodicStreamingJob.java
new file mode 100644
index 000..ee8a49b8583
--- /dev/null
+++
b/flink-end-to-end-tests/flink-api-test/src/main/java/org/apache/flink/runtime/tests/PeriodicStreamingJob.java
[jira] [Commented] (FLINK-10463) Null literal cannot be properly parsed in Java Table API function call
[
https://issues.apache.org/jira/browse/FLINK-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676968#comment-16676968
]
Xingcan Cui commented on FLINK-10463:
-
[~twalthr], sure. I'll cherry-pick the commit to versions 1.6.3 and 1.7.1.
> Null literal cannot be properly parsed in Java Table API function call
> --
>
> Key: FLINK-10463
> URL: https://issues.apache.org/jira/browse/FLINK-10463
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Reporter: Xingcan Cui
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.3, 1.8.0, 1.7.1
>
>
> For example, the expression `Null(STRING).regexpReplace('oo|ar', '')` throws
> the following exception.
> {code:java}
> org.apache.flink.table.api.ExpressionParserException: Could not parse
> expression at column 13: string matching regex
> `(?i)\Qas\E(?![_$\p{javaJavaIdentifierPart}])' expected but `.' found
> Null(STRING).regexpReplace('oo|ar', '')
> ^
> at
> org.apache.flink.table.expressions.ExpressionParser$.throwError(ExpressionParser.scala:576)
> at
> org.apache.flink.table.expressions.ExpressionParser$.parseExpression(ExpressionParser.scala:569)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10463) Null literal cannot be properly parsed in Java Table API function call
[
https://issues.apache.org/jira/browse/FLINK-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingcan Cui updated FLINK-10463:
Fix Version/s: 1.7.1
1.6.3
> Null literal cannot be properly parsed in Java Table API function call
> --
>
> Key: FLINK-10463
> URL: https://issues.apache.org/jira/browse/FLINK-10463
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Reporter: Xingcan Cui
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.6.3, 1.8.0, 1.7.1
>
>
> For example, the expression `Null(STRING).regexpReplace('oo|ar', '')` throws
> the following exception.
> {code:java}
> org.apache.flink.table.api.ExpressionParserException: Could not parse
> expression at column 13: string matching regex
> `(?i)\Qas\E(?![_$\p{javaJavaIdentifierPart}])' expected but `.' found
> Null(STRING).regexpReplace('oo|ar', '')
> ^
> at
> org.apache.flink.table.expressions.ExpressionParser$.throwError(ExpressionParser.scala:576)
> at
> org.apache.flink.table.expressions.ExpressionParser$.parseExpression(ExpressionParser.scala:569)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (FLINK-10720) Add stress deployment end-to-end test
[
https://issues.apache.org/jira/browse/FLINK-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Richter closed FLINK-10720.
--
Resolution: Implemented
Merged in:
master: 660bb687c1
release-1.7: 380df0559b
> Add stress deployment end-to-end test
> -
>
> Key: FLINK-10720
> URL: https://issues.apache.org/jira/browse/FLINK-10720
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> In order to test Flink's scalability, I suggest to add an end-to-end test
> which tests the deployment of a job which is very demanding. The job should
> have large {{TaskDeploymentDescriptors}} (e.g. a job using union state or
> having a high degree of parallelism). That way we can test that the
> serialization overhead of the TDDs does not affect the health of the cluster
> (e.g. heartbeats are not affected because the serialization does not happen
> in the main thread).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10531) State TTL RocksDb backend end-to-end test failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676957#comment-16676957
]
Kostas Kloudas commented on FLINK-10531:
It is a test instability.
The way the test works is the following. For a given state:
1) it keeps the state with TTL
2) and keeps all the updates to that state irrespective of their timestamp
(e.g. all the elements added in a ListState)
To verify correctness:
1) it fetches the state from Flink, as cleaned up by the internal TTL mechanism
2) it reconstructs the expected state from the stored updates by taking the
timestamp
of the latest update (ts) discarding elements with timestamp ts-ttl
As you can see from the stacktrace in the error from Travis, the latest update
has timestamp ts=1538918066021
while there are elements in the list with timestamps greater than ts (e.g.
1538918066136). This means that the internal
clock on that machine went backwards, so Flink's TTL may have removed elements
that appear in the expected state
of the test, as it takes as current timestamp the ts=1538918066021.
The fix is simply to assume that (for the test), processing time increases
monotonically and ignore "updates from the past".
> State TTL RocksDb backend end-to-end test failed on Travis
> --
>
> Key: FLINK-10531
> URL: https://issues.apache.org/jira/browse/FLINK-10531
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.6.1
>Reporter: Till Rohrmann
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: test-stability
>
> The {{State TTL RocksDb backend end-to-end test}} end-to-end test failed on
> Travis.
> https://travis-ci.org/apache/flink/jobs/438226190
> https://api.travis-ci.org/v3/job/438226190/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10463) Null literal cannot be properly parsed in Java Table API function call
[
https://issues.apache.org/jira/browse/FLINK-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676955#comment-16676955
]
Timo Walther commented on FLINK-10463:
--
[~xccui] can you also cherry-pick bug fixes to the last two release branches if
backporting is not a big effort? thanks.
> Null literal cannot be properly parsed in Java Table API function call
> --
>
> Key: FLINK-10463
> URL: https://issues.apache.org/jira/browse/FLINK-10463
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Reporter: Xingcan Cui
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> For example, the expression `Null(STRING).regexpReplace('oo|ar', '')` throws
> the following exception.
> {code:java}
> org.apache.flink.table.api.ExpressionParserException: Could not parse
> expression at column 13: string matching regex
> `(?i)\Qas\E(?![_$\p{javaJavaIdentifierPart}])' expected but `.' found
> Null(STRING).regexpReplace('oo|ar', '')
> ^
> at
> org.apache.flink.table.expressions.ExpressionParser$.throwError(ExpressionParser.scala:576)
> at
> org.apache.flink.table.expressions.ExpressionParser$.parseExpression(ExpressionParser.scala:569)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (FLINK-10531) State TTL RocksDb backend end-to-end test failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas reassigned FLINK-10531:
--
Assignee: Kostas Kloudas (was: Renjie Liu)
> State TTL RocksDb backend end-to-end test failed on Travis
> --
>
> Key: FLINK-10531
> URL: https://issues.apache.org/jira/browse/FLINK-10531
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.6.1
>Reporter: Till Rohrmann
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: test-stability
>
> The {{State TTL RocksDb backend end-to-end test}} end-to-end test failed on
> Travis.
> https://travis-ci.org/apache/flink/jobs/438226190
> https://api.travis-ci.org/v3/job/438226190/log.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10720) Add stress deployment end-to-end test
[
https://issues.apache.org/jira/browse/FLINK-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676948#comment-16676948
]
ASF GitHub Bot commented on FLINK-10720:
asfgit closed pull request #6994: [FLINK-10720][tests] Add deployment
end-to-end stress test with many …
URL: https://github.com/apache/flink/pull/6994
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/DataStreamAllroundTestJobFactory.java
b/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/DataStreamAllroundTestJobFactory.java
index 3c8d0ad537f..8bd649cc5eb 100644
---
a/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/DataStreamAllroundTestJobFactory.java
+++
b/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/DataStreamAllroundTestJobFactory.java
@@ -88,7 +88,7 @@
* Total duration is (sliding_window_operator.num_events) *
(sequence_generator_source.event_time.clock_progress).
*
*/
-class DataStreamAllroundTestJobFactory {
+public class DataStreamAllroundTestJobFactory {
private static final ConfigOption TEST_SEMANTICS = ConfigOptions
.key("test.semantics")
.defaultValue("exactly-once")
@@ -201,7 +201,7 @@
.key("tumbling_window_operator.num_events")
.defaultValue(20L);
- static void setupEnvironment(StreamExecutionEnvironment env,
ParameterTool pt) throws Exception {
+ public static void setupEnvironment(StreamExecutionEnvironment env,
ParameterTool pt) throws Exception {
// set checkpointing semantics
String semantics = pt.get(TEST_SEMANTICS.key(),
TEST_SEMANTICS.defaultValue());
diff --git a/flink-end-to-end-tests/flink-heavy-deployment-stress-test/pom.xml
b/flink-end-to-end-tests/flink-heavy-deployment-stress-test/pom.xml
new file mode 100644
index 000..fa9444dec1b
--- /dev/null
+++ b/flink-end-to-end-tests/flink-heavy-deployment-stress-test/pom.xml
@@ -0,0 +1,81 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-end-to-end-tests
+ 1.8-SNAPSHOT
+ ..
+
+
+ flink-heavy-deployment-stress-test
+ flink-heavy-deployment-stress-test
+ jar
+
+
+
+ org.apache.flink
+ flink-streaming-java_2.11
+ ${project.version}
+
+
+ org.apache.flink
+ flink-datastream-allround-test
+ ${project.version}
+
+
+ org.apache.flink
+ flink-test-utils_2.11
+ ${project.version}
+ compile
+
+
+
+
+
+
+ org.apache.maven.plugins
+ maven-shade-plugin
+
+
+
HeavyDeploymentStressTestProgram
+ package
+
+ shade
+
+
+
HeavyDeploymentStressTestProgram
+
+
+
org.apache.flink.deployment.HeavyDeploymentStressTestProgram
+
+
+
+
+
+
+
+
+
+
diff --git
a/flink-end-to-end-tests/flink-heavy-deployment-stress-test/src/main/java/org/apache/flink/deployment/HeavyDeploymentStressTestProgram.java
[GitHub] asfgit closed pull request #6994: [FLINK-10720][tests] Add deployment end-to-end stress test with many …
asfgit closed pull request #6994: [FLINK-10720][tests] Add deployment
end-to-end stress test with many …
URL: https://github.com/apache/flink/pull/6994
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/DataStreamAllroundTestJobFactory.java
b/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/DataStreamAllroundTestJobFactory.java
index 3c8d0ad537f..8bd649cc5eb 100644
---
a/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/DataStreamAllroundTestJobFactory.java
+++
b/flink-end-to-end-tests/flink-datastream-allround-test/src/main/java/org/apache/flink/streaming/tests/DataStreamAllroundTestJobFactory.java
@@ -88,7 +88,7 @@
* Total duration is (sliding_window_operator.num_events) *
(sequence_generator_source.event_time.clock_progress).
*
*/
-class DataStreamAllroundTestJobFactory {
+public class DataStreamAllroundTestJobFactory {
private static final ConfigOption TEST_SEMANTICS = ConfigOptions
.key("test.semantics")
.defaultValue("exactly-once")
@@ -201,7 +201,7 @@
.key("tumbling_window_operator.num_events")
.defaultValue(20L);
- static void setupEnvironment(StreamExecutionEnvironment env,
ParameterTool pt) throws Exception {
+ public static void setupEnvironment(StreamExecutionEnvironment env,
ParameterTool pt) throws Exception {
// set checkpointing semantics
String semantics = pt.get(TEST_SEMANTICS.key(),
TEST_SEMANTICS.defaultValue());
diff --git a/flink-end-to-end-tests/flink-heavy-deployment-stress-test/pom.xml
b/flink-end-to-end-tests/flink-heavy-deployment-stress-test/pom.xml
new file mode 100644
index 000..fa9444dec1b
--- /dev/null
+++ b/flink-end-to-end-tests/flink-heavy-deployment-stress-test/pom.xml
@@ -0,0 +1,81 @@
+
+
+http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
+
+ 4.0.0
+
+
+ org.apache.flink
+ flink-end-to-end-tests
+ 1.8-SNAPSHOT
+ ..
+
+
+ flink-heavy-deployment-stress-test
+ flink-heavy-deployment-stress-test
+ jar
+
+
+
+ org.apache.flink
+ flink-streaming-java_2.11
+ ${project.version}
+
+
+ org.apache.flink
+ flink-datastream-allround-test
+ ${project.version}
+
+
+ org.apache.flink
+ flink-test-utils_2.11
+ ${project.version}
+ compile
+
+
+
+
+
+
+ org.apache.maven.plugins
+ maven-shade-plugin
+
+
+
HeavyDeploymentStressTestProgram
+ package
+
+ shade
+
+
+
HeavyDeploymentStressTestProgram
+
+
+
org.apache.flink.deployment.HeavyDeploymentStressTestProgram
+
+
+
+
+
+
+
+
+
+
diff --git
a/flink-end-to-end-tests/flink-heavy-deployment-stress-test/src/main/java/org/apache/flink/deployment/HeavyDeploymentStressTestProgram.java
b/flink-end-to-end-tests/flink-heavy-deployment-stress-test/src/main/java/org/apache/flink/deployment/HeavyDeploymentStressTestProgram.java
new file mode 100644
index 000..d65583f98af
--- /dev/null
+++
[jira] [Commented] (FLINK-10720) Add stress deployment end-to-end test
[
https://issues.apache.org/jira/browse/FLINK-10720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676943#comment-16676943
]
ASF GitHub Bot commented on FLINK-10720:
StefanRRichter commented on issue #6994: [FLINK-10720][tests] Add deployment
end-to-end stress test with many …
URL: https://github.com/apache/flink/pull/6994#issuecomment-436309544
Thanks @tillrohrmann! Merging.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add stress deployment end-to-end test
> -
>
> Key: FLINK-10720
> URL: https://issues.apache.org/jira/browse/FLINK-10720
> Project: Flink
> Issue Type: Sub-task
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> In order to test Flink's scalability, I suggest to add an end-to-end test
> which tests the deployment of a job which is very demanding. The job should
> have large {{TaskDeploymentDescriptors}} (e.g. a job using union state or
> having a high degree of parallelism). That way we can test that the
> serialization overhead of the TDDs does not affect the health of the cluster
> (e.g. heartbeats are not affected because the serialization does not happen
> in the main thread).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] StefanRRichter commented on issue #6994: [FLINK-10720][tests] Add deployment end-to-end stress test with many …
StefanRRichter commented on issue #6994: [FLINK-10720][tests] Add deployment end-to-end stress test with many … URL: https://github.com/apache/flink/pull/6994#issuecomment-436309544 Thanks @tillrohrmann! Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-10805) test_confluent_schema_registry.sh and test_sql_client.sh end-to-end test broken
[
https://issues.apache.org/jira/browse/FLINK-10805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10805:
---
Labels: pull-request-available test-stability (was: test-stability)
> test_confluent_schema_registry.sh and test_sql_client.sh end-to-end test
> broken
> ---
>
> Key: FLINK-10805
> URL: https://issues.apache.org/jira/browse/FLINK-10805
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available, test-stability
>
> The end-to-end test {{test_confluent_schema_registry.sh}} and
> {{test_sql_client.sh}} seem to be broken after fixing FLINK-10600.
> I suspect that the kafka-common.sh import has not been adapted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] tillrohrmann opened a new pull request #7035: [BP-1.7][FLINK-10805] Fix failing end-to-end tests
tillrohrmann opened a new pull request #7035: [BP-1.7][FLINK-10805] Fix failing end-to-end tests URL: https://github.com/apache/flink/pull/7035 Backport of #7034 for `release-1.7`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] tillrohrmann opened a new pull request #7034: [FLINK-10805] Fix failing end-to-end tests
tillrohrmann opened a new pull request #7034: [FLINK-10805] Fix failing end-to-end tests URL: https://github.com/apache/flink/pull/7034 ## What is the purpose of the change Fix test_confluent_schema_registry.sh and test_sql_client.sh end-to-end tests which failed because of missing arguments for kafka-common.sh and the newly introduce set -e flag. ## Verifying this change - Manually tested ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10805) test_confluent_schema_registry.sh and test_sql_client.sh end-to-end test broken
[
https://issues.apache.org/jira/browse/FLINK-10805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676937#comment-16676937
]
ASF GitHub Bot commented on FLINK-10805:
tillrohrmann opened a new pull request #7034: [FLINK-10805] Fix failing
end-to-end tests
URL: https://github.com/apache/flink/pull/7034
## What is the purpose of the change
Fix test_confluent_schema_registry.sh and test_sql_client.sh end-to-end
tests which failed
because of missing arguments for kafka-common.sh and the newly introduce set
-e flag.
## Verifying this change
- Manually tested
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> test_confluent_schema_registry.sh and test_sql_client.sh end-to-end test
> broken
> ---
>
> Key: FLINK-10805
> URL: https://issues.apache.org/jira/browse/FLINK-10805
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: pull-request-available, test-stability
>
> The end-to-end test {{test_confluent_schema_registry.sh}} and
> {{test_sql_client.sh}} seem to be broken after fixing FLINK-10600.
> I suspect that the kafka-common.sh import has not been adapted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10771) Replace hard code of job graph file path with config option for FileJobGraphRetriever
[
https://issues.apache.org/jira/browse/FLINK-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676916#comment-16676916
]
vinoyang commented on FLINK-10771:
--
Yes, In AbstractYarnClusterDescriptor, we can explicitly set the value
"job.graph" for "internal.jobgraph-path" and then uniformly retrieve it from
the configuration object.
> Replace hard code of job graph file path with config option for
> FileJobGraphRetriever
> -
>
> Key: FLINK-10771
> URL: https://issues.apache.org/jira/browse/FLINK-10771
> Project: Flink
> Issue Type: Improvement
> Components: YARN
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>
> There is a config option to configure the value :
> {code:java}
> public static final ConfigOption JOB_GRAPH_FILE_PATH = ConfigOptions
>.key("internal.jobgraph-path")
>.defaultValue("job.graph");
> {code}
> However, we used the default hard code in AbstractYarnClusterDescriptor.
> This is just a preliminary refactoring, and I finally recommend that we use
> Zookeeper-based storage. It has been implemented in FLINK-10292.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-8985) End-to-end test: CLI
[ https://issues.apache.org/jira/browse/FLINK-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676906#comment-16676906 ] ASF GitHub Bot commented on FLINK-8985: --- twalthr commented on issue #5863: [FLINK-8985][e2etest] initial support for End-to-end CLI test, excluding YARN test URL: https://github.com/apache/flink/pull/5863#issuecomment-436296096 Thanks for the PR @walterddr. I think the test looks good to merge. I will add some additional safety checks and rebase this code to the current master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > End-to-end test: CLI > > > Key: FLINK-8985 > URL: https://issues.apache.org/jira/browse/FLINK-8985 > Project: Flink > Issue Type: Sub-task > Components: Client, Tests >Affects Versions: 1.5.0 >Reporter: Till Rohrmann >Assignee: Rong Rong >Priority: Major > Labels: pull-request-available > > We should add end-to-end test which verifies that all client commands are > working correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-8985) End-to-end test: CLI
[ https://issues.apache.org/jira/browse/FLINK-8985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-8985: -- Labels: pull-request-available (was: ) > End-to-end test: CLI > > > Key: FLINK-8985 > URL: https://issues.apache.org/jira/browse/FLINK-8985 > Project: Flink > Issue Type: Sub-task > Components: Client, Tests >Affects Versions: 1.5.0 >Reporter: Till Rohrmann >Assignee: Rong Rong >Priority: Major > Labels: pull-request-available > > We should add end-to-end test which verifies that all client commands are > working correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] twalthr commented on issue #5863: [FLINK-8985][e2etest] initial support for End-to-end CLI test, excluding YARN test
twalthr commented on issue #5863: [FLINK-8985][e2etest] initial support for End-to-end CLI test, excluding YARN test URL: https://github.com/apache/flink/pull/5863#issuecomment-436296096 Thanks for the PR @walterddr. I think the test looks good to merge. I will add some additional safety checks and rebase this code to the current master. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (FLINK-10463) Null literal cannot be properly parsed in Java Table API function call
[
https://issues.apache.org/jira/browse/FLINK-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingcan Cui resolved FLINK-10463.
-
Resolution: Fixed
Fixed in 1.8.0 5a4e5e9dcaa83368862d27bd80429235c02ed66e
> Null literal cannot be properly parsed in Java Table API function call
> --
>
> Key: FLINK-10463
> URL: https://issues.apache.org/jira/browse/FLINK-10463
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Reporter: Xingcan Cui
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> For example, the expression `Null(STRING).regexpReplace('oo|ar', '')` throws
> the following exception.
> {code:java}
> org.apache.flink.table.api.ExpressionParserException: Could not parse
> expression at column 13: string matching regex
> `(?i)\Qas\E(?![_$\p{javaJavaIdentifierPart}])' expected but `.' found
> Null(STRING).regexpReplace('oo|ar', '')
> ^
> at
> org.apache.flink.table.expressions.ExpressionParser$.throwError(ExpressionParser.scala:576)
> at
> org.apache.flink.table.expressions.ExpressionParser$.parseExpression(ExpressionParser.scala:569)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10463) Null literal cannot be properly parsed in Java Table API function call
[
https://issues.apache.org/jira/browse/FLINK-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingcan Cui updated FLINK-10463:
Fix Version/s: 1.8.0
> Null literal cannot be properly parsed in Java Table API function call
> --
>
> Key: FLINK-10463
> URL: https://issues.apache.org/jira/browse/FLINK-10463
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Reporter: Xingcan Cui
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.8.0
>
>
> For example, the expression `Null(STRING).regexpReplace('oo|ar', '')` throws
> the following exception.
> {code:java}
> org.apache.flink.table.api.ExpressionParserException: Could not parse
> expression at column 13: string matching regex
> `(?i)\Qas\E(?![_$\p{javaJavaIdentifierPart}])' expected but `.' found
> Null(STRING).regexpReplace('oo|ar', '')
> ^
> at
> org.apache.flink.table.expressions.ExpressionParser$.throwError(ExpressionParser.scala:576)
> at
> org.apache.flink.table.expressions.ExpressionParser$.parseExpression(ExpressionParser.scala:569)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10463) Null literal cannot be properly parsed in Java Table API function call
[
https://issues.apache.org/jira/browse/FLINK-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676893#comment-16676893
]
ASF GitHub Bot commented on FLINK-10463:
asfgit closed pull request #6888: [FLINK-10463][table] Null literal cannot be
properly parsed in Java Table API function call
URL: https://github.com/apache/flink/pull/6888
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/table/functions.md b/docs/dev/table/functions.md
index e47b31772b3..e652c304e93 100644
--- a/docs/dev/table/functions.md
+++ b/docs/dev/table/functions.md
@@ -4186,7 +4186,7 @@ CAST(value AS type)
Returns a new value being cast to type type. See the
supported types here.
-E.g., CAST('42' AS INT) returns 42.
+E.g., CAST('42' AS INT) returns 42; CAST(NULL AS
VARCHAR) returns NULL of type VARCHAR.
@@ -4211,7 +4211,7 @@ ANY.cast(TYPE)
Returns a new ANY being cast to type TYPE. See the
supported types here.
-E.g., '42'.cast(INT) returns 42.
+E.g., '42'.cast(INT) returns 42;
Null(STRING) returns NULL of type STRING.
@@ -4236,7 +4236,7 @@ ANY.cast(TYPE)
Returns a new ANY being cast to type TYPE. See the
supported types here.
-E.g., "42".cast(Types.INT) returns 42.
+E.g., "42".cast(Types.INT) returns 42;
Null(Types.STRING) returns NULL of type STRING.
diff --git a/docs/dev/table/tableApi.md b/docs/dev/table/tableApi.md
index 6c4e1be0cf7..3f599a8f5a2 100644
--- a/docs/dev/table/tableApi.md
+++ b/docs/dev/table/tableApi.md
@@ -1738,11 +1738,13 @@ term = product , [ ( "+" | "-" ) , product ] ;
product = unary , [ ( "*" | "/" | "%") , unary ] ;
-unary = [ "!" | "-" ] , composite ;
+unary = [ "!" | "-" | "+" ] , composite ;
-composite = over | nullLiteral | suffixed | atom ;
+composite = over | suffixed | nullLiteral | prefixed | atom ;
-suffixed = interval | cast | as | if | functionCall ;
+suffixed = interval | suffixAs | suffixCast | suffixIf | suffixDistinct |
suffixFunctionCall ;
+
+prefixed = prefixAs | prefixCast | prefixIf | prefixDistinct |
prefixFunctionCall ;
interval = timeInterval | rowInterval ;
@@ -1750,15 +1752,27 @@ timeInterval = composite , "." , ("year" | "years" |
"quarter" | "quarters" | "m
rowInterval = composite , "." , "rows" ;
-cast = composite , ".cast(" , dataType , ")" ;
+suffixCast = composite , ".cast(" , dataType , ")" ;
+
+prefixCast = "cast(" , expression , dataType , ")" ;
dataType = "BYTE" | "SHORT" | "INT" | "LONG" | "FLOAT" | "DOUBLE" | "BOOLEAN"
| "STRING" | "DECIMAL" | "SQL_DATE" | "SQL_TIME" | "SQL_TIMESTAMP" |
"INTERVAL_MONTHS" | "INTERVAL_MILLIS" | ( "MAP" , "(" , dataType , "," ,
dataType , ")" ) | ( "PRIMITIVE_ARRAY" , "(" , dataType , ")" ) | (
"OBJECT_ARRAY" , "(" , dataType , ")" ) ;
-as = composite , ".as(" , fieldReference , ")" ;
+suffixAs = composite , ".as(" , fieldReference , ")" ;
+
+prefixAs = "as(" , expression, fieldReference , ")" ;
+
+suffixIf = composite , ".?(" , expression , "," , expression , ")" ;
+
+prefixIf = "?(" , expression , "," , expression , "," , expression , ")" ;
+
+suffixDistinct = composite , "distinct.()" ;
+
+prefixDistinct = functionIdentifier , ".distinct" , [ "(" , [ expression , {
"," , expression } ] , ")" ] ;
-if = composite , ".?(" , expression , "," , expression , ")" ;
+suffixFunctionCall = composite , "." , functionIdentifier , [ "(" , [
expression , { "," , expression } ] , ")" ] ;
-functionCall = composite , "." , functionIdentifier , [ "(" , [ expression , {
"," , expression } ] , ")" ] ;
+prefixFunctionCall = functionIdentifier , [ "(" , [ expression , { "," ,
expression } ] , ")" ] ;
atom = ( "(" , expression , ")" ) | literal | fieldReference ;
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
index 0b84f9490a1..7fd9309b5db 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
@@ -455,7 +455,7 @@ object ExpressionParser extends JavaTokenParsers with
PackratParsers {
// suffix/prefix composite
- lazy val composite: PackratParser[Expression] = over | nullLiteral |
suffixed | prefixed | atom |
+ lazy val composite: PackratParser[Expression] = over | suffixed |
nullLiteral | prefixed | atom |
failure("Composite expression expected.")
// unary ops
[GitHub] asfgit closed pull request #6888: [FLINK-10463][table] Null literal cannot be properly parsed in Java Table API function call
asfgit closed pull request #6888: [FLINK-10463][table] Null literal cannot be
properly parsed in Java Table API function call
URL: https://github.com/apache/flink/pull/6888
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/dev/table/functions.md b/docs/dev/table/functions.md
index e47b31772b3..e652c304e93 100644
--- a/docs/dev/table/functions.md
+++ b/docs/dev/table/functions.md
@@ -4186,7 +4186,7 @@ CAST(value AS type)
Returns a new value being cast to type type. See the
supported types here.
-E.g., CAST('42' AS INT) returns 42.
+E.g., CAST('42' AS INT) returns 42; CAST(NULL AS
VARCHAR) returns NULL of type VARCHAR.
@@ -4211,7 +4211,7 @@ ANY.cast(TYPE)
Returns a new ANY being cast to type TYPE. See the
supported types here.
-E.g., '42'.cast(INT) returns 42.
+E.g., '42'.cast(INT) returns 42;
Null(STRING) returns NULL of type STRING.
@@ -4236,7 +4236,7 @@ ANY.cast(TYPE)
Returns a new ANY being cast to type TYPE. See the
supported types here.
-E.g., "42".cast(Types.INT) returns 42.
+E.g., "42".cast(Types.INT) returns 42;
Null(Types.STRING) returns NULL of type STRING.
diff --git a/docs/dev/table/tableApi.md b/docs/dev/table/tableApi.md
index 6c4e1be0cf7..3f599a8f5a2 100644
--- a/docs/dev/table/tableApi.md
+++ b/docs/dev/table/tableApi.md
@@ -1738,11 +1738,13 @@ term = product , [ ( "+" | "-" ) , product ] ;
product = unary , [ ( "*" | "/" | "%") , unary ] ;
-unary = [ "!" | "-" ] , composite ;
+unary = [ "!" | "-" | "+" ] , composite ;
-composite = over | nullLiteral | suffixed | atom ;
+composite = over | suffixed | nullLiteral | prefixed | atom ;
-suffixed = interval | cast | as | if | functionCall ;
+suffixed = interval | suffixAs | suffixCast | suffixIf | suffixDistinct |
suffixFunctionCall ;
+
+prefixed = prefixAs | prefixCast | prefixIf | prefixDistinct |
prefixFunctionCall ;
interval = timeInterval | rowInterval ;
@@ -1750,15 +1752,27 @@ timeInterval = composite , "." , ("year" | "years" |
"quarter" | "quarters" | "m
rowInterval = composite , "." , "rows" ;
-cast = composite , ".cast(" , dataType , ")" ;
+suffixCast = composite , ".cast(" , dataType , ")" ;
+
+prefixCast = "cast(" , expression , dataType , ")" ;
dataType = "BYTE" | "SHORT" | "INT" | "LONG" | "FLOAT" | "DOUBLE" | "BOOLEAN"
| "STRING" | "DECIMAL" | "SQL_DATE" | "SQL_TIME" | "SQL_TIMESTAMP" |
"INTERVAL_MONTHS" | "INTERVAL_MILLIS" | ( "MAP" , "(" , dataType , "," ,
dataType , ")" ) | ( "PRIMITIVE_ARRAY" , "(" , dataType , ")" ) | (
"OBJECT_ARRAY" , "(" , dataType , ")" ) ;
-as = composite , ".as(" , fieldReference , ")" ;
+suffixAs = composite , ".as(" , fieldReference , ")" ;
+
+prefixAs = "as(" , expression, fieldReference , ")" ;
+
+suffixIf = composite , ".?(" , expression , "," , expression , ")" ;
+
+prefixIf = "?(" , expression , "," , expression , "," , expression , ")" ;
+
+suffixDistinct = composite , "distinct.()" ;
+
+prefixDistinct = functionIdentifier , ".distinct" , [ "(" , [ expression , {
"," , expression } ] , ")" ] ;
-if = composite , ".?(" , expression , "," , expression , ")" ;
+suffixFunctionCall = composite , "." , functionIdentifier , [ "(" , [
expression , { "," , expression } ] , ")" ] ;
-functionCall = composite , "." , functionIdentifier , [ "(" , [ expression , {
"," , expression } ] , ")" ] ;
+prefixFunctionCall = functionIdentifier , [ "(" , [ expression , { "," ,
expression } ] , ")" ] ;
atom = ( "(" , expression , ")" ) | literal | fieldReference ;
diff --git
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
index 0b84f9490a1..7fd9309b5db 100644
---
a/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
+++
b/flink-libraries/flink-table/src/main/scala/org/apache/flink/table/expressions/ExpressionParser.scala
@@ -455,7 +455,7 @@ object ExpressionParser extends JavaTokenParsers with
PackratParsers {
// suffix/prefix composite
- lazy val composite: PackratParser[Expression] = over | nullLiteral |
suffixed | prefixed | atom |
+ lazy val composite: PackratParser[Expression] = over | suffixed |
nullLiteral | prefixed | atom |
failure("Composite expression expected.")
// unary ops
diff --git
a/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/expressions/ScalarFunctionsTest.scala
b/flink-libraries/flink-table/src/test/scala/org/apache/flink/table/expressions/ScalarFunctionsTest.scala
index
[jira] [Updated] (FLINK-10805) test_confluent_schema_registry.sh and test_sql_client.sh end-to-end test broken
[
https://issues.apache.org/jira/browse/FLINK-10805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-10805:
--
Description:
The end-to-end test {{test_confluent_schema_registry.sh}} and
{{test_sql_client.sh}} seem to be broken after fixing FLINK-10600.
I suspect that the kafka-common.sh import has not been adapted.
was:
The end-to-end test {{test_confluent_schema_registry.sh}} and
{{test_sql_client.sh}} seem to be broken after fixing FLINK-16000.
I suspect that the kafka-common.sh import has not been adapted.
> test_confluent_schema_registry.sh and test_sql_client.sh end-to-end test
> broken
> ---
>
> Key: FLINK-10805
> URL: https://issues.apache.org/jira/browse/FLINK-10805
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests
>Affects Versions: 1.7.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Major
> Labels: test-stability
>
> The end-to-end test {{test_confluent_schema_registry.sh}} and
> {{test_sql_client.sh}} seem to be broken after fixing FLINK-10600.
> I suspect that the kafka-common.sh import has not been adapted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10771) Replace hard code of job graph file path with config option for FileJobGraphRetriever
[
https://issues.apache.org/jira/browse/FLINK-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676883#comment-16676883
]
Till Rohrmann commented on FLINK-10771:
---
Or better to set a value in the {{Configuration}} in the
{{AbstractYarnClusterDescriptor}}.
> Replace hard code of job graph file path with config option for
> FileJobGraphRetriever
> -
>
> Key: FLINK-10771
> URL: https://issues.apache.org/jira/browse/FLINK-10771
> Project: Flink
> Issue Type: Improvement
> Components: YARN
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>
> There is a config option to configure the value :
> {code:java}
> public static final ConfigOption JOB_GRAPH_FILE_PATH = ConfigOptions
>.key("internal.jobgraph-path")
>.defaultValue("job.graph");
> {code}
> However, we used the default hard code in AbstractYarnClusterDescriptor.
> This is just a preliminary refactoring, and I finally recommend that we use
> Zookeeper-based storage. It has been implemented in FLINK-10292.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9699) Add api to replace registered table
[ https://issues.apache.org/jira/browse/FLINK-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676878#comment-16676878 ] ASF GitHub Bot commented on FLINK-9699: --- yanghua commented on issue #6236: [FLINK-9699] [table] Add api to replace registered table URL: https://github.com/apache/flink/pull/6236#issuecomment-436288232 @hequn8128 I agree with you. Actually, I have given a similar suggestion about `CREATE VIEW` in SQL client. See [FLINK-10420](https://issues.apache.org/jira/browse/FLINK-10420?filter=-1). cc @twalthr This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add api to replace registered table > --- > > Key: FLINK-9699 > URL: https://issues.apache.org/jira/browse/FLINK-9699 > Project: Flink > Issue Type: Improvement >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] yanghua commented on issue #6236: [FLINK-9699] [table] Add api to replace registered table
yanghua commented on issue #6236: [FLINK-9699] [table] Add api to replace registered table URL: https://github.com/apache/flink/pull/6236#issuecomment-436288232 @hequn8128 I agree with you. Actually, I have given a similar suggestion about `CREATE VIEW` in SQL client. See [FLINK-10420](https://issues.apache.org/jira/browse/FLINK-10420?filter=-1). cc @twalthr This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10384) Add Sinh math function supported in Table API and SQL
[ https://issues.apache.org/jira/browse/FLINK-10384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676875#comment-16676875 ] vinoyang commented on FLINK-10384: -- [~pnovi...@gmail.com] I think this issue could be closed. > Add Sinh math function supported in Table API and SQL > - > > Key: FLINK-10384 > URL: https://issues.apache.org/jira/browse/FLINK-10384 > Project: Flink > Issue Type: Sub-task > Components: Table API SQL >Reporter: vinoyang >Assignee: vinoyang >Priority: Minor > Labels: pull-request-available > > like FLINK-10340 for adding Cosh math function -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10771) Replace hard code of job graph file path with config option for FileJobGraphRetriever
[
https://issues.apache.org/jira/browse/FLINK-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676866#comment-16676866
]
vinoyang commented on FLINK-10771:
--
OK, since it may be used, why don't we refactor it as: if it's not explicitly
set, then use the default value of the config option (rather than hard-coded
strings) uniformly?
> Replace hard code of job graph file path with config option for
> FileJobGraphRetriever
> -
>
> Key: FLINK-10771
> URL: https://issues.apache.org/jira/browse/FLINK-10771
> Project: Flink
> Issue Type: Improvement
> Components: YARN
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>
> There is a config option to configure the value :
> {code:java}
> public static final ConfigOption JOB_GRAPH_FILE_PATH = ConfigOptions
>.key("internal.jobgraph-path")
>.defaultValue("job.graph");
> {code}
> However, we used the default hard code in AbstractYarnClusterDescriptor.
> This is just a preliminary refactoring, and I finally recommend that we use
> Zookeeper-based storage. It has been implemented in FLINK-10292.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-10805) test_confluent_schema_registry.sh and test_sql_client.sh end-to-end test broken
Till Rohrmann created FLINK-10805:
-
Summary: test_confluent_schema_registry.sh and test_sql_client.sh
end-to-end test broken
Key: FLINK-10805
URL: https://issues.apache.org/jira/browse/FLINK-10805
Project: Flink
Issue Type: Bug
Components: E2E Tests
Affects Versions: 1.7.0
Reporter: Till Rohrmann
Assignee: Till Rohrmann
The end-to-end test {{test_confluent_schema_registry.sh}} and
{{test_sql_client.sh}} seem to be broken after fixing FLINK-16000.
I suspect that the kafka-common.sh import has not been adapted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10802) Add metrics on vertex in job graph picture show
[ https://issues.apache.org/jira/browse/FLINK-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676850#comment-16676850 ] Till Rohrmann commented on FLINK-10802: --- Hi [~lining], I think this could be useful feature but we might also overload the job vertex view. I guess it is all about how the user activates this view. > Add metrics on vertex in job graph picture show > --- > > Key: FLINK-10802 > URL: https://issues.apache.org/jira/browse/FLINK-10802 > Project: Flink > Issue Type: Improvement > Components: Webfrontend >Reporter: lining >Assignee: lining >Priority: Major > Attachments: image-2018-11-06-18-12-16-914.png > > > Hi. Now, if someone want see job metrics, have to click job overview. if we > can add some import metrics on graph, can impore it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10463) Null literal cannot be properly parsed in Java Table API function call
[
https://issues.apache.org/jira/browse/FLINK-10463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676846#comment-16676846
]
ASF GitHub Bot commented on FLINK-10463:
hequn8128 commented on issue #6888: [FLINK-10463][table] Null literal cannot be
properly parsed in Java Table API function call
URL: https://github.com/apache/flink/pull/6888#issuecomment-436276465
Thank you all for the review and suggestions!
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Null literal cannot be properly parsed in Java Table API function call
> --
>
> Key: FLINK-10463
> URL: https://issues.apache.org/jira/browse/FLINK-10463
> Project: Flink
> Issue Type: Bug
> Components: Table API SQL
>Reporter: Xingcan Cui
>Assignee: Hequn Cheng
>Priority: Major
> Labels: pull-request-available
>
> For example, the expression `Null(STRING).regexpReplace('oo|ar', '')` throws
> the following exception.
> {code:java}
> org.apache.flink.table.api.ExpressionParserException: Could not parse
> expression at column 13: string matching regex
> `(?i)\Qas\E(?![_$\p{javaJavaIdentifierPart}])' expected but `.' found
> Null(STRING).regexpReplace('oo|ar', '')
> ^
> at
> org.apache.flink.table.expressions.ExpressionParser$.throwError(ExpressionParser.scala:576)
> at
> org.apache.flink.table.expressions.ExpressionParser$.parseExpression(ExpressionParser.scala:569)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] hequn8128 commented on issue #6888: [FLINK-10463][table] Null literal cannot be properly parsed in Java Table API function call
hequn8128 commented on issue #6888: [FLINK-10463][table] Null literal cannot be properly parsed in Java Table API function call URL: https://github.com/apache/flink/pull/6888#issuecomment-436276465 Thank you all for the review and suggestions! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10771) Replace hard code of job graph file path with config option for FileJobGraphRetriever
[
https://issues.apache.org/jira/browse/FLINK-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676841#comment-16676841
]
Till Rohrmann commented on FLINK-10771:
---
But from the implementation of {{FileJobGraphRetriever}} you cannot tell that
it won't be used. At the moment this might be right but some people might use
the {{StandaloneJobClusterEntrypoint}} differently. Thus, I would not change
this.
> Replace hard code of job graph file path with config option for
> FileJobGraphRetriever
> -
>
> Key: FLINK-10771
> URL: https://issues.apache.org/jira/browse/FLINK-10771
> Project: Flink
> Issue Type: Improvement
> Components: YARN
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>
> There is a config option to configure the value :
> {code:java}
> public static final ConfigOption JOB_GRAPH_FILE_PATH = ConfigOptions
>.key("internal.jobgraph-path")
>.defaultValue("job.graph");
> {code}
> However, we used the default hard code in AbstractYarnClusterDescriptor.
> This is just a preliminary refactoring, and I finally recommend that we use
> Zookeeper-based storage. It has been implemented in FLINK-10292.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9699) Add api to replace registered table
[ https://issues.apache.org/jira/browse/FLINK-9699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676842#comment-16676842 ] ASF GitHub Bot commented on FLINK-9699: --- hequn8128 commented on issue #6236: [FLINK-9699] [table] Add api to replace registered table URL: https://github.com/apache/flink/pull/6236#issuecomment-436275448 @zjffdu Thanks for the PR. I think this feature is very useful. I haven't taken a deep look at the code. Considering the api, how about change these `register()` functions with replace parameter to `registerOrReplaceXXX()` ? It makes our api more alignment with SQL, such as CREATE OR REPLACE. What do you guys think here? @twalthr @yanghua This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add api to replace registered table > --- > > Key: FLINK-9699 > URL: https://issues.apache.org/jira/browse/FLINK-9699 > Project: Flink > Issue Type: Improvement >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] hequn8128 commented on issue #6236: [FLINK-9699] [table] Add api to replace registered table
hequn8128 commented on issue #6236: [FLINK-9699] [table] Add api to replace registered table URL: https://github.com/apache/flink/pull/6236#issuecomment-436275448 @zjffdu Thanks for the PR. I think this feature is very useful. I haven't taken a deep look at the code. Considering the api, how about change these `register()` functions with replace parameter to `registerOrReplaceXXX()` ? It makes our api more alignment with SQL, such as CREATE OR REPLACE. What do you guys think here? @twalthr @yanghua This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaoxuan-wang commented on issue #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md
shaoxuan-wang commented on issue #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md URL: https://github.com/apache/flink/pull/7025#issuecomment-436263199 > @shaoxuan-wang regardless the duplication of this pr(#7025 ) and #7024 I'd like to point out two concerns on this thread. > > 1. would it be better to fix the issue by replacing site link with relative path?(thus keep link issue isolated among versions) > 2. what version(s) should this fix back port to? (I guess all versions with `README.md` link to the site url are broken) > > x. there is another incorrect path of "IntelliJ Setup" in `pom.xml` as modify within #7024 . Thanks @TisonKun . I did not notice there is an existing PR.@xndai can you please close this PR as #7024 already addressed problem reported in FLINK-10797 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10797) "IntelliJ Setup" link is broken in Readme.md
[ https://issues.apache.org/jira/browse/FLINK-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676792#comment-16676792 ] ASF GitHub Bot commented on FLINK-10797: shaoxuan-wang commented on issue #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md URL: https://github.com/apache/flink/pull/7025#issuecomment-436263199 > @shaoxuan-wang regardless the duplication of this pr(#7025 ) and #7024 I'd like to point out two concerns on this thread. > > 1. would it be better to fix the issue by replacing site link with relative path?(thus keep link issue isolated among versions) > 2. what version(s) should this fix back port to? (I guess all versions with `README.md` link to the site url are broken) > > x. there is another incorrect path of "IntelliJ Setup" in `pom.xml` as modify within #7024 . Thanks @TisonKun . I did not notice there is an existing PR.@xndai can you please close this PR as #7024 already addressed problem reported in FLINK-10797 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "IntelliJ Setup" link is broken in Readme.md > > > Key: FLINK-10797 > URL: https://issues.apache.org/jira/browse/FLINK-10797 > Project: Flink > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Xiening Dai >Assignee: TisonKun >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > The link points to > https://github.com/apache/flink/blob/master/docs/internals/ide_setup.md#intellij-idea > which is a 404 not found. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (FLINK-10795) STDDEV_POP error
[
https://issues.apache.org/jira/browse/FLINK-10795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676791#comment-16676791
]
Hequn Cheng commented on FLINK-10795:
-
Hi, I had looked into this issue. I think it is a precision issue.
As for sql, {{ROUND(STDDEV_POP(CAST(lon AS DOUBLE)),2)}}, {{STDDEV_POP}} will
be converted to {{SQRT((SUM(x * x) - SUM( x ) * SUM( x ) / COUNT( x )) / COUNT(
x ))}} in {{AggregateReduceFunctionsRule}} during optimization. However, due to
precision issue, the result of {{SUM(x * x) - SUM( x ) * SUM( x ) / COUNT( x
)}} would be negative which making sqrt return a value of NaN. {{ROUND}} throws
an exception taking NaN as an input.
We may not be able to solve the precision problem. However, I think we can find
ways to avoid the exception. For example, we can change {{SQRT((SUM(x * x) -
SUM( x ) * SUM( x ) / COUNT( x )) / COUNT( x ))}} to
SQRT({color:red}ABS{color}((SUM(x * x) - SUM( x ) * SUM( x ) / COUNT( x ))) /
COUNT( x )) by adding an abs(). What do you guys think?
> STDDEV_POP error
>
>
> Key: FLINK-10795
> URL: https://issues.apache.org/jira/browse/FLINK-10795
> Project: Flink
> Issue Type: Improvement
> Components: Table API SQL
>Affects Versions: 1.6.2
>Reporter: Flavio Pompermaier
>Assignee: winifredtang
>Priority: Major
> Attachments: FlinkTableApiError.java, test.tsv
>
>
> if using STDDEV_POP in the attached job the following error is thrown (with
> Flink 1.6.1):
>
> {code:java}
> Exception in thread "main"
> org.apache.flink.runtime.client.JobExecutionException:
> java.lang.NumberFormatException
> at
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:623)
> at org.apache.flink.client.LocalExecutor.executePlan(LocalExecutor.java:235)
> at
> org.apache.flink.api.java.LocalEnvironment.execute(LocalEnvironment.java:91)
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:816)
> at org.apache.flink.api.java.DataSet.collect(DataSet.java:413)
> at org.apache.flink.api.java.DataSet.print(DataSet.java:1652)
> at
> it.okkam.datalinks.batch.flink.operations.FlinkTableApiError.main(FlinkTableApiError.java:466)
> Caused by: java.lang.NumberFormatException
> at java.math.BigDecimal.(BigDecimal.java:494)
> at java.math.BigDecimal.(BigDecimal.java:383)
> at java.math.BigDecimal.(BigDecimal.java:806)
> at java.math.BigDecimal.valueOf(BigDecimal.java:1274)
> at org.apache.calcite.runtime.SqlFunctions.sround(SqlFunctions.java:1242)
> at DataSetCalcRule$6909.flatMap(Unknown Source)
> at
> org.apache.flink.table.runtime.FlatMapRunner.flatMap(FlatMapRunner.scala:52)
> at
> org.apache.flink.table.runtime.FlatMapRunner.flatMap(FlatMapRunner.scala:31)
> at
> org.apache.flink.runtime.operators.chaining.ChainedFlatMapDriver.collect(ChainedFlatMapDriver.java:80)
> at
> org.apache.flink.runtime.operators.util.metrics.CountingCollector.collect(CountingCollector.java:35)
> at DataSetSingleRowJoinRule$6450.join(Unknown Source)
> at
> org.apache.flink.table.runtime.MapJoinLeftRunner.flatMap(MapJoinLeftRunner.scala:35)
> at
> org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver.java:109)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:503)
> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (FLINK-10419) ClassNotFoundException while deserializing user exceptions from checkpointing
[
https://issues.apache.org/jira/browse/FLINK-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-10419:
---
Labels: pull-request-available (was: )
> ClassNotFoundException while deserializing user exceptions from checkpointing
> -
>
> Key: FLINK-10419
> URL: https://issues.apache.org/jira/browse/FLINK-10419
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination, State Backends, Checkpointing
>Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.6.0, 1.6.1, 1.7.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> It seems that somewhere in the operator's failure handling, we hand a
> user-code exception to the checkpoint coordinator via Java serialization but
> it will then fail during the de-serialization because the class is not
> available. This will result in the following error shadowing the real one:
> {code}
> java.lang.ClassNotFoundException:
> org.apache.flink.streaming.connectors.kafka.FlinkKafka011Exception
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.flink.util.InstantiationUtil.resolveClass(InstantiationUtil.java:76)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1859)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1745)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2033)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2278)
> at
> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:557)
> at java.lang.Throwable.readObject(Throwable.java:914)
> at sun.reflect.GeneratedMethodAccessor158.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)
> at
> org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation.readObject(RemoteRpcInvocation.java:222)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:502)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:489)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:477)
> at
> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:58)
> at
> org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation.deserializeMethodInvocation(RemoteRpcInvocation.java:118)
> at
> org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation.getMethodName(RemoteRpcInvocation.java:59)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:214)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:162)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
>
[jira] [Commented] (FLINK-10419) ClassNotFoundException while deserializing user exceptions from checkpointing
[
https://issues.apache.org/jira/browse/FLINK-10419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676768#comment-16676768
]
ASF GitHub Bot commented on FLINK-10419:
NicoK opened a new pull request #7033: [FLINK-10419][checkpoint] fix
ClassNotFoundException while deserializing user exceptions
URL: https://github.com/apache/flink/pull/7033
## What is the purpose of the change
`DeclineCheckpoint` tries to make sure that it does wrap any exception which
can be and/or contain user code into a `SerializedThrowable`. It also contains
special handling of a few exceptions that are not wrapped into
`SerializedThrowable` (`CheckpointDeclineException` sub-classes are claimed to
not contain user-code) but if these carry user-code exceptions as *suppressed*
exceptions, the deserialization at the JobManager will fail and the RPC call
will not be processed at all - we thus rely on another method to actually get
rid of the checkpoint, e.g. a timeout.
## Brief change log
- remove the special handling of `CheckpointDeclineException` sub-classes
## Verifying this change
This change is already covered by existing tests, such as checkpoint tests
and IT cases.
I also manually verified that no code is relying on checks such as
`instanceof CheckpointDeclineException` or the explicitly mentioned
sub-classes, so that it is ok to replace these with `SerializedThrowable` in
the RPC.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: **no**
- The serializers: **no**
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: **yes**
- The S3 file system connector: **no**
## Documentation
- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **JavaDocs**
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> ClassNotFoundException while deserializing user exceptions from checkpointing
> -
>
> Key: FLINK-10419
> URL: https://issues.apache.org/jira/browse/FLINK-10419
> Project: Flink
> Issue Type: Bug
> Components: Distributed Coordination, State Backends, Checkpointing
>Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.6.0, 1.6.1, 1.7.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.7.0
>
>
> It seems that somewhere in the operator's failure handling, we hand a
> user-code exception to the checkpoint coordinator via Java serialization but
> it will then fail during the de-serialization because the class is not
> available. This will result in the following error shadowing the real one:
> {code}
> java.lang.ClassNotFoundException:
> org.apache.flink.streaming.connectors.kafka.FlinkKafka011Exception
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher.loadClass(Launcher.java:338)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at
> org.apache.flink.util.InstantiationUtil.resolveClass(InstantiationUtil.java:76)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1859)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1745)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2033)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2278)
> at
> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:557)
> at java.lang.Throwable.readObject(Throwable.java:914)
> at sun.reflect.GeneratedMethodAccessor158.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
> at
>
[GitHub] NicoK opened a new pull request #7033: [FLINK-10419][checkpoint] fix ClassNotFoundException while deserializing user exceptions
NicoK opened a new pull request #7033: [FLINK-10419][checkpoint] fix ClassNotFoundException while deserializing user exceptions URL: https://github.com/apache/flink/pull/7033 ## What is the purpose of the change `DeclineCheckpoint` tries to make sure that it does wrap any exception which can be and/or contain user code into a `SerializedThrowable`. It also contains special handling of a few exceptions that are not wrapped into `SerializedThrowable` (`CheckpointDeclineException` sub-classes are claimed to not contain user-code) but if these carry user-code exceptions as *suppressed* exceptions, the deserialization at the JobManager will fail and the RPC call will not be processed at all - we thus rely on another method to actually get rid of the checkpoint, e.g. a timeout. ## Brief change log - remove the special handling of `CheckpointDeclineException` sub-classes ## Verifying this change This change is already covered by existing tests, such as checkpoint tests and IT cases. I also manually verified that no code is relying on checks such as `instanceof CheckpointDeclineException` or the explicitly mentioned sub-classes, so that it is ok to replace these with `SerializedThrowable` in the RPC. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: **yes** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **JavaDocs** This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-10804) Transfer suppressed exceptions with SerializedThrowable
Nico Kruber created FLINK-10804:
---
Summary: Transfer suppressed exceptions with SerializedThrowable
Key: FLINK-10804
URL: https://issues.apache.org/jira/browse/FLINK-10804
Project: Flink
Issue Type: Improvement
Components: Distributed Coordination
Affects Versions: 1.7.0
Reporter: Nico Kruber
Currently, {{SerializedThrowable}} mimics the original exception including the
causes but not the suppressed exceptions. You may argue whether these should
really be transferred as well or whether it would be enough to have them in the
logging of the component which sent the message.
However, it may be surprising to silently drop suppressed exceptions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-10797) "IntelliJ Setup" link is broken in Readme.md
[ https://issues.apache.org/jira/browse/FLINK-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676750#comment-16676750 ] ASF GitHub Bot commented on FLINK-10797: TisonKun edited a comment on issue #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md URL: https://github.com/apache/flink/pull/7025#issuecomment-436249401 @shaoxuan-wang regardless the duplication of this pr(#7025 ) and #7024 I'd like to point out two concerns on this thread. 1. would it be better to fix the issue by replacing site link with relative path?(thus keep link issue isolated among versions) 2. what version(s) should this fix back port to? (I guess all versions with `README.md` link to the site url are broken) x. there is another incorrect path of "IntelliJ Setup" in `pom.xml` as modify within #7024 . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "IntelliJ Setup" link is broken in Readme.md > > > Key: FLINK-10797 > URL: https://issues.apache.org/jira/browse/FLINK-10797 > Project: Flink > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Xiening Dai >Assignee: TisonKun >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > The link points to > https://github.com/apache/flink/blob/master/docs/internals/ide_setup.md#intellij-idea > which is a 404 not found. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] TisonKun edited a comment on issue #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md
TisonKun edited a comment on issue #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md URL: https://github.com/apache/flink/pull/7025#issuecomment-436249401 @shaoxuan-wang regardless the duplication of this pr(#7025 ) and #7024 I'd like to point out two concerns on this thread. 1. would it be better to fix the issue by replacing site link with relative path?(thus keep link issue isolated among versions) 2. what version(s) should this fix back port to? (I guess all versions with `README.md` link to the site url are broken) x. there is another incorrect path of "IntelliJ Setup" in `pom.xml` as modify within #7024 . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10797) "IntelliJ Setup" link is broken in Readme.md
[ https://issues.apache.org/jira/browse/FLINK-10797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676747#comment-16676747 ] ASF GitHub Bot commented on FLINK-10797: TisonKun commented on issue #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md URL: https://github.com/apache/flink/pull/7025#issuecomment-436249401 @shaoxuan-wang regardless the duplication of this pr(#7025 ) and #7024 I'd like to point out two concerns on this thread. 1. would it be better to fix the issue by replacing site link with relative path?(thus keep link issue isolated among versions) 2. what version(s) should this fix back port to? x. there is another incorrect path of "IntelliJ Setup" in `pom.xml` as modify within #7024 . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > "IntelliJ Setup" link is broken in Readme.md > > > Key: FLINK-10797 > URL: https://issues.apache.org/jira/browse/FLINK-10797 > Project: Flink > Issue Type: Bug >Affects Versions: 1.7.0 >Reporter: Xiening Dai >Assignee: TisonKun >Priority: Major > Labels: pull-request-available > Fix For: 1.7.0 > > > The link points to > https://github.com/apache/flink/blob/master/docs/internals/ide_setup.md#intellij-idea > which is a 404 not found. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] TisonKun commented on issue #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md
TisonKun commented on issue #7025: [FLINK-10797] "IntelliJ Setup" link is broken in README.md URL: https://github.com/apache/flink/pull/7025#issuecomment-436249401 @shaoxuan-wang regardless the duplication of this pr(#7025 ) and #7024 I'd like to point out two concerns on this thread. 1. would it be better to fix the issue by replacing site link with relative path?(thus keep link issue isolated among versions) 2. what version(s) should this fix back port to? x. there is another incorrect path of "IntelliJ Setup" in `pom.xml` as modify within #7024 . This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10353) Restoring a KafkaProducer with Semantic.EXACTLY_ONCE from a savepoint written with Semantic.AT_LEAST_ONCE fails with NPE
[
https://issues.apache.org/jira/browse/FLINK-10353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676695#comment-16676695
]
ASF GitHub Bot commented on FLINK-10353:
StefanRRichter commented on issue #7010: [FLINK-10353][kafka] Support change of
transactional semantics in Kaf…
URL: https://github.com/apache/flink/pull/7010#issuecomment-436235901
Addressed the comments. Please have another look.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Restoring a KafkaProducer with Semantic.EXACTLY_ONCE from a savepoint written
> with Semantic.AT_LEAST_ONCE fails with NPE
>
>
> Key: FLINK-10353
> URL: https://issues.apache.org/jira/browse/FLINK-10353
> Project: Flink
> Issue Type: Improvement
> Components: Kafka Connector
>Affects Versions: 1.5.3, 1.6.0
>Reporter: Konstantin Knauf
>Assignee: Stefan Richter
>Priority: Critical
> Labels: pull-request-available
>
> If a KafkaProducer with {{Semantic.EXACTLY_ONCE}} is restored from a
> savepoint written with {{Semantic.AT_LEAST_ONCE}} the job fails on restore
> with the NPE below. This makes it impossible to upgrade an AT_LEAST_ONCE
> pipeline to an EXACTL_ONCE pipeline statefully.
> {quote}
> java.lang.NullPointerException
> at java.util.Hashtable.put(Hashtable.java:460)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initTransactionalProducer(FlinkKafkaProducer011.java:955)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.recoverAndCommit(FlinkKafkaProducer011.java:733)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.recoverAndCommit(FlinkKafkaProducer011.java:93)
> at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.recoverAndCommitInternal(TwoPhaseCommitSinkFunction.java:373)
> at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:333)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:867)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:254)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:748){quote}
> The reason is, that for {{Semantic.AT_LEAST_ONCE}} the snapshotted state of
> the {{TwoPhaseCommitFunction}} is of the form
> "TransactionHolder\{handle=KafkaTransactionState [transactionalId=null,
> producerId=-1, epoch=-1], transactionStartTime=1537175471175}".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] StefanRRichter commented on issue #7010: [FLINK-10353][kafka] Support change of transactional semantics in Kaf…
StefanRRichter commented on issue #7010: [FLINK-10353][kafka] Support change of transactional semantics in Kaf… URL: https://github.com/apache/flink/pull/7010#issuecomment-436235901 Addressed the comments. Please have another look. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676692#comment-16676692 ] ASF GitHub Bot commented on FLINK-10600: yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-436235491 @shaoxuan-wang Can you share your error log with us? The test cases related to kafka connector are not very stable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
yanghua commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-436235491 @shaoxuan-wang Can you share your error log with us? The test cases related to kafka connector are not very stable. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] shaoxuan-wang commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors
shaoxuan-wang commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-436234220 @pnowojski @yanghuaI got this test failure when I rebase the flink master to my own repository which is connected with travis-ci.org. And I did not find the origin travis test result in this thread. Is it possible this is a flaky test? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-10600) Provide End-to-end test cases for modern Kafka connectors
[ https://issues.apache.org/jira/browse/FLINK-10600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676690#comment-16676690 ] ASF GitHub Bot commented on FLINK-10600: shaoxuan-wang commented on issue #6924: [FLINK-10600] Provide End-to-end test cases for modern Kafka connectors URL: https://github.com/apache/flink/pull/6924#issuecomment-436234220 @pnowojski @yanghuaI got this test failure when I rebase the flink master to my own repository which is connected with travis-ci.org. And I did not find the origin travis test result in this thread. Is it possible this is a flaky test? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Provide End-to-end test cases for modern Kafka connectors > - > > Key: FLINK-10600 > URL: https://issues.apache.org/jira/browse/FLINK-10600 > Project: Flink > Issue Type: Sub-task > Components: Kafka Connector >Reporter: vinoyang >Assignee: vinoyang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.7.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (FLINK-10801) Fix sql client integrate elasticsearch connector test failure
[
https://issues.apache.org/jira/browse/FLINK-10801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
vinoyang updated FLINK-10801:
-
Description:
It usually reports :
{code:java}
FAIL SQL Client Elasticsearch Upsert: Output hash mismatch. Got
6187222e109ee9222e6b2f117742070c, expected 982cb32908def9801e781381c1b8a8db.
head hexdump of actual:
000 { \n " h i t s " : { \n
010 " t o t a l " : 3 , \n
020 " m a x _ s c o r e "
030 : 1 . 0 , \n " h i t s
040 " : [ \n { \n
050 " _ i n d e x " :
060 " m y _ u s e r s " , \n
070 " _ t y p e " : "
080 u s e r " , \n "
090 _ i d " : " 1 _ B o b "
0a0 , \n " _ s c o r
0b0 e " : 1 . 0 , \n
0ba
{code}
the actual hash means :
{code:java}
{
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [
{
"_index" : "my_users",
"_type" : "user",
"_id" : "1_Bob ",
"_score" : 1.0,
"_source" : {
"user_id" : 1,
"user_name" : "Bob ",
"user_count" : 1
}
},
{
"_index" : "my_users",
"_type" : "user",
"_id" : "22_Alice",
"_score" : 1.0,
"_source" : {
"user_id" : 22,
"user_name" : "Alice",
"user_count" : 1
}
},
{
"_index" : "my_users",
"_type" : "user",
"_id" : "42_Greg ",
"_score" : 1.0,
"_source" : {
"user_id" : 42,
"user_name" : "Greg ",
"user_count" : 3
}
}
]
}
}
{code}
the expected hash code means :
{code:java}
{
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [
{
"_index" : "my_users",
"_type" : "user",
"_id" : "1_Bob ",
"_score" : 1.0,
"_source" : {
"user_id" : 1,
"user_name" : "Bob ",
"user_count" : 2
}
},
{
"_index" : "my_users",
"_type" : "user",
"_id" : "22_Alice",
"_score" : 1.0,
"_source" : {
"user_id" : 22,
"user_name" : "Alice",
"user_count" : 1
}
},
{
"_index" : "my_users",
"_type" : "user",
"_id" : "42_Greg ",
"_score" : 1.0,
"_source" : {
"user_id" : 42,
"user_name" : "Greg ",
"user_count" : 3
}
}
]
}
}
{code}
It seems that the user count for "Bob" is off by 1.
The speculation is due to the premature acquisition of aggregated statistics
from Elasticsearch.
> Fix sql client integrate elasticsearch connector test failure
> -
>
> Key: FLINK-10801
> URL: https://issues.apache.org/jira/browse/FLINK-10801
> Project: Flink
> Issue Type: Bug
> Components: E2E Tests
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>
> It usually reports :
> {code:java}
> FAIL SQL Client Elasticsearch Upsert: Output hash mismatch. Got
> 6187222e109ee9222e6b2f117742070c, expected 982cb32908def9801e781381c1b8a8db.
> head hexdump of actual:
> 000 { \n " h i t s " : { \n
> 010 " t o t a l " : 3 , \n
> 020 " m a x _ s c o r e "
> 030 : 1 . 0 , \n " h i t s
> 040 " : [ \n { \n
> 050 " _ i n d e x " :
> 060 " m y _ u s e r s " , \n
> 070 " _ t y p e " : "
> 080 u s e r " , \n "
> 090 _ i d " : " 1 _ B o b "
> 0a0 , \n " _ s c o r
> 0b0 e " : 1 . 0 , \n
> 0ba
> {code}
> the actual hash means :
> {code:java}
> {
> "hits" : {
> "total" : 3,
> "max_score" : 1.0,
> "hits" : [
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "1_Bob ",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 1,
> "user_name" : "Bob ",
> "user_count" : 1
> }
> },
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "22_Alice",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 22,
> "user_name" : "Alice",
> "user_count" : 1
> }
> },
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "42_Greg ",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 42,
> "user_name" : "Greg ",
> "user_count" : 3
> }
> }
> ]
> }
> }
> {code}
> the expected hash code means :
> {code:java}
> {
> "hits" : {
> "total" : 3,
> "max_score" : 1.0,
> "hits" : [
> {
> "_index" : "my_users",
> "_type" : "user",
> "_id" : "1_Bob ",
> "_score" : 1.0,
> "_source" : {
> "user_id" : 1,
> "user_name" : "Bob ",
> "user_count" : 2
> }
> },
>
