[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479040#comment-16479040
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5847
Closing, because it's already merged.
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.6.0, 1.5.1
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479041#comment-16479041
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou closed the pull request at:
https://github.com/apache/flink/pull/5847
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.6.0, 1.5.1
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477369#comment-16477369
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5847
Sorry for the delay @sihuazhou I have a few minor touchups that I'll apply
while merging.
And thanks for discovering this and fixing it.
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475552#comment-16475552
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5847
cc @StefanRRichter could you please have a look at this...
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16475487#comment-16475487
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5847
cc @tillrohrmann could you please take a look at this?
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473491#comment-16473491
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5847
Could anyone have a look at this...
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469015#comment-16469015
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5847
cc @StephanEwen Could you please have a look at this? I think it's already
for another review.
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467312#comment-16467312
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5847
Hi, @aljoscha will this PR go into 1.5?
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462176#comment-16462176
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5847
Hi @aljoscha does this PR look good to you now?
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458736#comment-16458736
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5847
@aljoscha Thanks for your review, I have addressed your comments.
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451673#comment-16451673
]
Sihua Zhou commented on FLINK-9174:
---
Hi [~aljoscha], could you please have a look at this? Do you think this should
be an issue?
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Critical
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438307#comment-16438307
]
ASF GitHub Bot commented on FLINK-9174:
---
Github user sihuazhou commented on the issue:
https://github.com/apache/flink/pull/5847
Hi, @StephanEwen could you please have a look at this? If you think this is
a a non-issue, I'd like to close this. (sorry for pinging you here, it because
I saw your some comments about ```UserFacingListState``` in
[5356](https://github.com/apache/flink/pull/5356))
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-9174) The type of state created in ProccessWindowFunction.proccess() is inconsistency
[
https://issues.apache.org/jira/browse/FLINK-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438306#comment-16438306
]
ASF GitHub Bot commented on FLINK-9174:
---
GitHub user sihuazhou opened a pull request:
https://github.com/apache/flink/pull/5847
[FLINK-9174][datastream]Fix the type of state created in
ProccessWindowFunction.proccess() is inconsistency
## What is the purpose of the change
This PR fixes the type of state created in
```ProccessWindowFunction.proccess()``` is inconsistency.
Problem detail,
```java
context.windowState().getListState(); // return type is HeapListState or
RocksDBListState
context.globalState().getListState(); // return type is UserFacingListState
This cause the problem in the following code,
```java
Iterable iterableState = listState.get();
if (terableState.iterator().hasNext()) {
for (T value : iterableState) {
value.setRetracting(true);
collector.collect(value);
}
state.clear();
}
```
If the listState is created from context.globalState() then it's fine, but
when it created from context.windowState() this will cause NPE. I met this in
1.3.2 but I found it also affect 1.5.0.
## Brief change log
- modify ```WindowOperator#PerWindowStateStore``` to ensure the type of
state created from `context.windowState().createXXXState()` is consistency with
the state created from `context.globalState().createXXXState()`
## Verifying this change
- add a unit test in ```
WindowOperatorTest#testStateTypeIsConsistencyCreatedFromWindowStateAndGlobalState()```
to guard this change.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sihuazhou/flink
fixStateTypeInconsistencyForWindowOperator
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5847.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5847
commit d92122575c150f7d0b6b7ed48824c490d4de0b64
Author: sihuazhou
Date: 2018-04-14T10:02:24Z
Fix the type of state created from windowState & globalState inconsistency.
> The type of state created in ProccessWindowFunction.proccess() is
> inconsistency
> ---
>
> Key: FLINK-9174
> URL: https://issues.apache.org/jira/browse/FLINK-9174
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
>Affects Versions: 1.5.0
>Reporter: Sihua Zhou
>Assignee: Sihua Zhou
>Priority: Major
> Fix For: 1.5.0
>
>
> The type of state created from windowState and globalState in
> {{ProcessWindowFunction.process()}} is inconsistency. For detail,
> {code}
> context.windowState().getListState(); // return type is HeapListState or
> RocksDBListState
> context.globalState().getListState(); // return type is UserFacingListState
> {code}
> This cause the problem in the following code,
> {code}
> Iterable iterableState = listState.get();
> if (terableState.iterator().hasNext()) {
>for (T value : iterableState) {
> value.setRetracting(true);
> collector.collect(value);
>}
>state.clear();
> }
> {code}
> If the {{listState}} is created from {{context.globalState()}} is fine, but
> when it created from {{context.windowState()}} this will cause NPE. I met
> this in 1.3.2 but I found it also affect 1.5.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
