[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513199#comment-15513199
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2533


> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512918#comment-15512918
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2533
  
I see, keeping the serializers for now makes probably sense.
It just seems that there are also user functions in there (like fold, etc) 
- those should probably be removed. May mean that we have to inject them back 
into the state descriptor later.

Orthogonal issue, so +1 for this change.

Merging this...


> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512855#comment-15512855
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/2533
  
@StephanEwen at least in the RocksDB backend we could remove user code 
completely. Right now, the only thing that needs to be serialized is the 
TypeSerializer from the ValueDescriptor. It is used in a check that users can 
not provide a descriptor with a different TypeSerializer than the one that was 
used initially. We might think about removing this to support versioning of 
TypeSerializers, but how can we somehow enforce compatibility between them?


> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512838#comment-15512838
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

Github user StefanRRichter commented on a diff in the pull request:

https://github.com/apache/flink/pull/2533#discussion_r80010321
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java
 ---
@@ -266,18 +265,20 @@ public void 
restorePartitionedState(List state) throws Exc
for (int i = 0; i < numKvStates; ++i) {
String stateName = inView.readUTF();
 
-   ObjectInputStream ois = new 
ObjectInputStream(inView);
+   TypeSerializer namespaceSerializer =
+   
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
+   TypeSerializer stateSerializer =
+   
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
 
-   TypeSerializer namespaceSerializer = 
(TypeSerializer) ois.readObject();
-   TypeSerializer stateSerializer = 
(TypeSerializer) ois.readObject();
-   StateTable stateTable = new 
StateTable(stateSerializer,
+   StateTable stateTable = new StateTable(
+   stateSerializer,
namespaceSerializer,
keyGroupRange);
stateTables.put(stateName, stateTable);
kvStatesById.put(i, stateName);
}
 
-   for (int keyGroupIndex = 
keyGroupRange.getStartKeyGroup(); keyGroupIndex <= 
keyGroupRange.getEndKeyGroup(); keyGroupIndex++) {
+   for (int keyGroupIndex = 
keyGroupRange.getStartKeyGroup();  keyGroupIndex <= 
keyGroupRange.getEndKeyGroup(); ++keyGroupIndex) {
--- End diff --

I think i wanted to break this because it exceeds the line limit but then 
decided against it because IntelliJ messed up the formatting for loops. Nothing 
wrong there at all.


> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512773#comment-15512773
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/2533#discussion_r80005604
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/state/heap/HeapKeyedStateBackend.java
 ---
@@ -266,18 +265,20 @@ public void 
restorePartitionedState(List state) throws Exc
for (int i = 0; i < numKvStates; ++i) {
String stateName = inView.readUTF();
 
-   ObjectInputStream ois = new 
ObjectInputStream(inView);
+   TypeSerializer namespaceSerializer =
+   
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
+   TypeSerializer stateSerializer =
+   
InstantiationUtil.deserializeObject(fsDataInputStream, userCodeClassLoader);
 
-   TypeSerializer namespaceSerializer = 
(TypeSerializer) ois.readObject();
-   TypeSerializer stateSerializer = 
(TypeSerializer) ois.readObject();
-   StateTable stateTable = new 
StateTable(stateSerializer,
+   StateTable stateTable = new StateTable(
+   stateSerializer,
namespaceSerializer,
keyGroupRange);
stateTables.put(stateName, stateTable);
kvStatesById.put(i, stateName);
}
 
-   for (int keyGroupIndex = 
keyGroupRange.getStartKeyGroup(); keyGroupIndex <= 
keyGroupRange.getEndKeyGroup(); keyGroupIndex++) {
+   for (int keyGroupIndex = 
keyGroupRange.getStartKeyGroup();  keyGroupIndex <= 
keyGroupRange.getEndKeyGroup(); ++keyGroupIndex) {
--- End diff --

Was this wrong before?


> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512702#comment-15512702
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2533
  
Fix looks good.
Better would probably be to not even have the user code in the checkpoint 
at all.
Can we do that?


> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[Stefan Richter (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512553#comment-15512553
 ] 

Stefan Richter commented on FLINK-4603:
---

Currently, we should still keep the UserCodeClassLoader around in the RocksDB 
backend because we still need to serialize the StateDescriptor, which contains 
the TypeSerializer, so that users can not accidentally create StateDescriptors 
with a wrong TypeSerializer. However, we should consider that TypeSerializer 
can be exchanged (ensuring their compatibility), e.g. to allow different 
serialization versions.

> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512541#comment-15512541
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/2533
  
Please review @tillrohrmann or @aljoscha 


> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512540#comment-15512540
 ] 

ASF GitHub Bot commented on FLINK-4603:
---

GitHub user StefanRRichter opened a pull request:

https://github.com/apache/flink/pull/2533

[FLINK-4603] Fixes: KeyedStateBackend cannot restore user code classes

This PR fixes [FLINK-4603] and introduces a test to protect better against 
future regression.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StefanRRichter/flink backend-classloader-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2533


commit d6b8b0112c6a4cf3f2cbf5eb758599e15d796aab
Author: Stefan Richter 
Date:   2016-09-21T12:55:58Z

[FLINK-4603] KeyedStateBackend can restore user code classes

commit 78b2a4f048bd62e55471a384169304ca46bbbf60
Author: Stefan Richter 
Date:   2016-09-21T15:56:08Z

[FLINK-4603] Test case




> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Stefan Richter
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483601#comment-15483601
 ] 

Till Rohrmann commented on FLINK-4603:
--

That makes sense. 

> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477093#comment-15477093
 ] 

Aljoscha Krettek commented on FLINK-4603:
-

We wouldn't if we always keep everything in serialized form, as is the case for 
the RocksDB backend. In that case, the user code is only necessary when a state 
access comes in and when that happens we're already "in" the user code class 
loader.

> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476941#comment-15476941
 ] 

Till Rohrmann commented on FLINK-4603:
--

Wouldn't we even then need the user code class loader to load user code classes?

> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4603) KeyedStateBackend cannot restore user code classes

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-4603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476881#comment-15476881
 ] 

Aljoscha Krettek commented on FLINK-4603:
-

Yep, the user code {{ClassLoder}} can be retrieved from the {{Environment}} 
passed to {{AbstractStateBackend.restoreKeyedStateBackend()}}. A nicer solution 
would be to completely get rid of Java Serialization in the state serialization.

> KeyedStateBackend cannot restore user code classes
> --
>
> Key: FLINK-4603
> URL: https://issues.apache.org/jira/browse/FLINK-4603
> Project: Flink
>  Issue Type: Bug
>  Components: State Backends, Checkpointing
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Priority: Blocker
> Fix For: 1.2.0
>
>
> A user reported that he cannot restore keyed state which contains user code 
> classes. I suspect that we don't use the user code class loader to 
> deserialize the state.
> The solution seems to be to forward the user code class loader to the 
> {{KeyedStateBackends}} when restoring state.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)