[jira] [Commented] (FLINK-3400) RocksDB Backend does not work when not in Flink lib folder

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15150745#comment-15150745
 ] 

ASF GitHub Bot commented on FLINK-3400:
---

Github user aljoscha closed the pull request at:

https://github.com/apache/flink/pull/1644


> RocksDB Backend does not work when not in Flink lib folder
> --
>
> Key: FLINK-3400
> URL: https://issues.apache.org/jira/browse/FLINK-3400
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.0.0
>
>
> The problem is that the external process runner cannot load the 
> {{HDFSCopyFromLocal}} and {{HDFSCopyToLocal}} classes when the rocksdb 
> backend jar is not in the Flink lib folder. If rocksdb is only in the 
> user-code jar the code cannot be loaded when trying to start the external 
> copy runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3400) RocksDB Backend does not work when not in Flink lib folder

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149394#comment-15149394
 ] 

ASF GitHub Bot commented on FLINK-3400:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1644#issuecomment-184891874
  
Looks good, +1 to merge this


> RocksDB Backend does not work when not in Flink lib folder
> --
>
> Key: FLINK-3400
> URL: https://issues.apache.org/jira/browse/FLINK-3400
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.0.0
>
>
> The problem is that the external process runner cannot load the 
> {{HDFSCopyFromLocal}} and {{HDFSCopyToLocal}} classes when the rocksdb 
> backend jar is not in the Flink lib folder. If rocksdb is only in the 
> user-code jar the code cannot be loaded when trying to start the external 
> copy runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3400) RocksDB Backend does not work when not in Flink lib folder

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148827#comment-15148827
 ] 

ASF GitHub Bot commented on FLINK-3400:
---

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/1644

[FLINK-3400] Move RocksDB Copy Utils to flink-streaming-java

They are not specific to RocksDB, just utilities for copying local
folders to/from HDFS. Moving them to flink-streaming-java means that
they are always in the classpath of the TaskManager, not only in the
user-code jar when using RocksDB. If they are only in the user-code jar
the external process runner cannot find the class files, leading to
ClassNotFoundExceptions.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink rocksdb-fix-copy-utils

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1644.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1644


commit 6dfdf5d643d9df2598b25f61134e24ab37ecb7df
Author: Aljoscha Krettek 
Date:   2016-02-15T12:57:15Z

[FLINK-3400] Move RocksDB Copy Utils to flink-streaming-java

They are not specific to RocksDB, just utilities for copying local
folders to/from HDFS. Moving them to flink-streaming-java means that
they are always in the classpath of the TaskManager, not only in the
user-code jar when using RocksDB. If they are only in the user-code jar
the external process runner cannot find the class files, leading to
ClassNotFoundExceptions.




> RocksDB Backend does not work when not in Flink lib folder
> --
>
> Key: FLINK-3400
> URL: https://issues.apache.org/jira/browse/FLINK-3400
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.0.0
>
>
> The problem is that the external process runner cannot load the 
> {{HDFSCopyFromLocal}} and {{HDFSCopyToLocal}} classes when the rocksdb 
> backend jar is not in the Flink lib folder. If rocksdb is only in the 
> user-code jar the code cannot be loaded when trying to start the external 
> copy runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3400) RocksDB Backend does not work when not in Flink lib folder

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15147323#comment-15147323
 ] 

Aljoscha Krettek commented on FLINK-3400:
-

I see two possibilities for solving this:

 - Move the utilities to a package that is always in the classpath anyways. 
Candidates would be {{flink-core}}, {{flink-runtime}} or 
{{flink-streaming-java}}. I would gravitate towards {{flink-streaming-java}} 
here.
 - Figure out how to get the path to the user-code jar/jars and add that to the 
classpath of the started external process. Are the user-code jars always 
available in a directory on the machine where a Task is running.

I think I'll go with option 1) move to {{flink-streaming-java}}. But what do 
others think?

> RocksDB Backend does not work when not in Flink lib folder
> --
>
> Key: FLINK-3400
> URL: https://issues.apache.org/jira/browse/FLINK-3400
> Project: Flink
>  Issue Type: Bug
>  Components: Streaming
>Affects Versions: 1.0.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.0.0
>
>
> The problem is that the external process runner cannot load the 
> {{HDFSCopyFromLocal}} and {{HDFSCopyToLocal}} classes when the rocksdb 
> backend jar is not in the Flink lib folder. If rocksdb is only in the 
> user-code jar the code cannot be loaded when trying to start the external 
> copy runners.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)