[
https://issues.apache.org/jira/browse/BEAM-3370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16296999#comment-16296999
]
Łukasz Gajowy commented on BEAM-3370:
-------------------------------------
FYI: [~jasonkuster] [~chamikara]
> Add ability to stage directories with compiled classes to Flink
> ---------------------------------------------------------------
>
> Key: BEAM-3370
> URL: https://issues.apache.org/jira/browse/BEAM-3370
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
> Reporter: Łukasz Gajowy
> Assignee: Aljoscha Krettek
> Priority: Minor
>
> Currently, when _filesToStage_ contain a path to directory with resources,
> flink runner throws a {{"java.io.FileNotFoundException: <path_to_the_dir> (Is
> a directory)"}}. A way to include directory resources would be helpful.
> This "blocker" occurs while trying to run IOITs on flink runner, which
> basically makes it impossible/very inconvenient to run. When the tests are
> run via "mvn verify" command, a "test-classes" *directory* gets detected by
> detectClasspathResourcesToStage() method which in turn causes the above
> error.
> One way to solve this issue is to package the directories to jars with unique
> names and update the paths accordingly before staging the files on flink.
> Something similar is already done in the Dataflow runner
> ([GcsStager|https://github.com/apache/beam/blob/cd186a531aaff0b21cf009b034e1a41f7e7b64af/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java#L74]),
> more specifically in
> [PackageUtil|https://github.com/apache/beam/blob/cd186a531aaff0b21cf009b034e1a41f7e7b64af/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java#L280]
> class. We are able to run the tests on dataflow thanks to that.
> As I checked in a [small experiment of
> mine|https://github.com/lgajowy/beam/commits/spark-and-flink-run-tests],
> providing analogous change makes it possible to run the tests on a Flink
> cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
