tillrohrmann commented on a change in pull request #10550: [FLINK-15194]
Support registering directory as cache file in Yarn per-job mode
URL: https://github.com/apache/flink/pull/10550#discussion_r358895494
##########
File path:
flink-yarn-tests/src/test/java/org/apache/flink/yarn/testjob/YarnTestCacheJob.java
##########
@@ -68,16 +70,21 @@ public static JobGraph getDistributedCacheJobGraph() {
@Override
public void open(Configuration config) throws IOException {
// access cached file via RuntimeContext and
DistributedCache
- File cacheFile =
getRuntimeContext().getDistributedCache().getFile("cacheFile");
- FileInputStream inputStream = new
FileInputStream(cacheFile);
+ final File cacheFile =
getRuntimeContext().getDistributedCache().getFile("cacheFile");
+ final FileInputStream inputStream = new
FileInputStream(cacheFile);
properties = new Properties();
properties.load(inputStream);
checkArgument(properties.size() == 2, "The property
file should contains 2 pair of key values");
+
+ final File testDirectory =
getRuntimeContext().getDistributedCache().getFile(TEST_DIRECTORY_NAME);
+ if (!testDirectory.isDirectory()) {
+ throw new RuntimeException(String.format("%s is
not a directory!", testDirectory.getAbsolutePath()));
+ }
Review comment:
Do we have to test this feature via a Yarn test? Wouldn't it be easier to
test the `FileCache` more targeted? For example, it could be done similar to
`FileCacheDirectoriesTest#testDirectoryDownloadedFromBlob`. What we are
interested in is the unpacking logic. The other part, that a registered cached
filed is accessible on Yarn should already be tested by accessing the
`cache.properties` file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
