dawidwys commented on a change in pull request #14275: URL: https://github.com/apache/flink/pull/14275#discussion_r535260432
########## File path: flink-end-to-end-tests/flink-end-to-end-tests-common/src/main/java/org/apache/flink/tests/util/flink/FlinkContainer.java ########## @@ -256,31 +258,83 @@ public FlinkContainer build() throws IOException { .mapToObj(i -> "localhost") .collect(Collectors.toList())); - ImageFromDockerfile image = new ImageFromDockerfile("flink-dist", true) - .withDockerfileFromBuilder( - builder -> { - builder.from("openjdk:" + getJavaVersionSuffix()) - .copy(flinkDistName, "flink") - .copy(flinkDistName + "/conf/workers", "workers") - .cmd(FLINK_BIN + "/start-cluster.sh && tail -f /dev/null") - .build(); - } - ) - .withFileFromPath("workers", workersFile) - .withFileFromPath(flinkDistName, flinkDist); + // Building the docker image is split into two stages: + // 1. build a base image with an immutable flink-dist + // 2. based on the base image add any mutable files such as e.g. workers files + // + // This lets us save some time for archiving and copying big, immutable files + // between tests runs. + String baseImage = buildBaseImage(flinkDist, flinkDistName); + ImageFromDockerfile configuredImage = buildConfiguredImage( + flinkDistName, + workersFile, + baseImage); Optional<Path> logBackupDirectory = DISTRIBUTION_LOG_BACKUP_DIRECTORY.get(); if (!logBackupDirectory.isPresent()) { LOG.warn( "Property {} not set, logs will not be backed up in case of test failures.", DISTRIBUTION_LOG_BACKUP_DIRECTORY.getPropertyName()); } - return new FlinkContainer(image, numTaskManagers, logBackupDirectory.orElse(null)); + return new FlinkContainer( + configuredImage, + numTaskManagers, + logBackupDirectory.orElse(null)); + } catch (Exception e) { + throw new RuntimeException("Could not build the flink-dist image", e); } finally { temporaryFolder.delete(); } } + private ImageFromDockerfile buildConfiguredImage( + String flinkDistName, + Path workersFile, + String baseImage) { + return new ImageFromDockerfile( + "flink-dist-configured") + .withDockerfileFromBuilder( + builder -> builder.from(baseImage) + .copy("workers", flinkDistName + "/conf/workers") Review comment: Actually the order is correct. The first one is `source` the second `destination`. I am copying the `workers` file which I am including in the docker build context with `.withFileFromPath("workers", workersFile);` into the flink distribution folder to be picked up by the `start-cluster` script. I did find two other bugs with the `workers` file when double checking. 1. First one is that I should've copied it over to `flink/conf/workers` instead of `flinkDistName + "/conf/workers"` 2. I should not delete the temporary folder in the finally block in the `build` method as the file is copied to the container during the start up of the FlinkContainer which is already after the `build` method. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org