dawidwys commented on a change in pull request #14275:
URL: https://github.com/apache/flink/pull/14275#discussion_r535260432
##########
File path:
flink-end-to-end-tests/flink-end-to-end-tests-common/src/main/java/org/apache/flink/tests/util/flink/FlinkContainer.java
##########
@@ -256,31 +258,83 @@ public FlinkContainer build() throws IOException {
.mapToObj(i -> "localhost")
.collect(Collectors.toList()));
- ImageFromDockerfile image = new
ImageFromDockerfile("flink-dist", true)
- .withDockerfileFromBuilder(
- builder -> {
- builder.from("openjdk:"
+ getJavaVersionSuffix())
-
.copy(flinkDistName, "flink")
-
.copy(flinkDistName + "/conf/workers", "workers")
- .cmd(FLINK_BIN
+ "/start-cluster.sh && tail -f /dev/null")
- .build();
- }
- )
- .withFileFromPath("workers",
workersFile)
- .withFileFromPath(flinkDistName,
flinkDist);
+ // Building the docker image is split into two
stages:
+ // 1. build a base image with an immutable
flink-dist
+ // 2. based on the base image add any mutable
files such as e.g. workers files
+ //
+ // This lets us save some time for archiving
and copying big, immutable files
+ // between tests runs.
+ String baseImage = buildBaseImage(flinkDist,
flinkDistName);
+ ImageFromDockerfile configuredImage =
buildConfiguredImage(
+ flinkDistName,
+ workersFile,
+ baseImage);
Optional<Path> logBackupDirectory =
DISTRIBUTION_LOG_BACKUP_DIRECTORY.get();
if (!logBackupDirectory.isPresent()) {
LOG.warn(
"Property {} not set, logs will
not be backed up in case of test failures.",
DISTRIBUTION_LOG_BACKUP_DIRECTORY.getPropertyName());
}
- return new FlinkContainer(image,
numTaskManagers, logBackupDirectory.orElse(null));
+ return new FlinkContainer(
+ configuredImage,
+ numTaskManagers,
+ logBackupDirectory.orElse(null));
+ } catch (Exception e) {
+ throw new RuntimeException("Could not build the
flink-dist image", e);
} finally {
temporaryFolder.delete();
}
}
+ private ImageFromDockerfile buildConfiguredImage(
+ String flinkDistName,
+ Path workersFile,
+ String baseImage) {
+ return new ImageFromDockerfile(
+ "flink-dist-configured")
+ .withDockerfileFromBuilder(
+ builder -> builder.from(baseImage)
+ .copy("workers", flinkDistName
+ "/conf/workers")
Review comment:
Actually the order is correct. The first one is `source` the second
`destination`. I am copying the `workers` file which I am including in the
docker build context with `.withFileFromPath("workers", workersFile);` into the
flink distribution folder to be picked up by the `start-cluster` script.
I did find two bugs with the `workers` file when double checking.
1. First one is that I should've copied it over to `flink/conf/workers`
instead of `flinkDistName + "/conf/workers"`
2. I should not delete the temporary folder in the finally block in the
`build` method as the file is copied to the container during the start up of
the FlinkContainer which is already after the `build` method.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
