dawidwys commented on a change in pull request #14275:
URL: https://github.com/apache/flink/pull/14275#discussion_r535260432



##########
File path: 
flink-end-to-end-tests/flink-end-to-end-tests-common/src/main/java/org/apache/flink/tests/util/flink/FlinkContainer.java
##########
@@ -256,31 +258,83 @@ public FlinkContainer build() throws IOException {
                                                .mapToObj(i -> "localhost")
                                                .collect(Collectors.toList()));
 
-                               ImageFromDockerfile image = new 
ImageFromDockerfile("flink-dist", true)
-                                       .withDockerfileFromBuilder(
-                                               builder -> {
-                                                       builder.from("openjdk:" 
+ getJavaVersionSuffix())
-                                                               
.copy(flinkDistName, "flink")
-                                                               
.copy(flinkDistName + "/conf/workers", "workers")
-                                                               .cmd(FLINK_BIN 
+ "/start-cluster.sh && tail -f /dev/null")
-                                                               .build();
-                                               }
-                                       )
-                                       .withFileFromPath("workers", 
workersFile)
-                                       .withFileFromPath(flinkDistName, 
flinkDist);
+                               // Building the docker image is split into two 
stages:
+                               // 1. build a base image with an immutable 
flink-dist
+                               // 2. based on the base image add any mutable 
files such as e.g. workers files
+                               //
+                               // This lets us save some time for archiving 
and copying big, immutable files
+                               // between tests runs.
+                               String baseImage = buildBaseImage(flinkDist, 
flinkDistName);
+                               ImageFromDockerfile configuredImage = 
buildConfiguredImage(
+                                       flinkDistName,
+                                       workersFile,
+                                       baseImage);
 
                                Optional<Path> logBackupDirectory = 
DISTRIBUTION_LOG_BACKUP_DIRECTORY.get();
                                if (!logBackupDirectory.isPresent()) {
                                        LOG.warn(
                                                "Property {} not set, logs will 
not be backed up in case of test failures.",
                                                
DISTRIBUTION_LOG_BACKUP_DIRECTORY.getPropertyName());
                                }
-                               return new FlinkContainer(image, 
numTaskManagers, logBackupDirectory.orElse(null));
+                               return new FlinkContainer(
+                                       configuredImage,
+                                       numTaskManagers,
+                                       logBackupDirectory.orElse(null));
+                       } catch (Exception e) {
+                               throw new RuntimeException("Could not build the 
flink-dist image", e);
                        } finally {
                                temporaryFolder.delete();
                        }
                }
 
+               private ImageFromDockerfile buildConfiguredImage(
+                               String flinkDistName,
+                               Path workersFile,
+                               String baseImage) {
+                       return new ImageFromDockerfile(
+                               "flink-dist-configured")
+                               .withDockerfileFromBuilder(
+                                       builder -> builder.from(baseImage)
+                                               .copy("workers", flinkDistName 
+ "/conf/workers")

Review comment:
       Actually the order is correct. The first one is `source` the second 
`destination`. I am copying the `workers` file which I am including in the 
docker build context with `.withFileFromPath("workers", workersFile);` into the 
flink distribution folder to be picked up by the `start-cluster` script.
   
   I did find two other bugs with the `workers` file when double checking.
   
   1. First one is that I should've copied it over to `flink/conf/workers` 
instead of `flinkDistName + "/conf/workers"`
   2. I should not delete the temporary folder in the finally block in the 
`build` method as the file is copied to the container during the start up of 
the FlinkContainer which is already after the `build` method.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Reply via email to