[flink] branch release-1.13 updated: [FLINK-25468] Copy SST files if they cannot be hard linked in RocksDBHandle.restoreInstanceDirectoryFromPath

[trohrmann]

This is an automated email from the ASF dual-hosted git repository.

trohrmann pushed a commit to branch release-1.13
in repository https://gitbox.apache.org/repos/asf/flink.git


The following commit(s) were added to refs/heads/release-1.13 by this push:
     new eace77b  [FLINK-25468] Copy SST files if they cannot be hard linked in 
RocksDBHandle.restoreInstanceDirectoryFromPath
eace77b is described below

commit eace77b224e8981f12361b1eae3ec647f0fffdf2
Author: Till Rohrmann <trohrm...@apache.org>
AuthorDate: Tue Oct 19 23:27:10 2021 +0200

    [FLINK-25468] Copy SST files if they cannot be hard linked in 
RocksDBHandle.restoreInstanceDirectoryFromPath
    
    The RocksDBHandle.restoreInstanceDirectoryFromPath hard links SST files. If 
this operation does not
    work because the source directory is on a different file system then it 
will now copy the file over.
    
    This closes #18222.
---
 .../streaming/state/restore/RocksDBHandle.java     | 25 +++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git 
a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/restore/RocksDBHandle.java
 
b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/restore/RocksDBHandle.java
index 5a37db0..9e9a12d 100644
--- 
a/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/restore/RocksDBHandle.java
+++ 
b/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/restore/RocksDBHandle.java
@@ -202,12 +202,27 @@ class RocksDBHandle implements AutoCloseable {
             final String fileName = file.getFileName().toString();
             final Path targetFile = instanceRocksDBDirectory.resolve(fileName);
             if (fileName.endsWith(SST_FILE_SUFFIX)) {
-                // hardlink'ing the immutable sst-files.
-                Files.createLink(targetFile, file);
-            } else {
-                // true copy for all other files.
-                Files.copy(file, targetFile, 
StandardCopyOption.REPLACE_EXISTING);
+                try {
+                    // hardlink'ing the immutable sst-files.
+                    Files.createLink(targetFile, file);
+                    continue;
+                } catch (IOException ioe) {
+                    final String logMessage =
+                            String.format(
+                                    "Could not hard link sst file %s. Trying 
to copy it over. This might "
+                                            + "increase the recovery time. In 
order to avoid this, configure "
+                                            + "RocksDB's working directory and 
the local state directory to be on the same volume.",
+                                    fileName);
+                    if (logger.isDebugEnabled()) {
+                        logger.debug(logMessage, ioe);
+                    } else {
+                        logger.info(logMessage);
+                    }
+                }
             }
+
+            // true copy for all other files and files that could not be hard 
linked.
+            Files.copy(file, targetFile, StandardCopyOption.REPLACE_EXISTING);
         }
     }