luoluoyuyu commented on code in PR #19258:
URL: https://github.com/apache/iotdb/pull/19258#discussion_r4560860111


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileDirPathMapping.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.resource.tsfile;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.db.i18n.DataNodePipeMessages;
+
+import org.apache.tsfile.utils.FSUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Resolves the directory that holds the hardlinked or copied files of a 
TsFile.
+ *
+ * <p>The directory is always a subdirectory of the data directory that 
contains the TsFile, i.e.
+ * {@code 
<dataDir>/<pipe_hardlink_base_dir_name>/<pipe_hardlink_tsfile_dir_name>[/<pipeName>]}.
+ * Since a hardlink can only be created on the same file system as the source 
file, this keeps the
+ * hardlink on the same disk as the TsFile. IoTDB may place the TsFiles of one 
DataRegion on
+ * different disks, because every new TsFile is allocated by {@code 
TierManager}/{@code
+ * FolderManager} according to {@code dn_multi_dir_strategy}. Therefore the 
target directory can not
+ * be chosen per DataRegion or per pipe, but must be chosen per TsFile.
+ *
+ * <p>TsFiles on non-local storage (HDFS, object storage) can not be 
hardlinked nor copied by the
+ * local file system APIs. They are returned as is, i.e., pipe reads them in 
place instead of
+ * pinning a hardlink.
+ */
+class PipeTsFileDirPathMapping {
+
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(PipeTsFileDirPathMapping.class);
+
+  /** Canonical path of every local data directory -> canonical path of its 
pipe base directory. */
+  private final Map<String, String> dataDirPathToPipeBaseDirPath = new 
LinkedHashMap<>();
+
+  private final String pipeHardlinkBaseDirName;
+  private final String pipeHardlinkTsFileDirName;
+
+  PipeTsFileDirPathMapping(
+      final @Nonnull String[] localDataDirs,
+      final @Nonnull String pipeHardlinkBaseDirName,
+      final @Nonnull String pipeHardlinkTsFileDirName) {
+    this.pipeHardlinkBaseDirName = pipeHardlinkBaseDirName;
+    this.pipeHardlinkTsFileDirName = pipeHardlinkTsFileDirName;
+
+    for (final String dataDir : localDataDirs) {
+      final String canonicalDataDir = canonicalPath(dataDir);
+      dataDirPathToPipeBaseDirPath.put(
+          canonicalDataDir, canonicalDataDir + File.separator + 
pipeHardlinkBaseDirName);
+    }
+    LOGGER.info(
+        
DataNodePipeMessages.LOG_PIPE_HARDLINK_DIRECTORIES_ARE_RESOLVED_AS_ARG_1DAB3D0F,
+        dataDirPathToPipeBaseDirPath);
+  }
+
+  /**
+   * Given a TsFile, a .resource file or a .mod file, returns the hardlink or 
the copied file in the
+   * pipe directory on the same disk.
+   *
+   * <p>If the given file is not on a local file system, it is returned as is.
+   */
+  File getHardlinkOrCopiedFile(final File file, final @Nullable String 
pipeName)
+      throws IOException {
+    if (!isLocalFile(file)) {
+      return file;
+    }
+    // The file has already been moved into a pipe directory, e.g., the 
historical extractor pins
+    // the pipe file of another pipe. It must not be linked or copied again.
+    if (isInPipeTsFileDir(file)) {
+      return file;

Review Comment:
   Fixed in 0555f6b.
   
   `PipeTsFileDirPathMapping.getHardlinkOrCopiedFile` resolves the target 
directory first, and returns the input as is only when its parent is exactly 
the pipe directory of the asking scope (the public `pipe/tsfile` for a `null` 
pipe name, or `pipe/tsfile/<pipeName>` for a pipe). A file owned by another 
scope -- i.e. the assigner's public `pipe/tsfile/f.tsfile` handed over to a 
pipe -- is now linked into `pipe/tsfile/<pipeName>/f.tsfile`, so the pipe owns 
its own file and its own reference count, 
`increasePublicReference()`/`getCommonFilePath()` keeps bumping the real 
assigner file, and `cleanPipeTsFileDir` only ever records the directory of the 
pipe that is actually being dropped.
   
   Added `testPublicPipeFileIsLinkedIntoTheRequestingPipe`, which asserts the 
public file of a `null`-pipe pin is `pipe/tsfile/<rel>`, that a pipe gets 
`pipe/tsfile/<pipe>/<rel>`, that two pipes get different files, and that a file 
already in the asking scope is neither re-linked nor nested.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java:
##########
@@ -130,13 +136,14 @@ private File increaseFileReference(
       } else {
         resultFile =
             isTsFile
-                ? FileUtils.createHardLink(source, hardlinkOrCopiedFile)
+                ? FileUtils.createHardLinkOrCopy(source, hardlinkOrCopiedFile)

Review Comment:
   Fixed in 0555f6b, but by failing the pin instead of waiting.
   
   `createHardLinkOrCopy` now refuses the copy fallback when the source 
resource is not closed yet (`resource != null && !resource.isClosed()`) and 
throws an explicit error carrying the link failure as a suppressed exception. 
The TsFile is then extracted in a later round: `shouldExtractTsFileResource` 
keeps selecting unclosed/closing resources, and a failed pin does not advance 
the pipe progress index, so nothing is lost and no truncated copy is ever 
pinned or reused.
   
   I deliberately did not wait for the close inside `pinTsFileResource`: it is 
called under `tsFileManager.readLock()`, while the close path needs 
`resourceListLock.writeLock()`, so blocking there would deadlock the close that 
the wait depends on.
   
   Added `testUnclosedTsFileIsNotCopiedWhenHardlinkFails`: the pipe directory 
is blocked by a regular file to force a hardlink failure; an unclosed resource 
must fail with the new message (link failure suppressed), while the same setup 
with a closed resource takes the copy path and fails for a different reason.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to