Caideyipi commented on code in PR #19258:
URL: https://github.com/apache/iotdb/pull/19258#discussion_r4560639301


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileDirPathMapping.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.resource.tsfile;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.db.i18n.DataNodePipeMessages;
+
+import org.apache.tsfile.utils.FSUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Resolves the directory that holds the hardlinked or copied files of a 
TsFile.
+ *
+ * <p>The directory is always a subdirectory of the data directory that 
contains the TsFile, i.e.
+ * {@code 
<dataDir>/<pipe_hardlink_base_dir_name>/<pipe_hardlink_tsfile_dir_name>[/<pipeName>]}.
+ * Since a hardlink can only be created on the same file system as the source 
file, this keeps the
+ * hardlink on the same disk as the TsFile. IoTDB may place the TsFiles of one 
DataRegion on
+ * different disks, because every new TsFile is allocated by {@code 
TierManager}/{@code
+ * FolderManager} according to {@code dn_multi_dir_strategy}. Therefore the 
target directory can not
+ * be chosen per DataRegion or per pipe, but must be chosen per TsFile.
+ *
+ * <p>TsFiles on non-local storage (HDFS, object storage) can not be 
hardlinked nor copied by the
+ * local file system APIs. They are returned as is, i.e., pipe reads them in 
place instead of
+ * pinning a hardlink.
+ */
+class PipeTsFileDirPathMapping {
+
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(PipeTsFileDirPathMapping.class);
+
+  /** Canonical path of every local data directory -> canonical path of its 
pipe base directory. */
+  private final Map<String, String> dataDirPathToPipeBaseDirPath = new 
LinkedHashMap<>();
+
+  private final String pipeHardlinkBaseDirName;
+  private final String pipeHardlinkTsFileDirName;
+
+  PipeTsFileDirPathMapping(
+      final @Nonnull String[] localDataDirs,
+      final @Nonnull String pipeHardlinkBaseDirName,
+      final @Nonnull String pipeHardlinkTsFileDirName) {
+    this.pipeHardlinkBaseDirName = pipeHardlinkBaseDirName;
+    this.pipeHardlinkTsFileDirName = pipeHardlinkTsFileDirName;
+
+    for (final String dataDir : localDataDirs) {
+      final String canonicalDataDir = canonicalPath(dataDir);
+      dataDirPathToPipeBaseDirPath.put(
+          canonicalDataDir, canonicalDataDir + File.separator + 
pipeHardlinkBaseDirName);
+    }
+    LOGGER.info(
+        
DataNodePipeMessages.LOG_PIPE_HARDLINK_DIRECTORIES_ARE_RESOLVED_AS_ARG_1DAB3D0F,
+        dataDirPathToPipeBaseDirPath);
+  }
+
+  /**
+   * Given a TsFile, a .resource file or a .mod file, returns the hardlink or 
the copied file in the
+   * pipe directory on the same disk.
+   *
+   * <p>If the given file is not on a local file system, it is returned as is.
+   */
+  File getHardlinkOrCopiedFile(final File file, final @Nullable String 
pipeName)
+      throws IOException {
+    if (!isLocalFile(file)) {
+      return file;
+    }
+    // The file has already been moved into a pipe directory, e.g., the 
historical extractor pins
+    // the pipe file of another pipe. It must not be linked or copied again.
+    if (isInPipeTsFileDir(file)) {
+      return file;

Review Comment:
   [P1] Preserve the requested pipe's ownership when resolving an existing pipe 
file
   
   This also matches the assigner's public file, not just a file already owned 
by the requested `pipeName`. In ordinary realtime extraction, 
`PipeDataRegionAssigner` first pins `/data/pipe/tsfile/f.tsfile` with a null 
pipe name, and then shallow-copies that path into each pipe's event. Returning 
it unchanged here makes each private resource own that same public pathname. 
Meanwhile, `increasePublicReference()` calls `getCommonFilePath()` and pins 
`/data/pipe/f.tsfile`, so it does not increase the reference count of the 
actual assigner file. `onAssignedHook()` can therefore delete 
`/data/pipe/tsfile/f.tsfile` while the child events are still queued. The 
recorded cleanup directory is also the shared `tsfile` directory, so dropping 
one such pipe can move aside other pipes' files. Resolve to the requested 
pipe's directory unless the input already belongs to that exact ownership 
scope, and cover the null-pipe-to-named-pipe handoff in a regression test.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java:
##########
@@ -100,6 +99,13 @@ private File increaseFileReference(
       final @Nullable String pipeName,
       final @Nullable File sourceFile)
       throws IOException {
+    // Files on non-local file systems (e.g., HDFS or object storage) can not 
be hardlinked or
+    // copied by the local file system APIs. They are read in place, and must 
never be deleted by
+    // the reference count, so they are not tracked at all.
+    if (!isLocalFile(file)) {
+      return file;

Review Comment:
   [P1] Keep non-local source files protected until their events are released
   
   Returning success without pinning leaves queued historical events dependent 
on a source file that the storage engine can still remove. Historical 
extraction only holds the TsFileManager read lock during collection; after 
that, compaction can call `TsFileResource.remove()`, which deletes the original 
through its filesystem factory. The hardlink provides lifetime protection for 
local TsFiles; returning the original remote file does not provide equivalent 
protection. This new branch retains neither a separate file nor a 
storage-engine lifetime guard, so a remote TsFile can disappear before its 
event is supplied or read, even with a filesystem-aware parser. Avoiding 
deletion by the pipe's own reference counter does not protect against 
storage-engine deletion. Retain an appropriate source-file guard for the full 
event lifetime, or stage a stable copy before treating the pin as successful.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java:
##########
@@ -130,13 +136,14 @@ private File increaseFileReference(
       } else {
         resultFile =
             isTsFile
-                ? FileUtils.createHardLink(source, hardlinkOrCopiedFile)
+                ? FileUtils.createHardLinkOrCopy(source, hardlinkOrCopiedFile)

Review Comment:
   [P1] Wait for the source TsFile to finish closing before using a copy
   
   The new fallback is not equivalent to a hardlink for an unclosed TsFile. For 
an ordinary pipe, 
`PipeHistoricalDataRegionTsFileAndDeletionSource.flushTsFilesForExtraction()` 
calls `asyncCloseAllWorkingTsFileProcessors()` and immediately calls 
`pinTsFileResource()` for selected closing resources. If hardlink creation 
fails, this call now copies the file while its final flush or 
`writer.endFile()` may still be pending. Later, `waitForTsFileClose()` waits 
for the original resource to close, but the copied file never receives those 
final writes, and subsequent reference acquisitions reuse the existing copy. 
This can permanently pin and transfer a truncated TsFile. Coordinate the 
fallback with the source-close lifecycle at the caller, and add a case with a 
delayed close and forced hardlink failure.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileDirPathMapping.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.resource.tsfile;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.db.i18n.DataNodePipeMessages;
+
+import org.apache.tsfile.utils.FSUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Resolves the directory that holds the hardlinked or copied files of a 
TsFile.
+ *
+ * <p>The directory is always a subdirectory of the data directory that 
contains the TsFile, i.e.
+ * {@code 
<dataDir>/<pipe_hardlink_base_dir_name>/<pipe_hardlink_tsfile_dir_name>[/<pipeName>]}.
+ * Since a hardlink can only be created on the same file system as the source 
file, this keeps the
+ * hardlink on the same disk as the TsFile. IoTDB may place the TsFiles of one 
DataRegion on
+ * different disks, because every new TsFile is allocated by {@code 
TierManager}/{@code
+ * FolderManager} according to {@code dn_multi_dir_strategy}. Therefore the 
target directory can not
+ * be chosen per DataRegion or per pipe, but must be chosen per TsFile.
+ *
+ * <p>TsFiles on non-local storage (HDFS, object storage) can not be 
hardlinked nor copied by the
+ * local file system APIs. They are returned as is, i.e., pipe reads them in 
place instead of
+ * pinning a hardlink.
+ */
+class PipeTsFileDirPathMapping {
+
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(PipeTsFileDirPathMapping.class);
+
+  /** Canonical path of every local data directory -> canonical path of its 
pipe base directory. */
+  private final Map<String, String> dataDirPathToPipeBaseDirPath = new 
LinkedHashMap<>();
+
+  private final String pipeHardlinkBaseDirName;
+  private final String pipeHardlinkTsFileDirName;
+
+  PipeTsFileDirPathMapping(
+      final @Nonnull String[] localDataDirs,
+      final @Nonnull String pipeHardlinkBaseDirName,
+      final @Nonnull String pipeHardlinkTsFileDirName) {
+    this.pipeHardlinkBaseDirName = pipeHardlinkBaseDirName;
+    this.pipeHardlinkTsFileDirName = pipeHardlinkTsFileDirName;
+
+    for (final String dataDir : localDataDirs) {
+      final String canonicalDataDir = canonicalPath(dataDir);
+      dataDirPathToPipeBaseDirPath.put(
+          canonicalDataDir, canonicalDataDir + File.separator + 
pipeHardlinkBaseDirName);
+    }
+    LOGGER.info(
+        
DataNodePipeMessages.LOG_PIPE_HARDLINK_DIRECTORIES_ARE_RESOLVED_AS_ARG_1DAB3D0F,
+        dataDirPathToPipeBaseDirPath);
+  }
+
+  /**
+   * Given a TsFile, a .resource file or a .mod file, returns the hardlink or 
the copied file in the
+   * pipe directory on the same disk.
+   *
+   * <p>If the given file is not on a local file system, it is returned as is.
+   */
+  File getHardlinkOrCopiedFile(final File file, final @Nullable String 
pipeName)
+      throws IOException {
+    if (!isLocalFile(file)) {
+      return file;
+    }
+    // The file has already been moved into a pipe directory, e.g., the 
historical extractor pins
+    // the pipe file of another pipe. It must not be linked or copied again.
+    if (isInPipeTsFileDir(file)) {
+      return file;
+    }
+    return new File(getPipeTsFileDirPath(file, pipeName), 
getRelativeFilePath(file));
+  }
+
+  /**
+   * Returns whether the given file is on a local file system. {@link File} 
collapses the {@code
+   * scheme://} of a remote path into {@code scheme:/}, so the prefixes are 
checked explicitly in
+   * addition to {@link FSUtils#isLocal(File)}.
+   */
+  static boolean isLocalFile(final File file) {
+    if (!FSUtils.isLocal(file)) {
+      return false;
+    }
+    final String path = file.getPath();
+    return !path.startsWith("hdfs:/") && !path.startsWith("os:/");

Review Comment:
   [P2] Recognize normalized remote prefixes on Windows too
   
   For the plain `java.io.File` used by `testNonLocalFileIsReturnedAsIs`, 
`FSUtils.isLocal(File)` returns true because its File overload recognizes 
remote filesystem subclasses rather than URI prefixes. The explicit prefix 
check then needs to handle the platform's normalized separators. On Windows, 
`new 
File("hdfs://namenode:9000/iotdb/data/sequence/root.db/0/0/f4.tsfile").getPath()`
 produces `hdfs:\namenode:9000\iotdb\data\sequence\root.db\0\0\f4.tsfile`, so 
`startsWith("hdfs:/")` is false (confirmed with the Windows JDK). The file is 
classified as local and sent through local directory resolution instead of 
being returned unchanged, breaking the new test on Windows. Normalize 
separators before checking the prefix, or use a platform-independent 
remote-path check; the `os:/` branch needs the same treatment.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileDirPathMapping.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.resource.tsfile;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.db.i18n.DataNodePipeMessages;
+
+import org.apache.tsfile.utils.FSUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Resolves the directory that holds the hardlinked or copied files of a 
TsFile.
+ *
+ * <p>The directory is always a subdirectory of the data directory that 
contains the TsFile, i.e.
+ * {@code 
<dataDir>/<pipe_hardlink_base_dir_name>/<pipe_hardlink_tsfile_dir_name>[/<pipeName>]}.
+ * Since a hardlink can only be created on the same file system as the source 
file, this keeps the
+ * hardlink on the same disk as the TsFile. IoTDB may place the TsFiles of one 
DataRegion on
+ * different disks, because every new TsFile is allocated by {@code 
TierManager}/{@code
+ * FolderManager} according to {@code dn_multi_dir_strategy}. Therefore the 
target directory can not
+ * be chosen per DataRegion or per pipe, but must be chosen per TsFile.
+ *
+ * <p>TsFiles on non-local storage (HDFS, object storage) can not be 
hardlinked nor copied by the
+ * local file system APIs. They are returned as is, i.e., pipe reads them in 
place instead of
+ * pinning a hardlink.
+ */
+class PipeTsFileDirPathMapping {
+
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(PipeTsFileDirPathMapping.class);
+
+  /** Canonical path of every local data directory -> canonical path of its 
pipe base directory. */
+  private final Map<String, String> dataDirPathToPipeBaseDirPath = new 
LinkedHashMap<>();
+
+  private final String pipeHardlinkBaseDirName;
+  private final String pipeHardlinkTsFileDirName;
+
+  PipeTsFileDirPathMapping(
+      final @Nonnull String[] localDataDirs,
+      final @Nonnull String pipeHardlinkBaseDirName,
+      final @Nonnull String pipeHardlinkTsFileDirName) {
+    this.pipeHardlinkBaseDirName = pipeHardlinkBaseDirName;
+    this.pipeHardlinkTsFileDirName = pipeHardlinkTsFileDirName;
+
+    for (final String dataDir : localDataDirs) {
+      final String canonicalDataDir = canonicalPath(dataDir);
+      dataDirPathToPipeBaseDirPath.put(
+          canonicalDataDir, canonicalDataDir + File.separator + 
pipeHardlinkBaseDirName);
+    }
+    LOGGER.info(
+        
DataNodePipeMessages.LOG_PIPE_HARDLINK_DIRECTORIES_ARE_RESOLVED_AS_ARG_1DAB3D0F,
+        dataDirPathToPipeBaseDirPath);
+  }
+
+  /**
+   * Given a TsFile, a .resource file or a .mod file, returns the hardlink or 
the copied file in the
+   * pipe directory on the same disk.
+   *
+   * <p>If the given file is not on a local file system, it is returned as is.
+   */
+  File getHardlinkOrCopiedFile(final File file, final @Nullable String 
pipeName)
+      throws IOException {
+    if (!isLocalFile(file)) {
+      return file;

Review Comment:
   [P2] Provide a filesystem-aware transfer path for returned remote files
   
   The existing file-transfer sinks cannot read an HDFS/object-storage `File` 
in place. `IoTDBDataRegionSyncSink.transferFilePieces()` and the Air Gap sink 
use `new RandomAccessFile(file, "r")`; the async sink does the same in 
`PipeTransferTsFileHandler.openNextFile()`. `RandomAccessFile` opens an 
OS-local pathname and does not dispatch through the remote File subclass or 
TsFile filesystem factory. Consequently, a remote TsFile that reaches a normal 
TsFile transfer still fails with a local file-open error, despite this branch 
reporting successful resource acquisition. Stage the file locally, use a 
filesystem-aware reader, or select a supported parsing path before returning 
remote files to those sinks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to