luoluoyuyu commented on code in PR #19258:
URL: https://github.com/apache/iotdb/pull/19258#discussion_r4560861480
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java:
##########
@@ -100,6 +99,13 @@ private File increaseFileReference(
final @Nullable String pipeName,
final @Nullable File sourceFile)
throws IOException {
+ // Files on non-local file systems (e.g., HDFS or object storage) can not
be hardlinked or
+ // copied by the local file system APIs. They are read in place, and must
never be deleted by
+ // the reference count, so they are not tracked at all.
+ if (!isLocalFile(file)) {
+ return file;
Review Comment:
Fixed in 0555f6b by rejecting such files instead of reporting them as pinned.
A non-local TsFile can neither be hardlinked nor copied, so there is no way
to give the event the lifetime protection you describe, and the sinks cannot
read the file either. Returning it as is was the only alternative to failing,
and that is exactly the silent-loss case you point out.
`getHardlinkOrCopiedFile` now throws
`EXCEPTION_CAN_NOT_PIN_THE_FILE_ARG_FOR_PIPE_BECAUSE_IT_IS_NOT_ON_A_LOCAL_FILE_SYSTEM_76C15CA6`,
which surfaces immediately as a failed pin instead of a later remote open
error. `unpinTsFileResource` returns early for such files, and
`increaseFileReference`'s comment documents that they are never tracked.
Supporting remote source TsFiles end to end (local staging or a
filesystem-aware reader/parser) is a separate feature and out of scope here.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileDirPathMapping.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.resource.tsfile;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.db.i18n.DataNodePipeMessages;
+
+import org.apache.tsfile.utils.FSUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Resolves the directory that holds the hardlinked or copied files of a
TsFile.
+ *
+ * <p>The directory is always a subdirectory of the data directory that
contains the TsFile, i.e.
+ * {@code
<dataDir>/<pipe_hardlink_base_dir_name>/<pipe_hardlink_tsfile_dir_name>[/<pipeName>]}.
+ * Since a hardlink can only be created on the same file system as the source
file, this keeps the
+ * hardlink on the same disk as the TsFile. IoTDB may place the TsFiles of one
DataRegion on
+ * different disks, because every new TsFile is allocated by {@code
TierManager}/{@code
+ * FolderManager} according to {@code dn_multi_dir_strategy}. Therefore the
target directory can not
+ * be chosen per DataRegion or per pipe, but must be chosen per TsFile.
+ *
+ * <p>TsFiles on non-local storage (HDFS, object storage) can not be
hardlinked nor copied by the
+ * local file system APIs. They are returned as is, i.e., pipe reads them in
place instead of
+ * pinning a hardlink.
+ */
+class PipeTsFileDirPathMapping {
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PipeTsFileDirPathMapping.class);
+
+ /** Canonical path of every local data directory -> canonical path of its
pipe base directory. */
+ private final Map<String, String> dataDirPathToPipeBaseDirPath = new
LinkedHashMap<>();
+
+ private final String pipeHardlinkBaseDirName;
+ private final String pipeHardlinkTsFileDirName;
+
+ PipeTsFileDirPathMapping(
+ final @Nonnull String[] localDataDirs,
+ final @Nonnull String pipeHardlinkBaseDirName,
+ final @Nonnull String pipeHardlinkTsFileDirName) {
+ this.pipeHardlinkBaseDirName = pipeHardlinkBaseDirName;
+ this.pipeHardlinkTsFileDirName = pipeHardlinkTsFileDirName;
+
+ for (final String dataDir : localDataDirs) {
+ final String canonicalDataDir = canonicalPath(dataDir);
+ dataDirPathToPipeBaseDirPath.put(
+ canonicalDataDir, canonicalDataDir + File.separator +
pipeHardlinkBaseDirName);
+ }
+ LOGGER.info(
+
DataNodePipeMessages.LOG_PIPE_HARDLINK_DIRECTORIES_ARE_RESOLVED_AS_ARG_1DAB3D0F,
+ dataDirPathToPipeBaseDirPath);
+ }
+
+ /**
+ * Given a TsFile, a .resource file or a .mod file, returns the hardlink or
the copied file in the
+ * pipe directory on the same disk.
+ *
+ * <p>If the given file is not on a local file system, it is returned as is.
+ */
+ File getHardlinkOrCopiedFile(final File file, final @Nullable String
pipeName)
+ throws IOException {
+ if (!isLocalFile(file)) {
+ return file;
Review Comment:
Addressed by rejecting non-local files (see the comment on
`PipeTsFileResourceManager`): a remote TsFile never reaches the sinks any more,
so `transferFilePieces()` / `PipeTransferTsFileHandler.openNextFile()` can no
longer be handed a `File` that `RandomAccessFile` cannot open. Staging the file
locally or introducing a filesystem-aware reader would be a new feature and is
left out of this fix.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileDirPathMapping.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.resource.tsfile;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.db.i18n.DataNodePipeMessages;
+
+import org.apache.tsfile.utils.FSUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Resolves the directory that holds the hardlinked or copied files of a
TsFile.
+ *
+ * <p>The directory is always a subdirectory of the data directory that
contains the TsFile, i.e.
+ * {@code
<dataDir>/<pipe_hardlink_base_dir_name>/<pipe_hardlink_tsfile_dir_name>[/<pipeName>]}.
+ * Since a hardlink can only be created on the same file system as the source
file, this keeps the
+ * hardlink on the same disk as the TsFile. IoTDB may place the TsFiles of one
DataRegion on
+ * different disks, because every new TsFile is allocated by {@code
TierManager}/{@code
+ * FolderManager} according to {@code dn_multi_dir_strategy}. Therefore the
target directory can not
+ * be chosen per DataRegion or per pipe, but must be chosen per TsFile.
+ *
+ * <p>TsFiles on non-local storage (HDFS, object storage) can not be
hardlinked nor copied by the
+ * local file system APIs. They are returned as is, i.e., pipe reads them in
place instead of
+ * pinning a hardlink.
+ */
+class PipeTsFileDirPathMapping {
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PipeTsFileDirPathMapping.class);
+
+ /** Canonical path of every local data directory -> canonical path of its
pipe base directory. */
+ private final Map<String, String> dataDirPathToPipeBaseDirPath = new
LinkedHashMap<>();
+
+ private final String pipeHardlinkBaseDirName;
+ private final String pipeHardlinkTsFileDirName;
+
+ PipeTsFileDirPathMapping(
+ final @Nonnull String[] localDataDirs,
+ final @Nonnull String pipeHardlinkBaseDirName,
+ final @Nonnull String pipeHardlinkTsFileDirName) {
+ this.pipeHardlinkBaseDirName = pipeHardlinkBaseDirName;
+ this.pipeHardlinkTsFileDirName = pipeHardlinkTsFileDirName;
+
+ for (final String dataDir : localDataDirs) {
+ final String canonicalDataDir = canonicalPath(dataDir);
+ dataDirPathToPipeBaseDirPath.put(
+ canonicalDataDir, canonicalDataDir + File.separator +
pipeHardlinkBaseDirName);
+ }
+ LOGGER.info(
+
DataNodePipeMessages.LOG_PIPE_HARDLINK_DIRECTORIES_ARE_RESOLVED_AS_ARG_1DAB3D0F,
+ dataDirPathToPipeBaseDirPath);
+ }
+
+ /**
+ * Given a TsFile, a .resource file or a .mod file, returns the hardlink or
the copied file in the
+ * pipe directory on the same disk.
+ *
+ * <p>If the given file is not on a local file system, it is returned as is.
+ */
+ File getHardlinkOrCopiedFile(final File file, final @Nullable String
pipeName)
+ throws IOException {
+ if (!isLocalFile(file)) {
+ return file;
+ }
+ // The file has already been moved into a pipe directory, e.g., the
historical extractor pins
+ // the pipe file of another pipe. It must not be linked or copied again.
+ if (isInPipeTsFileDir(file)) {
+ return file;
+ }
+ return new File(getPipeTsFileDirPath(file, pipeName),
getRelativeFilePath(file));
+ }
+
+ /**
+ * Returns whether the given file is on a local file system. {@link File}
collapses the {@code
+ * scheme://} of a remote path into {@code scheme:/}, so the prefixes are
checked explicitly in
+ * addition to {@link FSUtils#isLocal(File)}.
+ */
+ static boolean isLocalFile(final File file) {
+ if (!FSUtils.isLocal(file)) {
+ return false;
+ }
+ final String path = file.getPath();
+ return !path.startsWith("hdfs:/") && !path.startsWith("os:/");
Review Comment:
Fixed in 0555f6b.
`isLocalFile` now normalizes the path before the prefix check:
```java
final String path = file.getPath().replace('\\',
'/').toLowerCase(Locale.ROOT);
return !path.startsWith("hdfs:/") && !path.startsWith("os:/");
```
so `hdfs:\namenode:9000\iotdb\...` and an upper-cased scheme are both
rejected. `testNonLocalFileIsRejected` covers the `hdfs://`, the `os://` and
the Windows-normalized `hdfs:\namenode:9000\...` forms.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileDirPathMapping.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.resource.tsfile;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.db.i18n.DataNodePipeMessages;
+
+import org.apache.tsfile.utils.FSUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * Resolves the directory that holds the hardlinked or copied files of a
TsFile.
+ *
+ * <p>The directory is always a subdirectory of the data directory that
contains the TsFile, i.e.
+ * {@code
<dataDir>/<pipe_hardlink_base_dir_name>/<pipe_hardlink_tsfile_dir_name>[/<pipeName>]}.
+ * Since a hardlink can only be created on the same file system as the source
file, this keeps the
+ * hardlink on the same disk as the TsFile. IoTDB may place the TsFiles of one
DataRegion on
+ * different disks, because every new TsFile is allocated by {@code
TierManager}/{@code
+ * FolderManager} according to {@code dn_multi_dir_strategy}. Therefore the
target directory can not
+ * be chosen per DataRegion or per pipe, but must be chosen per TsFile.
+ *
+ * <p>TsFiles on non-local storage (HDFS, object storage) can not be
hardlinked nor copied by the
+ * local file system APIs. They are returned as is, i.e., pipe reads them in
place instead of
+ * pinning a hardlink.
+ */
+class PipeTsFileDirPathMapping {
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(PipeTsFileDirPathMapping.class);
+
+ /** Canonical path of every local data directory -> canonical path of its
pipe base directory. */
+ private final Map<String, String> dataDirPathToPipeBaseDirPath = new
LinkedHashMap<>();
+
+ private final String pipeHardlinkBaseDirName;
+ private final String pipeHardlinkTsFileDirName;
+
+ PipeTsFileDirPathMapping(
+ final @Nonnull String[] localDataDirs,
+ final @Nonnull String pipeHardlinkBaseDirName,
+ final @Nonnull String pipeHardlinkTsFileDirName) {
+ this.pipeHardlinkBaseDirName = pipeHardlinkBaseDirName;
+ this.pipeHardlinkTsFileDirName = pipeHardlinkTsFileDirName;
+
+ for (final String dataDir : localDataDirs) {
+ final String canonicalDataDir = canonicalPath(dataDir);
+ dataDirPathToPipeBaseDirPath.put(
+ canonicalDataDir, canonicalDataDir + File.separator +
pipeHardlinkBaseDirName);
+ }
+ LOGGER.info(
+
DataNodePipeMessages.LOG_PIPE_HARDLINK_DIRECTORIES_ARE_RESOLVED_AS_ARG_1DAB3D0F,
+ dataDirPathToPipeBaseDirPath);
+ }
+
+ /**
+ * Given a TsFile, a .resource file or a .mod file, returns the hardlink or
the copied file in the
+ * pipe directory on the same disk.
+ *
+ * <p>If the given file is not on a local file system, it is returned as is.
+ */
+ File getHardlinkOrCopiedFile(final File file, final @Nullable String
pipeName)
+ throws IOException {
+ if (!isLocalFile(file)) {
+ return file;
+ }
+ // The file has already been moved into a pipe directory, e.g., the
historical extractor pins
+ // the pipe file of another pipe. It must not be linked or copied again.
+ if (isInPipeTsFileDir(file)) {
+ return file;
+ }
+ return new File(getPipeTsFileDirPath(file, pipeName),
getRelativeFilePath(file));
+ }
+
+ /**
+ * Returns whether the given file is on a local file system. {@link File}
collapses the {@code
+ * scheme://} of a remote path into {@code scheme:/}, so the prefixes are
checked explicitly in
+ * addition to {@link FSUtils#isLocal(File)}.
+ */
+ static boolean isLocalFile(final File file) {
+ if (!FSUtils.isLocal(file)) {
+ return false;
+ }
+ final String path = file.getPath();
+ return !path.startsWith("hdfs:/") && !path.startsWith("os:/");
Review Comment:
Fixed in 0555f6b.
`isLocalFile` now normalizes the path before the prefix check:
```java
final String path = file.getPath().replace('\\',
'/').toLowerCase(Locale.ROOT);
return !path.startsWith("hdfs:/") && !path.startsWith("os:/");
```
so `hdfs:\namenode:9000\iotdb\...` and an upper-cased scheme are both
rejected. `testNonLocalFileIsRejected` covers the `hdfs://` form, the `os://`
form and the Windows-normalized `hdfs:\namenode:9000\...` form.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]