OIiveirra commented on code in PR #69527:
URL: https://github.com/apache/doris/pull/69527#discussion_r4366828608
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/LocalTableValuedFunction.java:
##########
@@ -135,6 +143,38 @@ private void getFileListFromBackend() throws
AnalysisException {
}
}
+ /** Apply LOCAL admission to sinks that select the built-in local JNI
writer as well. */
+ public static void checkSinkEnabled(String tvfName, Map<String, String>
properties) {
+ String writerClass = properties.getOrDefault("writer_class", "");
+ // JNI consumes a C string, and ScannerLoader accepts slash-separated
class names.
+ int nul = writerClass.indexOf('\0');
+ if (nul >= 0) {
+ writerClass = writerClass.substring(0, nul);
+ }
+ if ("local".equalsIgnoreCase(tvfName)
+ ||
"org.apache.doris.writer.LocalFileJniWriter".equals(writerClass.replace('/',
'.'))) {
Review Comment:
The delegated ViewFS variant is fixed in
46237ec1740e6c62c8fedacd84cad85876972c2a. Native admission uses the actual
generated file's parent; FE cleanup checks its actual parent directory. The raw
and normalized checks follow the selected mount table and path-component link,
including trailing-source slashes, authority/default table selection, fallback
and multiple targets. Nested targets are checked before filesystem creation.
Unused local mounts do not block an unambiguous remote route.
Runtime-overloaded, regex, cyclic, inherited or ambiguous nested delegation
requires the existing LOCAL startup opt-in instead of assuming the inline
configuration proves remote storage. Enabled requests retain their existing
Hadoop behavior.
All 51 selected FE tests passed with fresh XML and unchanged sources,
including INSERT/EXPLAIN, direct binding, resource-derived mounts, off/on and
remote controls, and a global-LOAD command entry with cleanup enabled.
Factory/executor mocks verify no side effect before rejection. No real
directory deletion was executed; targeted Checkstyle passed.
##########
be/src/format_v2/table/lance_reader.cpp:
##########
@@ -35,11 +39,79 @@
#include "format_v2/lance/lance_reader_helper.h"
#include "format_v2/lance/lance_runtime_filter_helper.h"
#include "format_v2/lance/lance_session_manager.h"
+#include "io/fs/local_file_system.h"
#include "runtime/file_scan_profile.h"
#include "runtime/runtime_state.h"
namespace doris::format::lance {
+namespace {
+std::string expand_home_path(const std::string& path) {
+ if (path != "~" && !path.starts_with("~/")) {
+ return path;
+ }
+ // Match the SDK's std::env::home_dir: HOME first, then the current user's
passwd entry.
+ std::string home;
+ if (const char* value = std::getenv("HOME"); value != nullptr && value[0]
!= '\0') {
+ home = value;
+ } else {
+ std::vector<char> buffer(1024);
+ struct passwd entry {};
+ struct passwd* result = nullptr;
+ int error;
+ while ((error = getpwuid_r(getuid(), &entry, buffer.data(),
buffer.size(), &result)) ==
+ ERANGE) {
+ buffer.resize(buffer.size() * 2);
+ }
+ if (error == 0 && result != nullptr && entry.pw_dir != nullptr) {
+ home = entry.pw_dir;
+ }
+ }
+ if (home.empty()) {
+ return path;
+ }
+ return path == "~" ? home : (io::Path(home) / path.substr(2)).string();
+}
+
+Status resolve_dataset_uri(const TFileRangeDesc& range, const
TFileScanRangeParams& scan_params,
+ std::string* uri) {
+ const auto& requested_uri =
range.table_format_params.lance_params.dataset_uri;
+ // Modern plans tag each range; older plans put the type in the scan
parameters.
+ // Untagged catalog scans retain their original storage-provider selection.
+ const bool is_local = range.__isset.file_type
+ ? range.file_type == TFileType::FILE_LOCAL
+ : scan_params.__isset.file_type &&
+ scan_params.file_type ==
TFileType::FILE_LOCAL;
+ // LOCAL TVF uses version zero for both schema discovery and whole-dataset
scans.
+ // Catalog scans pin a positive snapshot version, including file://
catalogs whose
+ // operator-configured warehouse is independent of the LOCAL TVF secure
directory.
+ if (is_local && range.table_format_params.lance_params.version == 0) {
+ // LOCAL Lance bypasses glob because it opens a dataset directory.
Apply the same
+ // canonical secure-root check before either schema discovery or a
cached dataset open.
+ std::string local_input = requested_uri;
+ const auto colon = requested_uri.find(':');
+ const auto slash = requested_uri.find('/');
+ const bool bare_path =
+ colon == std::string::npos || (slash != std::string::npos &&
slash < colon);
+ if (bare_path && !requested_uri.empty() && requested_uri.find('\0') ==
std::string::npos &&
+ !config::user_files_secure_path.empty()) {
+ // Lance's PathAbs fallback resolves bare dot segments before
following symlinks.
+ // Anchor relative TVF paths to the canonical secure root, not the
process cwd.
+ std::string secure_root;
+ RETURN_IF_ERROR(io::global_local_filesystem()->canonicalize(
+ config::user_files_secure_path, &secure_root));
+ local_input = (io::Path(secure_root) /
expand_home_path(requested_uri))
+ .lexically_normal()
+ .string();
+ }
+ return io::global_local_filesystem()->canonicalize_local_file(
Review Comment:
Fixed in 46237ec1740e6c62c8fedacd84cad85876972c2a and mirrored to the
existing 4.1 PR. LOCAL/version-zero resolution now checks reachable dataset
descendants before either schema discovery or dataset/session/cache opens.
Every symlink target is canonicalized and confined to the secure root before
traversal; failures return a generic error. Iterative canonical-directory
identity deduplicates aliases/cycles without an arbitrary dataset-size quota.
All 17 selected BE filesystem/Lance tests passed. Real fixtures reject
external metadata/data directories, the latest manifest, a fragment file,
deeper and dangling links. In-root directory/file links and an in-root
directory cycle preserve schema and actual rows 1–4. This preflight scales with
the reachable dataset tree; remote and fixed-version catalog routing are
unchanged. No SDK or manifest-reference format change is included.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
