This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4b0e660b82f [refactor](catalog) drop hadoop source imports from
fe-core and fe-common (#66324)
4b0e660b82f is described below
commit 4b0e660b82fb3321fc966faf8a8d4e735ed0d3b5
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Aug 1 14:48:45 2026 +0800
[refactor](catalog) drop hadoop source imports from fe-core and fe-common
(#66324)
### What problem does this PR solve?
Issue Number: #65185
Related PR: #66004
Problem Summary:
Now that catalogs go through the connector/filesystem SPIs, fe-core and
fe-common no
longer have a reason to compile against hadoop, but a handful of source
imports were
still left over from the pre-SPI code. This removes them.
Scope is deliberately narrow: **source imports only**. The hadoop pom
dependencies stay
exactly as they are, because they remain reachable at runtime —
`ranger-plugins-common`
inherits from `org.apache.hadoop.conf.Configuration`, `hive-exec`
supplies the UDF base
class that `CREATE FUNCTION` resolves, and `fe-kerberos` uses
`UserGroupInformation`.
Dropping the jars is a separate, larger question and is not attempted
here.
After this PR, `grep -rn "^import org.apache.hadoop" fe/fe-core/src
fe/fe-common/src`
returns exactly one line: `RangerHiveAuditHandler`, which is a
sanctioned exception. The
type is imposed by Ranger's own API, and `ranger-plugins-common` puts
hadoop on the
classpath regardless, so removing that import would remove no
dependency. It is
documented as such in the code rather than worked around.
The six commits are independent steps and are easiest to review one at a
time:
| Commit | What moves |
|---|---|
| `4314ad2` | Dead and trivial imports: delete `CatalogConfigFileUtils`
(no caller; `fe-filesystem-hdfs*` already carries the port) and
`LocationPath.getTempWritePath` (no caller); inline two hadoop constants
that were plain strings; document the Ranger exception |
| `29649e2` | `hadoop-huaweicloud` moves from fe-core to the
`fe-filesystem-obs` plugin, together with the `huawei-obs-sdk`
repository declaration. The only FE reference is the `Class.forName`
probe in `ObsFileSystemProperties`, which must resolve against that
plugin's classloader to report the truth |
| `90339cd` | `FileSplitter` takes a new fe-core `FileBlockLocation`
instead of hadoop's `BlockLocation`. Only
`getOffset`/`getLength`/`getHosts` were ever read |
| `c55800a` | `LocationPath.toStorageLocation()` returns the existing
`org.apache.doris.filesystem.Location` instead of a hadoop `Path`;
`getPath()` is deleted in favour of the `fsIdentifier` the class already
computes. **See the behaviour-change note below** |
| `592e8aa` | The Azure OAuth2 backend map moves from `StorageAdapter`
into `fe-filesystem-azure`, which is where the legacy `AzureProperties`
owned it. This deletes ~120 lines of fe-core code that turned out to be
unreachable |
| `464efa4` | `StageUtil`'s `GlobExpander`/`GlobFilter` use is ported to
a package-private `GlobPatterns`, compiled with re2j — the same engine
hadoop's `GlobPattern` uses, and already a declared fe-core dependency |
Two notes that are easy to miss on review:
- `hadoop.fs.GlobFilter` reads like a wildcard predicate but its
constructor is also a
validator: it rejects `a[b`, `a{b`, a trailing backslash and `[z-a]`,
and `analyzeGlob`
surfaces that as a `DdlException`. Porting only the predicate would have
accepted those
globs and quietly listed an unintended object-store prefix instead of
failing the
statement, so the validation is ported with it.
- `StorageAdapter.getHadoopStorageConfig()` had no caller anywhere in
the tree; the only
consumer was the adapter's own Azure OAuth2 arm. That map is still
load-bearing (BE
routes Microsoft Fabric OneLake locations to `FILE_HDFS`, and
`hdfs_builder` feeds every
entry into its JNI hadoop builder), so it is preserved key-for-key
rather than dropped —
it just lives in the Azure plugin now.
---
.../doris/common/CatalogConfigFileUtils.java | 85 ------
fe/fe-core/pom.xml | 26 +-
.../org/apache/doris/catalog/HdfsStorageVault.java | 6 +-
.../ranger/hive/RangerHiveAuditHandler.java | 15 ++
.../org/apache/doris/cloud/stage/GlobPatterns.java | 298 +++++++++++++++++++++
.../org/apache/doris/cloud/stage/StageUtil.java | 14 +-
.../org/apache/doris/common/util/LocationPath.java | 27 +-
.../doris/datasource/scan/FileGroupInfo.java | 8 +-
.../doris/datasource/scan/FileQueryScanNode.java | 8 +-
.../doris/datasource/split/FileBlockLocation.java | 58 ++++
.../doris/datasource/split/FileSplitter.java | 13 +-
.../doris/datasource/storage/StorageAdapter.java | 229 +---------------
.../doris/nereids/load/NereidsFileGroupInfo.java | 8 +-
.../apache/doris/cloud/stage/StageGlobTest.java | 164 ++++++++++++
.../doris/datasource/split/FileSplitterTest.java | 62 ++++-
.../doris/planner/FederationBackendPolicyTest.java | 5 +-
fe/fe-filesystem/fe-filesystem-azure/pom.xml | 37 +++
.../azure/AzureFileSystemProperties.java | 49 +++-
.../azure/AzureFileSystemPropertiesTest.java | 72 +++++
fe/fe-filesystem/fe-filesystem-obs/pom.xml | 43 +++
20 files changed, 844 insertions(+), 383 deletions(-)
diff --git
a/fe/fe-common/src/main/java/org/apache/doris/common/CatalogConfigFileUtils.java
b/fe/fe-common/src/main/java/org/apache/doris/common/CatalogConfigFileUtils.java
deleted file mode 100644
index d8e9f2e5133..00000000000
---
a/fe/fe-common/src/main/java/org/apache/doris/common/CatalogConfigFileUtils.java
+++ /dev/null
@@ -1,85 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.common;
-
-import org.apache.commons.lang3.StringUtils;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.fs.Path;
-
-import java.io.File;
-import java.util.function.BiConsumer;
-import java.util.function.Supplier;
-
-public class CatalogConfigFileUtils {
-
- /**
- * Generic method to load Hadoop-style configuration files from a
directory.
- *
- * @param resourcesPath Comma-separated list of resource file names to
be loaded.
- * @param configDir Directory prefix where the configuration files
reside.
- * @param configSupplier Supplier that creates a new configuration
object.
- * @param addResourceMethod Method to add a resource file to the
configuration object.
- * @param <T> Type of the configuration.
- * @return A configuration object loaded with the given resource files.
- * @throws IllegalArgumentException if the resourcesPath is empty or if
any file does not exist.
- */
- private static <T> T loadConfigFromDir(String resourcesPath, String
configDir,
- Supplier<T> configSupplier,
- BiConsumer<T, Path>
addResourceMethod) {
- // Check if the provided resourcesPath is blank and throw an exception
if so.
- if (StringUtils.isBlank(resourcesPath)) {
- throw new IllegalArgumentException("Config resource path is
empty");
- }
-
- // Create a new configuration object.
- T conf = configSupplier.get();
-
- // Iterate over the comma-separated list of resource files.
- for (String resource : resourcesPath.split(",")) {
- // Construct the full path to the resource file.
- String resourcePath = configDir + resource.trim();
- File file = new File(resourcePath);
-
- // Check if the file exists and is a regular file; if not, throw
an exception.
- if (file.exists() && file.isFile()) {
- // Add the resource file to the configuration object.
- addResourceMethod.accept(conf, new Path(file.toURI()));
- } else {
- // Throw an exception if the file does not exist or is not a
regular file.
- throw new IllegalArgumentException("Config resource file does
not exist: " + resourcePath);
- }
- }
- return conf;
- }
-
- /**
- * Loads a Hadoop Configuration object from a list of files under the
specified config directory.
- *
- * @param resourcesPath Comma-separated list of file names to be loaded.
- * @return A Hadoop Configuration object.
- * @throws IllegalArgumentException if the input is invalid or files are
missing.
- */
- public static Configuration loadConfigurationFromHadoopConfDir(String
resourcesPath) {
- return loadConfigFromDir(
- resourcesPath,
- Config.hadoop_config_dir,
- Configuration::new,
- Configuration::addResource
- );
- }
-}
diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml
index f6a2d1fc462..6744db2f762 100644
--- a/fe/fe-core/pom.xml
+++ b/fe/fe-core/pom.xml
@@ -391,11 +391,13 @@ under the License.
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
- <!-- Runtime-only: hadoop-huaweicloud's OBSFileSystem
(reflection-loaded, so no compile-time
- reference reveals this) calls org.apache.commons.lang.StringUtils
while declaring no
- commons-lang of its own. This declaration is its only source on
the FE classpath;
- dropping it compiles green and then NoClassDefFoundErrors on OBS
access.
- scope=runtime keeps it off the compile classpath, so fe source
stays commons-lang(2.x)-free. -->
+ <!-- Runtime-only, and still LOAD-BEARING after hadoop-huaweicloud
moved out of this module:
+ OBSFileSystem calls org.apache.commons.lang.StringUtils while
declaring no commons-lang of its
+ own, and the plugins that now carry hadoop-huaweicloud
(fe-filesystem-obs, fe-connector-paimon)
+ do not declare commons-lang either — their child-first loaders
miss it and fall through to
+ fe/lib, i.e. to this declaration. Dropping it compiles green and
then NoClassDefFoundErrors on
+ OBS access. scope=runtime keeps it off the compile classpath, so
fe source stays
+ commons-lang(2.x)-free. -->
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
@@ -555,13 +557,6 @@ under the License.
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
</dependency>
- <!-- hadoop-huaweicloud provides OBSFileSystem; loaded via reflection
in OBSProperties
- with graceful S3A fallback — runtime only, not needed at compile
time -->
- <dependency>
- <groupId>com.huaweicloud</groupId>
- <artifactId>hadoop-huaweicloud</artifactId>
- <scope>runtime</scope>
- </dependency>
<!-- antl4 The version of antlr-runtime in trino parser is need to be
consistent with doris,
when upgrade doris antlr-runtime version, should take care of
trino-parser.-->
<dependency>
@@ -864,13 +859,6 @@ under the License.
<artifactId>fastutil-core</artifactId>
</dependency>
</dependencies>
- <repositories>
- <!-- for huawei obs sdk -->
- <repository>
- <id>huawei-obs-sdk</id>
-
<url>https://repo.huaweicloud.com/repository/maven/huaweicloudsdk/</url>
- </repository>
- </repositories>
<build>
<finalName>doris-fe</finalName>
<directory>${project.basedir}/target/</directory>
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java
index 5e2017092c8..c5851405958 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/HdfsStorageVault.java
@@ -32,7 +32,6 @@ import com.google.common.collect.ImmutableMap;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Maps;
import com.google.gson.annotations.SerializedName;
-import org.apache.hadoop.fs.CommonConfigurationKeysPublic;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -70,8 +69,9 @@ public class HdfsStorageVault extends StorageVault {
public static String HADOOP_FS_NAME = "fs.defaultFS";
public static String VAULT_PATH_PREFIX = "path_prefix";
public static String HADOOP_USER_NAME =
AuthenticationConfig.HADOOP_USER_NAME;
- public static String HADOOP_SECURITY_AUTHENTICATION =
- CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION;
+ // Literal of hadoop's
CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, inlined
+ // so fe-core carries no org.apache.hadoop import for a plain
configuration-key string.
+ public static String HADOOP_SECURITY_AUTHENTICATION =
"hadoop.security.authentication";
public static String HADOOP_KERBEROS_KEYTAB =
AuthenticationConfig.HADOOP_KERBEROS_KEYTAB;
public static String HADOOP_KERBEROS_PRINCIPAL =
AuthenticationConfig.HADOOP_KERBEROS_PRINCIPAL;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/authorizer/ranger/hive/RangerHiveAuditHandler.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/authorizer/ranger/hive/RangerHiveAuditHandler.java
index 4f5d3678531..73d73a3933b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/authorizer/ranger/hive/RangerHiveAuditHandler.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/catalog/authorizer/ranger/hive/RangerHiveAuditHandler.java
@@ -38,6 +38,21 @@ import java.util.List;
import java.util.Map;
import java.util.Set;
+/**
+ * ATTN: the {@code org.apache.hadoop.conf.Configuration} import below is a
SANCTIONED EXCEPTION to the
+ * fe-core hadoop-decoupling effort — do not "clean" it away.
+ *
+ * <p>The type is imposed by the Ranger library, not chosen by Doris: {@link
RangerDefaultAuditHandler}
+ * declares the constructor as {@code
RangerDefaultAuditHandler(org.apache.hadoop.conf.Configuration)}, and
+ * the only argument ever passed is {@code RangerBasePlugin.getConfig()},
whose type chain is
+ * {@code RangerPluginConfig -> RangerConfiguration ->
org.apache.hadoop.conf.Configuration}. Narrowing the
+ * parameter to {@code RangerPluginConfig} would make a grep look clean while
removing no dependency at all:
+ * ranger-plugins-common itself pulls hadoop-client-api/hadoop-client-runtime
onto the fe-core classpath.
+ *
+ * <p>Removing hadoop from here therefore requires moving the whole Ranger
authorizer out of fe-core, which
+ * contradicts the standing decision that this package is a generic
external-catalog authorizer kept in
+ * fe-core unchanged.
+ */
public class RangerHiveAuditHandler extends RangerDefaultAuditHandler {
public static final String ACCESS_TYPE_ROWFILTER = "ROW_FILTER";
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/cloud/stage/GlobPatterns.java
b/fe/fe-core/src/main/java/org/apache/doris/cloud/stage/GlobPatterns.java
new file mode 100644
index 00000000000..680c1126f72
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/stage/GlobPatterns.java
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.cloud.stage;
+
+import com.google.re2j.Pattern;
+import com.google.re2j.PatternSyntaxException;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Brace expansion and wildcard detection for the globs accepted by {@code
COPY INTO}.
+ *
+ * <p>Ported from hadoop's {@code org.apache.hadoop.fs.GlobExpander} and
+ * {@code org.apache.hadoop.fs.GlobPattern} (both Apache-2.0), which {@link
StageUtil} used
+ * directly until fe-core dropped its hadoop source imports. {@code StageUtil}
already carries
+ * local copies of the neighbouring {@code org.apache.hadoop.fs.Globber}
helpers, so these two
+ * complete the set.
+ *
+ * <p>Semantics are preserved exactly, down to compiling with re2j the way
hadoop's
+ * {@code GlobPattern} does. That matters twice over:
+ * <ul>
+ * <li>{@link #hasWildcard} decides whether a path component is a literal
prefix or the point
+ * where wildcard matching starts, which is what {@code
StageUtil.analyzeGlob} turns into
+ * the object-store listing prefix. Widening or narrowing it lists the
wrong keys.</li>
+ * <li>Both entry points reject malformed globs, and {@code analyzeGlob}
surfaces that as a
+ * {@code DdlException}. A laxer validator would silently accept e.g.
{@code a[b} and copy
+ * from an unintended prefix instead of failing the statement.</li>
+ * </ul>
+ *
+ * <p>Only what {@code StageUtil} calls is ported: hadoop's glob
<em>matching</em> is not, since
+ * the compiled pattern is used here purely to validate. Note this is a
different glob dialect
+ * from {@code org.apache.doris.common.GlobRegexUtil}, which treats {@code {}
} as literal
+ * characters and has no notion of brace groups.
+ */
+final class GlobPatterns {
+
+ private static final char BACKSLASH = '\\';
+
+ private GlobPatterns() {
+ }
+
+ /**
+ * Expands the glob into a set of patterns none of which has a slash
inside a curly bracket
+ * pair, e.g. {@code p{a/b,c/d}s} becomes {@code [pa/bs, pc/ds]}. Brace
groups without a slash
+ * are left alone for {@link #hasWildcard} to handle per component.
+ *
+ * @throws IOException if the glob ends with a dangling escape character
+ */
+ static List<String> expand(String filePattern) throws IOException {
+ List<String> fullyExpanded = new ArrayList<>();
+ List<StringWithOffset> toExpand = new ArrayList<>();
+ toExpand.add(new StringWithOffset(filePattern, 0));
+ while (!toExpand.isEmpty()) {
+ StringWithOffset path = toExpand.remove(0);
+ List<StringWithOffset> expanded = expandLeftmost(path);
+ if (expanded == null) {
+ fullyExpanded.add(path.string);
+ } else {
+ toExpand.addAll(0, expanded);
+ }
+ }
+ return fullyExpanded;
+ }
+
+ /**
+ * Whether the glob contains a wildcard, i.e. any of {@code * ? [ {}.
Escaped characters and
+ * the closing {@code ] } } do not count, matching hadoop's {@code
GlobPattern.hasWildcard()}.
+ *
+ * @throws IOException if the glob is malformed, mirroring the {@code
GlobFilter} constructor
+ */
+ static boolean hasWildcard(String glob) throws IOException {
+ try {
+ return compile(glob);
+ } catch (PatternSyntaxException e) {
+ throw new IOException("Illegal file pattern: " + e.getMessage(),
e);
+ }
+ }
+
+ /**
+ * Translates the glob to a regex exactly as hadoop's {@code
GlobPattern.set} does and compiles
+ * it for its validation side effect, returning whether a wildcard was
seen.
+ */
+ private static boolean compile(String glob) {
+ StringBuilder regex = new StringBuilder();
+ int setOpen = 0;
+ int curlyOpen = 0;
+ int len = glob.length();
+ boolean hasWildcard = false;
+
+ for (int i = 0; i < len; i++) {
+ char c = glob.charAt(i);
+
+ switch (c) {
+ case BACKSLASH:
+ if (++i >= len) {
+ error("Missing escaped character", glob, i);
+ }
+ regex.append(c).append(glob.charAt(i));
+ continue;
+ case '.':
+ case '$':
+ case '(':
+ case ')':
+ case '|':
+ case '+':
+ // escape regex special chars that are not glob special
chars
+ regex.append(BACKSLASH);
+ break;
+ case '*':
+ regex.append('.');
+ hasWildcard = true;
+ break;
+ case '?':
+ regex.append('.');
+ hasWildcard = true;
+ continue;
+ case '{': // start of a group
+ regex.append("(?:"); // non-capturing
+ curlyOpen++;
+ hasWildcard = true;
+ continue;
+ case ',':
+ regex.append(curlyOpen > 0 ? '|' : c);
+ continue;
+ case '}':
+ if (curlyOpen > 0) {
+ // end of a group
+ curlyOpen--;
+ regex.append(")");
+ continue;
+ }
+ break;
+ case '[':
+ if (setOpen > 0) {
+ error("Unclosed character class", glob, i);
+ }
+ setOpen++;
+ hasWildcard = true;
+ break;
+ case '^': // ^ inside [...] can be unescaped
+ if (setOpen == 0) {
+ regex.append(BACKSLASH);
+ }
+ break;
+ case '!': // [! needs to be translated to [^
+ regex.append(setOpen > 0 && '[' == glob.charAt(i - 1) ?
'^' : '!');
+ continue;
+ case ']':
+ // Many set errors like [][] could not be easily detected
here,
+ // as []], []-] and [-] are all valid POSIX glob and java
regex.
+ // We'll just let the regex compiler do the real work.
+ setOpen = 0;
+ break;
+ default:
+ }
+ regex.append(c);
+ }
+
+ if (setOpen > 0) {
+ error("Unclosed character class", glob, len);
+ }
+ if (curlyOpen > 0) {
+ error("Unclosed group", glob, len);
+ }
+ // Compiled for validation only: re2j rejects character-class errors
that the scan above
+ // deliberately leaves to it (see the ']' case), and hadoop surfaced
those the same way.
+ // DOTALL is not about matching here -- it never runs -- but re2j
echoes the flags into the
+ // PatternSyntaxException text, which ends up verbatim in the
user-facing DdlException.
+ Pattern.compile(regex.toString(), Pattern.DOTALL);
+ return hasWildcard;
+ }
+
+ /**
+ * Expands the leftmost outer curly bracket pair that contains a slash.
+ *
+ * @return the expansions, or null when there is no such pair
+ */
+ private static List<StringWithOffset> expandLeftmost(StringWithOffset
filePatternWithOffset)
+ throws IOException {
+ String filePattern = filePatternWithOffset.string;
+ int leftmost = leftmostOuterCurlyContainingSlash(filePattern,
filePatternWithOffset.offset);
+ if (leftmost == -1) {
+ return null;
+ }
+ int curlyOpen = 0;
+ StringBuilder prefix = new StringBuilder(filePattern.substring(0,
leftmost));
+ StringBuilder suffix = new StringBuilder();
+ List<String> alts = new ArrayList<>();
+ StringBuilder alt = new StringBuilder();
+ StringBuilder cur = prefix;
+ for (int i = leftmost; i < filePattern.length(); i++) {
+ char c = filePattern.charAt(i);
+ if (cur == suffix) {
+ cur.append(c);
+ } else if (c == '\\') {
+ i++;
+ if (i >= filePattern.length()) {
+ throw new IOException("Illegal file pattern: "
+ + "An escaped character does not present for glob "
+ + filePattern + " at " + i);
+ }
+ c = filePattern.charAt(i);
+ cur.append(c);
+ } else if (c == '{') {
+ if (curlyOpen++ == 0) {
+ alt.setLength(0);
+ cur = alt;
+ } else {
+ cur.append(c);
+ }
+ } else if (c == '}' && curlyOpen > 0) {
+ if (--curlyOpen == 0) {
+ alts.add(alt.toString());
+ alt.setLength(0);
+ cur = suffix;
+ } else {
+ cur.append(c);
+ }
+ } else if (c == ',') {
+ if (curlyOpen == 1) {
+ alts.add(alt.toString());
+ alt.setLength(0);
+ } else {
+ cur.append(c);
+ }
+ } else {
+ cur.append(c);
+ }
+ }
+ List<StringWithOffset> exp = new ArrayList<>();
+ for (String string : alts) {
+ exp.add(new StringWithOffset(prefix + string + suffix,
prefix.length()));
+ }
+ return exp;
+ }
+
+ /**
+ * @return the index of the leftmost opening curly bracket containing a
slash, or -1
+ */
+ private static int leftmostOuterCurlyContainingSlash(String filePattern,
int offset) throws IOException {
+ int curlyOpen = 0;
+ int leftmost = -1;
+ boolean seenSlash = false;
+ for (int i = offset; i < filePattern.length(); i++) {
+ char c = filePattern.charAt(i);
+ if (c == '\\') {
+ i++;
+ if (i >= filePattern.length()) {
+ throw new IOException("Illegal file pattern: "
+ + "An escaped character does not present for glob "
+ + filePattern + " at " + i);
+ }
+ } else if (c == '{') {
+ if (curlyOpen++ == 0) {
+ leftmost = i;
+ }
+ } else if (c == '}' && curlyOpen > 0) {
+ if (--curlyOpen == 0 && leftmost != -1 && seenSlash) {
+ return leftmost;
+ }
+ } else if (c == '/' && curlyOpen > 0) {
+ seenSlash = true;
+ }
+ }
+ return -1;
+ }
+
+ private static void error(String message, String pattern, int pos) {
+ throw new PatternSyntaxException(String.format("%s at pos %d",
message, pos), pattern);
+ }
+
+ private static class StringWithOffset {
+ private final String string;
+ private final int offset;
+
+ StringWithOffset(String string, int offset) {
+ this.string = string;
+ this.offset = offset;
+ }
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/cloud/stage/StageUtil.java
b/fe/fe-core/src/main/java/org/apache/doris/cloud/stage/StageUtil.java
index 8d306d83e66..79225bbf7ac 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/cloud/stage/StageUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/cloud/stage/StageUtil.java
@@ -45,8 +45,6 @@ import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.lang3.tuple.Triple;
-import org.apache.hadoop.fs.GlobExpander;
-import org.apache.hadoop.fs.GlobFilter;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -302,7 +300,7 @@ public class StageUtil {
return globs;
}
try {
- List<String> flattenedPatterns = GlobExpander.expand(glob);
+ List<String> flattenedPatterns = GlobPatterns.expand(glob);
for (String flattenedPattern : flattenedPatterns) {
try {
globs.addAll(analyzeFlattenedPattern(flattenedPattern));
@@ -326,8 +324,7 @@ public class StageUtil {
boolean sawWildcard = false;
for (int componentIdx = 0; componentIdx < components.size();
componentIdx++) {
String component = components.get(componentIdx);
- GlobFilter globFilter = new GlobFilter(component);
- if (globFilter.hasPattern()) {
+ if (GlobPatterns.hasWildcard(component)) {
if (componentIdx == components.size() - 1) {
List<Pair<String, Boolean>> pairs =
analyzeLastComponent(component);
if (pairs != null) {
@@ -356,8 +353,7 @@ public class StageUtil {
String sub = component.substring(1, component.length() - 1);
List<String> splits = splitByComma(sub);
for (String split : splits) {
- GlobFilter globFilter = new GlobFilter(split);
- if (globFilter.hasPattern()) {
+ if (GlobPatterns.hasWildcard(split)) {
results.add(Pair.of(getComponentPrefix(split), true));
} else {
results.add(Pair.of(unescapePathComponent(split), false));
@@ -412,7 +408,7 @@ public class StageUtil {
}
/*
- * Glob process method are referenced from {@link
org.apache.hadoop.fs.Globber}
+ * Glob process method are referenced from {@code
org.apache.hadoop.fs.Globber}
*/
private static String unescapePathComponent(String name) {
return name.replaceAll("\\\\(.)", "$1");
@@ -420,7 +416,7 @@ public class StageUtil {
private static List<String> getPathComponents(String path) {
ArrayList<String> ret = new ArrayList<>();
- for (String component :
path.split(org.apache.hadoop.fs.Path.SEPARATOR)) {
+ for (String component : path.split("/")) {
if (!component.isEmpty()) {
ret.add(component);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java
index 85badd97265..f6da8e0d85e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java
@@ -23,12 +23,12 @@ import org.apache.doris.datasource.storage.StorageRegistry;
import org.apache.doris.datasource.storage.StorageTypeId;
import org.apache.doris.datasource.storage.StorageUriUtils;
import org.apache.doris.filesystem.FileSystemType;
+import org.apache.doris.filesystem.Location;
import org.apache.doris.foundation.property.StoragePropertiesException;
import org.apache.doris.thrift.TFileType;
import com.google.common.base.Strings;
import org.apache.commons.lang3.StringUtils;
-import org.apache.hadoop.fs.Path;
import java.io.UnsupportedEncodingException;
import java.net.URI;
@@ -37,7 +37,6 @@ import java.nio.charset.StandardCharsets;
import java.nio.file.InvalidPathException;
import java.nio.file.Paths;
import java.util.Map;
-import java.util.UUID;
/**
* LocationPath is a utility class for parsing, validating, and normalizing
storage location URIs.
@@ -367,13 +366,6 @@ public class LocationPath {
return locationPath.getTFileTypeForBE();
}
- public static String getTempWritePath(String loc, String prefix) {
- // If prefix is relative, it is resolved under loc; if absolute, it is
used as the base path.
- Path tempRoot = new Path(loc, prefix);
- Path tempPath = new Path(tempRoot,
UUID.randomUUID().toString().replace("-", ""));
- return tempPath.toString();
- }
-
public TFileType getTFileTypeForBE() {
if (("abfs".equals(schema) || "abfss".equals(schema))
&& StorageUriUtils.isOneLakeLocation(normalizedLocation)) {
@@ -388,10 +380,19 @@ public class LocationPath {
/**
* The converted path is used for BE
*
+ * <p>Returns the normalized location verbatim. This used to route through
hadoop's
+ * {@code Path}, which rewrote the string on the way out: it collapsed
repeated slashes,
+ * dropped a trailing slash, resolved {@code .}/{@code ..} segments, and
turned an empty
+ * authority into a single slash ({@code hdfs:///a/b} became {@code
hdfs:/a/b}). None of that
+ * is wanted here — object stores key on the exact byte string, so {@code
a//b} and {@code a/b}
+ * name different objects, and collapsing them rewrites the key. This
matches
+ * {@link Location}, which stores what it was given, and Trino's {@code
io.trino.filesystem
+ * .Location}, whose javadoc likewise states it does not follow URI format
rules.
+ *
* @return BE scan range path
*/
- public Path toStorageLocation() {
- return new Path(normalizedLocation);
+ public Location toStorageLocation() {
+ return Location.of(normalizedLocation);
}
@@ -420,8 +421,4 @@ public class LocationPath {
public StorageAdapter getStorageAdapter() {
return storageAdapter;
}
-
- public Path getPath() {
- return new Path(normalizedLocation);
- }
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileGroupInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileGroupInfo.java
index 680989d959e..b1d70b467fc 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileGroupInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileGroupInfo.java
@@ -26,6 +26,7 @@ import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
import org.apache.doris.common.Pair;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.LocationPath;
import org.apache.doris.common.util.Util;
import org.apache.doris.load.BrokerFileGroup;
import org.apache.doris.planner.FileLoadScanNode;
@@ -47,11 +48,9 @@ import org.apache.doris.thrift.TUniqueKeyUpdateMode;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.Path;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
-import java.net.URI;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
@@ -413,8 +412,9 @@ public class FileGroupInfo {
rangeDesc.setColumnsFromPathKeys(columnsFromPathKeys);
rangeDesc.setColumnsFromPathIsNull(columnsFromPathIsNull);
if (getFileType() == TFileType.FILE_HDFS) {
- URI fileUri = new Path(fileStatus.path).toUri();
- rangeDesc.setFsName(fileUri.getScheme() + "://" +
fileUri.getAuthority());
+ // LocationPath parses scheme and authority into fsIdentifier
in exactly this
+ // "scheme://authority" shape, without pulling in hadoop.
+
rangeDesc.setFsName(LocationPath.of(fileStatus.path).getFsIdentifier());
}
} else {
// for stream load
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java
index 44a05404ed4..75bac73beaa 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/scan/FileQueryScanNode.java
@@ -77,7 +77,6 @@ import lombok.Getter;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
-import java.net.URI;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
@@ -583,8 +582,11 @@ public abstract class FileQueryScanNode extends
FileScanNode {
rangeDesc.setFileType(fileSplit.getLocationType());
rangeDesc.setPath(fileSplit.getPath().toStorageLocation().toString());
if (fileSplit.getLocationType() == TFileType.FILE_HDFS) {
- URI fileUri = fileSplit.getPath().getPath().toUri();
- rangeDesc.setFsName(fileUri.getScheme() + "://" +
fileUri.getAuthority());
+ // LocationPath already parsed scheme and authority into
fsIdentifier, in exactly this
+ // "scheme://authority" shape; re-deriving it through hadoop's
Path bought nothing. It also
+ // emitted the literal "hdfs://null" when a location carried no
authority, where
+ // fsIdentifier yields "hdfs://".
+ rangeDesc.setFsName(fileSplit.getPath().getFsIdentifier());
}
rangeDesc.setModificationTime(fileSplit.getModificationTime());
return rangeDesc;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/split/FileBlockLocation.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/split/FileBlockLocation.java
new file mode 100644
index 00000000000..033f2f4961d
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/split/FileBlockLocation.java
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.split;
+
+/**
+ * One block of a file, described by its offset, its length and the hosts
holding it.
+ *
+ * <p>This is the exact subset of {@code org.apache.hadoop.fs.BlockLocation}
that
+ * {@link FileSplitter} reads — {@code getOffset()}, {@code getLength()} and
{@code getHosts()} —
+ * reimplemented as a fe-core type so split planning carries no hadoop
dependency. The hadoop
+ * constructor's unused {@code names} argument is not carried over.
+ *
+ * <p>Null {@code hosts} is normalized to an empty array, matching hadoop:
{@code new
+ * BlockLocation(null, null, 0, len).getHosts()} returns {@code String[0]},
never null. Splitting a
+ * file with no block information relies on that — it builds a single
synthetic block with null
+ * hosts, and the resulting splits must carry an empty host array rather than
a null one.
+ */
+public class FileBlockLocation {
+
+ private static final String[] EMPTY_HOSTS = new String[0];
+
+ private final long offset;
+ private final long length;
+ private final String[] hosts;
+
+ public FileBlockLocation(String[] hosts, long offset, long length) {
+ this.hosts = hosts == null ? EMPTY_HOSTS : hosts;
+ this.offset = offset;
+ this.length = length;
+ }
+
+ public long getOffset() {
+ return offset;
+ }
+
+ public long getLength() {
+ return length;
+ }
+
+ public String[] getHosts() {
+ return hosts;
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/split/FileSplitter.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/split/FileSplitter.java
index 9ea50b95d5d..41bf5eaad6c 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/split/FileSplitter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/split/FileSplitter.java
@@ -26,7 +26,6 @@ import com.google.common.base.Preconditions;
import com.google.common.base.Verify;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.BlockLocation;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -94,7 +93,7 @@ public class FileSplitter {
public List<Split> splitFile(
LocationPath path,
long specifiedFileSplitSize,
- BlockLocation[] blockLocations,
+ FileBlockLocation[] blockLocations,
long length,
long modificationTime,
boolean splittable,
@@ -108,8 +107,8 @@ public class FileSplitter {
// Pass splitCreator.create() to set target file split size to
calculate split weight.
long targetFileSplitSize = specifiedFileSplitSize > 0 ?
specifiedFileSplitSize : maxSplitSize;
if (blockLocations == null) {
- blockLocations = new BlockLocation[1];
- blockLocations[0] = new BlockLocation(null, null, 0L, length);
+ blockLocations = new FileBlockLocation[1];
+ blockLocations[0] = new FileBlockLocation(null, 0L, length);
}
List<Split> result = Lists.newArrayList();
TFileCompressType compressType =
Util.inferFileCompressTypeByPath(path.getNormalizedLocation());
@@ -147,7 +146,7 @@ public class FileSplitter {
// split file by block
long start = 0;
ImmutableList.Builder<InternalBlock> blockBuilder =
ImmutableList.builder();
- for (BlockLocation blockLocation : blockLocations) {
+ for (FileBlockLocation blockLocation : blockLocations) {
// clamp the block range
long blockStart = Math.max(start, blockLocation.getOffset());
long blockEnd = Math.min(start + length, blockLocation.getOffset()
+ blockLocation.getLength());
@@ -215,7 +214,7 @@ public class FileSplitter {
}
}
- private int getBlockIndex(BlockLocation[] blkLocations, long offset) {
+ private int getBlockIndex(FileBlockLocation[] blkLocations, long offset) {
if (blkLocations == null || blkLocations.length == 0) {
return -1;
}
@@ -225,7 +224,7 @@ public class FileSplitter {
return i;
}
}
- BlockLocation last = blkLocations[blkLocations.length - 1];
+ FileBlockLocation last = blkLocations[blkLocations.length - 1];
long fileLength = last.getOffset() + last.getLength() - 1L;
throw new IllegalArgumentException(String.format("Offset %d is outside
of file (0..%d)", offset, fileLength));
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java
index 55583433a44..2c236218c8f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/storage/StorageAdapter.java
@@ -18,7 +18,6 @@
package org.apache.doris.datasource.storage;
import org.apache.doris.common.Config;
-import
org.apache.doris.datasource.property.common.AwsCredentialsProviderFactory;
import org.apache.doris.datasource.property.common.AwsCredentialsProviderMode;
import org.apache.doris.filesystem.properties.FileSystemProperties;
import org.apache.doris.filesystem.properties.HadoopStorageProperties;
@@ -29,17 +28,12 @@ import
org.apache.doris.foundation.security.ExecutionAuthenticator;
import org.apache.doris.fs.FileSystemPluginManager;
import com.google.common.collect.ImmutableMap;
-import com.google.common.collect.ImmutableSet;
-import org.apache.commons.lang3.BooleanUtils;
import org.apache.commons.lang3.StringUtils;
-import org.apache.hadoop.conf.Configuration;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
-import
software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider;
import java.util.ArrayList;
import java.util.HashMap;
-import java.util.LinkedHashSet;
import java.util.List;
import java.util.Locale;
import java.util.Map;
@@ -52,8 +46,11 @@ import java.util.Set;
*
* <p>One adapter wraps one {@link FileSystemProperties} binding produced by
* {@link FileSystemPluginManager#bindPrimary}/{@code bindAll}. Its public
surface mirrors the
- * legacy typed storage-properties contract exactly — backend map,
- * Hadoop configuration, storage name, schemas, type — so consumers can
migrate mechanically.
+ * legacy typed storage-properties contract exactly — backend map, storage
name, schemas, type
+ * — so consumers can migrate mechanically. Building a hadoop {@code
Configuration} is NOT part
+ * of that surface: the one legacy consumer (the Azure OAuth2 backend map) is
now served by
+ * {@code AzureFileSystemProperties.toMap()} inside fe-filesystem-azure, which
keeps fe-core
+ * source hadoop-free.
* Every known SPI-vs-fe-core drift from the master plan's §2.4 parity ledger
is reconciled here
* (or in the SPI implementation, where noted); each reconciliation carries a
* "align fe-core" comment referencing the ledger item.</p>
@@ -80,22 +77,6 @@ public final class StorageAdapter {
pluginManager = manager;
}
- private static final String SIMPLE_AWS_CREDENTIALS_PROVIDER =
- "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider";
- private static final String ASSUMED_ROLE_CREDENTIAL_PROVIDER =
- "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider";
-
- /**
- * Hadoop keys the facade re-derives for the S3 provider instead of taking
the SPI values:
- * fe-core gates them on accessKey-blank and on
Config.aws_credentials_provider_version,
- * which the SPI layer cannot see (align fe-core, ledger 2.4-3).
- */
- private static final Set<String> S3_CREDENTIAL_KEYS = ImmutableSet.of(
- "fs.s3a.aws.credentials.provider",
- "fs.s3a.assumed.role.arn",
- "fs.s3a.assumed.role.credentials.provider",
- "fs.s3a.assumed.role.external.id");
-
/**
* Legacy per-dialect alias prefix for {@code
force_parsing_by_standard_uri} (fe-core
* {@code AbstractS3CompatibleProperties} subclasses declared
@@ -123,14 +104,6 @@ public final class StorageAdapter {
private final AwsCredentialsProviderMode s3CredentialsMode;
/** Resolved once at construction (legacy bound it at init); read per file
in listing loops. */
private final String forceParsingByStandardUriValue;
- /**
- * Lazily built (legacy {@code X.of()} factories never built one; {@code
new Configuration()}
- * re-parses the Hadoop XML defaults and is too expensive for
per-statement bindings that
- * never read it, e.g. cloud COPY INTO). Volatile double-checked;
construction is a pure
- * function of final fields.
- */
- private volatile Configuration hadoopStorageConfig;
- private volatile boolean hadoopStorageConfigBuilt;
/**
* Adapters are shared across threads (catalog adapter map, FS caches), so
the lazily cached
* map must be safely published — legacy classes were immune (eager init
or fresh map per
@@ -313,25 +286,12 @@ public final class StorageAdapter {
return spi.storageFamilyName();
}
- /** Legacy schemas() (drives ensureDisableCache): provider-declared legacy
scheme set. */
+ /** Legacy schemas(): provider-declared legacy scheme set. */
public Set<String> schemas() {
// provider-declared legacy scheme set (ledger 2.4-6 note lives on
legacyCacheSchemes)
return spi.legacyCacheSchemes();
}
- /** Hadoop configuration equivalent of legacy getHadoopStorageConfig();
null for BROKER/HTTP. */
- public Configuration getHadoopStorageConfig() {
- if (!hadoopStorageConfigBuilt) {
- synchronized (this) {
- if (!hadoopStorageConfigBuilt) {
- hadoopStorageConfig = buildHadoopStorageConfig();
- hadoopStorageConfigBuilt = true;
- }
- }
- }
- return hadoopStorageConfig;
- }
-
/** Legacy isKerberos() — meaningful for the HDFS family only. */
public boolean isKerberos() {
return
spi.toHadoopProperties().map(HadoopStorageProperties::isKerberos).orElse(false);
@@ -476,7 +436,13 @@ public final class StorageAdapter {
// Align fe-core: Broker/Local/Http return the raw user
properties verbatim.
return origProps;
case AZURE:
- return azureBackendConfigProperties();
+ // Provider-owned, both auth types (the OAuth2 map is a hadoop
Configuration dump
+ // built inside fe-filesystem-azure). Routed out here so it
never reaches the
+ // S3-family alignment below, exactly as before.
+ return spi.toBackendProperties()
+ .orElseThrow(() -> new IllegalStateException(
+ "Provider " + providerKey + " exposes no
backend properties"))
+ .toMap();
default:
break;
}
@@ -493,20 +459,6 @@ public final class StorageAdapter {
return base;
}
- /**
- * Align fe-core, ledger 2.4-7: with OAuth2 the legacy AzureProperties
dumps the ENTIRE
- * Hadoop Configuration (hadoop defaults + fs.azure.* + user fs.*
passthrough +
- * disable-cache keys) as the backend map; shared-key uses the SPI's exact
7-key map.
- */
- private Map<String, String> azureBackendConfigProperties() {
- if (!isAzureOauth2()) {
- return spi.toBackendProperties().orElseThrow().toMap();
- }
- Map<String, String> dump = new HashMap<>();
- getHadoopStorageConfig().forEach(entry -> dump.put(entry.getKey(),
entry.getValue()));
- return dump;
- }
-
/**
* Backend-map reconciliation for every S3-compatible provider
(S3/OSS/OBS/COS/GCS/MinIO/Ozone).
*/
@@ -567,150 +519,6 @@ public final class StorageAdapter {
return aligned;
}
- private Configuration buildHadoopStorageConfig() {
- switch (type) {
- case BROKER:
- case HTTP:
- // Align fe-core: BrokerProperties/HttpProperties leave
hadoopStorageConfig null.
- return null;
- case LOCAL:
- return buildLocalHadoopConfig();
- default:
- break;
- }
- // Align fe-core, ledger 2.4-2: `new Configuration()` loads
core-default/core-site,
- // where the SPI map is a bare key-value view.
- Configuration conf = new Configuration();
- Map<String, String> spiMap = spi.toHadoopProperties()
- .orElseThrow(() -> new IllegalStateException(
- "Provider " + providerKey + " exposes no hadoop
properties"))
- .toHadoopConfigurationMap();
- if ("JFS".equals(providerKey)) {
- // fe-core builds the HDFS-family Configuration FROM the backend
map, so the JFS
- // auth-key alignment (see alignJfsBackendMap) must be
materialized here too.
- spiMap = alignJfsBackendMap(spiMap);
- }
- boolean isS3 = "S3".equals(providerKey);
- boolean skipGcsAnonProvider = isGcsAnonymous();
- for (Map.Entry<String, String> entry : spiMap.entrySet()) {
- if (isS3 && S3_CREDENTIAL_KEYS.contains(entry.getKey())) {
- // Re-derived below with fe-core Config gating (align fe-core,
ledger 2.4-3).
- continue;
- }
- if (skipGcsAnonProvider &&
"fs.s3a.aws.credentials.provider".equals(entry.getKey())) {
- // Align fe-core: GCSProperties never sets an anonymous s3a
credentials provider;
- // the SPI's AnonymousAWSCredentialsProvider extra is dropped
here.
- continue;
- }
- conf.set(entry.getKey(), entry.getValue());
- }
- if (isS3) {
- applyS3CredentialProviders(conf,
(S3CompatibleFileSystemProperties) spi);
- }
- if ("AZURE".equals(providerKey) && !isAzureOauth2()) {
- applyAzureAccountKeysFromConfig(conf);
- }
- appendUserFsConfig(conf);
- ensureDisableCache(conf);
- return conf;
- }
-
- private Configuration buildLocalHadoopConfig() {
- // Align fe-core LocalProperties.initializeHadoopStorageConfig (Local
has no SPI
- // hadoop view; the two impl keys are fe-core knowledge).
- Configuration conf = new Configuration();
- conf.set("fs.local.impl", "org.apache.hadoop.fs.LocalFileSystem");
- conf.set("fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem");
- appendUserFsConfig(conf);
- ensureDisableCache(conf);
- return conf;
- }
-
- /**
- * Align fe-core, ledger 2.4-3: S3Properties only emits assumed-role keys
when accessKey is
- * blank, and selects the provider chain per
Config.aws_credentials_provider_version — the
- * SPI cannot see fe-core Config, so these four keys are owned by the
facade.
- */
- private void applyS3CredentialProviders(Configuration conf,
S3CompatibleFileSystemProperties s3) {
- boolean v2 =
Config.aws_credentials_provider_version.equalsIgnoreCase("v2");
- if (StringUtils.isNotBlank(s3.getAccessKey())) {
- // Static credentials win; access/secret/session keys came from
the SPI map.
- conf.set("fs.s3a.aws.credentials.provider",
SIMPLE_AWS_CREDENTIALS_PROVIDER);
- return;
- }
- if (StringUtils.isNotBlank(s3.getRoleArn())) {
- conf.set("fs.s3a.assumed.role.arn", s3.getRoleArn());
- conf.set("fs.s3a.aws.credentials.provider",
ASSUMED_ROLE_CREDENTIAL_PROVIDER);
- conf.set("fs.s3a.assumed.role.credentials.provider",
- v2 ?
AwsCredentialsProviderFactory.getV2ClassName(s3CredentialsMode, false)
- :
InstanceProfileCredentialsProvider.class.getName());
- if (StringUtils.isNotBlank(s3.getExternalId())) {
- conf.set("fs.s3a.assumed.role.external.id",
s3.getExternalId());
- }
- return;
- }
- if (v2) {
- conf.set("fs.s3a.aws.credentials.provider",
-
AwsCredentialsProviderFactory.getV2ClassName(s3CredentialsMode, true));
- }
- // v1 + anonymous: fe-core sets nothing and leaves the hadoop default
untouched.
- }
-
- /**
- * Align fe-core AzureProperties.setHDFSAzureAccountKeys: shared-key
account keys are derived
- * from Config.azure_blob_host_suffixes (blob + dfs endpoints,
admin-extensible), not from the
- * SPI's static suffix list. Runs before appendUserFsConfig so explicit
user values still win.
- */
- private void applyAzureAccountKeysFromConfig(Configuration conf) {
- Map<String, String> backend =
spi.toBackendProperties().orElseThrow().toMap();
- String accountName = backend.get("AWS_ACCESS_KEY");
- String accountKey = backend.get("AWS_SECRET_KEY");
- Set<String> suffixes = new LinkedHashSet<>();
- if (Config.azure_blob_host_suffixes != null) {
- for (String suffix : Config.azure_blob_host_suffixes) {
- if (StringUtils.isBlank(suffix)) {
- continue;
- }
- String normalized = suffix.trim().toLowerCase(Locale.ROOT);
- if (normalized.startsWith(".")) {
- normalized = normalized.substring(1);
- }
- if (!normalized.isEmpty()) {
- suffixes.add(normalized);
- }
- }
- }
- for (String suffix : suffixes) {
- conf.set(String.format("fs.azure.account.key.%s.%s", accountName,
suffix), accountKey);
- }
- conf.set("fs.azure.account.key", accountKey);
- }
-
- /** Ledger 2.4-8: user fs.* keys with non-blank values pass through into
the configuration. */
- private void appendUserFsConfig(Configuration conf) {
- origProps.forEach((key, value) -> {
- if (key.startsWith("fs.") && StringUtils.isNotBlank(value)) {
- conf.set(key, value);
- }
- });
- }
-
- /**
- * Ledger 2.4-8: per-schema FileSystem cache disabling over the LEGACY
schemas() set, with an
- * explicit user value taking precedence — copied from
StorageProperties.ensureDisableCache.
- */
- private void ensureDisableCache(Configuration conf) {
- for (String schema : schemas()) {
- String key = "fs." + schema + ".impl.disable.cache";
- String userValue = origProps.get(key);
- if (StringUtils.isNotBlank(userValue)) {
- conf.setBoolean(key, BooleanUtils.toBoolean(userValue));
- } else {
- conf.setBoolean(key, true);
- }
- }
- }
-
/** fe-core S3Properties alias list for s3.credentials_provider_type (no
AWS_* alias). */
private String feCoreS3CredentialsProviderType() {
return firstNonBlank(origProps.get("s3.credentials_provider_type"),
@@ -718,17 +526,6 @@ public final class StorageAdapter {
origProps.get("iceberg.rest.credentials_provider_type"));
}
- private boolean isAzureOauth2() {
- return
"OAuth2".equalsIgnoreCase(origProps.getOrDefault("azure.auth_type",
"SharedKey"));
- }
-
- private boolean isGcsAnonymous() {
- if (!"GCS".equals(providerKey) || !(spi instanceof
S3CompatibleFileSystemProperties)) {
- return false;
- }
- return !((S3CompatibleFileSystemProperties)
spi).hasStaticCredentials();
- }
-
/**
* Align fe-core AzureProperties.initNormalizeAndCheckProps: the temporary
fe-core-only
* restriction that OAuth2 is supported only for the Iceberg REST catalog.
This check reads
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsFileGroupInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsFileGroupInfo.java
index b170d4e60c6..2713365cf28 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsFileGroupInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsFileGroupInfo.java
@@ -26,6 +26,7 @@ import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
import org.apache.doris.common.Pair;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.LocationPath;
import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.scan.FederationBackendPolicy;
import org.apache.doris.datasource.scan.FileGroupInfo;
@@ -48,11 +49,9 @@ import org.apache.doris.thrift.TUniqueKeyUpdateMode;
import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
-import org.apache.hadoop.fs.Path;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
-import java.net.URI;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
@@ -430,8 +429,9 @@ public class NereidsFileGroupInfo {
rangeDesc.setColumnsFromPathKeys(columnsFromPathKeys);
rangeDesc.setColumnsFromPathIsNull(columnsFromPathIsNull);
if (getFileType() == TFileType.FILE_HDFS) {
- URI fileUri = new Path(fileStatus.path).toUri();
- rangeDesc.setFsName(fileUri.getScheme() + "://" +
fileUri.getAuthority());
+ // LocationPath parses scheme and authority into fsIdentifier
in exactly this
+ // "scheme://authority" shape, without pulling in hadoop.
+
rangeDesc.setFsName(LocationPath.of(fileStatus.path).getFsIdentifier());
}
} else {
// for stream load
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/cloud/stage/StageGlobTest.java
b/fe/fe-core/src/test/java/org/apache/doris/cloud/stage/StageGlobTest.java
new file mode 100644
index 00000000000..7ab3386da78
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/cloud/stage/StageGlobTest.java
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.cloud.stage;
+
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.Pair;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Covers the glob analysis behind {@code COPY INTO ... FROM @stage('<glob>')}.
+ *
+ * <p>{@code analyzeGlob} turns a user glob into (prefix, hasWildcard) pairs:
the prefix is what
+ * gets listed from the object store, and the flag says whether the listing
still has to be
+ * filtered. Both matter for correctness, not just performance -- too short a
prefix lists a whole
+ * bucket, too long a prefix silently skips files the user asked for.
+ *
+ * <p>The expectations are literals rather than a comparison against hadoop's
{@code GlobExpander}
+ * / {@code GlobFilter}, which {@link GlobPatterns} was ported from.
Equivalence to those classes
+ * was established with a differential run over the cases below plus 20k
fuzzed globs (matching
+ * results, exception types and exception messages); pinning literals here
keeps that behaviour
+ * from drifting without making fe-core tests depend on hadoop again.
+ */
+public class StageGlobTest {
+
+ private static String render(List<Pair<String, Boolean>> pairs) {
+ return pairs.stream().map(p -> p.first + "|" +
p.second).collect(Collectors.joining(", "));
+ }
+
+ private static String analyze(String glob) throws DdlException {
+ return render(StageUtil.analyzeGlob("qid", glob));
+ }
+
+ @Test
+ public void literalGlobListsExactlyThatKey() throws DdlException {
+ // No wildcard anywhere: the whole path is the prefix and no filtering
is needed.
+ Assertions.assertEquals("a/b/c.csv|false", analyze("a/b/c.csv"));
+ // A backslash escape makes '*' an ordinary character, so this is
still a literal key.
+ Assertions.assertEquals("a*b|false", analyze("a\\*b"));
+ }
+
+ @Test
+ public void prefixStopsAtTheFirstWildcardCharacter() throws DdlException {
+ // The prefix must keep every leading literal component, otherwise the
listing widens to
+ // the whole bucket.
+ Assertions.assertEquals("data/|true", analyze("data/*.parquet"));
+ Assertions.assertEquals("dt=2026-07-31/|true",
analyze("dt=2026-07-31/*"));
+ // ... and it must also keep the literal head *inside* the first
wildcard component.
+ Assertions.assertEquals("part-|true", analyze("part-?????.orc"));
+ Assertions.assertEquals("dir/sub/pre|true",
analyze("dir/sub/pre*post/x"));
+ // Wildcard in the very first component leaves nothing to narrow the
listing with.
+ Assertions.assertEquals("|true", analyze("*.csv"));
+ Assertions.assertEquals("|true", analyze("[abc]/x"));
+ }
+
+ @Test
+ public void nullAndEmptyGlobsDifferInWhetherFilteringIsNeeded() throws
DdlException {
+ // No pattern at all -> list everything under the stage and keep it
all.
+ Assertions.assertEquals("|true", analyze(null));
+ // An empty pattern is a literal, so nothing needs filtering.
+ Assertions.assertEquals("|false", analyze(""));
+ }
+
+ @Test
+ public void braceGroupWithSlashBecomesOneListingPerAlternative() throws
DdlException {
+ // This is the reason brace expansion runs before prefix analysis:
each alternative is a
+ // different object-store prefix, so one glob must produce two
listings.
+ Assertions.assertEquals("logs/2026/01/|true, logs/2026/02/|true",
+ analyze("logs/{2026/01,2026/02}/*.csv"));
+ Assertions.assertEquals("a/b|false, c/d|false", analyze("{a/b,c/d}"));
+ }
+
+ @Test
+ public void braceGroupWithoutSlashStaysInsideOneComponent() throws
DdlException {
+ // No slash inside the braces -> no expansion; the component is simply
wildcard-bearing.
+ Assertions.assertEquals("pre|true", analyze("pre{a,b}post"));
+ Assertions.assertEquals("|true", analyze("{a,b}.csv"));
+ }
+
+ @Test
+ public void malformedGlobsAreRejectedRatherThanListedFromAWrongPrefix() {
+ // Validation parity is the point: without it these would silently
degrade into a prefix
+ // listing instead of failing the statement.
+ assertRejected("a[b", "Unclosed character class");
+ assertRejected("a{b", "Unclosed group");
+ assertRejected("a\\", "An escaped character does not present");
+ // Character-class errors are left to the regex engine, exactly as
hadoop did.
+ assertRejected("[z-a]", "invalid character class range");
+ }
+
+ private static void assertRejected(String glob, String expectedFragment) {
+ DdlException thrown = Assertions.assertThrows(DdlException.class,
+ () -> StageUtil.analyzeGlob("qid", glob));
+ Assertions.assertTrue(thrown.getMessage().contains(expectedFragment),
+ "expected <" + expectedFragment + "> in: " +
thrown.getMessage());
+ Assertions.assertTrue(thrown.getMessage().contains("Failed to analyze
glob: " + glob),
+ thrown.getMessage());
+ }
+
+ @Test
+ public void hasWildcardCountsOnlyTheFourGlobMetacharacters() throws
Exception {
+ Assertions.assertFalse(GlobPatterns.hasWildcard("abc"));
+ Assertions.assertFalse(GlobPatterns.hasWildcard(""));
+ Assertions.assertTrue(GlobPatterns.hasWildcard("a*"));
+ Assertions.assertTrue(GlobPatterns.hasWildcard("a?"));
+ Assertions.assertTrue(GlobPatterns.hasWildcard("a[b]"));
+ Assertions.assertTrue(GlobPatterns.hasWildcard("a{b}"));
+ // An escaped metacharacter is a literal ...
+ Assertions.assertFalse(GlobPatterns.hasWildcard("a\\*b"));
+ // ... and the closing/separator characters are not wildcards on their
own.
+ Assertions.assertFalse(GlobPatterns.hasWildcard("a,b"));
+ Assertions.assertFalse(GlobPatterns.hasWildcard("a}b"));
+ Assertions.assertFalse(GlobPatterns.hasWildcard("a]b"));
+ }
+
+ @Test
+ public void expandFlattensOnlyBraceGroupsContainingASlash() throws
Exception {
+ // The four examples hadoop's GlobExpander javadoc pins, kept as the
port's contract.
+ Assertions.assertEquals(List.of("pa/bs", "pc/ds"),
GlobPatterns.expand("p{a/b,c/d}s"));
+ Assertions.assertEquals(List.of("a/b", "c/d", "{e,f}"),
GlobPatterns.expand("{a/b,c/d,{e,f}}"));
+ Assertions.assertEquals(List.of("{a,b}/b", "{a,b}/c/d", "{a,b}/e/f"),
+ GlobPatterns.expand("{a,b}/{b,{c/d,e/f}}"));
+ Assertions.assertEquals(List.of("{a,b}/c/d"),
GlobPatterns.expand("{a,b}/{c/\\d}"));
+ // Slash-free groups are left for per-component wildcard handling.
+ Assertions.assertEquals(List.of("{a,b}.csv"),
GlobPatterns.expand("{a,b}.csv"));
+ Assertions.assertEquals(List.of("no-braces"),
GlobPatterns.expand("no-braces"));
+ }
+
+ /**
+ * Pins a PRE-EXISTING defect so the hadoop-removal port is provably
behaviour-preserving --
+ * this is documentation of current behaviour, not an endorsement of it.
+ *
+ * <p>{@code x/{y,z}} should yield two prefixes but yields only {@code
x/y}: the group has no
+ * slash, so it survives expansion and reaches {@code
StageUtil.splitByComma}, whose closing
+ * {@code if (start != str.length() - 1)} drops the final alternative
whenever it is a single
+ * character. The bug is in Doris's own helper, not in the ported hadoop
code, and predates
+ * this change; fixing it changes which files COPY INTO reads and belongs
in its own commit.
+ */
+ @Test
+ public void knownDefectSingleCharTrailingAlternativeIsDropped() throws
DdlException {
+ Assertions.assertEquals("x/y|false", analyze("x/{y,z}"));
+ // Two characters instead of one and the alternative survives,
confirming the cause.
+ Assertions.assertEquals("x/y|false, x/zz|false", analyze("x/{y,zz}"));
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/split/FileSplitterTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/split/FileSplitterTest.java
index 9931edb19cc..fcfd336b476 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/split/FileSplitterTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/split/FileSplitterTest.java
@@ -20,7 +20,6 @@ package org.apache.doris.datasource.split;
import org.apache.doris.common.util.LocationPath;
import org.apache.doris.spi.Split;
-import org.apache.hadoop.fs.BlockLocation;
import org.junit.Assert;
import org.junit.Test;
@@ -36,7 +35,7 @@ public class FileSplitterTest {
@Test
public void testNonSplittableCompressedFileProducesSingleSplit() throws
Exception {
LocationPath loc = LocationPath.of("hdfs://example.com/path/file.gz");
- BlockLocation[] locations = new BlockLocation[]{new
BlockLocation(null, new String[]{"h1"}, 0L, 10 * MB)};
+ FileBlockLocation[] locations = new FileBlockLocation[]{new
FileBlockLocation(new String[]{"h1"}, 0L, 10 * MB)};
FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB,
DEFAULT_INITIAL_SPLITS);
List<Split> splits = fileSplitter.splitFile(
loc,
@@ -58,7 +57,7 @@ public class FileSplitterTest {
@Test
public void testEmptyBlockLocationsProducesSingleSplitAndNullHosts()
throws Exception {
LocationPath loc = LocationPath.of("hdfs://example.com/path/file");
- BlockLocation[] locations = new BlockLocation[0];
+ FileBlockLocation[] locations = new FileBlockLocation[0];
FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB,
DEFAULT_INITIAL_SPLITS);
List<Split> splits = fileSplitter.splitFile(
loc,
@@ -81,7 +80,7 @@ public class FileSplitterTest {
public void
testSplittableSingleBigBlockProducesExpectedSplitsWithInitialSmallChunks()
throws Exception {
LocationPath loc = LocationPath.of("hdfs://example.com/path/bigfile");
long length = 200 * MB;
- BlockLocation[] locations = new BlockLocation[]{new
BlockLocation(null, new String[]{"h1"}, 0L, length)};
+ FileBlockLocation[] locations = new FileBlockLocation[]{new
FileBlockLocation(new String[]{"h1"}, 0L, length)};
// set maxInitialSplits to 2 to force the first two splits to be small.
FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB, 2);
List<Split> splits = fileSplitter.splitFile(
@@ -110,13 +109,48 @@ public class FileSplitterTest {
Assert.assertEquals(0, fileSplitter.getRemainingInitialSplitNum());
}
+ @Test
+ public void testNullBlockLocationsSplitLikeOneWholeFileBlock() throws
Exception {
+ // The only production caller (TVFScanNode) always passes null block
locations, so this is the
+ // shape that actually ships. Null must degrade to a single synthetic
block spanning the whole
+ // file -- not to "no splits" and not to one unsplit range -- which is
why the expected sizes
+ // below are identical to
testSplittableSingleBigBlockProducesExpectedSplitsWithInitialSmallChunks
+ // with the same splitter settings. A broken synthetic block would
silently change how many
+ // ranges the BE receives for every TVF scan.
+ LocationPath loc = LocationPath.of("hdfs://example.com/path/bigfile");
+ long length = 200 * MB;
+ FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB, 2);
+ List<Split> splits = fileSplitter.splitFile(
+ loc,
+ 0L,
+ null,
+ length,
+ 0L,
+ true,
+ Collections.emptyList(),
+ FileSplit.FileSplitCreator.DEFAULT);
+
+ long[] expected = new long[]{32 * MB, 32 * MB, 64 * MB, 36 * MB, 36 *
MB};
+ Assert.assertEquals(expected.length, splits.size());
+ long sum = 0L;
+ for (int i = 0; i < expected.length; i++) {
+ FileSplit s = (FileSplit) splits.get(i);
+ Assert.assertEquals(expected[i], s.getLength());
+ sum += s.getLength();
+ // No locality information is available, so no split may claim a
host.
+ Assert.assertNotNull(s.getHosts());
+ Assert.assertEquals(0, s.getHosts().length);
+ }
+ Assert.assertEquals(length, sum);
+ }
+
@Test
public void testMultiBlockSplitsAndHostPreservation() throws Exception {
LocationPath loc =
LocationPath.of("hdfs://example.com/path/twoblocks");
long len = 96 * MB;
- BlockLocation[] locations = new BlockLocation[]{
- new BlockLocation(null, new String[]{"h1"}, 0L, 48 * MB),
- new BlockLocation(null, new String[]{"h2"}, 48 * MB, 48 * MB)
+ FileBlockLocation[] locations = new FileBlockLocation[]{
+ new FileBlockLocation(new String[]{"h1"}, 0L, 48 * MB),
+ new FileBlockLocation(new String[]{"h2"}, 48 * MB, 48 * MB)
};
FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB, 0);
List<Split> splits = fileSplitter.splitFile(
@@ -141,9 +175,9 @@ public class FileSplitterTest {
public void testZeroLengthBlockIsSkipped() throws Exception {
LocationPath loc =
LocationPath.of("hdfs://example.com/path/zeroblock");
long length = 10 * MB;
- BlockLocation[] locations = new BlockLocation[]{
- new BlockLocation(null, new String[]{"h1"}, 0L, 0L),
- new BlockLocation(null, new String[]{"h1"}, 0L, 10 * MB)
+ FileBlockLocation[] locations = new FileBlockLocation[]{
+ new FileBlockLocation(new String[]{"h1"}, 0L, 0L),
+ new FileBlockLocation(new String[]{"h1"}, 0L, 10 * MB)
};
FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB,
DEFAULT_INITIAL_SPLITS);
List<Split> splits = fileSplitter.splitFile(
@@ -164,7 +198,7 @@ public class FileSplitterTest {
@Test
public void testNonSplittableFlagDecrementsCounter() throws Exception {
LocationPath loc = LocationPath.of("hdfs://example.com/path/file.gz");
- BlockLocation[] locations = new BlockLocation[]{new
BlockLocation(null, new String[]{"h1"}, 0L, 10 * MB)};
+ FileBlockLocation[] locations = new FileBlockLocation[]{new
FileBlockLocation(new String[]{"h1"}, 0L, 10 * MB)};
FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB, 2);
List<Split> splits = fileSplitter.splitFile(
loc,
@@ -181,7 +215,7 @@ public class FileSplitterTest {
@Test
public void testNullRemainingInitialSplitIsAllowed() throws Exception {
LocationPath loc = LocationPath.of("hdfs://example.com/path/somefile");
- BlockLocation[] locations = new BlockLocation[]{new
BlockLocation(null, new String[]{"h1"}, 0L, 10 * MB)};
+ FileBlockLocation[] locations = new FileBlockLocation[]{new
FileBlockLocation(new String[]{"h1"}, 0L, 10 * MB)};
FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB,
DEFAULT_INITIAL_SPLITS);
List<Split> splits = fileSplitter.splitFile(
loc,
@@ -198,7 +232,7 @@ public class FileSplitterTest {
@Test
public void testZeroLengthFileProducesNoSplits() throws Exception {
LocationPath loc =
LocationPath.of("hdfs://example.com/path/emptyfile");
- BlockLocation[] locations = new BlockLocation[]{new
BlockLocation(null, new String[]{"h1"}, 0L, 0L)};
+ FileBlockLocation[] locations = new FileBlockLocation[]{new
FileBlockLocation(new String[]{"h1"}, 0L, 0L)};
FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB,
DEFAULT_INITIAL_SPLITS);
// Non-splittable zero-length file
List<Split> splits = fileSplitter.splitFile(
@@ -222,7 +256,7 @@ public class FileSplitterTest {
@Test
public void testSmallFileNoSplit() throws Exception {
LocationPath loc = LocationPath.of("hdfs://example.com/path/small");
- BlockLocation[] locations = new BlockLocation[]{new
BlockLocation(null, new String[]{"h1"}, 0L, 2 * MB)};
+ FileBlockLocation[] locations = new FileBlockLocation[]{new
FileBlockLocation(new String[]{"h1"}, 0L, 2 * MB)};
FileSplitter fileSplitter = new FileSplitter(32 * MB, 64 * MB,
DEFAULT_INITIAL_SPLITS);
List<Split> splits = fileSplitter.splitFile(
loc,
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/planner/FederationBackendPolicyTest.java
b/fe/fe-core/src/test/java/org/apache/doris/planner/FederationBackendPolicyTest.java
index 9c04b73d61c..34ec8355a96 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/planner/FederationBackendPolicyTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/planner/FederationBackendPolicyTest.java
@@ -32,7 +32,6 @@ import org.apache.doris.system.SystemInfoService;
import com.google.common.collect.ArrayListMultimap;
import com.google.common.collect.ListMultimap;
import com.google.common.collect.Multimap;
-import org.apache.hadoop.fs.Path;
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
@@ -155,10 +154,10 @@ public class FederationBackendPolicyTest {
for (Split split : assignedSplits) {
FileSplit fileSplit = (FileSplit) split;
++totalSplitNum;
- if (fileSplit.getPath().getPath().equals(new
Path("hdfs://HDFS8000871/usr/hive/warehouse/clickbench.db/hits_orc/part-00000-3e24f7d5-f658-4a80-a168-7b215c5a35bf-c000.snappy.orc")))
{
+ if
(fileSplit.getPath().getNormalizedLocation().equals("hdfs://HDFS8000871/usr/hive/warehouse/clickbench.db/hits_orc/part-00000-3e24f7d5-f658-4a80-a168-7b215c5a35bf-c000.snappy.orc"))
{
Assert.assertEquals("172.30.0.100", backend.getHost());
checkedLocalSplit.add(true);
- } else if (fileSplit.getPath().getPath().equals(new
Path("hdfs://HDFS8000871/usr/hive/warehouse/clickbench.db/hits_orc/part-00003-3e24f7d5-f658-4a80-a168-7b215c5a35bf-c000.snappy.orc")))
{
+ } else if
(fileSplit.getPath().getNormalizedLocation().equals("hdfs://HDFS8000871/usr/hive/warehouse/clickbench.db/hits_orc/part-00003-3e24f7d5-f658-4a80-a168-7b215c5a35bf-c000.snappy.orc"))
{
Assert.assertEquals("172.30.0.106", backend.getHost());
checkedLocalSplit.add(true);
}
diff --git a/fe/fe-filesystem/fe-filesystem-azure/pom.xml
b/fe/fe-filesystem/fe-filesystem-azure/pom.xml
index fae9d3a230f..cb76d8c2f09 100644
--- a/fe/fe-filesystem/fe-filesystem-azure/pom.xml
+++ b/fe/fe-filesystem/fe-filesystem-azure/pom.xml
@@ -52,6 +52,43 @@ under the License.
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
+ <!-- OAuth2 backend map only (toMap): the legacy fe-core
AzureProperties handed BE a whole
+ org.apache.hadoop.conf.Configuration, so the map carries hadoop's
core-default.xml
+ plus any core-site.xml on the FE classpath (start_fe.sh puts both
${DORIS_HOME}/conf
+ and ${HADOOP_CONF_DIR} there). Reproducing that key-for-key needs
the real class.
+
+ Bundled (not provided), matching
fe-filesystem-hdfs/-jfs/-oss-hdfs, so the plugin does
+ not depend on the host shipping hadoop. Note this copy is a
FALLBACK, not the active
+ one: FileSystemPluginManager.FS_PARENT_FIRST_PREFIXES lists
"org.apache.hadoop.", so
+ ChildFirstClassLoader resolves hadoop from the parent whenever
fe/lib carries it and
+ only reaches these jars through the findClass fallback when it
does not. That ordering
+ is what keeps a single Class object in play today, i.e. no
classloader split-brain.
+
+ Logging exclusions mirror fe-filesystem-hdfs-base: the assembly
drops org.slf4j:* and
+ org.apache.logging.log4j:*, but log4j:log4j (1.x) is a different
groupId and would
+ otherwise land in the plugin lib/ and fight the host's logging.
-->
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ <exclusions>
+ <exclusion>
+ <groupId>commons-collections</groupId>
+ <artifactId>commons-collections</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-compress</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.slf4j</groupId>
+ <artifactId>slf4j-log4j12</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
<!-- Azure SDK (versions managed by azure-sdk-bom in fe/pom.xml
dependencyManagement) -->
<dependency>
diff --git
a/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProperties.java
b/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProperties.java
index c7a65df4b69..33b611c7b76 100644
---
a/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProperties.java
+++
b/fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystemProperties.java
@@ -27,7 +27,9 @@ import
org.apache.doris.foundation.property.ConnectorPropertiesUtils;
import org.apache.doris.foundation.property.ConnectorProperty;
import org.apache.doris.foundation.property.ParamRules;
+import org.apache.commons.lang3.BooleanUtils;
import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
import java.lang.reflect.Field;
import java.net.URI;
@@ -235,7 +237,7 @@ public final class AzureFileSystemProperties
// S3-style params. Mirrors legacy
AzureProperties.getBackendConfigProperties();
// native-SDK OAuth2 support may replace this in the future.
if (isOauth2Auth()) {
- return toHadoopConfigurationMap();
+ return oauth2BackendProperties();
}
Map<String, String> s3Props = new HashMap<>();
s3Props.put("AWS_ENDPOINT", endpoint);
@@ -275,6 +277,51 @@ public final class AzureFileSystemProperties
return Collections.unmodifiableMap(cfg);
}
+ /**
+ * BE property map for OAuth2, key-for-key equal to what legacy fe-core
+ * {@code AzureProperties.getBackendConfigProperties()} produced: it
dumped its whole
+ * {@link Configuration}, so on top of {@link #toHadoopConfigurationMap()}
the map also carries
+ * hadoop's {@code core-default.xml} and any {@code core-site.xml}
reachable on the FE
+ * classpath ({@code start_fe.sh} puts {@code ${DORIS_HOME}/conf} and
{@code ${HADOOP_CONF_DIR}}
+ * there, so this is a real operator-facing channel, not just hadoop's
built-in defaults).
+ *
+ * <p>The live consumer is Microsoft Fabric OneLake: {@code
LocationPath.getTFileTypeForBE()}
+ * routes {@code abfs[s]://...dfs.fabric.microsoft.com} to {@code
FILE_HDFS}, and BE's
+ * {@code hdfs_builder.cpp} feeds every entry of this map into its JNI
hadoop builder — which is
+ * how the {@code fs.azure.account.oauth2.*} keys reach the ABFS connector.
+ *
+ * <p>Ordering is load-bearing and mirrors the legacy sequence: hadoop
defaults first, then the
+ * provider's own {@code fs.azure.*} view, then user {@code fs.*}
passthrough (so an explicit
+ * user value wins), then cache-disable normalization last.
+ *
+ * <p>The plugin bundles hadoop, but {@code
FileSystemPluginManager.FS_PARENT_FIRST_PREFIXES}
+ * makes {@code org.apache.hadoop.} parent-first, so whenever the FE host
ships hadoop this
+ * resolves to the host copy and the bundled one is only a {@code
findClass} fallback. That also
+ * means the XML defaults below come from wherever the context classloader
finds them — hadoop's
+ * {@code Configuration} looks up {@code core-default.xml}/{@code
core-site.xml} through the
+ * TCCL, not through this class's loader. A host that stops shipping
hadoop would therefore need
+ * the TCCL pinned to the plugin loader here, not just the bundled jars.
+ */
+ private Map<String, String> oauth2BackendProperties() {
+ Configuration conf = new Configuration();
+ toHadoopConfigurationMap().forEach(conf::set);
+ rawProperties.forEach((key, value) -> {
+ if (key.startsWith("fs.") && StringUtils.isNotBlank(value)) {
+ conf.set(key, value);
+ }
+ });
+ for (String scheme : legacyCacheSchemes()) {
+ String key = "fs." + scheme + ".impl.disable.cache";
+ String userValue = rawProperties.get(key);
+ // An explicit user value wins but is normalized to true/false
("yes"/"1" would reach BE
+ // verbatim through the fs.* passthrough above otherwise).
+ conf.setBoolean(key, StringUtils.isNotBlank(userValue) ?
BooleanUtils.toBoolean(userValue) : true);
+ }
+ Map<String, String> dump = new HashMap<>();
+ conf.forEach(entry -> dump.put(entry.getKey(), entry.getValue()));
+ return Collections.unmodifiableMap(dump);
+ }
+
public String getEndpoint() {
return endpoint;
}
diff --git
a/fe/fe-filesystem/fe-filesystem-azure/src/test/java/org/apache/doris/filesystem/azure/AzureFileSystemPropertiesTest.java
b/fe/fe-filesystem/fe-filesystem-azure/src/test/java/org/apache/doris/filesystem/azure/AzureFileSystemPropertiesTest.java
index 23283968b3e..889d4c14dd0 100644
---
a/fe/fe-filesystem/fe-filesystem-azure/src/test/java/org/apache/doris/filesystem/azure/AzureFileSystemPropertiesTest.java
+++
b/fe/fe-filesystem/fe-filesystem-azure/src/test/java/org/apache/doris/filesystem/azure/AzureFileSystemPropertiesTest.java
@@ -117,6 +117,78 @@ class AzureFileSystemPropertiesTest {
Assertions.assertEquals("azure", backendMap.get("provider"));
Assertions.assertEquals("true", backendMap.get("use_path_style"));
Assertions.assertFalse(backendMap.keySet().stream().anyMatch(keyName
-> keyName.startsWith("AZURE_")));
+ // The hadoop Configuration dump is the OAuth2 arm ONLY: SharedKey
must stay the exact
+ // 7-key S3-style map, so a regression that applies the dump
unconditionally shows up here.
+ Assertions.assertEquals(7, backendMap.size(), backendMap.toString());
+ }
+
+ /**
+ * Pins the OAuth2 backend map that moved here from fe-core {@code
StorageAdapter} (which built a
+ * hadoop {@code Configuration} and dumped it). BE's live consumer is
Microsoft Fabric OneLake:
+ * {@code abfs[s]://...dfs.fabric.microsoft.com} routes to {@code
FILE_HDFS} and every entry is
+ * fed into BE's JNI hadoop builder, so dropping the hadoop-resolved keys
silently changes what
+ * the ABFS connector is configured with.
+ */
+ @Test
+ void toBackendProperties_oauth2DumpsHadoopResolvedConfig() {
+ AzureFileSystemProperties properties =
AzureFileSystemProperties.of(Map.of(
+ "azure.endpoint", "account.blob.core.windows.net",
+ "azure.auth_type", "OAuth2",
+ "azure.oauth2_account_host", "myaccount.dfs.core.windows.net",
+ "azure.oauth2_client_id", "client-id",
+ "azure.oauth2_client_secret", "client-secret",
+ "azure.oauth2_server_uri",
"https://login.microsoftonline.com/tenant/oauth2/token";));
+
+ Map<String, String> backendMap =
properties.toBackendProperties().orElseThrow().toMap();
+
+ // 1. The OAuth config the ABFS connector actually authenticates with.
+ Assertions.assertEquals("OAuth",
+
backendMap.get("fs.azure.account.auth.type.myaccount.dfs.core.windows.net"));
+
Assertions.assertEquals("org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
+
backendMap.get("fs.azure.account.oauth.provider.type.myaccount.dfs.core.windows.net"));
+ Assertions.assertEquals("client-id",
+
backendMap.get("fs.azure.account.oauth2.client.id.myaccount.dfs.core.windows.net"));
+ Assertions.assertEquals("client-secret",
+
backendMap.get("fs.azure.account.oauth2.client.secret.myaccount.dfs.core.windows.net"));
+
Assertions.assertEquals("https://login.microsoftonline.com/tenant/oauth2/token";,
+
backendMap.get("fs.azure.account.oauth2.client.endpoint.myaccount.dfs.core.windows.net"));
+
+ // 2. hadoop core-default.xml is merged in. This is the whole reason
the module compiles
+ // against hadoop-common: a plain key-value map would carry the
OAuth keys above but none
+ // of these, and the change would be invisible to every other
assertion.
+ Assertions.assertEquals("file:///", backendMap.get("fs.defaultFS"));
+
Assertions.assertTrue(backendMap.containsKey("hadoop.security.authentication"),
backendMap.toString());
+ Assertions.assertTrue(backendMap.size() > 100,
+ "expected a resolved hadoop config, got " + backendMap.size()
+ " keys");
+
+ // 3. Never S3-style: OAuth2 has no AK/SK the BE S3 adapter could
consume.
+ Assertions.assertFalse(backendMap.containsKey("AWS_ACCESS_KEY"),
backendMap.toString());
+ Assertions.assertFalse(backendMap.containsKey("provider"),
backendMap.toString());
+ }
+
+ @Test
+ void
toBackendProperties_oauth2PassesUserFsKeysThroughAndNormalizesCacheFlags() {
+ AzureFileSystemProperties properties =
AzureFileSystemProperties.of(Map.of(
+ "azure.endpoint", "account.blob.core.windows.net",
+ "azure.auth_type", "OAuth2",
+ "azure.oauth2_account_host", "myaccount.dfs.core.windows.net",
+ "azure.oauth2_client_id", "client-id",
+ "azure.oauth2_client_secret", "client-secret",
+ "azure.oauth2_server_uri",
"https://login.microsoftonline.com/tenant/oauth2/token";,
+ // arbitrary user fs.* key, not azure-scoped: legacy passed
the whole fs.* family through
+ "fs.azure.readaheadqueue.depth", "8",
+ // explicit cache flag in a spelling only BooleanUtils
understands
+ "fs.abfss.impl.disable.cache", "no"));
+
+ Map<String, String> backendMap =
properties.toBackendProperties().orElseThrow().toMap();
+
+ Assertions.assertEquals("8",
backendMap.get("fs.azure.readaheadqueue.depth"));
+ // Explicit user value wins, but normalized to true/false — "no" must
not reach BE verbatim.
+ Assertions.assertEquals("false",
backendMap.get("fs.abfss.impl.disable.cache"));
+ // The other three legacy schemes keep the default.
+ Assertions.assertEquals("true",
backendMap.get("fs.abfs.impl.disable.cache"));
+ Assertions.assertEquals("true",
backendMap.get("fs.wasb.impl.disable.cache"));
+ Assertions.assertEquals("true",
backendMap.get("fs.wasbs.impl.disable.cache"));
}
@Test
diff --git a/fe/fe-filesystem/fe-filesystem-obs/pom.xml
b/fe/fe-filesystem/fe-filesystem-obs/pom.xml
index f4498f9d980..fdfbc0ae9ae 100644
--- a/fe/fe-filesystem/fe-filesystem-obs/pom.xml
+++ b/fe/fe-filesystem/fe-filesystem-obs/pom.xml
@@ -64,6 +64,41 @@ under the License.
<artifactId>huaweicloud-sdk-iam</artifactId>
<version>${huaweicloud-sdk-iam.version}</version>
</dependency>
+ <!-- Supplies org.apache.hadoop.fs.obs.OBSFileSystem. Nothing here
calls it: this module reaches OBS
+ through the native esdk SDK above, and the ONLY reference is the
initialize=false
+ Class.forName probe in ObsFileSystemProperties, which decides
whether toHadoopConfigurationMap
+ emits fs.obs.impl=OBSFileSystem or falls back to
fs.obs.impl=S3AFileSystem. The probe resolves
+ against THIS module's own classloader, so the jar has to sit in
this plugin for the probe to
+ tell the truth. Consumers that actually instantiate the class
carry their own copy
+ (fe-connector-paimon, be-java-extensions/hadoop-deps);
fe-connector-iceberg needs none because
+ it normalizes obs:// to s3 and goes through S3AFileSystem.
+ runtime scope keeps it off the compile classpath; plugin-zip.xml
packs the runtime closure.
+
+ com/obs/* DUPLICATION — read before touching versions or artifact
names. The -hw-46 jar is a
+ fat jar that inlines the whole esdk SDK (762 com/obs/* classes,
identical to the
+ esdk-obs-java-optimised 3.21.8.2 content), and
esdk-obs-java-bundle above ships 511 of which
+ 510 collide. That is TWO copies of com.obs.services.ObsClient in
this plugin's lib/, at
+ DIFFERENT versions (3.21.11 declared above vs 3.21.8.2 inlined
here), and ObsObjStorage
+ compiles against the former.
DirectoryPluginRuntimeManager.collectJars sorts lib/ jars
+ lexicographically, so "esdk-obs-java-bundle-*" precedes
"hadoop-huaweicloud-*" and the
+ declared 3.21.11 wins — correct, but only because of that
ordering. If either artifact is
+ renamed or the esdk coordinates change, re-check that the bundle
still sorts first.
+ The transitive esdk-obs-java-optimised is excluded below: it is a
third, redundant copy of
+ the same classes already inlined in the fat jar, and the only one
that could shadow the
+ declared bundle if the sort order ever shifted.
+ NOTE: the -hw-46 artifact is NOT in Maven Central; it comes from
the huawei-obs-sdk repo
+ declared in <repositories> below (fe-connector-paimon declares
the same repo for the same dep). -->
+ <dependency>
+ <groupId>com.huaweicloud</groupId>
+ <artifactId>hadoop-huaweicloud</artifactId>
+ <scope>runtime</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.huaweicloud</groupId>
+ <artifactId>esdk-obs-java-optimised</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
@@ -76,6 +111,14 @@ under the License.
</dependency>
</dependencies>
+ <repositories>
+ <!-- for huawei obs sdk (hadoop-huaweicloud is not published to Maven
Central) -->
+ <repository>
+ <id>huawei-obs-sdk</id>
+
<url>https://repo.huaweicloud.com/repository/maven/huaweicloudsdk/</url>
+ </repository>
+ </repositories>
+
<build>
<finalName>doris-fe-filesystem-obs</finalName>
<plugins>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
