OIiveirra commented on code in PR #69528:
URL: https://github.com/apache/doris/pull/69528#discussion_r4368441150
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/LocalTableValuedFunction.java:
##########
@@ -135,6 +143,38 @@ private void getFileListFromBackend() throws
AnalysisException {
}
}
+ /** Apply LOCAL admission to sinks that select the built-in local JNI
writer as well. */
+ public static void checkSinkEnabled(String tvfName, Map<String, String>
properties) {
+ String writerClass = properties.getOrDefault("writer_class", "");
+ // JNI consumes a C string, and ScannerLoader accepts slash-separated
class names.
+ int nul = writerClass.indexOf('\0');
+ if (nul >= 0) {
+ writerClass = writerClass.substring(0, nul);
+ }
+ if ("local".equalsIgnoreCase(tvfName)
Review Comment:
Fixed in 02f2ddded5c1f13c9a59cca1b04b02c582f4c831. Disabled LOCAL writes now
reach the actual Hadoop consumer guard, including implicit core-site/hdfs-site
provider overrides. The runtime registers defaults before freezing its
configuration and rejects local, unknown and overloaded routes before provider
initialization. BE also requires the matching patched runtime and binds cached
handlers to exact parameters/authority and the captured Context. The startup
opt-in for explicitly enabled writes is retained.
Validation: final standard FE/BE targeted suites passed with fresh XML and
unchanged source fingerprints (100 FE / 15 BE). Real pinned libhdfs controls
reject inherited local providers, overloaded providers and inherited local
child mounts before file IO; cold-JVM controls cover resource registration and
all default-URI entrypoints. No storage-data read/write/delete was used in
these runtime controls. The same mechanism is synchronized across #69526,
#69527 and #69528.
##########
fe/fe-core/src/main/java/org/apache/doris/tablefunction/LocalTableValuedFunction.java:
##########
@@ -135,6 +150,332 @@ private void getFileListFromBackend() throws
AnalysisException {
}
}
+ /** Apply LOCAL admission to the effective sink before planning or
filesystem side effects. */
+ public static void checkSinkEnabled(String tvfName, Map<String, String>
properties) {
+ if ("local".equalsIgnoreCase(tvfName) ||
isLocalWriter(properties.getOrDefault("writer_class", ""))) {
+ checkEnabled();
+ }
+ if ("hdfs".equalsIgnoreCase(tvfName)) {
+ if (!properties.containsKey("writer_class")) {
+ checkHdfsSinkEnabled(properties);
+ }
+ if
(Boolean.parseBoolean(properties.getOrDefault("delete_existing_files",
"false"))) {
+ checkHadoopFileSystemEnabled(
+
BrokerUtil.extractParentDirectory(properties.getOrDefault("file_path", "")),
properties);
+ }
+ }
+ }
+
+ private static boolean isLocalWriter(String writer) {
+ // ScannerLoader accepts slashes, and JNI passes the class name as a C
string.
+ return
"org.apache.doris.writer.LocalFileJniWriter".equals(nativeStringPrefix(writer).replace('/',
'.'));
+ }
+
+ /** Check the NameNode and provider used by the native FILE_HDFS writer. */
+ public static void checkHdfsSinkEnabled(Map<String, String> properties) {
+ checkHdfsSinkEnabled(properties.getOrDefault("file_path", ""),
properties);
+ }
+
+ public static void checkHdfsSinkEnabled(String outputPrefix, Map<String,
String> properties) {
+ String nameNode =
nativeStringPrefix(properties.getOrDefault("fs.defaultFS", "")).trim();
+ if ("default".equals(nameNode)) {
+ // libhdfs delegates this selector to the BE's defaults, which may
select local storage.
+ checkEnabled();
+ }
+ // libhdfs prefixes bare NameNodes with hdfs:// rather than using the
output path's scheme.
+ checkHadoopSchemeEnabled(nameNode.contains("://") ?
uriScheme(nameNode) : "hdfs", properties);
+ if (!Config.enable_local_tvf) {
+ try {
+ URI filesystem = new URI(nameNode.contains("://") ? nameNode :
"hdfs://" + nameNode);
+ // BE appends a generated filename directly to this prefix,
then convert_path
+ // removes the NameNode. Resolve its parent, not an exact
mount-point filename.
+ String generated = outputPrefix + "doris-tvf-file";
+ String rawNameNode = properties.getOrDefault("fs.defaultFS",
"");
+ if (!rawNameNode.isEmpty() && generated.contains(rawNameNode))
{
+ generated = generated.substring(rawNameNode.length());
+ }
+ if (!generated.startsWith("/")) {
+ generated = "/" + generated;
+ }
+ String path =
hadoopPathUri(nativeStringPrefix(generated)).getPath();
+ path = path.substring(0, path.lastIndexOf('/') + 1);
+ if (path.length() > 1) {
+ path = path.substring(0, path.length() - 1);
+ }
+ checkViewFsEnabled(filesystem, path, properties, new
HashSet<>());
+ } catch (URISyntaxException e) {
+ // Invalid NameNodes remain subject to the existing storage
validation.
+ }
+ }
+ }
+
+ /** Check Hadoop's path/default-filesystem selection before FE-side
cleanup. */
+ public static void checkHadoopFileSystemEnabled(String path, Map<String,
String> properties) {
Review Comment:
Fixed in 02f2ddded5c1f13c9a59cca1b04b02c582f4c831. Hadoop TVF reads are
guarded at FE listing and the actual BE consumer, including cached native
connections and Avro schema/data readers. The actual frozen configuration
resolves inherited/overloaded providers and ViewFS mounts; each real path is
rechecked instead of trusting FE-only properties. S3A bucket-option and JobConf
configuration copies preserve the request guard. Unmarked catalogs retain their
previous behavior; local reads use constrained local().
Validation: final standard FE/BE targeted suites passed with fresh XML and
unchanged source fingerprints (100 FE / 15 BE). Real pinned libhdfs controls
reject inherited local providers, overloaded providers and inherited local
child mounts before file IO; cold-JVM controls cover resource registration and
all default-URI entrypoints. No storage-data read/write/delete was used in
these runtime controls. The same mechanism is synchronized across #69526,
#69527 and #69528.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
