hubgeter opened a new pull request, #70802: URL: https://github.com/apache/doris/pull/70802
### Background Doris binds storage properties in file queries, exports and loads before opening a filesystem. For OUTFILE, the FE chooses the writer and passes a path and an optional suffix to the BE, which generates the output filename. ### What problem does this PR solve? Issue Number: None Related PR: #24334 (introduced the OUTFILE suffix option) PR Type: Bugfix Problem Summary: A statement could override the Hadoop implementation or redirect a remote filesystem to local storage. OUTFILE also accepted path separators in its suffix. The FE now checks options supplied by the statement before binding storage, verifies that the provider matches the requested operation, and constrains the suffix to a single filename component. ### Details - **Change map:** 205 changed lines of FE code cover statement options, provider selection and output validation. The other 141 changed lines cover FE and SQL cases, including an existing HDFS export fixture. - **Frequency:** Confirmed by tracing statement analysis and writer selection in the source. Runtime frequency was not measured. - **Root cause:** HDFS bindings accepted broad Hadoop configuration namespaces. OUTFILE checked the URI prefix before properties selected the backend, and the writer appended the suffix verbatim. - **Why existing tests missed it:** OutFileTest checked the HDFS default URI and object-store destinations. The file-suffix and ViewFS regression cases exercised valid extensions and remote mounts. They lacked negative cases for provider selection, mount targets and filename boundaries. - **Fix:** Check raw SQL options before resource expansion and provider binding. Allow standard HA providers and direct remote ViewFS links. Check resource usage before reading its configuration. Match readers and writers to the resolved provider, including the filesystem used for pre-write deletion. - **Other paths and branches:** The checks cover file TVFs, INSERT INTO TVF, OUTFILE and the shared WITH storage clause for EXPORT and LOAD. Named broker destinations retain broker-side path handling. Source inspection found the original Hadoop passthrough and unchecked suffix on branch-3.1, branch-4.0, branch-4.1 and branch-4.2. - **Behavior change:** Unsupported per-statement Hadoop options produce an analysis error. Native local OUTFILE requires the canonical local URI and the FE local-output setting. Compound extensions such as `csv.gz` remain valid. ### Testing - Checkstyle 9.3 with the repository configuration passed on all 16 changed Java files. The FE metadata dependency check, Groovy syntax parsing of both changed suites, and `git diff --check` passed. - Added FE tests for accepted HA and ViewFS settings, rejected property overrides, storage-kind mismatches, and valid and invalid suffixes. Added SQL cases covering a file query, OUTFILE, INSERT INTO TVF and EXPORT. - No local build, FE unit test execution or cluster regression run was performed. The Maven validation phase could not resolve uncached dependency metadata offline. The style checker and metadata dependency check ran independently. ### Release note File statements reject unsupported Hadoop overrides and incompatible storage providers. Move custom Hadoop settings into operator-managed configuration. OUTFILE suffixes accept letters, digits, dots, underscores, plus signs and hyphens. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes. Update the file TVF and OUTFILE references with the statement options and allowed suffix characters. - Pre-submission status: Partially verified: static checks passed. FE unit and SQL runtime coverage await CI. ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Implementation, tests and PR description. - Human review: Pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
