goutamadwant opened a new issue, #12935:
URL: https://github.com/apache/seatunnel/issues/12935
### Search before asking
- [x] Searched open and closed issues and PRs for `CSV preamble`, `CSV skip
header`, and `skip_header_row_number`.
### What happened
CSV files can contain report metadata or other physical preamble lines
before their column header. With `enable_file_split=false`,
`csv_use_header_line=true`, and a positive `skip_header_row_number`, the shared
file-base reader constructs the header-aware CSV parser before skipping those
physical lines.
Commons CSV reads the first record as the header during construction and can
buffer the remaining input. The subsequent direct `BufferedReader.readLine()`
can therefore return EOF even when the file has enough lines. A preamble
containing unmatched quotes can also be parsed as CSV instead of being skipped.
This is separate from schema inference and from skipping the same header
twice in #9948 / #11606. An explicit schema is supplied here, and the skip
count covers only lines before the header.
### SeaTunnel Version
Reproduced on `dev` at `bab2d089ce81793871ddedcd81d4930564c7dddc`
(`3.0.1-SNAPSHOT`). Released versions were not separately tested.
### Reproduction and before/after example
Save this synthetic file as `/tmp/preamble.csv`:
```csv
generated report
name,id,age
Alice,1,10
Bob,2,20
```
Use the source settings below. The configured schema deliberately differs
from the header order to verify header-name mapping.
```hocon
env {
parallelism = 1
job.mode = "BATCH"
}
source {
LocalFile {
path = "/tmp/preamble.csv"
file_format_type = csv
enable_file_split = false
csv_use_header_line = true
skip_header_row_number = 1
schema {
fields {
id = int
name = string
age = int
}
}
}
}
sink {
Assert {
rules {
row_rules = [
{ rule_type = MIN_ROW, rule_value = 2 },
{ rule_type = MAX_ROW, rule_value = 2 }
]
}
}
}
```
**Before:** the reader fails with FILE-02 / an IOException indicating that
the file has fewer lines than expected to skip, despite sufficient input.
Header-only input with a zero skip count works.
**After the proposed fix:** skip `generated report` first, parse
`name,id,age` as the header, and emit exactly these rows in configured schema
order:
```text
[1, Alice, 10]
[2, Bob, 20]
```
The unit regression invokes the public `CsvReadStrategy.readProcess` path
and asserts these exact values. The engine fixture additionally covers a UTF-8
BOM, two preamble lines including an unmatched quote, and a quoted multiline
STRING.
### Running Command
The reader-level reproduction was run before modifying production code:
```sh
./mvnw -o -pl seatunnel-connectors-v2/connector-file/connector-file-base
-Dtest=CsvReadStrategyTest -Dskip.spotless=true test
```
Of 18 invocations, six errored in the affected header/skip scenarios before
the fix. All 18 passed afterward.
The config above illustrates the one-line preamble case; the executed engine
validation used the two-line preamble fixture and command under Testing below.
### Error Exception
```text
FileConnectorException: ErrorCode:[FILE-02], ErrorDescription:[Data
deserialization failed]
Caused by: java.io.IOException: File [preamble] has fewer lines than
expected to skip.
```
### Proposed fix and advantages
Move physical-line skipping before CSV parser construction in
`CsvReadStrategy`, while keeping the reader in try-with-resources before either
operation. Do not reapply skips to byte-range splits, which are handled during
enumeration. Use a long counter to match the existing long-valued option.
This allows ingestion of CSV exports with preambles without preprocessing
the input, preserves header-name mapping and multiline records, and keeps input
streams closed on initialization failure. The implementation is confined to the
shared file-base reader, its existing tests, LocalFile E2E coverage, and EN/ZH
docs.
### Compatibility / breaking changes
No public API, option, default, or dependency changes. No migration is
required. Zero-skip and headerless behavior remain covered. The separate
byte-range enumeration/header policy is unchanged; this fix does not claim to
add preamble-plus-header support to that path.
### Testing
Host JVM: Java 11 (Temurin 11.0.19). The Zeta E2E container uses
`eclipse-temurin:11-jdk`. Java 11 matches the current dev baseline; Java 8 is
not a supported target of this baseline.
- Reader regression: 18 invocations passed after the fix, including
LF/CRLF/CR, BOM, reordered headers, multiline values, no-header/zero-skip
controls, empty input, EOF handling, stream closure, and byte-range skip
protection.
- CSV format module: 15 tests passed.
- File-base module: 369 tests passed, including the 18 reader invocations;
zero failures/errors/skips.
- Zeta LocalFile E2E: one invocation passed, executing both the new preamble
fixture and the existing header-only control. Both jobs reached FINISHED.
- Scoped 65-module package and verify builds, Spotless, and `git diff
--check` passed.
```sh
./mvnw -o -pl
seatunnel-connectors-v2/connector-file/connector-file-base,seatunnel-formats/seatunnel-format-csv
test
RUN_ENGINES=zeta ./mvnw -o -pl
seatunnel-core/seatunnel-starter,seatunnel-connectors-v2/connector-file/connector-file-local,seatunnel-connectors-v2/connector-assert,seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-local-e2e
-am -DskipUT=true -DskipIT=false
-Dit.test=LocalFileIT#testLocalFileCsvWithPreamble
-Dfailsafe.failIfNoSpecifiedTests=false -DfailIfNoTests=false
-De2e.dependency.skip=false verify
```
These are local results. Full repository CI, Flink/Spark E2E, and remote
file-store validation have not been run for this patch.
### Zeta or Flink or Spark Version
Zeta built from the same dev revision.
### Java or Scala Version
Java 11.0.19; project-default Scala 2.12.15 build profile.
### Implementation branch
https://github.com/apache/seatunnel/compare/dev...goutamadwant:seatunnel:fix/csv-preamble-before-header
### Are you willing to submit PR?
- [x] Yes, I will submit the fix from the linked branch.
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
