kz930 opened a new issue, #7211: URL: https://github.com/apache/texera/issues/7211
### What happened? - `customDelimiter` on `CSVScanSourceOpDesc`, `ParallelCSVScanSourceOpDesc` and `CSVOldScanSourceOpDesc` is an `Option[String]` with no validation — no `@Size`, no `@Pattern`, no length check, no assertion — and the UI offers a free-text field. - Every consumer narrows it with `.charAt(0)`, because univocity's `setDelimiter` and scala-csv's `DefaultCSVFormat.delimiter` both take a `Char`. Characters beyond the first are dropped without a word. Driving each operator's own `sourceSchema()` over a three-column `;`-separated file: | Delimiter | Columns inferred | | --- | --- | | `;` | `id, name, score` | | `;;` | `id, name, score` — the second `;` ignored | | `;abc` | `id, name, score` — `abc` ignored | | `,;` | `id;name;score` — one column | | `,` (control) | `id;name;score` — identical to `,;` | All three operators behave identically. The last two rows are the decisive pair: `,;` and a bare `,` give byte-identical results, so the second character never participates — were both characters delimiters, `,;` would have produced three columns. The user-visible consequence: typing `;abc` produces a workflow that runs to completion with correct-looking output, and nothing ever says `abc` was discarded. Typing `,;` produces a single wrong column with no hint that only the first character counted. Expected: either the field rejects more than one character, or it states that only the first is used. ### How to reproduce? 1. Point a CSV File Scan at a `;`-separated file and set Delimiter to `;abc`. 2. The schema resolves to the same three columns as `;` — the extra characters vanish silently. 3. Set it to `,;` and the schema collapses to one column, exactly as a bare `,` would. ### Version/Branch 1.3.0-incubating-SNAPSHOT (main) ### Relevant log output ```shell (no error or warning is produced — the extra characters are dropped silently) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
