goldmedal commented on code in PR #12263:
URL: https://github.com/apache/datafusion/pull/12263#discussion_r1741127326
##########
datafusion/core/src/datasource/physical_plan/csv.rs:
##########
@@ -1210,6 +1244,44 @@ mod tests {
crate::assert_batches_eq!(expected, &result);
}
+ #[tokio::test]
+ async fn test_terminator() {
Review Comment:
I got two issues when creating the sqllogictests case:
1. I prepared a test file breaking the line by `CR`. However, it seems that
object store doesn't support this by default.
```
External error: query failed: DataFusion error: Object Store error: Generic
LocalFileSystem error: Requested range was invalid
```
2. I tried to create an external table with the terminator config like
```
statement ok
CREATE EXTERNAL TABLE stored_table_with_cf_terminator (
col1 TEXT,
col2 TEXT
) STORED AS CSV
LOCATION '../core/tests/data/cr_terminator.csv'
OPTIONS ('format.terminator' '\r', 'format.has_header' 'true');
```
However, there are some issues for parsing the value.
```
External error: statement failed: DataFusion error: Invalid or Unsupported
Configuration: Error parsing \r as u8. Non-ASCII string provided
```
It seems that `\r` is parsed to two different ASCII but it is one. 🤔
I tried to select `\r` directly
```
> select '\t';
+-----------+
| Utf8(" ") |
+-----------+
| |
+-----------+
1 row(s) fetched.
Elapsed 0.003 seconds.
```
It works fine but it's not valid for the config value. Is there any way to
correctly create `\r` character for the config value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
