asereda-gs commented on issue #1032: [CALCITE-2769] CSV Adapter does not handle - Empty and malformed csv lines, space before or after comma URL: https://github.com/apache/calcite/pull/1032#issuecomment-471768930 Proposed solution looks very specific. What people think about the following alternative: 1. Add default method to [Source](https://github.com/apache/calcite/blob/9721283bd0ce46a337f51a3691585cca8003e399/core/src/main/java/org/apache/calcite/util/Source.java) interface: ```java default Stream<String> lines() { // TODO add close() logic return new BufferedReader(reader()).lines(); } ``` 2. Add / Replace constructor of [CsvSchema](https://github.com/apache/calcite/blob/master/example/csv/src/main/java/org/apache/calcite/adapter/csv/CsvSchema.java) to accept a list (stream) of sources (instead of parent dir as File) : ```java public CsvSchema(Stream<Source> sources, CsvTable.Flavor flavor) { } ``` If user wants to modify / filter lines based on particular logic they can wrap existing source and override `lines()` method: ```java return lines().map(String::trim()).filter(line -> !line.isEmpty()); ```` It also has the advantage of accepting generic sources of data like HDFS.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
