zeroshade commented on code in PR #1508:
URL: https://github.com/apache/arrow-go/pull/1508#discussion_r4553086675
##########
arrow/csv/transformer.go:
##########
@@ -304,3 +296,38 @@ func (w *Writer) transformColToStringArr(typ
arrow.DataType, col arrow.Array, st
}
return res, nil
}
+
+// encodeListItems formats list items as {item1,item2,...}, quoting items with
+// CSV rules when they are empty or contain characters that would be ambiguous
+// inside the list, so that the reader can parse them back. An unquoted empty
+// item would be lost: {} is an empty list. Quoting braces keeps the items
+// intact for readers that strip every leading and trailing brace.
+//
+// A quoted item is never read back as null, so null items are written as the
+// null value unquoted, and non-null items equal to it are quoted. With an
+// empty null value, a list whose only item is null is written as {}.
+func encodeListItems(values []string, list arrow.Array, nullValue string)
string {
+ var b strings.Builder
+ b.WriteByte('{')
+ for i, v := range values {
+ if i > 0 {
+ b.WriteByte(',')
+ }
+ if list.IsNull(i) || (!listItemNeedsQuotes(v) && v !=
nullValue) {
+ b.WriteString(v)
+ continue
+ }
+ b.WriteByte('"')
+ b.WriteString(strings.ReplaceAll(v, `"`, `""`))
+ b.WriteByte('"')
+ }
+ b.WriteByte('}')
+ return b.String()
+}
+
+func listItemNeedsQuotes(v string) bool {
+ if v == "" || strings.ContainsAny(v, ",\"\r\n{}") {
Review Comment:
Good question. It doesn't need to: the separator inside a list value is
always a comma, in both the writer and the reader, and `WithComma` only changes
the separator between fields. If an item contains the field separator, for
example `a;b` with `WithComma(';')`, the whole list field gets quoted by
`encoding/csv` in the usual way, so it still reads back.
To pin that down I added `TestCSVListWithCustomComma` in 1bc9c8f2, which
round-trips list items containing `;` and `,` with `;` as the field separator,
and a sentence on `WithComma` saying it doesn't apply inside list values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
