Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/23052#discussion_r236584201
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
---
@@ -169,13 +169,18 @@ private[csv] class CsvOutputWriter(
context: TaskAttemptContext,
params: CSVOptions) extends OutputWriter with Logging {
- private val charset = Charset.forName(params.charset)
+ private var univocityGenerator: Option[UnivocityGenerator] = None
--- End diff --
We have not observe any race conditions so far. Instances of
`UnivocityGenerator` are created per-tasks as well as `OutputStreamWriter`s.
They share instances of schema and CSVOptions but we do not modify them while
writing. Inside of each `UnivocityGenerator`, we create an instance of
`CsvWriter` but I almost absolutely sure they do not share anything internally.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
