HyukjinKwon commented on a change in pull request #30970:
URL: https://github.com/apache/spark/pull/30970#discussion_r549975019
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeCsvJsonExprs.scala
##########
@@ -98,12 +98,12 @@ object OptimizeCsvJsonExprs extends Rule[LogicalPlan] {
child
case g @ GetStructField(j @ JsonToStructs(schema: StructType, _, _, _),
ordinal, _)
- if schema.length > 1 =>
+ if schema.length > 1 && j.options.isEmpty =>
Review comment:
Maybe:
```suggestion
if schema.length > 1 && j.options.isEmpty =>
// Options here should be empty because the optimization should not
be enabled
// for some options. For example, when the parse mode is failfast it
should not
// optimize, and should force to parse the whole input JSON with
failing fast for
// an invalid input.
// To be more conservative, it does not optimize when any option is
set for now.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
