Repository: spark
Updated Branches:
refs/heads/master 3c0e9ce94 -> 57eddc718
[SPARK-25886][SQL][MINOR] Improve error message of `FailureSafeParser` and
`from_avro` in FAILFAST mode
## What changes were proposed in this pull request?
Currently in `FailureSafeParser` and `from_avro`, the exception is created with
such code
```
throw new SparkException("Malformed records are detected in record parsing. " +
s"Parse Mode: ${FailFastMode.name}.", e.cause)
```
1. The cause part should be `e` instead of `e.cause`
2. If `e` contains non-null message, it should be shown in
`from_json`/`from_csv`/`from_avro`, e.g.
```
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('1' (code
49)): was expecting a colon to separate field name and value
at [Source: (InputStreamReader); line: 1, column: 7]
```
3.Kindly show hint for trying PERMISSIVE in error message.
## How was this patch tested?
Unit test.
Closes #22895 from gengliangwang/improve_error_msg.
Authored-by: Gengliang Wang <[email protected]>
Signed-off-by: hyukjinkwon <[email protected]>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/57eddc71
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/57eddc71
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/57eddc71
Branch: refs/heads/master
Commit: 57eddc7182ece0030f6d0cc02339c0b8d8c0be5c
Parents: 3c0e9ce
Author: Gengliang Wang <[email protected]>
Authored: Wed Oct 31 20:22:57 2018 +0800
Committer: hyukjinkwon <[email protected]>
Committed: Wed Oct 31 20:22:57 2018 +0800
----------------------------------------------------------------------
.../main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala | 2 +-
.../org/apache/spark/sql/catalyst/util/FailureSafeParser.scala | 3 ++-
2 files changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/57eddc71/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
----------------------------------------------------------------------
diff --git
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
index ae61587..5656ac7 100644
---
a/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
+++
b/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala
@@ -102,7 +102,7 @@ case class AvroDataToCatalyst(
case FailFastMode =>
throw new SparkException("Malformed records are detected in record
parsing. " +
s"Current parse Mode: ${FailFastMode.name}. To process malformed
records as null " +
- "result, try setting the option 'mode' as 'PERMISSIVE'.",
e.getCause)
+ "result, try setting the option 'mode' as 'PERMISSIVE'.", e)
case _ =>
throw new AnalysisException(unacceptableModeMessage(parseMode.name))
}
http://git-wip-us.apache.org/repos/asf/spark/blob/57eddc71/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
----------------------------------------------------------------------
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
index fecfff5..76745b1 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
@@ -73,7 +73,8 @@ class FailureSafeParser[IN](
Iterator.empty
case FailFastMode =>
throw new SparkException("Malformed records are detected in record
parsing. " +
- s"Parse Mode: ${FailFastMode.name}.", e.cause)
+ s"Parse Mode: ${FailFastMode.name}. To process malformed records
as null " +
+ "result, try setting the option 'mode' as 'PERMISSIVE'.", e)
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
