cloud-fan commented on a change in pull request #23665: [SPARK-26745][SQL] Skip
empty lines in JSON-derived DataFrames when skipParsing optimization in effect
URL: https://github.com/apache/spark/pull/23665#discussion_r251455360
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/FailureSafeParser.scala
##########
@@ -55,11 +56,15 @@ class FailureSafeParser[IN](
def parse(input: IN): Iterator[InternalRow] = {
try {
- if (skipParsing) {
- Iterator.single(InternalRow.empty)
- } else {
- rawParser.apply(input).toIterator.map(row => toResultRow(Some(row), ()
=> null))
- }
+ if (skipParsing) {
+ if (unparsedRecordIsNonEmpty(input)) {
Review comment:
> we introduced 3rd category - hidden (or invisible for users) input
This is a good definition. So there are 3 kinds of records: valid, bad and
hidden.
The definition of bad records is query-dependent. If the corrected column is
not read, then this is still a valid record.
`df.show()` displays valid records, and bad records if the mode is
permissive. This means the result of `df.show()` depends on a config. If we
want to make `df.count()` consistent with `df.show()`, maybe we should let it
depend on the same config?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
