This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 078f50b16f0 [SPARK-38826][SQL][DOCS] dropFieldIfAllNull option does
not work for empty JSON struct
078f50b16f0 is described below
commit 078f50b16f07db6df169254fac36f8c41201ca61
Author: morvenhuang <[email protected]>
AuthorDate: Sun Apr 17 08:51:20 2022 -0500
[SPARK-38826][SQL][DOCS] dropFieldIfAllNull option does not work for empty
JSON struct
### What changes were proposed in this pull request?
Make a minor correction for the documentation of json option
`dropFieldIfAllNull`
### Why are the changes needed?
The `dropFieldIfAllNull` option actually does not work for empty json
struct due to [SPARK-8093](https://issues.apache.org/jira/browse/SPARK-8093),
the empty struct will be dropped anyway.
We should update the doc, otherwise it would be confusing.
### Does this PR introduce _any_ user-facing change?
Yes
### How was this patch tested?
I've built the documentation locally and tested my change.
Closes #36111 from morvenhuang/SPARK-38826.
Authored-by: morvenhuang <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
---
docs/sql-data-sources-json.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/sql-data-sources-json.md b/docs/sql-data-sources-json.md
index b5f27aacf41..f75efd1108a 100644
--- a/docs/sql-data-sources-json.md
+++ b/docs/sql-data-sources-json.md
@@ -235,7 +235,7 @@ Data source options of JSON can be set via:
<tr>
<td><code>dropFieldIfAllNull</code></td>
<td><code>false</code></td>
- <td>Whether to ignore column of all null values or empty array/struct
during schema inference.</td>
+ <td>Whether to ignore column of all null values or empty array during
schema inference.</td>
<td>read</td>
</tr>
<tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
