Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/8469#discussion_r38140769
--- Diff: docs/sql-programming-guide.md ---
@@ -2057,6 +2057,12 @@ options.
- The canonical name of SQL/DataFrame functions are now lower case (e.g.
sum vs SUM).
- It has been determined that using the DirectOutputCommitter when
speculation is enabled is unsafe
and thus this output committer will not be used when speculation is on,
independent of configuration.
+ - JSON data source will not automatically load new files that are created
by other applications
+ (i.e. files that are not inserted to the dataset through Spark SQL).
+ For a JSON persistent table (i.e. the metadata of the table is stored
in Hive Metastore),
+ users can use `REFRESH TABLE` SQL command or `HiveContext`'s
`refreshTable` method
+ to include those new files to the table. For a DataFrame representing a
JSON dataset, users need to recreate
+ the DataFrame and the new DataFrame will include new files.
--- End diff --
In the release note, we need to add `JSON data source will not
automatically load new files that are created by other applications (i.e. files
that are not inserted to the dataset through Spark SQL). [SPARK-10287]`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
