Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/5001#discussion_r26354593
--- Diff: docs/sql-programming-guide.md ---
@@ -907,9 +907,132 @@ SELECT * FROM parquetTable
</div>
+### Partition discovery
+
+Table partitioning is a common optimization approach used in systems like
Hive. In a partitioned
+table, data are usually stored in different directories, with partitioning
column values encoded in
+the path of each partition directory. The Parquet data source is now able
to discover and infer
+partitioning information automatically. For exmaple, we can store all our
previously used
+population data into a partitioned table using the following directory
structure, with two extra
+columns, `sex` and `country` as partitioning columns:
+
+{% highlight text %}
+
+path
+âââ to
+ âââ table
+ âââ sex=0
--- End diff --
I used to use "male" and "female", but then I tried to illustrate partition
discovery is also able to infer numeric column data types as well as strings.
However, I forgot to update the printed schema below.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
