Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11509#discussion_r55081339
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala ---
@@ -464,215 +378,140 @@ abstract class OutputWriter {
}
}
-/**
- * ::Experimental::
- * A [[BaseRelation]] that provides much of the common code required for
relations that store their
- * data to an HDFS compatible filesystem.
- *
- * For the read path, similar to [[PrunedFilteredScan]], it can eliminate
unneeded columns and
- * filter using selected predicates before producing an RDD containing all
matching tuples as
- * [[Row]] objects. In addition, when reading from Hive style partitioned
tables stored in file
- * systems, it's able to discover partitioning information from the paths
of input directories, and
- * perform partition pruning before start reading the data. Subclasses of
[[HadoopFsRelation()]]
- * must override one of the four `buildScan` methods to implement the read
path.
- *
- * For the write path, it provides the ability to write to both
non-partitioned and partitioned
- * tables. Directory layout of the partitioned tables is compatible with
Hive.
- *
- * @constructor This constructor is for internal uses only. The
[[PartitionSpec]] argument is for
- * implementing metastore table conversion.
- *
- * @param maybePartitionSpec An [[HadoopFsRelation]] can be created with
an optional
- * [[PartitionSpec]], so that partition discovery can be skipped.
- *
- * @since 1.4.0
- */
-@Experimental
-abstract class HadoopFsRelation private[sql](
- maybePartitionSpec: Option[PartitionSpec],
- parameters: Map[String, String])
- extends BaseRelation with FileRelation with Logging {
+case class HadoopFsRelation(
+ sqlContext: SQLContext,
+ location: FileCatalog,
+ partitionSchema: StructType,
--- End diff --
The `partitionSchema` is immutable and only tells you the names / types of
the partitioning columns. The `partitionSpec` actually enumerates all the
possible values along with the files for each set of values. This can change
overtime, even for a single instance of a DataFrame.
We should think about reducing the redundancy here if possible though as
today the spec also includes another copy of the schema
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
