vinothchandar commented on a change in pull request #1722:
URL: https://github.com/apache/hudi/pull/1722#discussion_r450165106
##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala
##########
@@ -58,26 +60,28 @@ class DefaultSource extends RelationProvider
throw new HoodieException("'path' must be specified.")
}
+ // Try to create hoodie table meta client from the give path
+ // TODO: Smarter path handling
+ val metaClient = try {
+ val conf = sqlContext.sparkContext.hadoopConfiguration
+ Option(new HoodieTableMetaClient(conf, path.get, true))
Review comment:
would n't be problematic if `path` is a glob and not the actual basePath
of the table? COW/Snapshot query can for e.g do this and I think we should
handle the same for MOR as well.
##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DataSourceOptions.scala
##########
@@ -65,7 +66,7 @@ object DataSourceReadOptions {
* This eases migration from old configs to new configs.
*/
def translateViewTypesToQueryTypes(optParams: Map[String, String]) :
Map[String, String] = {
- val translation = Map(VIEW_TYPE_READ_OPTIMIZED_OPT_VAL ->
QUERY_TYPE_SNAPSHOT_OPT_VAL,
+ val translation = Map(VIEW_TYPE_READ_OPTIMIZED_OPT_VAL ->
QUERY_TYPE_READ_OPTIMIZED_OPT_VAL,
Review comment:
No.. there are no more views.. we did a renaming exercise to clear
things up as "query types" .. with that there should be no confusion.. our docs
are consistent with this as well.. On COW there is in fact no RO view.. so
this change has to be done differently, if you need for MOR..
##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala
##########
@@ -58,26 +60,28 @@ class DefaultSource extends RelationProvider
throw new HoodieException("'path' must be specified.")
}
+ // Try to create hoodie table meta client from the give path
+ // TODO: Smarter path handling
+ val metaClient = try {
+ val conf = sqlContext.sparkContext.hadoopConfiguration
+ Option(new HoodieTableMetaClient(conf, path.get, true))
+ } catch {
+ case e: HoodieException => Option.empty
Review comment:
can just error out there?
##########
File path: hudi-spark/src/main/scala/org/apache/hudi/DefaultSource.scala
##########
@@ -123,4 +127,25 @@ class DefaultSource extends RelationProvider
}
override def shortName(): String = "hudi"
+
+ private def getReadOptimizedView(sqlContext: SQLContext,
Review comment:
we can rename to something like `getFilteredBaseFileRelation()`. Again,
don't want to bring back view nomenclature into the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
