xkrogen commented on a change in pull request #33328:
URL: https://github.com/apache/spark/pull/33328#discussion_r671319290
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
##########
@@ -244,7 +244,11 @@ private[hive] class HiveMetastoreCatalog(sparkSession:
SparkSession) extends Log
paths = rootPath.toString :: Nil,
userSpecifiedSchema = Option(updatedTable.dataSchema),
bucketSpec = None,
- options = options,
+ // Do not interpret the 'path' option at all when tables are
read using the Hive
+ // source, since the URIs will already have been read from the
table's LOCATION.
+ // The `Map() ++` is necessary to force materialization since
the return of
+ // `filterKeys` is a view which is not serializable
+ options = Map() ++
options.filterKeys(!_.equalsIgnoreCase("path")),
Review comment:
Unfortunately `toMap` is insufficient to materialize the view, check out
the Scaladoc/implementation:
```
/** Returns this $coll as an immutable map.
*
* A new map will not be built; lazy collections will stay lazy.
*/
@deprecatedOverriding("Immutable maps should do nothing on toMap except
return themselves cast as a map.", "2.11.0")
override def toMap[T, U](implicit ev: (K, V) <:< (T, U)): immutable.Map[T,
U] =
self.asInstanceOf[immutable.Map[T, U]]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
