Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/15382#discussion_r83899709
  
    --- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
    @@ -741,7 +741,7 @@ private[sql] class SQLConf extends Serializable with 
CatalystConf with Logging {
     
       def variableSubstituteDepth: Int = getConf(VARIABLE_SUBSTITUTE_DEPTH)
     
    -  def warehousePath: String = new Path(getConf(WAREHOUSE_PATH)).toString
    +  def warehousePath: String = 
Utils.resolveURI(getConf(WAREHOUSE_PATH)).toString
    --- End diff --
    
    You are right, the difference was that the _default_ before was always a 
local path. Right now ends up defaulting to an HDFS path (when using with HDFS 
for example). But explicitly setting "/foo/bar" has been interpreted as a HDFS 
path in the past and that would change. Hm, OK I see your point about only 
applying this to the default value.
    
    Looking at `hive.metastore.warehouse.dir`, that seems to accept a path 
without scheme and that's supposed to be read as an HDFS path, I suppose.
    
    The catch remains Windows paths. It should always be possible to write 
these as URIs, but, then becomes mandatory to write them as a URI, if I make 
this new change. That's a downside. That's only if you need to specify a local, 
non-default path on Windows, which might be a fairly corner case.
    
    I can make the change to have it available for consideration here which I 
assume @vanzin  would favor, and I can see the logic of. How about @avulanov ? 
I know this has dragged on but I want to get it right. It's complex.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to