shahidki31 commented on a change in pull request #22502: [SPARK-25474][SQL] 
Support `spark.sql.statistics.fallBackToHdfs` in data source tables
URL: https://github.com/apache/spark/pull/22502#discussion_r314297097
 
 

 ##########
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala
 ##########
 @@ -71,7 +70,13 @@ case class HadoopFsRelation(
 
   override def sizeInBytes: Long = {
     val compressionFactor = sqlContext.conf.fileCompressionFactor
-    (location.sizeInBytes * compressionFactor).toLong
+    val defaultSize = (location.sizeInBytes * compressionFactor).toLong
+    location match {
+      case cfi: CatalogFileIndex if 
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled =>
 
 Review comment:
   `Yes. I have prepared some tests to illustrate this issue. These tests can 
be passed before this commit:`
   
   @wangyum First and 3rd will pass after the PR. 2nd test is a bug which fixed 
in the commit. 
   
   Btw, 1st and 3rd tests are not CatalogFileIndex, which any way won't come to 
this flow

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to