[GitHub] [spark] cloud-fan commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command

2019-08-15 Thread GitBox
cloud-fan commented on issue #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN 
command
URL: https://github.com/apache/spark/pull/24759#issuecomment-521895096
 
 
   I'm OK to stay with the `TreeNode` string methods for now, but I do think we 
should refactor it to a visitor pattern in the future. It's more flexible and 
maintainable. We can put the related code in one place instead of spreading it 
to many places.
   
   BTW I have a concern about subqueries. How do we handle nested subqueries? 
It looks like the current format will flatten the nested subqueries, and put 
them all together in the `subqueries` section.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] felixcheung commented on issue #25442: [SPARK-28722][ML] Change sequential label sorting in StringIndexer fit to parallel

2019-08-15 Thread GitBox
felixcheung commented on issue #25442: [SPARK-28722][ML] Change sequential 
label sorting in StringIndexer fit to parallel
URL: https://github.com/apache/spark/pull/25442#issuecomment-521893411
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] 
fallback to hdfs when relation table stats is not available
URL: https://github.com/apache/spark/pull/25460#issuecomment-521892398
 
 
   To @shahidki31 . 
   
   @maropu and @cloud-fan meant the corner case when the table size is equal to 
the user configuration value (not 8.0EB). Let say we set the configuration to 
1GB and we have a static table T1 whose size happens to be 1GB. In that case, 
every query on that tables might invoke this function. Although it's a very 
special case, but it's a regression.
   
   So, @cloud-fan and @maropu suggested to close this PR and proceed with 
#24715 .
   
   I'm +1 for that suggestion because that is the correct way.
   
   I know that you are worrying that #24715 doesn't resolve 8.0EB issue. 
However, that should be covered by your UTs in the previous PR. In the worst 
case, some of your code might be reverted. However, your test cases should 
survive there. It's your contribution. I believe @wangyum 's PR will pass your 
existing test cases in addition to his new test code. That's the way we make 
Apache Spark stronger.
   
   How do you think about this, @shahidki31 ? It's a way of collaboration.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] 
fallback to hdfs when relation table stats is not available
URL: https://github.com/apache/spark/pull/25460#issuecomment-521892398
 
 
   To @shahidki31 . 
   
   @maropu and @cloud-fan meant the corner case when the table size is equal to 
the user configuration value (not 8.0EB). Let say we set the configuration to 
1GB and we have a static table T1 whose size happens to be 1GB. In that case, 
every query on that tables might invoke this function. Although it's a very 
special case, but it's a regression.
   
   So, @cloud-fan and @maropu suggested to close this PR and proceed with 
#24715 .
   
   I'm +1 for that suggestion because that is the correct way.
   
   I know that you are worrying that #24715 doesn't resolve 8.0EB issue. 
However, that should be covered by your UTs in the previous PR. In the worst 
case, some of your code might be reverted. However, your test cases should 
survive there. It's your contribution. I believe @wangyum 's PR will pass your 
test cases in addition to his new test code. That's the way we make Apache 
Spark stronger.
   
   How do you think about this, @shahidki31 ? It's a way of collaboration.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] 
fallback to hdfs when relation table stats is not available
URL: https://github.com/apache/spark/pull/25460#issuecomment-521892398
 
 
   To @shahidki31 . 
   
   @maropu and @cloud-fan meant the corner case when the table size is equal to 
the user configuration value (not 8.0EB). Let say we set the configuration to 
1GB and we have a static table T1 whose size happens to be 1GB. In that case, 
every query on that tables might invoke this function. Although it's a very 
special case, but it's a regression.
   
   So, @cloud-fan and @maropu suggested to close this PR and proceed with 
#24715 .
   
   I'm +1 for that because that is the correct way.
   
   I know that you are worrying that #24715 doesn't resolve 8.0EB issue. 
However, that should be covered by your UTs in the previous PR. In the worst 
case, some of your code might be reverted. However, your test case should be 
there. It's your contribution. I believe @wangyum 's PR will pass your test 
cases in addition to his new test code. That's the way we make Apache Spark 
stronger.
   
   How do you think about this, @shahidki31 ? It's a way of collaboration.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun edited a comment on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] 
fallback to hdfs when relation table stats is not available
URL: https://github.com/apache/spark/pull/25460#issuecomment-521892398
 
 
   To @shahidki31 . 
   
   @maropu and @cloud-fan meant the corner case when the table size is equal to 
the user configuration value (not 8.0EB). Let say we set the configuration to 
1GB and we have a static table T1 whose size happens to be 1GB. In that case, 
every query on that tables might invoke this function. Although it's a very 
special case, but it's a regression.
   
   So, @cloud-fan and @maropu suggested to close this PR and proceed with 
#24715 .
   
   I'm +1 for that suggestion because that is the correct way.
   
   I know that you are worrying that #24715 doesn't resolve 8.0EB issue. 
However, that should be covered by your UTs in the previous PR. In the worst 
case, some of your code might be reverted. However, your test case should be 
there. It's your contribution. I believe @wangyum 's PR will pass your test 
cases in addition to his new test code. That's the way we make Apache Spark 
stronger.
   
   How do you think about this, @shahidki31 ? It's a way of collaboration.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback to hdfs when relation table stats is not available

2019-08-15 Thread GitBox
dongjoon-hyun commented on issue #25460: [SPARK-25474][SQL][FOLLOW-UP] fallback 
to hdfs when relation table stats is not available
URL: https://github.com/apache/spark/pull/25460#issuecomment-521892398
 
 
   To @shahidki31 . 
   
   @maropu and @cloud-fan meant the corner case when the table size is equal to 
the user configuration value (not 8.0EB). Let say we set the configuration to 
1GB and we have a static table T1 whose size happens to be 1GB. In that case, 
every query on that tables might invoke this functions. Although it's a very 
special case, but it's a regression.
   
   So, @cloud-fan and @maropu suggested to close this PR and proceed with 
#24715 .
   
   I'm +1 for that because that is the correct way.
   
   I know that you are worrying that #24715 doesn't resolve 8.0EB issue. 
However, that should be covered by your UTs in the previous PR. In the worst 
case, some of your code might be reverted. However, your test case should be 
there. It's your contribution. I believe @wangyum 's PR will pass your test 
cases in addition to his new test code. That's the way we make Apache Spark 
stronger.
   
   How do you think about this, @shahidki31 ? It's a way of collaboration.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314589735
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
 ##
 @@ -345,14 +345,16 @@ object CommandUtils extends Logging {
 !path.getName.startsWith(stagingDir) && DataSourceUtils.isDataPath(path)
   }
 
-  def getSizeInBytesFallBackToHdfs(session: SparkSession, path: Path, 
defaultSize: Long): Long = {
+  def getSizeInBytesFallBackToHdfs(session: SparkSession, table: 
CatalogTable): Long = {
 try {
   val hadoopConf = session.sessionState.newHadoopConf()
-  path.getFileSystem(hadoopConf).getContentSummary(path).getLength
+  val tablePath = new Path(table.location)
+  val fs: FileSystem = tablePath.getFileSystem(hadoopConf)
+  fs.getContentSummary(tablePath).getLength
 } catch {
   case NonFatal(e) =>
-logWarning(s"Failed to get table size from hdfs. Using the default 
size, $defaultSize.", e)
-defaultSize
+logWarning(s"Failed to get table size from HDFS. Using the default 
data size.", e)
 
 Review comment:
   I think we should format it if need print `defaultSize`: 
`org.apache.spark.util.Utils.bytesToString`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25394: [SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent permanent function in dropFunction

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25394: [SPARK-28671][SQL] Throw 
NoSuchPermanentFunctionException for a non-exsistent permanent function in 
dropFunction
URL: https://github.com/apache/spark/pull/25394#issuecomment-521891682
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25394: [SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent permanent function in dropFunction

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25394: [SPARK-28671][SQL] Throw 
NoSuchPermanentFunctionException for a non-exsistent permanent function in 
dropFunction
URL: https://github.com/apache/spark/pull/25394#issuecomment-521891684
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14258/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25394: [SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent permanent function in dropFunction

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25394: [SPARK-28671][SQL] Throw 
NoSuchPermanentFunctionException for a non-exsistent permanent function in 
dropFunction
URL: https://github.com/apache/spark/pull/25394#issuecomment-521891684
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14258/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add 
`date_part` function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#issuecomment-521891635
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14257/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add `date_part` 
function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#issuecomment-521891635
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14257/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25410: [SPARK-28690][SQL] Add 
`date_part` function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#issuecomment-521891632
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25394: [SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent permanent function in dropFunction

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25394: [SPARK-28671][SQL] Throw 
NoSuchPermanentFunctionException for a non-exsistent permanent function in 
dropFunction
URL: https://github.com/apache/spark/pull/25394#issuecomment-521891682
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25410: [SPARK-28690][SQL] Add `date_part` 
function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#issuecomment-521891632
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] PavithraRamachandran commented on a change in pull request #25394: [SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent permanent function in dropFunction

2019-08-15 Thread GitBox
PavithraRamachandran commented on a change in pull request #25394: 
[SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent 
permanent function in dropFunction
URL: https://github.com/apache/spark/pull/25394#discussion_r314588644
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ##
 @@ -1114,7 +1114,7 @@ class SessionCatalog(
   }
   externalCatalog.dropFunction(db, name.funcName)
 } else if (!ignoreIfNotExists) {
-  throw new NoSuchFunctionException(db = db, func = identifier.toString)
+  throw new NoSuchPermanentFunctionException(db = db, func = 
identifier.toString)
 
 Review comment:
   cc @maropu @HyukjinKwon @viirya  I have changed the exception in alter 
function .


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25394: [SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent permanent function in dropFunction

2019-08-15 Thread GitBox
SparkQA commented on issue #25394: [SPARK-28671][SQL] Throw 
NoSuchPermanentFunctionException for a non-exsistent permanent function in 
dropFunction
URL: https://github.com/apache/spark/pull/25394#issuecomment-521890621
 
 
   **[Test build #109187 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109187/testReport)**
 for PR 25394 at commit 
[`ee90d02`](https://github.com/apache/spark/commit/ee90d02cbdcaad5a5b411d21376f1e77f02fceda).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
SparkQA commented on issue #25410: [SPARK-28690][SQL] Add `date_part` function 
for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#issuecomment-521890624
 
 
   **[Test build #109186 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109186/testReport)**
 for PR 25410 at commit 
[`2fa25b1`](https://github.com/apache/spark/commit/2fa25b1d64bd75ee4f56be88f36944f03f623cc5).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on issue #22570: [SPARK-25553][BUILD] Add EmptyInterpolatedStringChecker to scalastyle config

2019-08-15 Thread GitBox
wangyum commented on issue #22570: [SPARK-25553][BUILD] Add 
EmptyInterpolatedStringChecker to scalastyle config
URL: https://github.com/apache/spark/pull/22570#issuecomment-521890373
 
 
   @srowen @maropu Do you think we can add it for Spark 3.0?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521890133
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14256/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521890125
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521890133
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14256/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521890125
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] PavithraRamachandran commented on a change in pull request #25394: [SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent permanent function in dropFunction

2019-08-15 Thread GitBox
PavithraRamachandran commented on a change in pull request #25394: 
[SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent 
permanent function in dropFunction
URL: https://github.com/apache/spark/pull/25394#discussion_r314588046
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ##
 @@ -1114,7 +1114,7 @@ class SessionCatalog(
   }
   externalCatalog.dropFunction(db, name.funcName)
 } else if (!ignoreIfNotExists) {
-  throw new NoSuchFunctionException(db = db, func = identifier.toString)
+  throw new NoSuchPermanentFunctionException(db = db, func = 
identifier.toString)
 
 Review comment:
   
![image](https://user-images.githubusercontent.com/51401130/63145056-846a9580-c013-11e9-9f05-8964c83a1cd5.png)
   In catalog dropTempFunction is present to handle temp function .
   in dropTempFunction  - **NoSuchTempFunctionException** exception is getting 
thrown
   both functions throws different exceptions , is the required to change in  
dropTempFunction?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] PavithraRamachandran commented on a change in pull request #25394: [SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent permanent function in dropFunction

2019-08-15 Thread GitBox
PavithraRamachandran commented on a change in pull request #25394: 
[SPARK-28671][SQL] Throw NoSuchPermanentFunctionException for a non-exsistent 
permanent function in dropFunction
URL: https://github.com/apache/spark/pull/25394#discussion_r314588046
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
 ##
 @@ -1114,7 +1114,7 @@ class SessionCatalog(
   }
   externalCatalog.dropFunction(db, name.funcName)
 } else if (!ignoreIfNotExists) {
-  throw new NoSuchFunctionException(db = db, func = identifier.toString)
+  throw new NoSuchPermanentFunctionException(db = db, func = 
identifier.toString)
 
 Review comment:
   
![image](https://user-images.githubusercontent.com/51401130/63145056-846a9580-c013-11e9-9f05-8964c83a1cd5.png)
   In catalog dropTempFunction is present to handle temp function .
   in dropTempFunction  - **NoSuchTempFunctionException** exception is getting 
thrown
   both functions throws different exceptions , is it required to change in  
dropTempFunction?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314587934
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
 ##
 @@ -619,3 +619,34 @@ object DataSourceStrategy {
 (nonconvertiblePredicates ++ unhandledPredicates, pushedFilters, 
handledFilters)
   }
 }
+
+
+/**
+ * Support for recalculating table statistics if table statistics are not 
available.
 
 Review comment:
   nit: `Defines default table statistics if `?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314587352
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
 ##
 @@ -345,14 +345,16 @@ object CommandUtils extends Logging {
 !path.getName.startsWith(stagingDir) && DataSourceUtils.isDataPath(path)
   }
 
-  def getSizeInBytesFallBackToHdfs(session: SparkSession, path: Path, 
defaultSize: Long): Long = {
+  def getSizeInBytesFallBackToHdfs(session: SparkSession, table: 
CatalogTable): Long = {
 try {
   val hadoopConf = session.sessionState.newHadoopConf()
-  path.getFileSystem(hadoopConf).getContentSummary(path).getLength
+  val tablePath = new Path(table.location)
+  val fs: FileSystem = tablePath.getFileSystem(hadoopConf)
+  fs.getContentSummary(tablePath).getLength
 } catch {
   case NonFatal(e) =>
-logWarning(s"Failed to get table size from hdfs. Using the default 
size, $defaultSize.", e)
-defaultSize
+logWarning(s"Failed to get table size from HDFS. Using the default 
data size.", e)
 
 Review comment:
   btw, we don't need to print `defaultSize` anymore?
   ```
   val defaultSize = session.sessionState.conf.defaultSizeInBytes
   logWarning(s"Failed to get table size from hdfs. Using the default 
size, $defaultSize.", e)
   defaultSize
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
SparkQA commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to 
track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521889074
 
 
   **[Test build #109184 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109184/testReport)**
 for PR 25368 at commit 
[`45cbbd0`](https://github.com/apache/spark/commit/45cbbd04408251e14a9157d1a5b93ae6a8e91401).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
SparkQA commented on issue #24715: [SPARK-25474][SQL] Data source tables 
support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#issuecomment-521889057
 
 
   **[Test build #109185 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109185/testReport)**
 for PR 24715 at commit 
[`7d22c15`](https://github.com/apache/spark/commit/7d22c157201a457d0a6954be7295ef3cfdd99113).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command

2019-08-15 Thread GitBox
dilipbiswal commented on a change in pull request #24759: 
[SPARK-27395][SQL][WIP] Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314587245
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala
 ##
 @@ -64,6 +66,15 @@ abstract class SparkPlan extends QueryPlan[SparkPlan] with 
Logging with Serializ
 
   protected def sparkContext = sqlContext.sparkContext
 
+  protected def wholestageCodegenIdStr(codegenId: Option[Int]): String = {
+codegenId.map("[codegen id : " + _ + "]").getOrElse("")
+  }
+
+  protected def operatorIdStr(
 
 Review comment:
   @cloud-fan will remove.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314587227
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
 ##
 @@ -650,4 +650,44 @@ class StatisticsCollectionSuite extends 
StatisticsCollectionTestBase with Shared
   }
 }
   }
+
+  test("Data source tables support fallback to HDFS for size estimation") {
+// Non-partitioned table
+withTempDir { dir =>
+  Seq(false, true).foreach { fallBackToHDFSForStats =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFSForStats") {
+  withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION 
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+  }
+}
+  }
+}
+
+// Partitioned table
+Seq(false, true).foreach { fallBackToHDFSForStats =>
+  withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFSForStats") {
+withTable("spark_25474") {
+  withTempDir { dir =>
+spark.sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command

2019-08-15 Thread GitBox
dilipbiswal commented on a change in pull request #24759: 
[SPARK-27395][SQL][WIP] Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314587233
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
 ##
 @@ -784,15 +787,17 @@ case class WholeStageCodegenExec(child: SparkPlan)(val 
codegenStageId: Int)
   verbose: Boolean,
   prefix: String = "",
   addSuffix: Boolean = false,
-  maxFields: Int): Unit = {
+  maxFields: Int,
+  printNodeId: Boolean): Unit = {
 child.generateTreeString(
   depth,
   lastChildren,
   append,
   verbose,
-  s"*($codegenStageId) ",
+  if (printNodeId) "*" else s"*($codegenStageId) ",
 
 Review comment:
   @cloud-fan ok


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command

2019-08-15 Thread GitBox
dilipbiswal commented on a change in pull request #24759: 
[SPARK-27395][SQL][WIP] Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314587254
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
 ##
 @@ -127,10 +131,14 @@ class QueryExecution(
 ReuseExchange(sparkSession.sessionState.conf),
 ReuseSubquery(sparkSession.sessionState.conf))
 
-  def simpleString: String = withRedaction {
+  def simpleString[T <: QueryPlan[T]]: String = withRedaction {
 
 Review comment:
   @cloud-fan not required... thanks .. will remove.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command

2019-08-15 Thread GitBox
dilipbiswal commented on a change in pull request #24759: 
[SPARK-27395][SQL][WIP] Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314587247
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala
 ##
 @@ -64,6 +66,15 @@ abstract class SparkPlan extends QueryPlan[SparkPlan] with 
Logging with Serializ
 
   protected def sparkContext = sqlContext.sparkContext
 
+  protected def wholestageCodegenIdStr(codegenId: Option[Int]): String = {
 
 Review comment:
   @cloud-fan yeah.. sorry.. will remove.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dilipbiswal commented on a change in pull request #24759: [SPARK-27395][SQL][WIP] Improve EXPLAIN command

2019-08-15 Thread GitBox
dilipbiswal commented on a change in pull request #24759: 
[SPARK-27395][SQL][WIP] Improve EXPLAIN command
URL: https://github.com/apache/spark/pull/24759#discussion_r314587222
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala
 ##
 @@ -17,6 +17,8 @@
 
 package org.apache.spark.sql.execution.aggregate
 
+import scala.collection.mutable
 
 Review comment:
   @cloud-fan will remove.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #24715: [SPARK-25474][SQL] Data source 
tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#issuecomment-521888589
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14255/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314587076
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
 ##
 @@ -619,3 +620,35 @@ object DataSourceStrategy {
 (nonconvertiblePredicates ++ unhandledPredicates, pushedFilters, 
handledFilters)
   }
 }
+
+
+/**
+ * Support for recalculating table statistics if table statistics are not 
available.
+ */
+class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan] {
+
+  private val sessionConf = session.sessionState.conf
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #24715: [SPARK-25474][SQL] Data source tables 
support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#issuecomment-521888584
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314587062
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
 ##
 @@ -619,3 +620,35 @@ object DataSourceStrategy {
 (nonconvertiblePredicates ++ unhandledPredicates, pushedFilters, 
handledFilters)
   }
 }
+
+
+/**
+ * Support for recalculating table statistics if table statistics are not 
available.
+ */
+class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan] {
+
+  private val sessionConf = session.sessionState.conf
+
+  override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+// For the data source table, we only recalculate the table statistics 
when it creates
+// the CatalogFileIndex using defaultSizeInBytes. See SPARK-25474 for more 
details.
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #24715: [SPARK-25474][SQL] Data source 
tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#issuecomment-521888584
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #24715: [SPARK-25474][SQL] Data source tables 
support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#issuecomment-521888589
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14255/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314587051
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala
 ##
 @@ -345,14 +345,16 @@ object CommandUtils extends Logging {
 !path.getName.startsWith(stagingDir) && DataSourceUtils.isDataPath(path)
   }
 
-  def getSizeInBytesFallBackToHdfs(session: SparkSession, path: Path, 
defaultSize: Long): Long = {
+  def getSizeInBytesFallBackToHdfs(session: SparkSession, table: 
CatalogTable): Long = {
 try {
   val hadoopConf = session.sessionState.newHadoopConf()
-  path.getFileSystem(hadoopConf).getContentSummary(path).getLength
+  val tablePath = new Path(table.location)
+  val fs: FileSystem = tablePath.getFileSystem(hadoopConf)
+  fs.getContentSummary(tablePath).getLength
 } catch {
   case NonFatal(e) =>
-logWarning(s"Failed to get table size from hdfs. Using the default 
size, $defaultSize.", e)
-defaultSize
+logWarning(s"Failed to get table size from HDFS. Using the default 
data size.", e)
 
 Review comment:
   nit: Drops `s` in the head.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314587042
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
 ##
 @@ -619,3 +620,35 @@ object DataSourceStrategy {
 (nonconvertiblePredicates ++ unhandledPredicates, pushedFilters, 
handledFilters)
   }
 }
+
+
+/**
+ * Support for recalculating table statistics if table statistics are not 
available.
+ */
+class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan] {
+
+  private val sessionConf = session.sessionState.conf
+
+  override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+// For the data source table, we only recalculate the table statistics 
when it creates
+// the CatalogFileIndex using defaultSizeInBytes. See SPARK-25474 for more 
details.
+case logical @ LogicalRelation(_, _, Some(table), _)
+  if sessionConf.fallBackToHdfsForStatsEnabled && table.stats.isEmpty &&
+sessionConf.manageFilesourcePartitions &&
+table.tracksPartitionsInCatalog && table.partitionColumnNames.nonEmpty 
=>
+  val sizeInBytes = CommandUtils.getSizeInBytesFallBackToHdfs(session, 
table)
+  val withStats = table.copy(stats = Some(CatalogStatistics(sizeInBytes = 
BigInt(sizeInBytes
+  logical.copy(catalogTable = Some(withStats))
+
+case relation: HiveTableRelation
 
 Review comment:
   @advancedxy Already work on 
this:https://github.com/apache/spark/pull/25306/commits/c86a27b2a5e286733ad305de1d7e42d1373b3a3b


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314587025
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
 ##
 @@ -650,4 +650,44 @@ class StatisticsCollectionSuite extends 
StatisticsCollectionTestBase with Shared
   }
 }
   }
+
+  test("Data source tables support fallback to HDFS for size estimation") {
+// Non-partitioned table
+withTempDir { dir =>
+  Seq(false, true).foreach { fallBackToHDFSForStats =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFSForStats") {
+  withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION 
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+  }
+}
+  }
+}
+
+// Partitioned table
+Seq(false, true).foreach { fallBackToHDFSForStats =>
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] brkyvz commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
brkyvz commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to 
track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521888516
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] MaxGekk commented on a change in pull request #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
MaxGekk commented on a change in pull request #25410: [SPARK-28690][SQL] Add 
`date_part` function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#discussion_r314587020
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ##
 @@ -1409,48 +1409,7 @@ class AstBuilder(conf: SQLConf) extends 
SqlBaseBaseVisitor[AnyRef] with Logging
* Create a Extract expression.
*/
   override def visitExtract(ctx: ExtractContext): Expression = withOrigin(ctx) 
{
-ctx.field.getText.toUpperCase(Locale.ROOT) match {
-  case "MILLENNIUM" | "MILLENNIA" | "MIL" | "MILS" =>
-Millennium(expression(ctx.source))
-  case "CENTURY" | "CENTURIES" | "C" | "CENT" =>
-Century(expression(ctx.source))
-  case "DECADE" | "DECADES" | "DEC" | "DECS" =>
-Decade(expression(ctx.source))
-  case "YEAR" | "Y" | "YEARS" | "YR" | "YRS" =>
-Year(expression(ctx.source))
-  case "ISOYEAR" =>
-IsoYear(expression(ctx.source))
-  case "QUARTER" | "QTR" =>
-Quarter(expression(ctx.source))
-  case "MONTH" | "MON" | "MONS" | "MONTHS" =>
-Month(expression(ctx.source))
-  case "WEEK" | "W" | "WEEKS" =>
-WeekOfYear(expression(ctx.source))
-  case "DAY" | "D" | "DAYS" =>
-DayOfMonth(expression(ctx.source))
-  case "DAYOFWEEK" =>
-DayOfWeek(expression(ctx.source))
-  case "DOW" =>
-Subtract(DayOfWeek(expression(ctx.source)), Literal(1))
-  case "ISODOW" =>
-Add(WeekDay(expression(ctx.source)), Literal(1))
-  case "DOY" =>
-DayOfYear(expression(ctx.source))
-  case "HOUR" | "H" | "HOURS" | "HR" | "HRS" =>
-Hour(expression(ctx.source))
-  case "MINUTE" | "M" | "MIN" | "MINS" | "MINUTES" =>
-Minute(expression(ctx.source))
-  case "SECOND" | "S" | "SEC" | "SECONDS" | "SECS" =>
-Second(expression(ctx.source))
-  case "MILLISECONDS" | "MSEC" | "MSECS" | "MILLISECON" | "MSECONDS" | 
"MS" =>
-Milliseconds(expression(ctx.source))
-  case "MICROSECONDS" | "USEC" | "USECS" | "USECONDS" | "MICROSECON" | 
"US" =>
-Microseconds(expression(ctx.source))
-  case "EPOCH" =>
-Epoch(expression(ctx.source))
-  case other =>
-throw new ParseException(s"Literals of type '$other' are currently not 
supported.", ctx)
 
 Review comment:
   ... but I can pass `ctx: ParserRuleContext` to `DatePart`'s constructor. I 
am just not sure that is is good practice. WDYT?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] brkyvz commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
brkyvz commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to 
track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521888502
 
 
   LGTM pending tests


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314587006
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
 ##
 @@ -650,4 +650,44 @@ class StatisticsCollectionSuite extends 
StatisticsCollectionTestBase with Shared
   }
 }
   }
+
+  test("Data source tables support fallback to HDFS for size estimation") {
+// Non-partitioned table
+withTempDir { dir =>
+  Seq(false, true).foreach { fallBackToHDFSForStats =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFSForStats") {
 
 Review comment:
   // fallBackToHDFSForStats = true: The table stats will be recalculated by 
DetermineTableStats
   // fallBackToHDFSForStats = false: The table stats will be recalculated by 
[FileIndex](https://github.com/apache/spark/blob/b276788d57b270d455ef6a7c5ed6cf8a74885dde/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L103)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] brkyvz commented on a change in pull request #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
brkyvz commented on a change in pull request #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#discussion_r314586621
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalog/v2/CatalogManager.scala
 ##
 @@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalog.v2
+
+import scala.collection.mutable
+import scala.util.control.NonFatal
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * A thread-safe manager for [[CatalogPlugin]]s. It tracks all the registered 
catalogs, and allow
+ * the caller to look up a catalog by name.
+ */
+class CatalogManager(conf: SQLConf) extends Logging {
+
+  private val catalogs = mutable.HashMap.empty[String, CatalogPlugin]
+
+  def catalog(name: String): CatalogPlugin = synchronized {
+catalogs.getOrElseUpdate(name, Catalogs.load(name, conf))
+  }
+
+  def defaultCatalog: Option[CatalogPlugin] = {
+conf.defaultV2Catalog.flatMap { catalogName =>
+  try {
+Some(catalog(catalogName))
+  } catch {
+case NonFatal(e) =>
+  logError(s"Cannot load default v2 catalog: $catalogName", e)
+  None
+  }
+}
+  }
+
+  def v2SessionCatalog: Option[CatalogPlugin] = {
+try {
+  Some(catalog(CatalogManager.SESSION_CATALOG_NAME))
+} catch {
+  case NonFatal(e) =>
+logError("Cannot load v2 session catalog", e)
+None
+}
+  }
+
+  private def getDefaultNamespace(c: CatalogPlugin) = c match {
+case c: SupportsNamespaces => c.defaultNamespace()
+case _ => Array.empty[String]
+  }
+
+  private var _currentNamespace = {
+// The builtin catalog use "default" as the default database.
 
 Review comment:
   Oh, wait, I got the defaultNamespace setting in `SupportsNamespaces` wrong. 
I thought it was actually tracked there. This looks good to me. Sorry.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] MaxGekk commented on a change in pull request #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
MaxGekk commented on a change in pull request #25410: [SPARK-28690][SQL] Add 
`date_part` function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#discussion_r314586539
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
 ##
 @@ -1963,3 +1963,77 @@ case class Epoch(child: Expression, timeZoneId: 
Option[String] = None)
 defineCodeGen(ctx, ev, c => s"$dtu.getEpoch($c, $zid)")
   }
 }
+
+@ExpressionDescription(
+  usage = "_FUNC_(field, source) - Extracts a part of the date/timestamp.",
+  arguments = """
+Arguments:
+  * field - selects which part of the source should be extracted. 
Supported string values are:
+["MILLENNIUM", "MILLENNIA", "MIL", "MILS",
+ "CENTURY", "CENTURIES", "C", "CENT",
+ "DECADE", "DECADES", "DEC", "DECS",
+ "YEAR", "Y", "YEARS", "YR", "YRS",
+ "ISOYEAR",
+ "QUARTER", "QTR",
+ "MONTH", "MON", "MONS", "MONTHS",
+ "WEEK", "W", "WEEKS",
+ "DAY", "D", "DAYS",
+ "DAYOFWEEK", "DOW", "ISODOW", "DOY",
+ "HOUR", "H", "HOURS", "HR", "HRS",
+ "MINUTE", "M", "MIN", "MINS", "MINUTES",
+ "SECOND", "S", "SEC", "SECONDS", "SECS",
+ "MILLISECONDS", "MSEC", "MSECS", "MILLISECON", "MSECONDS", 
"MS",
+ "MICROSECONDS", "USEC", "USECS", "USECONDS", "MICROSECON", 
"US",
+ "EPOCH"]
 
 Review comment:
   Sounds good. I will do that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] MaxGekk commented on a change in pull request #25410: [SPARK-28690][SQL] Add `date_part` function for timestamps/dates

2019-08-15 Thread GitBox
MaxGekk commented on a change in pull request #25410: [SPARK-28690][SQL] Add 
`date_part` function for timestamps/dates
URL: https://github.com/apache/spark/pull/25410#discussion_r314586294
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
 ##
 @@ -1409,48 +1409,7 @@ class AstBuilder(conf: SQLConf) extends 
SqlBaseBaseVisitor[AnyRef] with Logging
* Create a Extract expression.
*/
   override def visitExtract(ctx: ExtractContext): Expression = withOrigin(ctx) 
{
-ctx.field.getText.toUpperCase(Locale.ROOT) match {
-  case "MILLENNIUM" | "MILLENNIA" | "MIL" | "MILS" =>
-Millennium(expression(ctx.source))
-  case "CENTURY" | "CENTURIES" | "C" | "CENT" =>
-Century(expression(ctx.source))
-  case "DECADE" | "DECADES" | "DEC" | "DECS" =>
-Decade(expression(ctx.source))
-  case "YEAR" | "Y" | "YEARS" | "YR" | "YRS" =>
-Year(expression(ctx.source))
-  case "ISOYEAR" =>
-IsoYear(expression(ctx.source))
-  case "QUARTER" | "QTR" =>
-Quarter(expression(ctx.source))
-  case "MONTH" | "MON" | "MONS" | "MONTHS" =>
-Month(expression(ctx.source))
-  case "WEEK" | "W" | "WEEKS" =>
-WeekOfYear(expression(ctx.source))
-  case "DAY" | "D" | "DAYS" =>
-DayOfMonth(expression(ctx.source))
-  case "DAYOFWEEK" =>
-DayOfWeek(expression(ctx.source))
-  case "DOW" =>
-Subtract(DayOfWeek(expression(ctx.source)), Literal(1))
-  case "ISODOW" =>
-Add(WeekDay(expression(ctx.source)), Literal(1))
-  case "DOY" =>
-DayOfYear(expression(ctx.source))
-  case "HOUR" | "H" | "HOURS" | "HR" | "HRS" =>
-Hour(expression(ctx.source))
-  case "MINUTE" | "M" | "MIN" | "MINS" | "MINUTES" =>
-Minute(expression(ctx.source))
-  case "SECOND" | "S" | "SEC" | "SECONDS" | "SECS" =>
-Second(expression(ctx.source))
-  case "MILLISECONDS" | "MSEC" | "MSECS" | "MILLISECON" | "MSECONDS" | 
"MS" =>
-Milliseconds(expression(ctx.source))
-  case "MICROSECONDS" | "USEC" | "USECS" | "USECONDS" | "MICROSECON" | 
"US" =>
-Microseconds(expression(ctx.source))
-  case "EPOCH" =>
-Epoch(expression(ctx.source))
-  case other =>
-throw new ParseException(s"Literals of type '$other' are currently not 
supported.", ctx)
 
 Review comment:
   `ParseException` requires either `ctx: ParserRuleContext` or `val start: 
Origin, val stop: Origin` that are not available to me at the point. And it is 
still `ParseException` in the output: 
https://github.com/apache/spark/pull/25410/files#diff-6f4edc80e2cc973e748705e85a6053b4R514


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25455: [SPARK-28737][CORE] Update Jersey to 2.29

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25455: [SPARK-28737][CORE] Update 
Jersey to 2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521887429
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109160/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation

2019-08-15 Thread GitBox
maropu commented on a change in pull request #24715: [SPARK-25474][SQL] Data 
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r314586339
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StatisticsCollectionSuite.scala
 ##
 @@ -650,4 +650,44 @@ class StatisticsCollectionSuite extends 
StatisticsCollectionTestBase with Shared
   }
 }
   }
+
+  test("Data source tables support fallback to HDFS for size estimation") {
+// Non-partitioned table
+withTempDir { dir =>
+  Seq(false, true).foreach { fallBackToHDFSForStats =>
+withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFSForStats") {
+  withTable("spark_25474") {
+sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION 
'${dir.toURI}'")
+
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
+
+assert(getCatalogTable("spark_25474").stats.isEmpty)
+val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
+assert(relation.stats.sizeInBytes === getDataSize(dir))
+  }
+}
+  }
+}
+
+// Partitioned table
+Seq(false, true).foreach { fallBackToHDFSForStats =>
+  withSQLConf(SQLConf.ENABLE_FALL_BACK_TO_HDFS_FOR_STATS.key -> 
s"$fallBackToHDFSForStats") {
+withTable("spark_25474") {
+  withTempDir { dir =>
+spark.sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
 
 Review comment:
   nit: don't need `spark.`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25455: [SPARK-28737][CORE] Update Jersey to 2.29

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25455: [SPARK-28737][CORE] Update Jersey to 
2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521887426
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25455: [SPARK-28737][CORE] Update Jersey to 2.29

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25455: [SPARK-28737][CORE] Update 
Jersey to 2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521887426
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25455: [SPARK-28737][CORE] Update Jersey to 2.29

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25455: [SPARK-28737][CORE] Update Jersey to 
2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521887429
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109160/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25455: [SPARK-28737][CORE] Update Jersey to 2.29

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25455: [SPARK-28737][CORE] Update Jersey to 
2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521857550
 
 
   **[Test build #109160 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109160/testReport)**
 for PR 25455 at commit 
[`57add37`](https://github.com/apache/spark/commit/57add373ba6cc95730f3f57f9e5b2deb13ab121a).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25455: [SPARK-28737][CORE] Update Jersey to 2.29

2019-08-15 Thread GitBox
SparkQA commented on issue #25455: [SPARK-28737][CORE] Update Jersey to 2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521887091
 
 
   **[Test build #109160 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109160/testReport)**
 for PR 25455 at commit 
[`57add37`](https://github.com/apache/spark/commit/57add373ba6cc95730f3f57f9e5b2deb13ab121a).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25371: 
[SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF 
test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-521885915
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109167/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive

2019-08-15 Thread GitBox
SparkQA commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For 
nettyBlockTransferService, if IOException occurred while create client, check 
whether relative executor is alive before retry #24533
URL: https://github.com/apache/spark/pull/25469#issuecomment-521886226
 
 
   **[Test build #109183 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109183/testReport)**
 for PR 25469 at commit 
[`e2dbe4b`](https://github.com/apache/spark/commit/e2dbe4bca387542e2043abc0801190531e805684).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] 
Convert and port 'pgSQL/join.sql' into UDF test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-521885915
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109167/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson 
Databind to 2.9.9.3
URL: https://github.com/apache/spark/pull/25451#issuecomment-521885912
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14254/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump 
Jackson Databind to 2.9.9.3
URL: https://github.com/apache/spark/pull/25451#issuecomment-521885912
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14254/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] 
Convert and port 'pgSQL/join.sql' into UDF test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-521885910
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson 
Databind to 2.9.9.3
URL: https://github.com/apache/spark/pull/25451#issuecomment-521885911
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump 
Jackson Databind to 2.9.9.3
URL: https://github.com/apache/spark/pull/25451#issuecomment-521885911
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25371: 
[SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF 
test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-521885910
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For 
nettyBlockTransferService, if IOException occurred while create client, check 
whether relative executor is alive before retry #24533
URL: https://github.com/apache/spark/pull/25469#issuecomment-521885858
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For 
nettyBlockTransferService, if IOException occurred while create client, check 
whether relative executor is alive before retry #24533
URL: https://github.com/apache/spark/pull/25469#issuecomment-521885863
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14253/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] 
Convert and port 'pgSQL/join.sql' into UDF test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-521857469
 
 
   **[Test build #109167 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109167/testReport)**
 for PR 25371 at commit 
[`35649d2`](https://github.com/apache/spark/commit/35649d24b326d33f3db3e01ce95028bb7b151040).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative exe

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25469: 
[SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException 
occurred while create client, check whether relative executor is alive before 
retry #24533
URL: https://github.com/apache/spark/pull/25469#issuecomment-521885863
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14253/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative exe

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25469: 
[SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException 
occurred while create client, check whether relative executor is alive before 
retry #24533
URL: https://github.com/apache/spark/pull/25469#issuecomment-521885858
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert and port 'pgSQL/join.sql' into UDF test base

2019-08-15 Thread GitBox
SparkQA commented on issue #25371: [SPARK-28393][SQL][PYTHON][TESTS] Convert 
and port 'pgSQL/join.sql' into UDF test base
URL: https://github.com/apache/spark/pull/25371#issuecomment-521885791
 
 
   **[Test build #109167 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109167/testReport)**
 for PR 25371 at commit 
[`35649d2`](https://github.com/apache/spark/commit/35649d24b326d33f3db3e01ce95028bb7b151040).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25463: [SPARK-28744][SQL][TEST] 
rename SharedSQLContext to SharedSparkSession
URL: https://github.com/apache/spark/pull/25463#issuecomment-521885478
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109173/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AngersZhuuuu commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is

2019-08-15 Thread GitBox
AngersZh commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For 
nettyBlockTransferService, if IOException occurred while create client, check 
whether relative executor is alive before retry #24533
URL: https://github.com/apache/spark/pull/25469#issuecomment-521885687
 
 
   > @AngersZh ok, the other prs hit the same errors, so they are not 
related to this pr.
   
   Got it, thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25463: [SPARK-28744][SQL][TEST] 
rename SharedSQLContext to SharedSparkSession
URL: https://github.com/apache/spark/pull/25463#issuecomment-521885476
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25463: [SPARK-28744][SQL][TEST] rename 
SharedSQLContext to SharedSparkSession
URL: https://github.com/apache/spark/pull/25463#issuecomment-521885476
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25463: [SPARK-28744][SQL][TEST] rename 
SharedSQLContext to SharedSparkSession
URL: https://github.com/apache/spark/pull/25463#issuecomment-521885478
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109173/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521885162
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109168/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25463: [SPARK-28744][SQL][TEST] rename 
SharedSQLContext to SharedSparkSession
URL: https://github.com/apache/spark/pull/25463#issuecomment-521858760
 
 
   **[Test build #109173 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109173/testReport)**
 for PR 25463 at commit 
[`8bc622f`](https://github.com/apache/spark/commit/8bc622f7332ad7970e54624b6d5b5aa184df3510).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521885161
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521857475
 
 
   **[Test build #109168 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109168/testReport)**
 for PR 25368 at commit 
[`45cbbd0`](https://github.com/apache/spark/commit/45cbbd04408251e14a9157d1a5b93ae6a8e91401).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25423: 
[SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521884974
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109164/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive

2019-08-15 Thread GitBox
maropu commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For 
nettyBlockTransferService, if IOException occurred while create client, check 
whether relative executor is alive before retry #24533
URL: https://github.com/apache/spark/pull/25469#issuecomment-521885308
 
 
   @AngersZh ok, the other prs hit the same errors, so they are not related 
to this pr.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For nettyBlockTransferService, if IOException occurred while create client, check whether relative executor is alive

2019-08-15 Thread GitBox
maropu commented on issue #25469: [SPARK-27637][Shuffle][FLLOW-UP]For 
nettyBlockTransferService, if IOException occurred while create client, check 
whether relative executor is alive before retry #24533
URL: https://github.com/apache/spark/pull/25469#issuecomment-521885185
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25463: [SPARK-28744][SQL][TEST] rename SharedSQLContext to SharedSparkSession

2019-08-15 Thread GitBox
SparkQA commented on issue #25463: [SPARK-28744][SQL][TEST] rename 
SharedSQLContext to SharedSparkSession
URL: https://github.com/apache/spark/pull/25463#issuecomment-521885268
 
 
   **[Test build #109173 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109173/testReport)**
 for PR 25463 at commit 
[`8bc622f`](https://github.com/apache/spark/commit/8bc622f7332ad7970e54624b6d5b5aa184df3510).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
 * `abstract class DockerJDBCIntegrationSuite extends SharedSparkSession 
with Eventually `
 * `class OracleIntegrationSuite extends DockerJDBCIntegrationSuite with 
SharedSparkSession `
 * `class OrcFilterSuite extends OrcTest with SharedSparkSession `


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521885161
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25368: [SPARK-28635][SQL] create 
CatalogManager to track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521885162
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109168/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25423: 
[SPARK-28701][test-java11][k8s] adding java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521884969
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump 
Jackson Databind to 2.9.9.3
URL: https://github.com/apache/spark/pull/25451#issuecomment-521884684
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109161/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to track registered v2 catalogs

2019-08-15 Thread GitBox
SparkQA commented on issue #25368: [SPARK-28635][SQL] create CatalogManager to 
track registered v2 catalogs
URL: https://github.com/apache/spark/pull/25368#issuecomment-521885024
 
 
   **[Test build #109168 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109168/testReport)**
 for PR 25368 at commit 
[`45cbbd0`](https://github.com/apache/spark/commit/45cbbd04408251e14a9157d1a5b93ae6a8e91401).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
SparkQA commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind 
to 2.9.9.3
URL: https://github.com/apache/spark/pull/25451#issuecomment-521885021
 
 
   **[Test build #109182 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109182/testReport)**
 for PR 25451 at commit 
[`f6c2f4a`](https://github.com/apache/spark/commit/f6c2f4acb6f9eb57eae23190b2be93002e1c0f6a).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
SparkQA removed a comment on issue #25423: [SPARK-28701][test-java11][k8s] 
adding java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521857496
 
 
   **[Test build #109164 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109164/testReport)**
 for PR 25423 at commit 
[`4d99b96`](https://github.com/apache/spark/commit/4d99b961e0f1f8ad983c041e8b4180cf94bfbba5).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] maropu commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
maropu commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 
2.9.9.3
URL: https://github.com/apache/spark/pull/25451#issuecomment-521884890
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521884974
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109164/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
AmplabJenkins removed a comment on issue #25451: [SPARK-28728][BUILD] Bump 
Jackson Databind to 2.9.9.3
URL: https://github.com/apache/spark/pull/25451#issuecomment-521884680
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521884969
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25423: [SPARK-28701][test-java11][k8s] adding java11 support for pull request builds

2019-08-15 Thread GitBox
SparkQA commented on issue #25423: [SPARK-28701][test-java11][k8s] adding 
java11 support for pull request builds
URL: https://github.com/apache/spark/pull/25423#issuecomment-521884806
 
 
   **[Test build #109164 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109164/testReport)**
 for PR 25423 at commit 
[`4d99b96`](https://github.com/apache/spark/commit/4d99b961e0f1f8ad983c041e8b4180cf94bfbba5).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson Databind to 2.9.9.3

2019-08-15 Thread GitBox
AmplabJenkins commented on issue #25451: [SPARK-28728][BUILD] Bump Jackson 
Databind to 2.9.9.3
URL: https://github.com/apache/spark/pull/25451#issuecomment-521884680
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



  1   2   3   4   5   6   7   8   >