[GitHub] [spark] SparkQA commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
SparkQA commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715#issuecomment-506929868 **[Test build #107034 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107034/testReport)** for PR 24715 at commit [`22dd26e`](https://github.com/apache/spark/commit/22dd26ecd794ec6b10b4d8c7ee191c2d9d639071). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
AmplabJenkins commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715#issuecomment-506929715 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
AmplabJenkins removed a comment on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715#issuecomment-506929716 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12227/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
AmplabJenkins removed a comment on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715#issuecomment-506929715 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
AmplabJenkins commented on issue #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation URL: https://github.com/apache/spark/pull/24715#issuecomment-506929716 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12227/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data source tables support fallback to HDFS for size estimation
wangyum commented on a change in pull request #24715: [SPARK-25474][SQL] Data
source tables support fallback to HDFS for size estimation
URL: https://github.com/apache/spark/pull/24715#discussion_r298788939
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/columnar/InMemoryColumnarQuerySuite.scala
##
@@ -512,37 +512,46 @@ class InMemoryColumnarQuerySuite extends QueryTest with
SharedSQLContext {
SQLConf.DEFAULT_DATA_SOURCE_NAME.key -> "orc",
SQLConf.USE_V1_SOURCE_READER_LIST.key -> useV1SourceReaderList) {
withTempPath { workDir =>
- withTable("table1") {
-val workDirPath = workDir.getAbsolutePath
-val data = Seq(100, 200, 300, 400).toDF("count")
-data.write.orc(workDirPath)
-val dfFromFile = spark.read.orc(workDirPath).cache()
-val inMemoryRelation =
dfFromFile.queryExecution.optimizedPlan.collect {
- case plan: InMemoryRelation => plan
-}.head
-// InMemoryRelation's stats is file size before the underlying RDD
is materialized
-assert(inMemoryRelation.computeStats().sizeInBytes === 486)
-
-// InMemoryRelation's stats is updated after materializing RDD
-dfFromFile.collect()
-assert(inMemoryRelation.computeStats().sizeInBytes === 16)
-
-// test of catalog table
-val dfFromTable = spark.catalog.createTable("table1",
workDirPath).cache()
-val inMemoryRelation2 = dfFromTable.queryExecution.optimizedPlan.
- collect { case plan: InMemoryRelation => plan }.head
-
-// Even CBO enabled, InMemoryRelation's stats keeps as the file
size before table's
-// stats is calculated
-assert(inMemoryRelation2.computeStats().sizeInBytes === 486)
Review comment:
Hive table also has this issue:
```scala
import org.apache.spark.sql.execution.columnar.InMemoryRelation
val tempDir = "/tmp/spark/spark_25474"
spark.range(10).write.mode("overwrite").parquet(tempDir)
spark.sql(s"CREATE TABLE spark_25474 (id BIGINT) STORED AS parquet LOCATION
'$tempDir'")
spark.sql("DESC FORMATTED spark_25474").show(false)
val inMemoryRelation = spark.table("spark_25474").cache()
val optimizedPlan = inMemoryRelation.queryExecution.optimizedPlan
val inMemoryRelation = optimizedPlan.collect { case plan: InMemoryRelation
=> plan }.head
println(inMemoryRelation.computeStats().sizeInBytes)
```
```
scala> println(inMemoryRelation.computeStats().sizeInBytes)
9223372036854775807
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #24997: [SPARK-28198][PYTHON] Add mapPartitionsInPandas to allow an iterator of DataFrames
HyukjinKwon edited a comment on issue #24997: [SPARK-28198][PYTHON] Add mapPartitionsInPandas to allow an iterator of DataFrames URL: https://github.com/apache/spark/pull/24997#issuecomment-506926878 Ah, yes. That seems working - I gotta fix the example in PR description. I actually want to do something like what [`DataFrame.apply`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html) does (not [`DataFrame.transform`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.transform.html) which guarantees the same length output), for instance. Let me fix the example later tonight. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #24997: [SPARK-28198][PYTHON] Add mapPartitionsInPandas to allow an iterator of DataFrames
HyukjinKwon edited a comment on issue #24997: [SPARK-28198][PYTHON] Add mapPartitionsInPandas to allow an iterator of DataFrames URL: https://github.com/apache/spark/pull/24997#issuecomment-506926878 Ah, yes. That seems working - I gotta fix the example in PR description. I actually want to do something like what [`DataFrame.apply`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html) does (not [`DataFrame.transform`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.transform.html) which guarantees the same length), for instance. Let me fix the example later tonight. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #23276: [SPARK-26321][SQL] Improve the behavior of sql text splitting for the spark-sql command line
AmplabJenkins commented on issue #23276: [SPARK-26321][SQL] Improve the behavior of sql text splitting for the spark-sql command line URL: https://github.com/apache/spark/pull/23276#issuecomment-506928037 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
HyukjinKwon commented on a change in pull request #25006: [SPARK-28208][BUILD]
Upgrade to ORC 1.5.6 including closing the ORC readers
URL: https://github.com/apache/spark/pull/25006#discussion_r298788206
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
##
@@ -184,6 +184,7 @@ class OrcFileFormat
val requestedColIdsOrEmptyFile = OrcUtils.requestedColumnIds(
isCaseSensitive, dataSchema, requiredSchema, reader, conf)
+ reader.close()
Review comment:
Can we do:
```scala
val requestedColIdsOrEmptyFile = {
Utils.tryWithResource(OrcFile.createReader(filePath, readerOptions))
{ reader =>
OrcUtils.requestedColumnIds(
isCaseSensitive, dataSchema, requiredSchema, reader, conf)
}
}
```
or, `try` and `finally`?
?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506927560 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506927561 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12226/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506927561 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12226/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506927560 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #25003: [SPARK-28204][SQL][TESTS] Make separate two test cases for column pruning in binary files
HyukjinKwon closed pull request #25003: [SPARK-28204][SQL][TESTS] Make separate two test cases for column pruning in binary files URL: https://github.com/apache/spark/pull/25003 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
SparkQA commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506927278 **[Test build #107033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107033/testReport)** for PR 25006 at commit [`0b9e8f3`](https://github.com/apache/spark/commit/0b9e8f353967e9901c0c955711e6b4559e02281d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #25003: [SPARK-28204][SQL][TESTS] Make separate two test cases for column pruning in binary files
HyukjinKwon commented on issue #25003: [SPARK-28204][SQL][TESTS] Make separate two test cases for column pruning in binary files URL: https://github.com/apache/spark/pull/25003#issuecomment-506927268 Let me merge this one - it's just a cleanup for tests. Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
dongjoon-hyun commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506927238 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
dongjoon-hyun commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506927228 Interesting. It passed with `-Phadoop-3.2`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on issue #24997: [SPARK-28198][PYTHON] Add mapPartitionsInPandas to allow an iterator of DataFrames
HyukjinKwon edited a comment on issue #24997: [SPARK-28198][PYTHON] Add mapPartitionsInPandas to allow an iterator of DataFrames URL: https://github.com/apache/spark/pull/24997#issuecomment-506926878 Ah, yes. That seems working - I gotta fix the example in PR description. I actually want to do something like what [`DataFrame.apply`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html) too (not [`DataFrame.transform`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.transform.html) which guarantees the same length), for instance. Let me fix the example later tonight. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #24997: [SPARK-28198][PYTHON] Add mapPartitionsInPandas to allow an iterator of DataFrames
HyukjinKwon commented on issue #24997: [SPARK-28198][PYTHON] Add mapPartitionsInPandas to allow an iterator of DataFrames URL: https://github.com/apache/spark/pull/24997#issuecomment-506926878 Ah, yes. That seems working - I gotta fix the example in PR description. I actually want to implement [`DataFrame.apply`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html) too (not [`DataFrame.transform`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.transform.html) which guarantees the same length), for instance. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506923645 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506923645 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506923650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107030/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506923650 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107030/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
SparkQA removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506891725 **[Test build #107030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107030/testReport)** for PR 25006 at commit [`0b9e8f3`](https://github.com/apache/spark/commit/0b9e8f353967e9901c0c955711e6b4559e02281d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
SparkQA commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506923546 **[Test build #107030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107030/testReport)** for PR 25006 at commit [`0b9e8f3`](https://github.com/apache/spark/commit/0b9e8f353967e9901c0c955711e6b4559e02281d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin edited a comment on issue #24982: [SPARK-28181][CORE] Add a filter interface to KVStore to speed up the entities retrieve
LantaoJin edited a comment on issue #24982: [SPARK-28181][CORE] Add a filter interface to KVStore to speed up the entities retrieve URL: https://github.com/apache/spark/pull/24982#issuecomment-506919132 > SPARK-28183 would be way more efficient if you instead scanned the tasks based on the status you want, applying the offset and limit, and sorted based on a different property after that (because of offset and limit, you wouldn't have a lot of elements to sort). Thanks for your reply @vanzin . IIUC, this still couldn’t address the target. The target is getting all tasks with particular `status` from all tasks. Even FAILED, KILLED will be getting in front with alphabetical order, or SUCCESS in front with reverse, it still needs additional filtering in upper application. SPARK-28183 wants to add a possibility in `taskList` API to get particular status. And this one could avoid filtering after pagination. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LantaoJin commented on issue #24982: [SPARK-28181][CORE] Add a filter interface to KVStore to speed up the entities retrieve
LantaoJin commented on issue #24982: [SPARK-28181][CORE] Add a filter interface to KVStore to speed up the entities retrieve URL: https://github.com/apache/spark/pull/24982#issuecomment-506919132 > SPARK-28183 would be way more efficient if you instead scanned the tasks based on the status you want, applying the offset and limit, and sorted based on a different property after that (because of offset and limit, you wouldn't have a lot of elements to sort). Thanks for your reply. IIUC, this still couldn’t address the target. The target is getting all tasks with particular `status` from all tasks. Even FAILED, KILLED will be getting in front with alphabetical order, or SUCCESS in front with reverse, it still needs additional filtering in upper application. SPARK-28183 wants to add a possibility in `taskList` API to get particular status. And this one could avoid filtering after pagination. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
dongjoon-hyun commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506916543 I'll try to make a patch for the missing part and send to your branch, `omalley:spark-28208`, during weekend, @omalley . Thank you again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506915729 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506915730 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107032/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506915729 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506915730 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107032/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506901279 **[Test build #107032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107032/testReport)** for PR 25007 at commit [`4c3d692`](https://github.com/apache/spark/commit/4c3d6926069bea6e89f1a13d8faa46ce1180fcf0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506915587 **[Test build #107032 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107032/testReport)** for PR 25007 at commit [`4c3d692`](https://github.com/apache/spark/commit/4c3d6926069bea6e89f1a13d8faa46ce1180fcf0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506912199 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506912200 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107031/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506912200 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107031/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506912199 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506912033 **[Test build #107031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107031/testReport)** for PR 25007 at commit [`3083d86`](https://github.com/apache/spark/commit/3083d8686cf1e1650969dde3d0862bd8c53855f0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506896097 **[Test build #107031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107031/testReport)** for PR 25007 at commit [`3083d86`](https://github.com/apache/spark/commit/3083d8686cf1e1650969dde3d0862bd8c53855f0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mickjermsurawong-stripe opened a new pull request #25009: test
mickjermsurawong-stripe opened a new pull request #25009: test URL: https://github.com/apache/spark/pull/25009 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review https://spark.apache.org/contributing.html before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mickjermsurawong-stripe closed pull request #25009: test
mickjermsurawong-stripe closed pull request #25009: test URL: https://github.com/apache/spark/pull/25009 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mickjermsurawong-stripe closed pull request #25009: test
mickjermsurawong-stripe closed pull request #25009: test URL: https://github.com/apache/spark/pull/25009 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mickjermsurawong-stripe opened a new pull request #25009: test
mickjermsurawong-stripe opened a new pull request #25009: test URL: https://github.com/apache/spark/pull/25009 ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests) (If this patch involves UI changes, please attach a screenshot; otherwise, remove this) Please review https://spark.apache.org/contributing.html before opening a pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar
AmplabJenkins removed a comment on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar URL: https://github.com/apache/spark/pull/25008#issuecomment-506909288 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107027/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar
AmplabJenkins removed a comment on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar URL: https://github.com/apache/spark/pull/25008#issuecomment-506909284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar
AmplabJenkins commented on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar URL: https://github.com/apache/spark/pull/25008#issuecomment-506909288 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107027/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar
AmplabJenkins commented on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar URL: https://github.com/apache/spark/pull/25008#issuecomment-506909284 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar
SparkQA removed a comment on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar URL: https://github.com/apache/spark/pull/25008#issuecomment-506877347 **[Test build #107027 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107027/testReport)** for PR 25008 at commit [`8c285e5`](https://github.com/apache/spark/commit/8c285e57837bc1ccb87c0d894805ceb5d16f1299). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar
SparkQA commented on issue #25008: [SPARK-28213][SQL] Replace ColumnarBatchScan with equivilant from Columnar URL: https://github.com/apache/spark/pull/25008#issuecomment-506909081 **[Test build #107027 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107027/testReport)** for PR 25008 at commit [`8c285e5`](https://github.com/apache/spark/commit/8c285e57837bc1ccb87c0d894805ceb5d16f1299). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] vanzin commented on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service.
vanzin commented on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service. URL: https://github.com/apache/spark/pull/24817#issuecomment-506908580 I'll file a bug for the flaky test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] carsonwang commented on a change in pull request #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
carsonwang commented on a change in pull request #24978: [SPARK-28177][SQL]
Adjust post shuffle partition number in adaptive execution
URL: https://github.com/apache/spark/pull/24978#discussion_r298776668
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ReduceNumShufflePartitions.scala
##
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive.rule
+
+import scala.collection.mutable.ArrayBuffer
+import scala.concurrent.duration.Duration
+
+import org.apache.spark.MapOutputStatistics
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning,
UnknownPartitioning}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{ShuffledRowRDD, SparkPlan,
UnaryExecNode}
+import org.apache.spark.sql.execution.adaptive.{QueryStageExec,
ReusedQueryStageExec, ShuffleQueryStageExec}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * A rule to adjust the post shuffle partitions based on the map output
statistics.
+ *
+ * The strategy used to determine the number of post-shuffle partitions is
described as follows.
+ * To determine the number of post-shuffle partitions, we have a target input
size for a
+ * post-shuffle partition. Once we have size statistics of all pre-shuffle
partitions, we will do
+ * a pass of those statistics and pack pre-shuffle partitions with continuous
indices to a single
+ * post-shuffle partition until adding another pre-shuffle partition would
cause the size of a
+ * post-shuffle partition to be greater than the target size.
+ *
+ * For example, we have two stages with the following pre-shuffle partition
size statistics:
+ * stage 1: [100 MiB, 20 MiB, 100 MiB, 10MiB, 30 MiB]
+ * stage 2: [10 MiB, 10 MiB, 70 MiB, 5 MiB, 5 MiB]
+ * assuming the target input size is 128 MiB, we will have four post-shuffle
partitions,
+ * which are:
+ * - post-shuffle partition 0: pre-shuffle partition 0 (size 110 MiB)
+ * - post-shuffle partition 1: pre-shuffle partition 1 (size 30 MiB)
+ * - post-shuffle partition 2: pre-shuffle partition 2 (size 170 MiB)
+ * - post-shuffle partition 3: pre-shuffle partition 3 and 4 (size 50 MiB)
+ */
+case class ReduceNumShufflePartitions(conf: SQLConf) extends Rule[SparkPlan] {
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+val shuffleMetrics: Seq[MapOutputStatistics] = plan.collect {
+ case stage: ShuffleQueryStageExec =>
+val metricsFuture = stage.mapOutputStatisticsFuture
+assert(metricsFuture.isCompleted, "ShuffleQueryStageExec should
already be ready")
+ThreadUtils.awaitResult(metricsFuture, Duration.Zero)
+ case ReusedQueryStageExec(_, stage: ShuffleQueryStageExec, _) =>
+val metricsFuture = stage.mapOutputStatisticsFuture
+assert(metricsFuture.isCompleted, "ShuffleQueryStageExec should
already be ready")
+ThreadUtils.awaitResult(metricsFuture, Duration.Zero)
+}
+
+if (!plan.collectLeaves().forall(_.isInstanceOf[QueryStageExec])) {
Review comment:
I believe we are safe here. After checking `isAllQueryStage`, we will get
shuffle metrics from shuffle stages and we will only adjust num shuffle
partitions if num shuffle metrics > 0.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] carsonwang commented on a change in pull request #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
carsonwang commented on a change in pull request #24978: [SPARK-28177][SQL]
Adjust post shuffle partition number in adaptive execution
URL: https://github.com/apache/spark/pull/24978#discussion_r298776462
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ReduceNumShufflePartitions.scala
##
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive.rule
+
+import scala.collection.mutable.ArrayBuffer
+import scala.concurrent.duration.Duration
+
+import org.apache.spark.MapOutputStatistics
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning,
UnknownPartitioning}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{ShuffledRowRDD, SparkPlan,
UnaryExecNode}
+import org.apache.spark.sql.execution.adaptive.{QueryStageExec,
ReusedQueryStageExec, ShuffleQueryStageExec}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * A rule to adjust the post shuffle partitions based on the map output
statistics.
+ *
+ * The strategy used to determine the number of post-shuffle partitions is
described as follows.
+ * To determine the number of post-shuffle partitions, we have a target input
size for a
+ * post-shuffle partition. Once we have size statistics of all pre-shuffle
partitions, we will do
+ * a pass of those statistics and pack pre-shuffle partitions with continuous
indices to a single
+ * post-shuffle partition until adding another pre-shuffle partition would
cause the size of a
+ * post-shuffle partition to be greater than the target size.
+ *
+ * For example, we have two stages with the following pre-shuffle partition
size statistics:
+ * stage 1: [100 MiB, 20 MiB, 100 MiB, 10MiB, 30 MiB]
+ * stage 2: [10 MiB, 10 MiB, 70 MiB, 5 MiB, 5 MiB]
+ * assuming the target input size is 128 MiB, we will have four post-shuffle
partitions,
+ * which are:
+ * - post-shuffle partition 0: pre-shuffle partition 0 (size 110 MiB)
+ * - post-shuffle partition 1: pre-shuffle partition 1 (size 30 MiB)
+ * - post-shuffle partition 2: pre-shuffle partition 2 (size 170 MiB)
+ * - post-shuffle partition 3: pre-shuffle partition 3 and 4 (size 50 MiB)
+ */
+case class ReduceNumShufflePartitions(conf: SQLConf) extends Rule[SparkPlan] {
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+val shuffleMetrics: Seq[MapOutputStatistics] = plan.collect {
+ case stage: ShuffleQueryStageExec =>
+val metricsFuture = stage.mapOutputStatisticsFuture
+assert(metricsFuture.isCompleted, "ShuffleQueryStageExec should
already be ready")
+ThreadUtils.awaitResult(metricsFuture, Duration.Zero)
+ case ReusedQueryStageExec(_, stage: ShuffleQueryStageExec, _) =>
+val metricsFuture = stage.mapOutputStatisticsFuture
+assert(metricsFuture.isCompleted, "ShuffleQueryStageExec should
already be ready")
+ThreadUtils.awaitResult(metricsFuture, Duration.Zero)
+}
+
+if (!plan.collectLeaves().forall(_.isInstanceOf[QueryStageExec])) {
+ // If not all leaf nodes are query stages, it's not safe to reduce the
number of
+ // shuffle partitions, because we may break the assumption that all
children of a spark plan
+ // have same number of output partitions.
+ plan
+} else {
+ // `ShuffleQueryStageExec` gives null mapOutputStatistics when the input
RDD has 0 partitions,
+ // we should skip it when calculating the `partitionStartIndices`.
+ val validMetrics = shuffleMetrics.filter(_ != null)
+ if (validMetrics.nonEmpty) {
+val partitionStartIndices =
estimatePartitionStartIndices(validMetrics.toArray)
+// This transformation adds new nodes, so we must use `transformUp`
here.
+plan.transformUp {
+ // even for shuffle exchange whose input RDD has 0 partition, we
should still update its
+ // `partitionStartIndices`, so that all the leaf shuffles in a stage
have the same
+ // number of output
[GitHub] [spark] carsonwang commented on a change in pull request #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
carsonwang commented on a change in pull request #24978: [SPARK-28177][SQL]
Adjust post shuffle partition number in adaptive execution
URL: https://github.com/apache/spark/pull/24978#discussion_r298776389
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -1911,14 +1911,14 @@ class SQLConf extends Serializable with Logging {
def targetPostShuffleInputSize: Long =
getConf(SHUFFLE_TARGET_POSTSHUFFLE_INPUT_SIZE)
- def runtimeReoptimizationEnabled: Boolean =
getConf(RUNTIME_REOPTIMIZATION_ENABLED)
-
- def adaptiveExecutionEnabled: Boolean =
-getConf(ADAPTIVE_EXECUTION_ENABLED) &&
!getConf(RUNTIME_REOPTIMIZATION_ENABLED)
+ def adaptiveExecutionEnabled: Boolean = getConf(ADAPTIVE_EXECUTION_ENABLED)
def minNumPostShufflePartitions: Int =
getConf(SHUFFLE_MIN_NUM_POSTSHUFFLE_PARTITIONS)
+ def maxNumPostShufflePartitions: Int =
+
getConf(SHUFFLE_MAX_NUM_POSTSHUFFLE_PARTITIONS).getOrElse(numShufflePartitions)
Review comment:
We actually use this as the initial shuffle partition number, which will be
set in the `ShuffleExchangeExec`. So this is expected to be a reasonable value,
instead of Int.Max. This seems to be a little misleading if a user is not
aware it is used as the initial shuffle partition number.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Tonix517 commented on a change in pull request #24994: [SPARK-28133] Adding inverse hyperbolic functions in SQL
Tonix517 commented on a change in pull request #24994: [SPARK-28133] Adding
inverse hyperbolic functions in SQL
URL: https://github.com/apache/spark/pull/24994#discussion_r298775993
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
##
@@ -557,6 +578,27 @@ case class Sin(child: Expression) extends
UnaryMathExpression(math.sin, "SIN")
""")
case class Sinh(child: Expression) extends UnaryMathExpression(math.sinh,
"SINH")
+@ExpressionDescription(
+ usage = """
+_FUNC_(expr) - Returns inverse hyperbolic sine of `expr`.
+ """,
+ arguments = """
+Arguments:
+ * expr - hyperbolic angle
+ """,
+ examples = """
+Examples:
+ > SELECT _FUNC_(0);
+ 0.0
+ """)
+case class Asinh(child: Expression)
+ extends UnaryMathExpression((x: Double) => math.log(x + math.sqrt(x * x +
1.0)), "ASINH") {
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+defineCodeGen(ctx, ev, c =>
+ s"${ev.value} = java.lang.Math.log($c + java.lang.Math.sqrt($c * $c +
1.0));")
Review comment:
Also I just tried the code with several negative values, i got correct
return values.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] carsonwang commented on a change in pull request #24978: [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
carsonwang commented on a change in pull request #24978: [SPARK-28177][SQL]
Adjust post shuffle partition number in adaptive execution
URL: https://github.com/apache/spark/pull/24978#discussion_r298775948
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ReduceNumShufflePartitions.scala
##
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.adaptive.rule
+
+import scala.collection.mutable.ArrayBuffer
+import scala.concurrent.duration.Duration
+
+import org.apache.spark.MapOutputStatistics
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.plans.physical.{Partitioning,
UnknownPartitioning}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.execution.{ShuffledRowRDD, SparkPlan,
UnaryExecNode}
+import org.apache.spark.sql.execution.adaptive.{QueryStageExec,
ReusedQueryStageExec, ShuffleQueryStageExec}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.ThreadUtils
+
+/**
+ * A rule to adjust the post shuffle partitions based on the map output
statistics.
+ *
+ * The strategy used to determine the number of post-shuffle partitions is
described as follows.
+ * To determine the number of post-shuffle partitions, we have a target input
size for a
+ * post-shuffle partition. Once we have size statistics of all pre-shuffle
partitions, we will do
+ * a pass of those statistics and pack pre-shuffle partitions with continuous
indices to a single
+ * post-shuffle partition until adding another pre-shuffle partition would
cause the size of a
+ * post-shuffle partition to be greater than the target size.
+ *
+ * For example, we have two stages with the following pre-shuffle partition
size statistics:
+ * stage 1: [100 MiB, 20 MiB, 100 MiB, 10MiB, 30 MiB]
+ * stage 2: [10 MiB, 10 MiB, 70 MiB, 5 MiB, 5 MiB]
+ * assuming the target input size is 128 MiB, we will have four post-shuffle
partitions,
+ * which are:
+ * - post-shuffle partition 0: pre-shuffle partition 0 (size 110 MiB)
+ * - post-shuffle partition 1: pre-shuffle partition 1 (size 30 MiB)
+ * - post-shuffle partition 2: pre-shuffle partition 2 (size 170 MiB)
+ * - post-shuffle partition 3: pre-shuffle partition 3 and 4 (size 50 MiB)
+ */
+case class ReduceNumShufflePartitions(conf: SQLConf) extends Rule[SparkPlan] {
+
+ override def apply(plan: SparkPlan): SparkPlan = {
+val shuffleMetrics: Seq[MapOutputStatistics] = plan.collect {
+ case stage: ShuffleQueryStageExec =>
+val metricsFuture = stage.mapOutputStatisticsFuture
+assert(metricsFuture.isCompleted, "ShuffleQueryStageExec should
already be ready")
+ThreadUtils.awaitResult(metricsFuture, Duration.Zero)
+ case ReusedQueryStageExec(_, stage: ShuffleQueryStageExec, _) =>
+val metricsFuture = stage.mapOutputStatisticsFuture
+assert(metricsFuture.isCompleted, "ShuffleQueryStageExec should
already be ready")
+ThreadUtils.awaitResult(metricsFuture, Duration.Zero)
+}
+
+if (!plan.collectLeaves().forall(_.isInstanceOf[QueryStageExec])) {
Review comment:
Good point!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-506905051 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
AmplabJenkins removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-506905053 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107024/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-506905053 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107024/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
AmplabJenkins commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-506905051 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
SparkQA removed a comment on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-506864928 **[Test build #107024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107024/testReport)** for PR 21599 at commit [`8e9715c`](https://github.com/apache/spark/commit/8e9715c35e4d4c80b7a4d5bf412bdc15044763ba). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result
SparkQA commented on issue #21599: [SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result URL: https://github.com/apache/spark/pull/21599#issuecomment-506904792 **[Test build #107024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107024/testReport)** for PR 21599 at commit [`8e9715c`](https://github.com/apache/spark/commit/8e9715c35e4d4c80b7a4d5bf412bdc15044763ba). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506903343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107028/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506903341 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
SparkQA removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506883274 **[Test build #107028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107028/testReport)** for PR 25006 at commit [`0b9e8f3`](https://github.com/apache/spark/commit/0b9e8f353967e9901c0c955711e6b4559e02281d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506903343 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107028/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506903341 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
SparkQA commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506903182 **[Test build #107028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107028/testReport)** for PR 25006 at commit [`0b9e8f3`](https://github.com/apache/spark/commit/0b9e8f353967e9901c0c955711e6b4559e02281d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506901279 **[Test build #107032 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107032/testReport)** for PR 25007 at commit [`4c3d692`](https://github.com/apache/spark/commit/4c3d6926069bea6e89f1a13d8faa46ce1180fcf0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506900893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12225/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506900887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506900893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12225/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506900887 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Tonix517 commented on a change in pull request #24994: [SPARK-28133] Adding inverse hyperbolic functions in SQL
Tonix517 commented on a change in pull request #24994: [SPARK-28133] Adding
inverse hyperbolic functions in SQL
URL: https://github.com/apache/spark/pull/24994#discussion_r298770863
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
##
@@ -557,6 +578,27 @@ case class Sin(child: Expression) extends
UnaryMathExpression(math.sin, "SIN")
""")
case class Sinh(child: Expression) extends UnaryMathExpression(math.sinh,
"SINH")
+@ExpressionDescription(
+ usage = """
+_FUNC_(expr) - Returns inverse hyperbolic sine of `expr`.
+ """,
+ arguments = """
+Arguments:
+ * expr - hyperbolic angle
+ """,
+ examples = """
+Examples:
+ > SELECT _FUNC_(0);
+ 0.0
+ """)
+case class Asinh(child: Expression)
+ extends UnaryMathExpression((x: Double) => math.log(x + math.sqrt(x * x +
1.0)), "ASINH") {
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+defineCodeGen(ctx, ev, c =>
+ s"${ev.value} = java.lang.Math.log($c + java.lang.Math.sqrt($c * $c +
1.0));")
Review comment:
thanks for reminding me of FastMath lib. Maybe we can simply use that lib
here, without this hand-crafted calculation?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service.
AmplabJenkins removed a comment on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service. URL: https://github.com/apache/spark/pull/24817#issuecomment-506899623 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service.
AmplabJenkins commented on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service. URL: https://github.com/apache/spark/pull/24817#issuecomment-506899623 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service.
AmplabJenkins commented on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service. URL: https://github.com/apache/spark/pull/24817#issuecomment-506899626 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107025/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service.
AmplabJenkins removed a comment on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service. URL: https://github.com/apache/spark/pull/24817#issuecomment-506899626 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107025/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service.
SparkQA removed a comment on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service. URL: https://github.com/apache/spark/pull/24817#issuecomment-506871224 **[Test build #107025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107025/testReport)** for PR 24817 at commit [`e414f60`](https://github.com/apache/spark/commit/e414f6099d6151cfedd75f4ac1dedf6b73be2d2d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service.
SparkQA commented on issue #24817: [SPARK-27963][core] Allow dynamic allocation without a shuffle service. URL: https://github.com/apache/spark/pull/24817#issuecomment-506899361 **[Test build #107025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107025/testReport)** for PR 24817 at commit [`e414f60`](https://github.com/apache/spark/commit/e414f6099d6151cfedd75f4ac1dedf6b73be2d2d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mgaido91 commented on a change in pull request #24994: [SPARK-28133] Adding inverse hyperbolic functions in SQL
mgaido91 commented on a change in pull request #24994: [SPARK-28133] Adding
inverse hyperbolic functions in SQL
URL: https://github.com/apache/spark/pull/24994#discussion_r298769416
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
##
@@ -557,6 +578,27 @@ case class Sin(child: Expression) extends
UnaryMathExpression(math.sin, "SIN")
""")
case class Sinh(child: Expression) extends UnaryMathExpression(math.sinh,
"SINH")
+@ExpressionDescription(
+ usage = """
+_FUNC_(expr) - Returns inverse hyperbolic sine of `expr`.
+ """,
+ arguments = """
+Arguments:
+ * expr - hyperbolic angle
+ """,
+ examples = """
+Examples:
+ > SELECT _FUNC_(0);
+ 0.0
+ """)
+case class Asinh(child: Expression)
+ extends UnaryMathExpression((x: Double) => math.log(x + math.sqrt(x * x +
1.0)), "ASINH") {
+ override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
+defineCodeGen(ctx, ev, c =>
+ s"${ev.value} = java.lang.Math.log($c + java.lang.Math.sqrt($c * $c +
1.0));")
Review comment:
mmh..what do other DBs do in this case? I checked the implementation of
`FastMath` and it handles negatives...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506896097 **[Test build #107031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107031/testReport)** for PR 25007 at commit [`3083d86`](https://github.com/apache/spark/commit/3083d8686cf1e1650969dde3d0862bd8c53855f0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506895747 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506895753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12224/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506895753 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12224/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506895747 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506892858 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506892868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12223/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins removed a comment on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506892868 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/12223/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers
AmplabJenkins commented on issue #25006: [SPARK-28208][BUILD] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506892858 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506892201 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107029/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506892199 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile closed pull request #25005: [SPARK-28056.2] [PYTHON] [SQL] add docstring/doctest for SCALAR_ITER Pandas UDF
gatorsmile closed pull request #25005: [SPARK-28056.2] [PYTHON] [SQL] add docstring/doctest for SCALAR_ITER Pandas UDF URL: https://github.com/apache/spark/pull/25005 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA removed a comment on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506890051 **[Test build #107029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107029/testReport)** for PR 25007 at commit [`70f59db`](https://github.com/apache/spark/commit/70f59db45a4518159c613d3106533c4cc0ece05e). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506892199 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
SparkQA commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506892185 **[Test build #107029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107029/testReport)** for PR 25007 at commit [`70f59db`](https://github.com/apache/spark/commit/70f59db45a4518159c613d3106533c4cc0ece05e). * This patch **fails Java style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API
AmplabJenkins commented on issue #25007: [SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API URL: https://github.com/apache/spark/pull/25007#issuecomment-506892201 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107029/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25006: [SPARK-28208][BUILD][test-hadoop3.2][test-maven] Upgrade to ORC 1.5.6 including closing the ORC readers
SparkQA commented on issue #25006: [SPARK-28208][BUILD][test-hadoop3.2][test-maven] Upgrade to ORC 1.5.6 including closing the ORC readers URL: https://github.com/apache/spark/pull/25006#issuecomment-506891725 **[Test build #107030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/107030/testReport)** for PR 25006 at commit [`0b9e8f3`](https://github.com/apache/spark/commit/0b9e8f353967e9901c0c955711e6b4559e02281d). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
