[GitHub] [spark] SparkQA commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window
SparkQA commented on issue #27943: [SPARK-31179] Fast fail the connection while last connection failed in fast fail time window URL: https://github.com/apache/spark/pull/27943#issuecomment-604245789 **[Test build #120389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120389/testReport)** for PR 27943 at commit [`3ef1612`](https://github.com/apache/spark/commit/3ef16128e6ea095faa3a0cdabccfc7bc66dc7f7c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights
AmplabJenkins removed a comment on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights URL: https://github.com/apache/spark/pull/28031#issuecomment-604245612 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights
AmplabJenkins removed a comment on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights URL: https://github.com/apache/spark/pull/28031#issuecomment-604245617 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25108/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights
AmplabJenkins commented on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights URL: https://github.com/apache/spark/pull/28031#issuecomment-604245617 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25108/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights
AmplabJenkins commented on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights URL: https://github.com/apache/spark/pull/28031#issuecomment-604245612 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights
SparkQA commented on issue #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights URL: https://github.com/apache/spark/pull/28031#issuecomment-604245289 **[Test build #120399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120399/testReport)** for PR 28031 at commit [`ec879cb`](https://github.com/apache/spark/commit/ec879cbdff9cb593ffb665cc1855ea55949d29b9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #27994: [SPARK-31223][ML] Set seed in np.random to regenerate test data
zhengruifeng commented on issue #27994: [SPARK-31223][ML] Set seed in np.random to regenerate test data URL: https://github.com/apache/spark/pull/27994#issuecomment-604244459 Merged to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao opened a new pull request #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights
huaxingao opened a new pull request #28031: [SPARK-30934][ML][FOLLOW-UP] Update ml-guide to include MulticlassClassificationEvaluator weight support in highlights URL: https://github.com/apache/spark/pull/28031 ### What changes were proposed in this pull request? Update ml-guide to include ```MulticlassClassificationEvaluator``` weight support in highlights ### Why are the changes needed? ```MulticlassClassificationEvaluator``` weight support is very important, so should include it in highlights ### Does this PR introduce any user-facing change? Yes after:  ### How was this patch tested? manually build and check This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #27994: [SPARK-31223][ML] Set seed in np.random to regenerate test data
zhengruifeng closed pull request #27994: [SPARK-31223][ML] Set seed in np.random to regenerate test data URL: https://github.com/apache/spark/pull/27994 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #28016: [SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in write/read for ORC datasource
MaxGekk commented on a change in pull request #28016: [SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in write/read for ORC datasource URL: https://github.com/apache/spark/pull/28016#discussion_r398331763 ## File path: sql/core/v1.2/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java ## @@ -42,6 +43,7 @@ private DecimalColumnVector decimalData; private TimestampColumnVector timestampData; Review comment: Yes, it does, but: 1. `DateColumnVector` doesn't have a method similar to `asScratchTimestamp` in `TimestampColumnVector` 2. We don't need to build `java.sql.Date` from serialized days to perform rebasing. It is unnecessary overhead. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only
cloud-fan commented on issue #28003: [SPARK-31234][SQL] ResetCommand should reset config to sc.conf only URL: https://github.com/apache/spark/pull/28003#issuecomment-604243019 I think this is clearly a bug. If users set a static SQL config in the config file, this should not be affected by SET or RESET. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning
AmplabJenkins removed a comment on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning URL: https://github.com/apache/spark/pull/27588#issuecomment-604242125 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25101/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning
AmplabJenkins removed a comment on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning URL: https://github.com/apache/spark/pull/27588#issuecomment-604242119 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning
SparkQA commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning URL: https://github.com/apache/spark/pull/27588#issuecomment-604242105 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25101/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
AmplabJenkins removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604242076 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120394/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning
AmplabJenkins commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning URL: https://github.com/apache/spark/pull/27588#issuecomment-604242125 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25101/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning
AmplabJenkins commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning URL: https://github.com/apache/spark/pull/27588#issuecomment-604242119 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
AmplabJenkins commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604242076 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120394/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
AmplabJenkins removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604242069 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
AmplabJenkins commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604242069 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
SparkQA commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604241644 **[Test build #120394 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120394/testReport)** for PR 28025 at commit [`536107e`](https://github.com/apache/spark/commit/536107e98fa9b1f2d838b704064d20439f60ac61). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
SparkQA removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604230839 **[Test build #120394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120394/testReport)** for PR 28025 at commit [`536107e`](https://github.com/apache/spark/commit/536107e98fa9b1f2d838b704064d20439f60ac61). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28016: [SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in write/read for ORC datasource
cloud-fan commented on a change in pull request #28016: [SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in write/read for ORC datasource URL: https://github.com/apache/spark/pull/28016#discussion_r398329582 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala ## @@ -32,6 +32,7 @@ import org.apache.spark.sql.AnalysisException import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.expressions._ import org.apache.spark.sql.catalyst.util._ +import org.apache.spark.sql.execution.datasources.DaysWritable Review comment: unnecessary change This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28016: [SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in write/read for ORC datasource
cloud-fan commented on a change in pull request #28016: [SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in write/read for ORC datasource URL: https://github.com/apache/spark/pull/28016#discussion_r398329428 ## File path: sql/core/v1.2/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java ## @@ -42,6 +43,7 @@ private DecimalColumnVector decimalData; private TimestampColumnVector timestampData; Review comment: does ORC have `DateColumnVector`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
dongjoon-hyun commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604241530 Hi, @holdenk . If you are busy for now, shall we close this PR for now? ``` - PVs with local storage *** FAILED *** ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28016: [SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in write/read for ORC datasource
cloud-fan commented on a change in pull request #28016:
[SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in
write/read for ORC datasource
URL: https://github.com/apache/spark/pull/28016#discussion_r398329137
##
File path:
sql/core/v1.2/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnVector.java
##
@@ -130,7 +138,13 @@ public short getShort(int rowId) {
@Override
public int getInt(int rowId) {
-return (int) longData.vector[getRowIndex(rowId)];
+int index = getRowIndex(rowId);
+int value = (int) longData.vector[index];
Review comment:
nit: `int value = (int) longData.vector[getRowIndex(rowId)];`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
AmplabJenkins commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604241230 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25106/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
AmplabJenkins commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604241225 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
AmplabJenkins removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604241225 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
AmplabJenkins removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604241230 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25106/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
dongjoon-hyun commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-604240855 BTW, @xuanyuanking . Could you confirm the above question? - https://github.com/apache/spark/pull/27665#discussion_r398328411 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
AmplabJenkins removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604240730 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25102/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
SparkQA commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers URL: https://github.com/apache/spark/pull/27695#issuecomment-604240942 **[Test build #120398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120398/testReport)** for PR 27695 at commit [`2b3ad5b`](https://github.com/apache/spark/commit/2b3ad5bff2db4aa1f0c49503c3bffbb230cb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
SparkQA commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604240932 **[Test build #120397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120397/testReport)** for PR 28028 at commit [`5342fd7`](https://github.com/apache/spark/commit/5342fd7f9c02edb9ec8854a9fc03db44ff0c99c8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
dongjoon-hyun commented on a change in pull request #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r398328411
##
File path:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
What do you mean by `the config must be set`, @xuanyuanking ? What value do
you expect by default? Apparently, this seems to revert SPARK-25641 together
without mentioning SPARK-25641. In the PR, only SPARK-24355 is mentioned.
> No need to give a default value here, when it comes to here, the config
must be set.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
AmplabJenkins removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604240725 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
AmplabJenkins commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604240730 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25102/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
jiangxb1987 commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604240551 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
SparkQA commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604240705 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25102/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
AmplabJenkins commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604240725 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
dongjoon-hyun commented on a change in pull request #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r398328411
##
File path:
common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
What do you mean by `the config must be set`, @xuanyuanking ? What value do
you expect by default? Apparently, this is reverting SPARK-25641 without
mentioning SPARK-25641. In the PR, only SPARK-24355 is mentioned.
> No need to give a default value here, when it comes to here, the config
must be set.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28016: [SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in write/read for ORC datasource
cloud-fan commented on a change in pull request #28016:
[SPARK-31238][SQL][test-hive1.2] Rebase dates to/from Julian calendar in
write/read for ORC datasource
URL: https://github.com/apache/spark/pull/28016#discussion_r398327978
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DaysWritable.scala
##
@@ -35,11 +35,12 @@ import
org.apache.spark.sql.catalyst.util.DateTimeUtils.{rebaseGregorianToJulian
* @param julianDays The number of days since the epoch 1970-01-01 in
* Julian calendar.
*/
-private[hive] class DaysWritable(
+class DaysWritable(
var gregorianDays: Int,
var julianDays: Int)
extends DateWritable {
Review comment:
We probably need to duplicate this class in v1.2 and v2.3 source code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers
jiangxb1987 commented on issue #27695: [SPARK-30949][K8S][CORE] decouple requests and parallelism on kubernetes drivers URL: https://github.com/apache/spark/pull/27695#issuecomment-604239855 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #28001: [SPARK-31237][SQL][TESTS] Replace 3-letter time zones by zone offsets
cloud-fan commented on issue #28001: [SPARK-31237][SQL][TESTS] Replace 3-letter time zones by zone offsets URL: https://github.com/apache/spark/pull/28001#issuecomment-604239459 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28001: [SPARK-31237][SQL][TESTS] Replace 3-letter time zones by zone offsets
cloud-fan closed pull request #28001: [SPARK-31237][SQL][TESTS] Replace 3-letter time zones by zone offsets URL: https://github.com/apache/spark/pull/28001 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
SparkQA commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604238884 **[Test build #120396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120396/testReport)** for PR 27969 at commit [`978a594`](https://github.com/apache/spark/commit/978a594d8ba84797cf4dad1b40db8351dbfa10e4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
AmplabJenkins removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604238154 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120386/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
AmplabJenkins commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604238154 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120386/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
AmplabJenkins commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604238152 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
AmplabJenkins removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604238152 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
SparkQA commented on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604237833 **[Test build #120386 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120386/testReport)** for PR 28028 at commit [`5342fd7`](https://github.com/apache/spark/commit/5342fd7f9c02edb9ec8854a9fc03db44ff0c99c8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator
SparkQA removed a comment on issue #28028: [SPARK-31259][CORE] Fix log message about fetch request size in ShuffleBlockFetcherIterator URL: https://github.com/apache/spark/pull/28028#issuecomment-604188651 **[Test build #120386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120386/testReport)** for PR 28028 at commit [`5342fd7`](https://github.com/apache/spark/commit/5342fd7f9c02edb9ec8854a9fc03db44ff0c99c8). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604237223 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25105/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604237223 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25105/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604237221 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604237221 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
SparkQA commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604236864 **[Test build #120395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120395/testReport)** for PR 27969 at commit [`7501d2c`](https://github.com/apache/spark/commit/7501d2ce9d5c61e1daccd67077228ba8caf2ef31). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
SparkQA commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604234836 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25102/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning
SparkQA commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning URL: https://github.com/apache/spark/pull/27588#issuecomment-604234510 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25101/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ScrapCodes commented on a change in pull request #27966: [SPARK-31200][k8s] Switch https for debian mirrors, to avoid Mirror sync i…
ScrapCodes commented on a change in pull request #27966: [SPARK-31200][k8s] Switch https for debian mirrors, to avoid Mirror sync i… URL: https://github.com/apache/spark/pull/27966#discussion_r396935593 ## File path: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/sources.list ## @@ -0,0 +1,20 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# This file is required for switching to https url based mirrors, See SPARK-31200, for more info. +deb https://deb.debian.org/debian buster main Review comment: Not a dumb question, I should have mentioned it earlier. Yes, this was the one already being used. It is also suggested by https://www.debian.org/mirror/list , when the location is not known or this is the only official mirror with CDN support. >If your system moves around a lot, you may be best served by a "mirror" that is backed by a global CDN. The Debian project maintains deb.debian.org for this purpose and you can use this in your apt sources.list — consult the service's website for details. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
AmplabJenkins commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604231158 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25104/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
AmplabJenkins removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604231155 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ScrapCodes commented on issue #27966: [SPARK-31200][k8s] Switch https for debian mirrors, to avoid Mirror sync i…
ScrapCodes commented on issue #27966: [SPARK-31200][k8s] Switch https for debian mirrors, to avoid Mirror sync i… URL: https://github.com/apache/spark/pull/27966#issuecomment-604231057 Hi @srowen, gentle reminder! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
AmplabJenkins commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604231155 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
AmplabJenkins removed a comment on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604231158 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25104/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names
SparkQA commented on issue #28025: [SPARK-31186][PySpark][SQL] toPandas should not fail on duplicate column names URL: https://github.com/apache/spark/pull/28025#issuecomment-604230839 **[Test build #120394 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120394/testReport)** for PR 28025 at commit [`536107e`](https://github.com/apache/spark/commit/536107e98fa9b1f2d838b704064d20439f60ac61). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
dbtsai commented on a change in pull request #27728:
[SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested
Column Predicate Pushdown for Parquet
URL: https://github.com/apache/spark/pull/27728#discussion_r398318993
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
##
@@ -103,22 +107,42 @@ abstract class ParquetFilterSuite extends QueryTest with
ParquetTest with Shared
checkFilterPredicate(predicate, filterClass, Seq(Row(expected)))(df)
}
- private def checkBinaryFilterPredicate
- (predicate: Predicate, filterClass: Class[_ <: FilterPredicate],
expected: Seq[Row])
- (implicit df: DataFrame): Unit = {
-def checkBinaryAnswer(df: DataFrame, expected: Seq[Row]) = {
- assertResult(expected.map(_.getAs[Array[Byte]](0).mkString(",")).sorted)
{
-
df.rdd.map(_.getAs[Array[Byte]](0).mkString(",")).collect().toSeq.sorted
- }
+ /**
+ * Takes single level `inputDF` dataframe to generate multi-level nested
+ * dataframes as new test data.
+ */
+ private def withNestedDataFrame(inputDF: DataFrame)
+ (runTests: (DataFrame, String, Any => Any) => Unit): Unit = {
+assert(inputDF.schema.fields.length == 1)
+assert(!inputDF.schema.fields.head.dataType.isInstanceOf[StructType])
Review comment:
@cloud-fan schema checking in the code to avoid passing any type of
dataframe.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
dbtsai commented on a change in pull request #27728:
[SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested
Column Predicate Pushdown for Parquet
URL: https://github.com/apache/spark/pull/27728#discussion_r398310277
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
##
@@ -121,43 +121,81 @@ abstract class ParquetFilterSuite extends QueryTest with
ParquetTest with Shared
checkBinaryFilterPredicate(predicate, filterClass, Seq(Row(expected)))(df)
}
+ /**
+ * Takes single level `inputDF` dataframe to generate multi-level nested
+ * dataframes as new test data.
+ */
+ private def withNestedDataFrame(inputDF: DataFrame)
Review comment:
Okay, this is not easy since one of the test case is like
```scala
val dataFrame = spark.createDataFrame(rdd, StructType.fromDDL(s"a
decimal($precision, 2)"))
withNestedDataFrame(dataFrame) { case (inputDF, pushDownColName,
resultFun) =>
withParquetDataFrame(inputDF) { implicit df =>
val decimalAttr: Expression = df(pushDownColName).expr
assert(df(pushDownColName).expr.dataType ===
DecimalType(precision, 2))
```
, so the dataframe can not be constructed directly from
`withNestedDataFrame[T](data: Seq[T])`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
dongjoon-hyun commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-604229321 Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
dbtsai commented on a change in pull request #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#discussion_r398317799 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala ## @@ -103,22 +107,42 @@ abstract class ParquetFilterSuite extends QueryTest with ParquetTest with Shared checkFilterPredicate(predicate, filterClass, Seq(Row(expected)))(df) } - private def checkBinaryFilterPredicate Review comment: @cloud-fan Since `checkFilterPredicate` works for binary data in the current implementation, `checkBinaryFilterPredicate` can be deleted. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
cloud-fan commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-604228688 it is now. merge script runs slowly at my sides... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
dongjoon-hyun commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665#issuecomment-604227335 Hi, @cloud-fan . This seems to be not in `branch-3.0` yet. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
SparkQA removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604222600 **[Test build #120392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120392/testReport)** for PR 25748 at commit [`fe68184`](https://github.com/apache/spark/commit/fe68184f6decdeca2969d1be48ffaa71fc1acacb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
AmplabJenkins removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604226923 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120392/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
AmplabJenkins removed a comment on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604226921 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
AmplabJenkins commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604226923 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120392/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
AmplabJenkins commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604226921 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
SparkQA commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604226821 **[Test build #120392 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120392/testReport)** for PR 25748 at commit [`fe68184`](https://github.com/apache/spark/commit/fe68184f6decdeca2969d1be48ffaa71fc1acacb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
dbtsai commented on a change in pull request #27728:
[SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested
Column Predicate Pushdown for Parquet
URL: https://github.com/apache/spark/pull/27728#discussion_r398314267
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2049,6 +2049,17 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val NESTED_PREDICATE_PUSHDOWN_ENABLED =
+buildConf("spark.sql.optimizer.nestedPredicatePushdown.enabled")
+ .internal()
+ .doc("When true, Spark tries to push down predicates for nested columns
and or names " +
+"containing `dots` to data sources. Currently, Parquet implements both
optimizations " +
+"while ORC only supports predicates for names containing `dots`. The
other data sources" +
+"don't support this feature yet.")
+ .version("3.0.0")
+ .booleanConf
+ .createWithDefault(true)
Review comment:
Since the filter apis will be enhanced to support nested columns and column
name containing `dots`, it will be nice to introduce it in a major release.
It's a good idea! We can make another PR to turn this feature on for
specific data sources in a separate PR. This PR already grows too big.
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-604224794 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins removed a comment on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-604224801 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25103/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-604224801 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25103/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
AmplabJenkins commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-604224794 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604224533 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604224533 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604224541 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120382/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
AmplabJenkins removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604224541 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120382/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
cloud-fan closed pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group URL: https://github.com/apache/spark/pull/27665 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
SparkQA removed a comment on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604148325 **[Test build #120382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120382/testReport)** for PR 27969 at commit [`14d7b19`](https://github.com/apache/spark/commit/14d7b191521dbc70179cfbcd20aa585959963172). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
SparkQA commented on issue #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#issuecomment-604224438 **[Test build #120393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120393/testReport)** for PR 27728 at commit [`4883e68`](https://github.com/apache/spark/commit/4883e68f636b5e2b9b845f4e23909a2854d28063). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir
SparkQA commented on issue #27969: [SPARK-31170][SQL][test-hive1.2] Spark SQL Cli should respect hive-site.xml and spark.sql.warehouse.dir URL: https://github.com/apache/spark/pull/27969#issuecomment-604223996 **[Test build #120382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120382/testReport)** for PR 27969 at commit [`14d7b19`](https://github.com/apache/spark/commit/14d7b191521dbc70179cfbcd20aa585959963172). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
dbtsai commented on a change in pull request #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet URL: https://github.com/apache/spark/pull/27728#discussion_r398312781 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala ## @@ -121,43 +121,81 @@ abstract class ParquetFilterSuite extends QueryTest with ParquetTest with Shared checkBinaryFilterPredicate(predicate, filterClass, Seq(Row(expected)))(df) } + /** + * Takes single level `inputDF` dataframe to generate multi-level nested + * dataframes as new test data. + */ + private def withNestedDataFrame(inputDF: DataFrame) Review comment: Instead, I add schema checking in the code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei commented on issue #28030: [SPARK-31263][SHUFFLE] Enable yarn shuffle service to close the idle connections
turboFei commented on issue #28030: [SPARK-31263][SHUFFLE] Enable yarn shuffle service to close the idle connections URL: https://github.com/apache/spark/pull/28030#issuecomment-604223347 cc @Ngone51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] turboFei edited a comment on issue #28030: [SPARK-31263][SHUFFLE] Enable yarn shuffle service to close the idle connections
turboFei edited a comment on issue #28030: [SPARK-31263][SHUFFLE] Enable yarn shuffle service to close the idle connections URL: https://github.com/apache/spark/pull/28030#issuecomment-604220710 just keep consistent with: https://github.com/apache/spark/blob/b024a8a69e4ae45c6ded3dd3f9f27e73a0069891/core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala#L107 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
SparkQA commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604222600 **[Test build #120392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120392/testReport)** for PR 25748 at commit [`fe68184`](https://github.com/apache/spark/commit/fe68184f6decdeca2969d1be48ffaa71fc1acacb). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning
SparkQA commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning URL: https://github.com/apache/spark/pull/27588#issuecomment-604222537 **[Test build #120391 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120391/testReport)** for PR 27588 at commit [`7404cf8`](https://github.com/apache/spark/commit/7404cf8cff46ea94f71a38a3a819329f6f2273d6). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning
dongjoon-hyun commented on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning URL: https://github.com/apache/spark/pull/27588#issuecomment-604221764 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite
dongjoon-hyun commented on issue #25748: [SPARK-28904][K8S][TESTS] Create mount for PvTestSuite URL: https://github.com/apache/spark/pull/25748#issuecomment-604221645 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning
AmplabJenkins removed a comment on issue #27588: [BACKPORT] Backport of [SPARK-20628][CORE][K8S] Start to improve decommissioning URL: https://github.com/apache/spark/pull/27588#issuecomment-600955009 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24728/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #27728: [SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested Column Predicate Pushdown for Parquet
dbtsai commented on a change in pull request #27728:
[SPARK-25556][SPARK-17636][SPARK-31026][SPARK-31060][SQL][test-hive1.2] Nested
Column Predicate Pushdown for Parquet
URL: https://github.com/apache/spark/pull/27728#discussion_r398310277
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala
##
@@ -121,43 +121,81 @@ abstract class ParquetFilterSuite extends QueryTest with
ParquetTest with Shared
checkBinaryFilterPredicate(predicate, filterClass, Seq(Row(expected)))(df)
}
+ /**
+ * Takes single level `inputDF` dataframe to generate multi-level nested
+ * dataframes as new test data.
+ */
+ private def withNestedDataFrame(inputDF: DataFrame)
Review comment:
Okay, this is not easy since one of the test case is like
```scala
val dataFrame = spark.createDataFrame(rdd, StructType.fromDDL(s"a
decimal($precision, 2)"))
withNestedDataFrame(dataFrame) { case (inputDF, pushDownColName,
resultFun) =>
withParquetDataFrame(inputDF) { implicit df =>
val decimalAttr: Expression = df(pushDownColName).expr
assert(df(pushDownColName).expr.dataType ===
DecimalType(precision, 2))
```
, and the dataframe can not be constructed directly from
`withNestedDataFrame[T](data: Seq[T])`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
