[GitHub] [spark] imback82 commented on a change in pull request #28676: [WIP][SPARK-31869][SQL] BroadcastHashJoinExec can utilize the build side for its output partitioning

imback82 commented on a change in pull request #28676: URL: https://github.com/apache/spark/pull/28676#discussion_r446489214 ## File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala ## @@ -554,7 +554,7 @@ class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28866: [SPARK-31845][CORE][TESTS] Refactor DAGSchedulerSuite by introducing completeAndCheckAnswer and using completeNextStageWithFetc

AmplabJenkins removed a comment on pull request #28866: URL: https://github.com/apache/spark/pull/28866#issuecomment-650495643 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins commented on pull request #28866: [SPARK-31845][CORE][TESTS] Refactor DAGSchedulerSuite by introducing completeAndCheckAnswer and using completeNextStageWithFetchFailure

AmplabJenkins commented on pull request #28866: URL: https://github.com/apache/spark/pull/28866#issuecomment-650495636 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AngersZhuuuu commented on a change in pull request #27983: [SPARK-32105][SQL]Implement ScriptTransformation in sql/core

AngersZh commented on a change in pull request #27983: URL: https://github.com/apache/spark/pull/27983#discussion_r446486779 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/script/ScriptTransformationExec.scala ## @@ -0,0 +1,226 @@ +/* + * Licensed to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28866: [SPARK-31845][CORE][TESTS] Refactor DAGSchedulerSuite by introducing completeAndCheckAnswer and using completeNextStageWithFetc

AmplabJenkins removed a comment on pull request #28866: URL: https://github.com/apache/spark/pull/28866#issuecomment-650495636 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA removed a comment on pull request #28866: [SPARK-31845][CORE][TESTS] Refactor DAGSchedulerSuite by introducing completeAndCheckAnswer and using completeNextStageWithFetchFailu

SparkQA removed a comment on pull request #28866: URL: https://github.com/apache/spark/pull/28866#issuecomment-650475411 **[Test build #124556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124556/testReport)** for PR 28866 at commit

[GitHub] [spark] SparkQA commented on pull request #28866: [SPARK-31845][CORE][TESTS] Refactor DAGSchedulerSuite by introducing completeAndCheckAnswer and using completeNextStageWithFetchFailure

SparkQA commented on pull request #28866: URL: https://github.com/apache/spark/pull/28866#issuecomment-650495387 **[Test build #124556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124556/testReport)** for PR 28866 at commit

[GitHub] [spark] viirya commented on a change in pull request #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage

viirya commented on a change in pull request #27690: URL: https://github.com/apache/spark/pull/27690#discussion_r446485414 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala ## @@ -97,12 +99,38 @@ private[hive] trait SaveAsHiveFile

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

AmplabJenkins removed a comment on pull request #28895: URL: https://github.com/apache/spark/pull/28895#issuecomment-650492074 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

AmplabJenkins removed a comment on pull request #28895: URL: https://github.com/apache/spark/pull/28895#issuecomment-650492070 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

AmplabJenkins commented on pull request #28895: URL: https://github.com/apache/spark/pull/28895#issuecomment-650492070 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

SparkQA commented on pull request #28895: URL: https://github.com/apache/spark/pull/28895#issuecomment-650491867 **[Test build #124555 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124555/testReport)** for PR 28895 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

SparkQA removed a comment on pull request #28895: URL: https://github.com/apache/spark/pull/28895#issuecomment-650475412 **[Test build #124555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124555/testReport)** for PR 28895 at commit

[GitHub] [spark] imback82 commented on a change in pull request #28676: [WIP][SPARK-31869][SQL] BroadcastHashJoinExec can utilize the build side for its output partitioning

imback82 commented on a change in pull request #28676: URL: https://github.com/apache/spark/pull/28676#discussion_r446484892 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala ## @@ -60,6 +60,26 @@ case class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage

AmplabJenkins removed a comment on pull request #27690: URL: https://github.com/apache/spark/pull/27690#issuecomment-650488630 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage

AmplabJenkins commented on pull request #27690: URL: https://github.com/apache/spark/pull/27690#issuecomment-650488630 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage

SparkQA commented on pull request #27690: URL: https://github.com/apache/spark/pull/27690#issuecomment-650488481 **[Test build #124560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124560/testReport)** for PR 27690 at commit

[GitHub] [spark] moomindani commented on a change in pull request #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage

moomindani commented on a change in pull request #27690: URL: https://github.com/apache/spark/pull/27690#discussion_r446482526 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala ## @@ -124,11 +153,24 @@ private[hive] trait

[GitHub] [spark] moomindani commented on a change in pull request #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage

moomindani commented on a change in pull request #27690: URL: https://github.com/apache/spark/pull/27690#discussion_r446482475 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala ## @@ -124,11 +153,24 @@ private[hive] trait

[GitHub] [spark] moomindani commented on a change in pull request #27690: [SPARK-21514][SQL] Added a new option to use non-blobstore storage when writing into blobstore storage

moomindani commented on a change in pull request #27690: URL: https://github.com/apache/spark/pull/27690#discussion_r446482392 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/SaveAsHiveFile.scala ## @@ -97,7 +99,34 @@ private[hive] trait

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

AmplabJenkins removed a comment on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-650487591 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

SparkQA removed a comment on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-650487372 **[Test build #124559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124559/testReport)** for PR 28880 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

AmplabJenkins removed a comment on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-650487589 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

AmplabJenkins commented on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-650487589 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

SparkQA commented on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-650487582 **[Test build #124559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124559/testReport)** for PR 28880 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

AmplabJenkins removed a comment on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-650487532 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

AmplabJenkins commented on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-650487532 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28880: [SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly

SparkQA commented on pull request #28880: URL: https://github.com/apache/spark/pull/28880#issuecomment-650487372 **[Test build #124559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124559/testReport)** for PR 28880 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

AmplabJenkins commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-650481570 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

AmplabJenkins removed a comment on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-650481570 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

AmplabJenkins removed a comment on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650480608 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

AmplabJenkins removed a comment on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650480605 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

AmplabJenkins commented on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650480605 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

SparkQA removed a comment on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650478040 **[Test build #124558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124558/testReport)** for PR 28931 at commit

[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

SparkQA commented on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650480574 **[Test build #124558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124558/testReport)** for PR 28931 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

dongjoon-hyun commented on a change in pull request #28895: URL: https://github.com/apache/spark/pull/28895#discussion_r446476737 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -335,23 +335,6 @@ private[spark] abstract class

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

dongjoon-hyun commented on a change in pull request #28895: URL: https://github.com/apache/spark/pull/28895#discussion_r446476737 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -335,23 +335,6 @@ private[spark] abstract class

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

dongjoon-hyun commented on a change in pull request #28895: URL: https://github.com/apache/spark/pull/28895#discussion_r446476096 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -335,23 +335,6 @@ private[spark] abstract class

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

dongjoon-hyun commented on a change in pull request #28895: URL: https://github.com/apache/spark/pull/28895#discussion_r446476096 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -335,23 +335,6 @@ private[spark] abstract class

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

dongjoon-hyun commented on a change in pull request #28895: URL: https://github.com/apache/spark/pull/28895#discussion_r446476096 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -335,23 +335,6 @@ private[spark] abstract class

[GitHub] [spark] cchighman commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path

cchighman commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-650478832 > The PR has lots of changed lines which are actually not changed (indentation). Indentation is the one of style guides, and they didn't seem to violate the guide (that said,

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

dongjoon-hyun commented on a change in pull request #28895: URL: https://github.com/apache/spark/pull/28895#discussion_r446476096 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -335,23 +335,6 @@ private[spark] abstract class

[GitHub] [spark] HeartSaVioR edited a comment on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path

HeartSaVioR edited a comment on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-650477921 The PR has lots of changed lines which are actually not changed (indentation). Indentation is the one of style guides, and they didn't seem to violate the guide

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

dongjoon-hyun commented on a change in pull request #28895: URL: https://github.com/apache/spark/pull/28895#discussion_r446476096 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -335,23 +335,6 @@ private[spark] abstract class

[GitHub] [spark] AmplabJenkins commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

AmplabJenkins commented on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650478156 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

AmplabJenkins removed a comment on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650478156 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

SparkQA commented on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650478040 **[Test build #124558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124558/testReport)** for PR 28931 at commit

[GitHub] [spark] HeartSaVioR commented on pull request #28841: [SPARK-31962][SQL][SS] Provide option to load files after a specified date when reading from a folder path

HeartSaVioR commented on pull request #28841: URL: https://github.com/apache/spark/pull/28841#issuecomment-650477921 The PR has lots of changed lines which are actually not changed (indentation). Indentation is the one of style guides, and they didn't seem to violate the guide (that said,

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

dongjoon-hyun commented on a change in pull request #28931: URL: https://github.com/apache/spark/pull/28931#discussion_r446475568 ## File path: resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala ## @@ -107,7 +107,7 @@ private[spark] class

[GitHub] [spark] dongjoon-hyun commented on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

dongjoon-hyun commented on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650477700 ok to test This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28931: [SPARK-32103] Handle host/port split in IPV6 in YarnRMClient

AmplabJenkins removed a comment on pull request #28931: URL: https://github.com/apache/spark/pull/28931#issuecomment-650026318 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] holdenk commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-650477676 Just an FYI to folks I'm not as active on this PR as I would normally be as I'm waiting to see where the SPIP discussions go. I'll circle back to this next week.

[GitHub] [spark] holdenk commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

holdenk commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r446475360 ## File path: core/src/main/scala/org/apache/spark/SparkContext.scala ## @@ -57,7 +57,7 @@ import org.apache.spark.resource._ import

[GitHub] [spark] SparkQA commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

SparkQA commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-650477379 **[Test build #124557 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124557/testReport)** for PR 28708 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r446475030 ## File path: core/src/main/scala/org/apache/spark/SparkContext.scala ## @@ -57,7 +57,7 @@ import org.apache.spark.resource._ import

[GitHub] [spark] dongjoon-hyun commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-650477142 Retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun closed pull request #28897: [SPARK-32058][BUILD] Use Apache Hadoop 3.2.0 dependency by default

dongjoon-hyun closed pull request #28897: URL: https://github.com/apache/spark/pull/28897 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] dongjoon-hyun commented on pull request #28897: [SPARK-32058][BUILD] Use Apache Hadoop 3.2.0 dependency by default

dongjoon-hyun commented on pull request #28897: URL: https://github.com/apache/spark/pull/28897#issuecomment-650476825 Thank you all. Merged to master. This is an automated message from the Apache Git Service. To respond to

[GitHub] [spark] dongjoon-hyun closed pull request #28866: [SPARK-31845][CORE][TESTS] Refactor DAGSchedulerSuite by introducing completeAndCheckAnswer and using completeNextStageWithFetchFailure

dongjoon-hyun closed pull request #28866: URL: https://github.com/apache/spark/pull/28866 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28866: [SPARK-31845][SPARK-31843][CORE][TESTS] DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and DAGSchedulerSuite: For t

AmplabJenkins removed a comment on pull request #28866: URL: https://github.com/apache/spark/pull/28866#issuecomment-650475583 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28866: [SPARK-31845][SPARK-31843][CORE][TESTS] DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and DAGSchedulerSuite: For the patte

AmplabJenkins commented on pull request #28866: URL: https://github.com/apache/spark/pull/28866#issuecomment-650475583 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins commented on pull request #28919: [SPARK-32038][SQL][FOLLOWUP] Make the alias name pretty after float/double normalization

AmplabJenkins commented on pull request #28919: URL: https://github.com/apache/spark/pull/28919#issuecomment-650475578 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

AmplabJenkins removed a comment on pull request #28895: URL: https://github.com/apache/spark/pull/28895#issuecomment-650475560 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28919: [SPARK-32038][SQL][FOLLOWUP] Make the alias name pretty after float/double normalization

AmplabJenkins removed a comment on pull request #28919: URL: https://github.com/apache/spark/pull/28919#issuecomment-650475578 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

AmplabJenkins commented on pull request #28895: URL: https://github.com/apache/spark/pull/28895#issuecomment-650475560 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28919: [SPARK-32038][SQL][FOLLOWUP] Make the alias name pretty after float/double normalization

SparkQA commented on pull request #28919: URL: https://github.com/apache/spark/pull/28919#issuecomment-650475410 **[Test build #124554 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124554/testReport)** for PR 28919 at commit

[GitHub] [spark] SparkQA commented on pull request #28866: [SPARK-31845][SPARK-31843][CORE][TESTS] DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and DAGSchedulerSuite: For the pattern of

SparkQA commented on pull request #28866: URL: https://github.com/apache/spark/pull/28866#issuecomment-650475411 **[Test build #124556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124556/testReport)** for PR 28866 at commit

[GitHub] [spark] SparkQA commented on pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

SparkQA commented on pull request #28895: URL: https://github.com/apache/spark/pull/28895#issuecomment-650475412 **[Test build #124555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124555/testReport)** for PR 28895 at commit

[GitHub] [spark] dongjoon-hyun commented on pull request #28866: [SPARK-31845][SPARK-31843][CORE][TESTS] DAGSchedulerSuite: Reuse completeNextStageWithFetchFailure and DAGSchedulerSuite: For the patte

dongjoon-hyun commented on pull request #28866: URL: https://github.com/apache/spark/pull/28866#issuecomment-650475206 Retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun commented on pull request #28895: [SPARK-32055][CORE][SQL] Unify getReader and getReaderForRange in ShuffleManager

dongjoon-hyun commented on pull request #28895: URL: https://github.com/apache/spark/pull/28895#issuecomment-650475112 Retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun commented on pull request #28919: [SPARK-32038][SQL][FOLLOWUP] Make the alias name pretty after float/double normalization

dongjoon-hyun commented on pull request #28919: URL: https://github.com/apache/spark/pull/28919#issuecomment-650474740 Retest this please. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] dongjoon-hyun closed pull request #28927: [SPARK-32099][DOCS] Remove broken link in cloud integration documentation

dongjoon-hyun closed pull request #28927: URL: https://github.com/apache/spark/pull/28927 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] dongjoon-hyun closed pull request #28932: [SPARK-32088][PYTHON] Pin the timezone in timestamp_seconds doctest

dongjoon-hyun closed pull request #28932: URL: https://github.com/apache/spark/pull/28932 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] dongjoon-hyun closed pull request #28857: [SPARK-32023][Streaming]Generate spark streaming test jar with maven plugin maven-jar-plugin

dongjoon-hyun closed pull request #28857: URL: https://github.com/apache/spark/pull/28857 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] dongjoon-hyun commented on pull request #28857: [SPARK-32023][Streaming]Generate spark streaming test jar with maven plugin maven-jar-plugin

dongjoon-hyun commented on pull request #28857: URL: https://github.com/apache/spark/pull/28857#issuecomment-650473151 Oh, thank you for confirmation, @wankunde . Then, I close this PR and SPARK-32023. In any way, thank you for your contribution on this issue. Please don't hesitate to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is l

AmplabJenkins removed a comment on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-650472551 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

AmplabJenkins commented on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-650472551 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

SparkQA removed a comment on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-650446036 **[Test build #124553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124553/testReport)** for PR 28848 at commit

[GitHub] [spark] SparkQA commented on pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

SparkQA commented on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-650472356 **[Test build #124553 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124553/testReport)** for PR 28848 at commit

[GitHub] [spark] github-actions[bot] closed pull request #25721: [WIP][SPARK-29018][SQL] Implement Spark Thrift Server with it's own code base on PROTOCOL_VERSION_V9

github-actions[bot] closed pull request #25721: URL: https://github.com/apache/spark/pull/25721 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] github-actions[bot] closed pull request #27377: [SPARK-30666][Core][WIP] Reliable single-stage accumulators

github-actions[bot] closed pull request #27377: URL: https://github.com/apache/spark/pull/27377 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] HeartSaVioR edited a comment on pull request #28904: [SPARK-30462][SS] Streamline the logic on file stream source and sink to avoid memory issue

HeartSaVioR edited a comment on pull request #28904: URL: https://github.com/apache/spark/pull/28904#issuecomment-650460286 I've been running sustain tests (still running) and here's some observation: - SPARK-30946 + SPARK-30462 just wrote compact batch 7309 (73,100,000 entries)

[GitHub] [spark] HeartSaVioR commented on pull request #28904: [SPARK-30462][SS] Streamline the logic on file stream source and sink to avoid memory issue

HeartSaVioR commented on pull request #28904: URL: https://github.com/apache/spark/pull/28904#issuecomment-650460286 I've been running sustain tests (still running) and here's some observation: - SPARK-30946 + SPARK-30462 just wrote compact batch 7309 (73,100,000 entries) which size

[GitHub] [spark] AmplabJenkins commented on pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

AmplabJenkins commented on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-650446374 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is l

AmplabJenkins removed a comment on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-650446374 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

SparkQA commented on pull request #28848: URL: https://github.com/apache/spark/pull/28848#issuecomment-650446036 **[Test build #124553 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124553/testReport)** for PR 28848 at commit

[GitHub] [spark] wypoon commented on a change in pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

wypoon commented on a change in pull request #28848: URL: https://github.com/apache/spark/pull/28848#discussion_r446447794 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1939,24 +1941,24 @@ private[spark] class DAGScheduler(

[GitHub] [spark] wypoon commented on a change in pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

wypoon commented on a change in pull request #28848: URL: https://github.com/apache/spark/pull/28848#discussion_r446446304 ## File path: core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala ## @@ -764,6 +825,7 @@ class DAGSchedulerSuite extends SparkFunSuite

[GitHub] [spark] wypoon commented on a change in pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

wypoon commented on a change in pull request #28848: URL: https://github.com/apache/spark/pull/28848#discussion_r446446147 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -1933,29 +1946,43 @@ private[spark] class DAGScheduler(

[GitHub] [spark] wypoon commented on a change in pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

wypoon commented on a change in pull request #28848: URL: https://github.com/apache/spark/pull/28848#discussion_r446445975 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -170,13 +170,29 @@ private[spark] class DAGScheduler( */

[GitHub] [spark] wypoon commented on a change in pull request #28848: [SPARK-32003][CORE] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost

wypoon commented on a change in pull request #28848: URL: https://github.com/apache/spark/pull/28848#discussion_r446445843 ## File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala ## @@ -170,13 +170,29 @@ private[spark] class DAGScheduler( */

[GitHub] [spark] maropu commented on a change in pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector

maropu commented on a change in pull request #28863: URL: https://github.com/apache/spark/pull/28863#discussion_r446438925 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/OracleConnectionProvider.scala ## @@ -0,0 +1,63 @@ +/* +

[GitHub] [spark] dongjoon-hyun commented on pull request #28919: [SPARK-32038][SQL][FOLLOWUP] Make the alias name pretty after float/double normalization

dongjoon-hyun commented on pull request #28919: URL: https://github.com/apache/spark/pull/28919#issuecomment-650433094 Retest this please. I guess we are waiting for @HyukjinKwon 's request https://github.com/apache/spark/pull/28919#issuecomment-649213796 .

[GitHub] [spark] dongjoon-hyun commented on pull request #28897: [SPARK-32058][BUILD] Use Apache Hadoop 3.2.0 dependency by default

dongjoon-hyun commented on pull request #28897: URL: https://github.com/apache/spark/pull/28897#issuecomment-650428500 Thank you so much, @holdenk ! Yes, we can discuss and improve it separately later. This is an automated

[GitHub] [spark] holdenk commented on pull request #28897: [SPARK-32058][BUILD] Use Apache Hadoop 3.2.0 dependency by default

holdenk commented on pull request #28897: URL: https://github.com/apache/spark/pull/28897#issuecomment-650418721 LGTM, we can continue the PyPI discussion separately. This is an automated message from the Apache Git Service.

[GitHub] [spark] sap1ens commented on a change in pull request #28852: [SPARK-30616][SQL] Introduce TTL config option for SQL Metadata Cache

sap1ens commented on a change in pull request #28852: URL: https://github.com/apache/spark/pull/28852#discussion_r446422811 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveMetadataCacheSuite.scala ## @@ -126,4 +129,39 @@ class HiveMetadataCacheSuite

[GitHub] [spark] dongjoon-hyun commented on pull request #28897: [SPARK-32058][BUILD] Use Apache Hadoop 3.2.0 dependency by default

dongjoon-hyun commented on pull request #28897: URL: https://github.com/apache/spark/pull/28897#issuecomment-650408514 Thank you so much, @dbtsai ! This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] dbtsai commented on pull request #28897: [SPARK-32058][BUILD] Use Apache Hadoop 3.2.0 dependency by default

dbtsai commented on pull request #28897: URL: https://github.com/apache/spark/pull/28897#issuecomment-650403512 +1 from me. Users still have the option to use Hadoop 2.7, so I feel it's safe. This is an automated message

[GitHub] [spark] dongjoon-hyun commented on pull request #28897: [SPARK-32058][BUILD] Use Apache Hadoop 3.2.0 dependency by default

dongjoon-hyun commented on pull request #28897: URL: https://github.com/apache/spark/pull/28897#issuecomment-650401065 Gentle ping once again. This is an automated message from the Apache Git Service. To respond to the

[GitHub] [spark] mccheah commented on a change in pull request #28618: [SPARK-31801][API][SHUFFLE] Register map output metadata

mccheah commented on a change in pull request #28618: URL: https://github.com/apache/spark/pull/28618#discussion_r446401053 ## File path: core/src/main/scala/org/apache/spark/shuffle/MemoizingShuffleDataIO.scala ## @@ -0,0 +1,51 @@ +/* + * Licensed to the Apache Software

  1   2   3   4   >