[GitHub] [spark] holdenk commented on pull request #31299: [SPARK-32866][K8S] Fix docker cross-build

holdenk commented on pull request #31299: URL: https://github.com/apache/spark/pull/31299#issuecomment-765725578 So previously docker buildx automatically did the push without the push flag, but that's no longer the case, so yeah the current code doesn't work if you need to push it

[GitHub] [spark] SparkQA removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

SparkQA removed a comment on pull request #31298: URL: https://github.com/apache/spark/pull/31298#issuecomment-765687452 **[Test build #134384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134384/testReport)** for PR 31298 at commit

[GitHub] [spark] dongjoon-hyun opened a new pull request #31301: [SPARK-34208][BUILD] Upgrade ORC to 1.6.7

dongjoon-hyun opened a new pull request #31301: URL: https://github.com/apache/spark/pull/31301 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ###

[GitHub] [spark] SparkQA commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

SparkQA commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-765796859 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38975/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

AmplabJenkins removed a comment on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-765817460 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134390/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31301: [SPARK-34208][BUILD] Upgrade ORC to 1.6.7

AmplabJenkins removed a comment on pull request #31301: URL: https://github.com/apache/spark/pull/31301#issuecomment-765817463 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38976/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

AmplabJenkins removed a comment on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765817462 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134389/

[GitHub] [spark] AmplabJenkins commented on pull request #31301: [SPARK-34208][BUILD] Upgrade ORC to 1.6.7

AmplabJenkins commented on pull request #31301: URL: https://github.com/apache/spark/pull/31301#issuecomment-765817463 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38976/

[GitHub] [spark] AmplabJenkins commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

AmplabJenkins commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765817462 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134389/

[GitHub] [spark] LuciferYang commented on a change in pull request #31294: [SPARK-34202][SQL][TEST] Add ability to fetch spark release package from internal environment in HiveExternalCatalogVersionsS

LuciferYang commented on a change in pull request #31294: URL: https://github.com/apache/spark/pull/31294#discussion_r563004221 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala ## @@ -237,17 +237,20 @@ class

[GitHub] [spark] xuanyuanking commented on a change in pull request #31271: [SPARK-34185][DOCS] Review and fix issues in API docs

xuanyuanking commented on a change in pull request #31271: URL: https://github.com/apache/spark/pull/31271#discussion_r563006334 ## File path: project/SparkBuild.scala ## @@ -910,7 +910,7 @@ object Unidoc {

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31271: [SPARK-34185][DOCS] Review and fix issues in API docs

dongjoon-hyun commented on a change in pull request #31271: URL: https://github.com/apache/spark/pull/31271#discussion_r563008430 ## File path: core/src/main/scala/org/apache/spark/storage/FallbackStorage.scala ## @@ -90,7 +90,7 @@ private[storage] class FallbackStorage(conf:

[GitHub] [spark] wangyum opened a new pull request #31303: [SPARK-34211][SQL][TEST] Benchmark TPC-DS with 1GB scale factor

wangyum opened a new pull request #31303: URL: https://github.com/apache/spark/pull/31303 ### What changes were proposed in this pull request? This pr add a new Github action to benchmark TPC-DS with 1GB scale factor. ### Why are the changes needed? 1. To track

[GitHub] [spark] SparkQA removed a comment on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

SparkQA removed a comment on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765807931 **[Test build #134392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134392/testReport)** for PR 31300 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #30788: [SPARK-33726][SQL] Fix for Duplicate field names during Aggregation

SparkQA removed a comment on pull request #30788: URL: https://github.com/apache/spark/pull/30788#issuecomment-765808041 **[Test build #134393 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134393/testReport)** for PR 30788 at commit

[GitHub] [spark] SparkQA commented on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

SparkQA commented on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765869599 **[Test build #134392 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134392/testReport)** for PR 31300 at commit

[GitHub] [spark] SparkQA commented on pull request #30788: [SPARK-33726][SQL] Fix for Duplicate field names during Aggregation

SparkQA commented on pull request #30788: URL: https://github.com/apache/spark/pull/30788#issuecomment-765869630 **[Test build #134393 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134393/testReport)** for PR 30788 at commit

[GitHub] [spark] rxin commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

rxin commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765877931 > > Yes the question is also applied to RDD.pipe as well, but the serialization is done via `OutputStreamWriter.println` which is relatively "known" - `String.valueOf(T)` and

[GitHub] [spark] SparkQA commented on pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

SparkQA commented on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-765882743 **[Test build #134399 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134399/testReport)** for PR 31286 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check

AmplabJenkins commented on pull request #31203: URL: https://github.com/apache/spark/pull/31203#issuecomment-765737026 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134382/

[GitHub] [spark] AmplabJenkins commented on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

AmplabJenkins commented on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765737029 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38971/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

AmplabJenkins removed a comment on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765737029 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38971/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

AmplabJenkins removed a comment on pull request #31298: URL: https://github.com/apache/spark/pull/31298#issuecomment-765737030 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31299: [SPARK-32866][K8S] Fix docker cross-build

AmplabJenkins removed a comment on pull request #31299: URL: https://github.com/apache/spark/pull/31299#issuecomment-765737028 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38969/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check

AmplabJenkins removed a comment on pull request #31203: URL: https://github.com/apache/spark/pull/31203#issuecomment-765737026 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134382/

[GitHub] [spark] AmplabJenkins commented on pull request #31299: [SPARK-32866][K8S] Fix docker cross-build

AmplabJenkins commented on pull request #31299: URL: https://github.com/apache/spark/pull/31299#issuecomment-765737028 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38969/

[GitHub] [spark] AmplabJenkins commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

AmplabJenkins commented on pull request #31298: URL: https://github.com/apache/spark/pull/31298#issuecomment-765737030 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

SparkQA commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765742679 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38972/

[GitHub] [spark] yliou edited a comment on pull request #30788: [SPARK-33726][SQL] Fix for Duplicate field names during Aggregation

yliou edited a comment on pull request #30788: URL: https://github.com/apache/spark/pull/30788#issuecomment-765800322 Sure, I'll add a `assert` to `FixedLengthRowBasedKeyValueBatch#appendRow` in the follow-up PR. I also used your PR description suggestion at the top @attilapiros .

[GitHub] [spark] AmplabJenkins removed a comment on pull request #30788: [SPARK-33726][SQL] Fix for Duplicate field names during Aggregation

AmplabJenkins removed a comment on pull request #30788: URL: https://github.com/apache/spark/pull/30788#issuecomment-765845969 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38979/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

AmplabJenkins removed a comment on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765845976 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38978/

[GitHub] [spark] AmplabJenkins commented on pull request #30788: [SPARK-33726][SQL] Fix for Duplicate field names during Aggregation

AmplabJenkins commented on pull request #30788: URL: https://github.com/apache/spark/pull/30788#issuecomment-765845969 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38979/

[GitHub] [spark] AmplabJenkins commented on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

AmplabJenkins commented on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765845976 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38978/

[GitHub] [spark] viirya commented on a change in pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

viirya commented on a change in pull request #31296: URL: https://github.com/apache/spark/pull/31296#discussion_r563011361 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -2889,6 +2889,24 @@ class Dataset[T] private[sql](

[GitHub] [spark] rxin commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

rxin commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765875318 Agree with @HeartSaVioR here. I think it is more general if we just provide a UDF for this... it also doesn’t pollute the Dataset API with something so rarely used. 

[GitHub] [spark] SparkQA commented on pull request #31273: [WIP][Spark-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

SparkQA commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-765875254 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38982/

[GitHub] [spark] viirya edited a comment on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

viirya edited a comment on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765878887 > It is an issue because encoder only specifies how an object would map to the internal physical structure of the row, and by exposing this pipe API, we are exposing the

[GitHub] [spark] SparkQA commented on pull request #31301: [SPARK-34208][BUILD] Upgrade ORC to 1.6.7

SparkQA commented on pull request #31301: URL: https://github.com/apache/spark/pull/31301#issuecomment-765738207 **[Test build #134387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134387/testReport)** for PR 31301 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

AmplabJenkins commented on pull request #31298: URL: https://github.com/apache/spark/pull/31298#issuecomment-765796559 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38973/

[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

SparkQA commented on pull request #31298: URL: https://github.com/apache/spark/pull/31298#issuecomment-765796545 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38973/

[GitHub] [spark] SparkQA commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

SparkQA commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765796602 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38974/

[GitHub] [spark] AmplabJenkins commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

AmplabJenkins commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765796613 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38974/

[GitHub] [spark] SparkQA removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

SparkQA removed a comment on pull request #31298: URL: https://github.com/apache/spark/pull/31298#issuecomment-765740513 **[Test build #134388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134388/testReport)** for PR 31298 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

AmplabJenkins removed a comment on pull request #31298: URL: https://github.com/apache/spark/pull/31298#issuecomment-765796559 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38973/

[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

SparkQA commented on pull request #31298: URL: https://github.com/apache/spark/pull/31298#issuecomment-765808536 **[Test build #134388 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134388/testReport)** for PR 31298 at commit

[GitHub] [spark] SparkQA commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

SparkQA commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-765814803 **[Test build #134390 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134390/testReport)** for PR 31249 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

SparkQA removed a comment on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-765740562 **[Test build #134390 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134390/testReport)** for PR 31249 at commit

[GitHub] [spark] SparkQA commented on pull request #31294: [SPARK-34202][SQL][TEST] Add ability to fetch spark release package from internal environment in HiveExternalCatalogVersionsSuite

SparkQA commented on pull request #31294: URL: https://github.com/apache/spark/pull/31294#issuecomment-765841546 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38980/

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31271: [SPARK-34185][DOCS] Review and fix issues in API docs

dongjoon-hyun commented on a change in pull request #31271: URL: https://github.com/apache/spark/pull/31271#discussion_r563008575 ## File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala ## @@ -1268,7 +1268,7 @@ private[hive] object

[GitHub] [spark] AmplabJenkins commented on pull request #31284: [SPARK-34167][SQL]Reading parquet with IntDecimal written as a LongDecimal blows up

AmplabJenkins commented on pull request #31284: URL: https://github.com/apache/spark/pull/31284#issuecomment-765862877 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134391/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31294: [SPARK-34202][SQL][TEST] Add ability to fetch spark release package from internal environment in HiveExternalCatalogVersionsSui

AmplabJenkins removed a comment on pull request #31294: URL: https://github.com/apache/spark/pull/31294#issuecomment-765868953 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #31302: [SPARK-34210][SQL] After upgrading 3.0.1, Spark SQL access hive on HBase table access exception

AmplabJenkins commented on pull request #31302: URL: https://github.com/apache/spark/pull/31302#issuecomment-765869063 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #31294: [SPARK-34202][SQL][TEST] Add ability to fetch spark release package from internal environment in HiveExternalCatalogVersionsSuite

AmplabJenkins commented on pull request #31294: URL: https://github.com/apache/spark/pull/31294#issuecomment-765868953 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #31273: [WIP][Spark-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

SparkQA commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-765870075 **[Test build #134396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134396/testReport)** for PR 31273 at commit

[GitHub] [spark] viirya commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

viirya commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765879991 > Please just create an executable which prints out stdin (serialized data) and passes to the pipe API... I think it's the easiest way to realize. Ok, ok. I didn't like

[GitHub] [spark] viirya edited a comment on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

viirya edited a comment on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765879991 > Please just create an executable which prints out stdin (serialized data) and passes to the pipe API... I think it's the easiest way to realize. Ok, ok. I didn't

[GitHub] [spark] SparkQA commented on pull request #31286: [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

SparkQA commented on pull request #31286: URL: https://github.com/apache/spark/pull/31286#issuecomment-765882028 **[Test build #134398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134398/testReport)** for PR 31286 at commit

[GitHub] [spark] SparkQA commented on pull request #31273: [WIP][Spark-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

SparkQA commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-765884015 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38982/

[GitHub] [spark] SparkQA commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

SparkQA commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765714476 **[Test build #134386 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134386/testReport)** for PR 31296 at commit

[GitHub] [spark] gatorsmile commented on a change in pull request #26804: [SPARK-26346][BUILD][SQL] Upgrade Parquet to 1.11.1

gatorsmile commented on a change in pull request #26804: URL: https://github.com/apache/spark/pull/26804#discussion_r562959779 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala ## @@ -127,6 +127,9 @@ class

[GitHub] [spark] holdenk commented on a change in pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

holdenk commented on a change in pull request #31249: URL: https://github.com/apache/spark/pull/31249#discussion_r562990433 ## File path: core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala ## @@ -176,11 +182,21 @@ class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31284: [SPARK-34167][SQL]Reading parquet with IntDecimal written as a LongDecimal blows up

AmplabJenkins removed a comment on pull request #31284: URL: https://github.com/apache/spark/pull/31284#issuecomment-765807778 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38977/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31301: [SPARK-34208][BUILD] Upgrade ORC to 1.6.7

AmplabJenkins removed a comment on pull request #31301: URL: https://github.com/apache/spark/pull/31301#issuecomment-76580 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134387/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

AmplabJenkins removed a comment on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-765807781 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38975/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

AmplabJenkins removed a comment on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765794317 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

AmplabJenkins commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-765807781 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38975/

[GitHub] [spark] AmplabJenkins commented on pull request #31284: [SPARK-34167][SQL]Reading parquet with IntDecimal written as a LongDecimal blows up

AmplabJenkins commented on pull request #31284: URL: https://github.com/apache/spark/pull/31284#issuecomment-765807778 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38977/

[GitHub] [spark] AmplabJenkins commented on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

AmplabJenkins commented on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765807780 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134385/

[GitHub] [spark] AmplabJenkins commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

AmplabJenkins commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765807779 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134386/

[GitHub] [spark] AmplabJenkins commented on pull request #31301: [SPARK-34208][BUILD] Upgrade ORC to 1.6.7

AmplabJenkins commented on pull request #31301: URL: https://github.com/apache/spark/pull/31301#issuecomment-76580 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134387/

[GitHub] [spark] SparkQA commented on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

SparkQA commented on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765807931 **[Test build #134392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134392/testReport)** for PR 31300 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

AmplabJenkins removed a comment on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765807780 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134385/

[GitHub] [spark] SparkQA commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

SparkQA commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765811935 **[Test build #134389 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134389/testReport)** for PR 31296 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

SparkQA removed a comment on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765740837 **[Test build #134389 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134389/testReport)** for PR 31296 at commit

[GitHub] [spark] SparkQA commented on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

SparkQA commented on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765815065 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38978/

[GitHub] [spark] xuanyuanking commented on a change in pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

xuanyuanking commented on a change in pull request #31296: URL: https://github.com/apache/spark/pull/31296#discussion_r563011061 ## File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ## @@ -2889,6 +2889,24 @@ class Dataset[T] private[sql](

[GitHub] [spark] SparkQA commented on pull request #31303: [SPARK-34211][SQL][TEST] Benchmark TPC-DS with 1GB scale factor

SparkQA commented on pull request #31303: URL: https://github.com/apache/spark/pull/31303#issuecomment-765870140 **[Test build #134395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134395/testReport)** for PR 31303 at commit

[GitHub] [spark] viirya commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

viirya commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765875951 > Agree with @HeartSaVioR here. I think it is more general if we just provide a UDF for this... it also doesn’t pollute the Dataset API with something so rarely used. I'm

[GitHub] [spark] rxin commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

rxin commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765879297 > > It is an issue because encoder only specifies how an object would map to the internal physical structure of the row, and by exposing this pipe API, we are exposing the

[GitHub] [spark] HeartSaVioR edited a comment on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

HeartSaVioR edited a comment on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765879370 Please just create an executable which prints out stdin (serialized data) and passes to the pipe API... I think it's the easiest way to realize it.

[GitHub] [spark] HeartSaVioR commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

HeartSaVioR commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765879370 Please just create an executable which prints out stdin and passes to the pipe API... I think it's the easiest way to realize it.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31273: [WIP][Spark-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

AmplabJenkins removed a comment on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-765881666 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134396/

[GitHub] [spark] SparkQA removed a comment on pull request #31273: [WIP][Spark-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

SparkQA removed a comment on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-765870075 **[Test build #134396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134396/testReport)** for PR 31273 at commit

[GitHub] [spark] SparkQA commented on pull request #31273: [WIP][Spark-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

SparkQA commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-765881640 **[Test build #134396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134396/testReport)** for PR 31273 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #31273: [WIP][Spark-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

AmplabJenkins commented on pull request #31273: URL: https://github.com/apache/spark/pull/31273#issuecomment-765881666 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134396/

[GitHub] [spark] SparkQA commented on pull request #31303: [SPARK-34211][SQL][TEST] Benchmark TPC-DS with 1GB scale factor

SparkQA commented on pull request #31303: URL: https://github.com/apache/spark/pull/31303#issuecomment-765885121 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38983/

[GitHub] [spark] SparkQA commented on pull request #31299: [SPARK-32866][K8S] Fix docker cross-build

SparkQA commented on pull request #31299: URL: https://github.com/apache/spark/pull/31299#issuecomment-765708706 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38969/

[GitHub] [spark] AmplabJenkins commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check

AmplabJenkins commented on pull request #31203: URL: https://github.com/apache/spark/pull/31203#issuecomment-765713170 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38968/

[GitHub] [spark] AmplabJenkins commented on pull request #31297: [SPARK-34206][K8S] Make Guava Cache to ExecutorPodsLifecycleManager private field

AmplabJenkins commented on pull request #31297: URL: https://github.com/apache/spark/pull/31297#issuecomment-765713172 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38967/

[GitHub] [spark] AmplabJenkins commented on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

AmplabJenkins commented on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-765713173 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check

AmplabJenkins removed a comment on pull request #31203: URL: https://github.com/apache/spark/pull/31203#issuecomment-765713170 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38968/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31297: [SPARK-34206][K8S] Make Guava Cache to ExecutorPodsLifecycleManager private field

AmplabJenkins removed a comment on pull request #31297: URL: https://github.com/apache/spark/pull/31297#issuecomment-765713172 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38967/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31249: [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

AmplabJenkins removed a comment on pull request #31249: URL: https://github.com/apache/spark/pull/31249#issuecomment-765713171 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #31203: [SPARK-33212][FOLLOW-UP][BUILD] Bring back duplicate dependency check and add more strict Hadoop version check

SparkQA commented on pull request #31203: URL: https://github.com/apache/spark/pull/31203#issuecomment-765732872 **[Test build #134382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134382/testReport)** for PR 31203 at commit

[GitHub] [spark] SparkQA commented on pull request #31296: [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

SparkQA commented on pull request #31296: URL: https://github.com/apache/spark/pull/31296#issuecomment-765732721 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38972/

[GitHub] [spark] razajafri commented on pull request #31284: [SPARK-34167][SQL]Reading parquet with IntDecimal written as a LongDecimal blows up

razajafri commented on pull request #31284: URL: https://github.com/apache/spark/pull/31284#issuecomment-765743754 > I'm a bit confused by your description, it would be nice to add more detail. looking at the code I think what you are saying is that you read it as a long from the parquet

[GitHub] [spark] SparkQA removed a comment on pull request #31299: [SPARK-32866][K8S] Fix docker cross-build

SparkQA removed a comment on pull request #31299: URL: https://github.com/apache/spark/pull/31299#issuecomment-765690763 **[Test build #134383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134383/testReport)** for PR 31299 at commit

[GitHub] [spark] SparkQA removed a comment on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

SparkQA removed a comment on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765696475 **[Test build #134385 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134385/testReport)** for PR 31300 at commit

[GitHub] [spark] SparkQA commented on pull request #31300: [SPARK-34052][SQL][3.1] store SQL text for a temp view created using "CACHE TABLE .. AS SELECT ..."

SparkQA commented on pull request #31300: URL: https://github.com/apache/spark/pull/31300#issuecomment-765802531 **[Test build #134385 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134385/testReport)** for PR 31300 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

AmplabJenkins removed a comment on pull request #31298: URL: https://github.com/apache/spark/pull/31298#issuecomment-765808832 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134388/

  1   2   3   4   5   6   >