[GitHub] [spark] HyukjinKwon closed pull request #37987: Bump protobuf from 4.21.5 to 4.21.6 in /dev
HyukjinKwon closed pull request #37987: Bump protobuf from 4.21.5 to 4.21.6 in /dev URL: https://github.com/apache/spark/pull/37987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dependabot[bot] commented on pull request #37987: Bump protobuf from 4.21.5 to 4.21.6 in /dev
dependabot[bot] commented on PR #37987: URL: https://github.com/apache/spark/pull/37987#issuecomment-1257127701 OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting `@dependabot ignore this major version` or `@dependabot ignore this minor version`. If you change your mind, just re-open this PR and I'll resolve any conflicts on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dependabot[bot] opened a new pull request, #37987: Bump protobuf from 4.21.5 to 4.21.6 in /dev
dependabot[bot] opened a new pull request, #37987:
URL: https://github.com/apache/spark/pull/37987
Bumps [protobuf](https://github.com/protocolbuffers/protobuf) from 4.21.5 to
4.21.6.
Release notes
Sourced from https://github.com/protocolbuffers/protobuf/releases;>protobuf's
releases.
Protocol Buffers v3.0.0-beta-4
Version 3.0.0-beta-4
General
Added a deterministic serialization API for C++. The deterministic
serialization guarantees that given a binary, equal messages will be
serialized to the same bytes. This allows applications like MapReduce to
group equal messages based on the serialized bytes. The deterministic
serialization is, however, NOT canonical across languages; it is also
unstable across different builds with schema changes due to unknown fields.
Users who need canonical serialization, e.g. persistent storage in a
canonical form, fingerprinting, etc, should define their own
canonicalization specification and implement the serializer using reflection
APIs rather than relying on this API.
Added OneofOptions. You can now define custom options for oneof
groups.
import google/protobuf/descriptor.proto;
extend google.protobuf.OneofOptions {
optional int32 my_oneof_extension = 12345;
}
message Foo {
oneof oneof_group {
(my_oneof_extension) = 54321;
...
}
}
C++ (beta)
Introduced a deterministic serialization API in
CodedOutputStream::SetSerializationDeterministic(bool). See the notes about
deterministic serialization in the General section.
Added google::protobuf::Map::swap() to swap two map fields.
Fixed a memory leak when calling Reflection::ReleaseMessage() on a
message
allocated on arena.
Improved error reporting when parsing text format protos.
JSON
Added a new parser option to ignore unknown fields when parsing
JSON.
Added convenient methods for message to/from JSON conversion.
Various performance optimizations.
Java (beta)
File option java_generate_equals_and_hash is now deprecated.
equals() and
hashCode() methods are generated by default.
Added a new JSON printer option
omittingInsignificantWhitespace to produce
a more compact JSON output. The printer will pretty-print by default.
Updated Java runtime to be compatible with 2.5.0/2.6.1 generated
protos.
Python (beta)
... (truncated)
Commits
See full diff in https://github.com/protocolbuffers/protobuf/commits;>compare view
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that
have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI
passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and
block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it.
You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen the PR
or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen the PR
or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the PR or
upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default
for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the
default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the
default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the
default for future PRs for this repo and language
You can disable automated security fix PRs for this repo from the [Security
[GitHub] [spark] HyukjinKwon closed pull request #37710: [SPARK-40448][CONNECT] Spark Connect build as Driver Plugin with Shaded Dependencies
HyukjinKwon closed pull request #37710: [SPARK-40448][CONNECT] Spark Connect build as Driver Plugin with Shaded Dependencies URL: https://github.com/apache/spark/pull/37710 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37710: [SPARK-40448][CONNECT] Spark Connect build as Driver Plugin with Shaded Dependencies
HyukjinKwon commented on PR #37710: URL: https://github.com/apache/spark/pull/37710#issuecomment-1257127356 Merged to master. I will follow up and actively work on cleaning up and followup tasks from tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] amaliujia commented on pull request #37982: [SPARK-38717][SQL][3.3] Handle Hive's bucket spec case preserving behaviour
amaliujia commented on PR #37982: URL: https://github.com/apache/spark/pull/37982#issuecomment-1257114799 > > Is the HIVE metastore case-sensitivity documented somewhere or we have to run some code or play with hive directly to confirm the behavior? > > @amaliujia, it just came up with a query similar to the one in the PR description. This PR is the 3.3 backport version of the original: #36027 where you can find answer to your `Object` type question. Oh I didn't notice this is a backport PR. Thanks! Will check the original PR for more context. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #37986: [SPARK-40357][SQL] Migrate window type check failures onto error classes
AmplabJenkins commented on PR #37986: URL: https://github.com/apache/spark/pull/37986#issuecomment-1257107320 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #37979: [SPARK-40545][SQL][TESTS] Clean up `metastorePath` after `SparkSQLEnvSuite` execution
wangyum commented on PR #37979: URL: https://github.com/apache/spark/pull/37979#issuecomment-1257095684 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum closed pull request #37979: [SPARK-40545][SQL][TESTS] Clean up `metastorePath` after `SparkSQLEnvSuite` execution
wangyum closed pull request #37979: [SPARK-40545][SQL][TESTS] Clean up `metastorePath` after `SparkSQLEnvSuite` execution URL: https://github.com/apache/spark/pull/37979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #35990: [SPARK-38639][SQL] Support ignoreCorruptRecord flag to ensure querying broken sequence file table smoothly
github-actions[bot] commented on PR #35990: URL: https://github.com/apache/spark/pull/35990#issuecomment-1257089554 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #35927: [WIP] Simplify the rule of auto-generated alias name
github-actions[bot] commented on PR #35927: URL: https://github.com/apache/spark/pull/35927#issuecomment-1257089562 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #36046: [SPARK-38771][SQL] Adaptive Bloom filter Join
github-actions[bot] commented on PR #36046: URL: https://github.com/apache/spark/pull/36046#issuecomment-1257089538 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #36052: [SPARK-38777][YARN] Add `bin/spark-submit --kill / --status` support for yarn
github-actions[bot] commented on PR #36052: URL: https://github.com/apache/spark/pull/36052#issuecomment-1257089535 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #36789: [SPARK-39403] Add SPARK_SUBMIT_OPTS in spark-env.sh.template
github-actions[bot] closed pull request #36789: [SPARK-39403] Add SPARK_SUBMIT_OPTS in spark-env.sh.template URL: https://github.com/apache/spark/pull/36789 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #35969: [SPARK-38651][SQL] Add configuration to support writing out empty schemas in supported filebased datasources
github-actions[bot] commented on PR #35969: URL: https://github.com/apache/spark/pull/35969#issuecomment-1257089558 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #36005: [SPARK-38506][SQL] Push partial aggregation through join
github-actions[bot] commented on PR #36005: URL: https://github.com/apache/spark/pull/36005#issuecomment-1257089547 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #36030: Draft: [SPARK-38715] Configurable client ID for Kafka Spark SQL producer
github-actions[bot] commented on PR #36030: URL: https://github.com/apache/spark/pull/36030#issuecomment-1257089545 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #36378: [SPARK-39022][SQL] Fix combination of HAVING and SORT not being resolved correctly
github-actions[bot] closed pull request #36378: [SPARK-39022][SQL] Fix combination of HAVING and SORT not being resolved correctly URL: https://github.com/apache/spark/pull/36378 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #36304: [SPARK-38959][SQL] DS V2: Support runtime group filtering in row-level commands
github-actions[bot] closed pull request #36304: [SPARK-38959][SQL] DS V2: Support runtime group filtering in row-level commands URL: https://github.com/apache/spark/pull/36304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #36088: [SPARK-38805][SHUFFLE] Automatically remove an expired indexFilePath from the ESS shuffleIndexCache or the PBS indexCache to save
github-actions[bot] commented on PR #36088: URL: https://github.com/apache/spark/pull/36088#issuecomment-1257089532 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #36126: [SPARK-38843][SQL] Fix translate metadata col filters
github-actions[bot] closed pull request #36126: [SPARK-38843][SQL] Fix translate metadata col filters URL: https://github.com/apache/spark/pull/36126 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #36658: [SPARK-39278][CORE] Fix backward compatibility of alternative configs of Hadoop Filesystems to access
github-actions[bot] closed pull request #36658: [SPARK-39278][CORE] Fix backward compatibility of alternative configs of Hadoop Filesystems to access URL: https://github.com/apache/spark/pull/36658 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #36305: [SPARK-38987][shuffle] Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true
github-actions[bot] closed pull request #36305: [SPARK-38987][shuffle] Handle fallback when merged shuffle blocks are corrupted and spark.shuffle.detectCorrupt is set to true URL: https://github.com/apache/spark/pull/36305 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #36770: [SPARK-39382][WEBUI] UI show the duration of the failed task when the executor lost
github-actions[bot] closed pull request #36770: [SPARK-39382][WEBUI] UI show the duration of the failed task when the executor lost URL: https://github.com/apache/spark/pull/36770 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #36700: [SPARK-39318][SQL] Remove tpch-plan-stability WithStats golden files
github-actions[bot] closed pull request #36700: [SPARK-39318][SQL] Remove tpch-plan-stability WithStats golden files URL: https://github.com/apache/spark/pull/36700 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #36859: DTW: new distance measure for clustering
github-actions[bot] closed pull request #36859: DTW: new distance measure for clustering URL: https://github.com/apache/spark/pull/36859 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #36874: [SPARK-39475][SQL] Pull out complex join keys for shuffled join
github-actions[bot] commented on PR #36874: URL: https://github.com/apache/spark/pull/36874#issuecomment-1257089507 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a diff in pull request #37986: [SPARK-40357][SQL] Migrate window type check failures onto error classes
MaxGekk commented on code in PR #37986:
URL: https://github.com/apache/spark/pull/37986#discussion_r979293197
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##
@@ -421,7 +454,12 @@ sealed abstract class FrameLessOffsetWindowFunction
if (check.isFailure) {
check
} else if (!offset.foldable) {
- TypeCheckFailure(s"Offset expression '$offset' must be a literal.")
+ DataTypeMismatch(
+errorSubClass = "FRAME_LESS_OFFSET_WITHOUT_FOLDABLE",
+messageParameters = Map(
+ "offset" -> s"$offset"
Review Comment:
Since offset is an expression, please, wrap it by toSQLExpr()
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##
@@ -65,24 +65,31 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
-TypeCheckFailure(
- "Cannot use an UnspecifiedFrame. This should have been converted
during analysis. " +
-"Please file a bug report.")
+DataTypeMismatch(
+ errorSubClass = "UNSPECIFIED_FRAME"
+)
case f: SpecifiedWindowFrame if f.frameType == RangeFrame &&
!f.isUnbounded &&
orderSpec.isEmpty =>
-TypeCheckFailure(
- "A range window frame cannot be used in an unordered window
specification.")
+DataTypeMismatch(
+ errorSubClass = "RANGE_FRAME_WITHOUT_ORDER"
+)
case f: SpecifiedWindowFrame if f.frameType == RangeFrame &&
f.isValueBound &&
orderSpec.size > 1 =>
-TypeCheckFailure(
- s"A range window frame with value boundaries cannot be used in a
window specification " +
-s"with multiple order by expressions: ${orderSpec.mkString(",")}")
+DataTypeMismatch(
+ errorSubClass = "RANGE_FRAME_MULTI_ORDER",
+ messageParameters = Map(
+"orderSpec" -> s"${orderSpec.mkString(",")}"
+ )
+)
case f: SpecifiedWindowFrame if f.frameType == RangeFrame &&
f.isValueBound &&
!isValidFrameType(f.valueBoundary.head.dataType) =>
-TypeCheckFailure(
- s"The data type '${orderSpec.head.dataType.catalogString}' used in
the order " +
-"specification does not match the data type " +
-s"'${f.valueBoundary.head.dataType.catalogString}' which is used
in the range frame.")
+DataTypeMismatch(
+ errorSubClass = "RANGE_FRAME_INVALID_TYPE",
+ messageParameters = Map(
+"orderSpecType" -> s"${orderSpec.head.dataType.catalogString}",
+"valueBoundaryType" ->
s"${f.valueBoundary.head.dataType.catalogString}"
Review Comment:
The same, toSQLType
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##
@@ -65,24 +65,31 @@ case class WindowSpecDefinition(
override def checkInputDataTypes(): TypeCheckResult = {
frameSpecification match {
case UnspecifiedFrame =>
-TypeCheckFailure(
- "Cannot use an UnspecifiedFrame. This should have been converted
during analysis. " +
-"Please file a bug report.")
+DataTypeMismatch(
+ errorSubClass = "UNSPECIFIED_FRAME"
+)
Review Comment:
```suggestion
DataTypeMismatch(errorSubClass = "UNSPECIFIED_FRAME")
```
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##
@@ -262,11 +284,22 @@ case class SpecifiedWindowFrame(
private def checkBoundary(b: Expression, location: String): TypeCheckResult
= b match {
case _: SpecialFrameBoundary => TypeCheckSuccess
case e: Expression if !e.foldable =>
- TypeCheckFailure(s"Window frame $location bound '$e' is not a literal.")
+ DataTypeMismatch(
+errorSubClass = "SPECIFIED_WINDOW_FRAME_WITHOUT_FOLDABLE",
+messageParameters = Map(
+ "location" -> s"$location",
+ "expression" -> s"$e"
Review Comment:
Use toSQLExpr()
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:
##
@@ -262,11 +284,22 @@ case class SpecifiedWindowFrame(
private def checkBoundary(b: Expression, location: String): TypeCheckResult
= b match {
case _: SpecialFrameBoundary => TypeCheckSuccess
case e: Expression if !e.foldable =>
- TypeCheckFailure(s"Window frame $location bound '$e' is not a literal.")
+ DataTypeMismatch(
+errorSubClass = "SPECIFIED_WINDOW_FRAME_WITHOUT_FOLDABLE",
+messageParameters = Map(
+ "location" -> s"$location",
+ "expression" -> s"$e"
+)
+ )
case e: Expression if !frameType.inputType.acceptsType(e.dataType) =>
- TypeCheckFailure(
-s"The data type of the $location bound
[GitHub] [spark] lvshaokang opened a new pull request, #37986: [SPARK-40357][SQL] Migrate window type check failures onto error classes
lvshaokang opened a new pull request, #37986: URL: https://github.com/apache/spark/pull/37986 ### What changes were proposed in this pull request? In the PR, I propose to use error classes in the case of type check failure in window expressions. ### Why are the changes needed? Migration onto error classes unifies Spark SQL error messages. ### Does this PR introduce _any_ user-facing change? Yes. The PR changes user-facing error messages. ### How was this patch tested? ``` build/sbt "sql/testOnly *SQLQueryTestSuite" build/sbt "test:testOnly *Window*Suite" ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bersprockets commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979267219
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##
@@ -213,7 +213,16 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan]
{
case a: Aggregate if mayNeedtoRewrite(a) => rewrite(a)
}
- def rewrite(a: Aggregate): Aggregate = {
+ def rewrite(aOrig: Aggregate): Aggregate = {
+// Make children of distinct aggregations the same if they are only
+// different due to superficial reasons, e.g.:
+// "1 + col1" vs "col1 + 1", both should become "1 + col1"
+// or
+// "col1" vs "Col1", both should become "col1"
+// This could potentially reduce the number of distinct
+// aggregate groups, and therefore reduce the number of
+// projections in Expand (or eliminate the need for Expand)
+val a = reduceDistinctAggregateGroups(aOrig)
Review Comment:
Thanks! I am working on it, just working through some small complications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #37985: [SPARK-40548][BUILD] Upgrade rocksdbjni from 7.5.3 to 7.6.0
AmplabJenkins commented on PR #37985: URL: https://github.com/apache/spark/pull/37985#issuecomment-1256998639 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #37982: [SPARK-38717][SQL][3.3] Handle Hive's bucket spec case preserving behaviour
peter-toth commented on PR #37982: URL: https://github.com/apache/spark/pull/37982#issuecomment-1256996469 > Is the HIVE metastore case-sensitivity documented somewhere or we have to run some code or play with hive directly to confirm the behavior? It just came up with a query similar to the one in the PR description. This PR is the 3.3 backport version of the original: https://github.com/apache/spark/pull/36027 where you can find answer to your `Object` type question. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #37976: [SPARK-40544][SQL][TESTS] Restore the file appender log level threshold of the hive UTs to info
wangyum commented on PR #37976: URL: https://github.com/apache/spark/pull/37976#issuecomment-1256994302 Merged to master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum closed pull request #37976: [SPARK-40544][SQL][TESTS] Restore the file appender log level threshold of the hive UTs to info
wangyum closed pull request #37976: [SPARK-40544][SQL][TESTS] Restore the file appender log level threshold of the hive UTs to info URL: https://github.com/apache/spark/pull/37976 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] panbingkun opened a new pull request, #37985: [BUILD] Upgrade rocksdbjni from 7.5.3 to 7.6.0
panbingkun opened a new pull request, #37985: URL: https://github.com/apache/spark/pull/37985 ### What changes were proposed in this pull request? This PR aims to upgrade RocksDB JNI library from 7.5.3 to 7.6.0. ### Why are the changes needed? This version bring performance improvements(related to read) and some bug fix， [The Release Note](https://github.com/facebook/rocksdb/releases/tag/v7.6.0) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] EvgenyZamyatin commented on pull request #37967: [WIP] Scalable SkipGram-Word2Vec implementation
EvgenyZamyatin commented on PR #37967: URL: https://github.com/apache/spark/pull/37967#issuecomment-1256936762 How could I fix this error? The problem is due to the change of parallel collections in scala 2.13 I can fix it for scala 2.13, but how I could fix it for cross building? ``` [error] /home/runner/work/spark/spark/mllib/src/main/scala/org/apache/spark/mllib/feature/SkipGram.scala:47:64: object forkjoin is not a member of package concurrent [error] val support = new ForkJoinTaskSupport(new scala.concurrent.forkjoin.ForkJoinPool(cpus)) [error]^ [error] /home/runner/work/spark/spark/mllib/src/main/scala/org/apache/spark/mllib/feature/SkipGram.scala:49:22: value par is not a member of Seq[A] [error] did you mean map, max, or span? [error] val parr = arr.par [error] ^ [error] /home/runner/work/spark/spark/mllib/src/main/scala/org/apache/spark/mllib/feature/SkipGram.scala:56:64: object forkjoin is not a member of package concurrent [error] val support = new ForkJoinTaskSupport(new scala.concurrent.forkjoin.ForkJoinPool(cpus)) [error]^ [error] /home/runner/work/spark/spark/mllib/src/main/scala/org/apache/spark/mllib/feature/SkipGram.scala:58:22: value par is not a member of Seq[A] [error] did you mean map, max, or span? [error] val parr = arr.par [error] ^ [error] four errors found [error] (mllib / Compile / compileIncremental) Compilation failed [error] Total time: 429 s (07:09), completed Sep 22, 2022 5:09:20 PM ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] roczei commented on a diff in pull request #37679: [SPARK-35242][SQL] Support changing session catalog's default database
roczei commented on code in PR #37679:
URL: https://github.com/apache/spark/pull/37679#discussion_r979213153
##
core/src/main/resources/error/error-classes.json:
##
@@ -70,6 +70,11 @@
],
"sqlState" : "22008"
},
+ "DEFAULT_CATALOG_DATABASE_NOT_EXISTS" : {
Review Comment:
Ok, I have renamed it.
##
core/src/main/resources/error/error-classes.json:
##
@@ -70,6 +70,11 @@
],
"sqlState" : "22008"
},
+ "DEFAULT_CATALOG_DATABASE_NOT_EXISTS" : {
+"message" : [
+ "Default catalog database not exist, please create it
first or change default database to 'default'. "
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #37984: [SPARK-40322][DOCS][3.3] Fix all dead links in the docs
HyukjinKwon closed pull request #37984: [SPARK-40322][DOCS][3.3] Fix all dead links in the docs URL: https://github.com/apache/spark/pull/37984 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #37984: [SPARK-40322][DOCS][3.3] Fix all dead links in the docs
HyukjinKwon commented on PR #37984: URL: https://github.com/apache/spark/pull/37984#issuecomment-1256904136 Merged to branch-3.3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] panbingkun commented on a diff in pull request #37941: [SPARK-40501][SQL] Add PushProjectionThroughLimit for Optimizer
panbingkun commented on code in PR #37941:
URL: https://github.com/apache/spark/pull/37941#discussion_r979185389
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushProjectionThroughLimit.scala:
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.plans.logical.{GlobalLimit, LocalLimit,
LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.catalyst.trees.TreePattern.{LIMIT, PROJECT}
+
+/**
+ * Pushes Project operator to Limit operator.
Review Comment:
OK!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wangyum opened a new pull request, #37984: [SPARK-40322][DOCS][3.3] Fix all dead links in the docs
wangyum opened a new pull request, #37984: URL: https://github.com/apache/spark/pull/37984 This PR backports https://github.com/apache/spark/pull/37981 to branch-3.3. The original PR description: ### What changes were proposed in this pull request? This PR fixes any dead links in the documentation. ### Why are the changes needed? Correct the document. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? manual test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
