[spark] branch master updated (f47e0f8 -> 323a6e8)
This is an automated email from the ASF dual-hosted git repository. viirya pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f47e0f8 [SPARK-35261][SQL] Support static magic method for stateless Java ScalarFunction add 323a6e8 [SPARK-35232][SQL] Nested column pruning should retain column metadata No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/SchemaPruning.scala | 8 ++- .../catalyst/expressions/SchemaPruningSuite.scala | 14 + .../sql/connector/catalog/InMemoryTable.scala | 68 -- 3 files changed, 70 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b4ec9e2 -> f47e0f8)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b4ec9e2 [SPARK-35321][SQL] Don't register Hive permanent functions when creating Hive client add f47e0f8 [SPARK-35261][SQL] Support static magic method for stateless Java ScalarFunction No new revisions were added by this update. Summary of changes: .../catalog/functions/ScalarFunction.java | 60 ++--- .../spark/sql/catalyst/analysis/Analyzer.scala | 5 +- .../V2FunctionBenchmark-jdk11-results.txt | 44 ++ .../benchmarks/V2FunctionBenchmark-results.txt | 44 ++ .../connector/catalog/functions/JavaLongAdd.java | 130 ++ .../connector/catalog/functions/JavaStrLen.java| 48 +++ .../sql/connector/DataSourceV2FunctionSuite.scala | 22 +++ .../connector/functions/V2FunctionBenchmark.scala | 147 + 8 files changed, 461 insertions(+), 39 deletions(-) create mode 100644 sql/core/benchmarks/V2FunctionBenchmark-jdk11-results.txt create mode 100644 sql/core/benchmarks/V2FunctionBenchmark-results.txt create mode 100644 sql/core/src/test/java/test/org/apache/spark/sql/connector/catalog/functions/JavaLongAdd.java create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/connector/functions/V2FunctionBenchmark.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (b190e2b -> 5268a38)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from b190e2b [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match add 5268a38 [SPARK-35321][SQL][3.0] Don't register Hive permanent functions when creating Hive client No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hive/client/HiveClientImpl.scala | 4 ++-- .../scala/org/apache/spark/sql/hive/client/HiveShim.scala | 11 +++ 2 files changed, 13 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated (a4c34ba -> 6fbea6a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/spark.git. from a4c34ba [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match add 6fbea6a [SPARK-35321][SQL][3.1] Don't register Hive permanent functions when creating Hive client No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hive/client/HiveClientImpl.scala | 4 ++-- .../scala/org/apache/spark/sql/hive/client/HiveShim.scala | 11 +++ 2 files changed, 13 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (33fbf56 -> b4ec9e2)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 33fbf56 [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match add b4ec9e2 [SPARK-35321][SQL] Don't register Hive permanent functions when creating Hive client No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hive/client/HiveClientImpl.scala | 4 ++-- .../scala/org/apache/spark/sql/hive/client/HiveShim.scala | 11 +++ 2 files changed, 13 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match
This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 7733510 [SPARK-35288][SQL] StaticInvoke should find the method
without exact argument classes match
7733510 is described below
commit 7733510d0403625c41710d7e79f810117aac2ced
Author: Liang-Chi Hsieh
AuthorDate: Fri May 7 09:07:57 2021 -0700
[SPARK-35288][SQL] StaticInvoke should find the method without exact
argument classes match
### What changes were proposed in this pull request?
This patch proposes to make StaticInvoke able to find method with given
method name even the parameter types do not exactly match to argument classes.
### Why are the changes needed?
Unlike `Invoke`, `StaticInvoke` only tries to get the method with exact
argument classes. If the calling method's parameter types are not exactly
matched with the argument classes, `StaticInvoke` cannot find the method.
`StaticInvoke` should be able to find the method under the cases too.
### Does this PR introduce _any_ user-facing change?
Yes. `StaticInvoke` can find a method even the argument classes are not
exactly matched.
### How was this patch tested?
Unit test.
Closes #32413 from viirya/static-invoke.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Liang-Chi Hsieh
(cherry picked from commit 33fbf5647b4a5587c78ac51339c0cbc9d70547a4)
Signed-off-by: Liang-Chi Hsieh
---
.../sql/catalyst/expressions/objects/objects.scala | 56 --
.../expressions/ObjectExpressionsSuite.scala | 34 +++--
2 files changed, 60 insertions(+), 30 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
index 43e8105..fb4132a 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
@@ -139,6 +139,34 @@ trait InvokeLike extends Expression with NonSQLExpression {
}
}
}
+
+ final def findMethod(cls: Class[_], functionName: String, argClasses:
Seq[Class[_]]): Method = {
+// Looking with function name + argument classes first.
+try {
+ cls.getMethod(functionName, argClasses: _*)
+} catch {
+ case _: NoSuchMethodException =>
+// For some cases, e.g. arg class is Object, `getMethod` cannot find
the method.
+// We look at function name + argument length
+val m = cls.getMethods.filter { m =>
+ m.getName == functionName && m.getParameterCount == arguments.length
+}
+if (m.isEmpty) {
+ sys.error(s"Couldn't find $functionName on $cls")
+} else if (m.length > 1) {
+ // More than one matched method signature. Exclude synthetic one,
e.g. generic one.
+ val realMethods = m.filter(!_.isSynthetic)
+ if (realMethods.length > 1) {
+// Ambiguous case, we don't know which method to choose, just fail
it.
+sys.error(s"Found ${realMethods.length} $functionName on $cls")
+ } else {
+realMethods.head
+ }
+} else {
+ m.head
+}
+}
+ }
}
/**
@@ -230,7 +258,7 @@ case class StaticInvoke(
override def children: Seq[Expression] = arguments
lazy val argClasses = ScalaReflection.expressionJavaClasses(arguments)
- @transient lazy val method = cls.getDeclaredMethod(functionName, argClasses
: _*)
+ @transient lazy val method = findMethod(cls, functionName, argClasses)
override def eval(input: InternalRow): Any = {
invoke(null, method, arguments, input, dataType)
@@ -317,31 +345,7 @@ case class Invoke(
@transient lazy val method = targetObject.dataType match {
case ObjectType(cls) =>
- // Looking with function name + argument classes first.
- try {
-Some(cls.getMethod(encodedFunctionName, argClasses: _*))
- } catch {
-case _: NoSuchMethodException =>
- // For some cases, e.g. arg class is Object, `getMethod` cannot find
the method.
- // We look at function name + argument length
- val m = cls.getMethods.filter { m =>
-m.getName == encodedFunctionName && m.getParameterCount ==
arguments.length
- }
- if (m.isEmpty) {
-sys.error(s"Couldn't find $encodedFunctionName on $cls")
- } else if (m.length > 1) {
-// More than one matched method signature. Exclude synthetic one,
e.g. generic one.
-val realMethods = m.filter(!_.isSynthetic)
-if (realMethods.length > 1) {
- // Ambiguous case, we don't know
[spark] branch branch-3.0 updated: [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match
This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new b190e2b [SPARK-35288][SQL] StaticInvoke should find the method
without exact argument classes match
b190e2b is described below
commit b190e2b36dba9ad85c2b30a2b693aa9defd6de02
Author: Liang-Chi Hsieh
AuthorDate: Fri May 7 09:07:57 2021 -0700
[SPARK-35288][SQL] StaticInvoke should find the method without exact
argument classes match
### What changes were proposed in this pull request?
This patch proposes to make StaticInvoke able to find method with given
method name even the parameter types do not exactly match to argument classes.
### Why are the changes needed?
Unlike `Invoke`, `StaticInvoke` only tries to get the method with exact
argument classes. If the calling method's parameter types are not exactly
matched with the argument classes, `StaticInvoke` cannot find the method.
`StaticInvoke` should be able to find the method under the cases too.
### Does this PR introduce _any_ user-facing change?
Yes. `StaticInvoke` can find a method even the argument classes are not
exactly matched.
### How was this patch tested?
Unit test.
Closes #32413 from viirya/static-invoke.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Liang-Chi Hsieh
(cherry picked from commit 33fbf5647b4a5587c78ac51339c0cbc9d70547a4)
Signed-off-by: Liang-Chi Hsieh
---
.../sql/catalyst/expressions/objects/objects.scala | 56 --
.../expressions/ObjectExpressionsSuite.scala | 34 +++--
2 files changed, 60 insertions(+), 30 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
index 066188a..e5e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
@@ -141,6 +141,34 @@ trait InvokeLike extends Expression with NonSQLExpression {
}
}
}
+
+ final def findMethod(cls: Class[_], functionName: String, argClasses:
Seq[Class[_]]): Method = {
+// Looking with function name + argument classes first.
+try {
+ cls.getMethod(functionName, argClasses: _*)
+} catch {
+ case _: NoSuchMethodException =>
+// For some cases, e.g. arg class is Object, `getMethod` cannot find
the method.
+// We look at function name + argument length
+val m = cls.getMethods.filter { m =>
+ m.getName == functionName && m.getParameterCount == arguments.length
+}
+if (m.isEmpty) {
+ sys.error(s"Couldn't find $functionName on $cls")
+} else if (m.length > 1) {
+ // More than one matched method signature. Exclude synthetic one,
e.g. generic one.
+ val realMethods = m.filter(!_.isSynthetic)
+ if (realMethods.length > 1) {
+// Ambiguous case, we don't know which method to choose, just fail
it.
+sys.error(s"Found ${realMethods.length} $functionName on $cls")
+ } else {
+realMethods.head
+ }
+} else {
+ m.head
+}
+}
+ }
}
/**
@@ -232,7 +260,7 @@ case class StaticInvoke(
override def children: Seq[Expression] = arguments
lazy val argClasses = ScalaReflection.expressionJavaClasses(arguments)
- @transient lazy val method = cls.getDeclaredMethod(functionName, argClasses
: _*)
+ @transient lazy val method = findMethod(cls, functionName, argClasses)
override def eval(input: InternalRow): Any = {
invoke(null, method, arguments, input, dataType)
@@ -319,31 +347,7 @@ case class Invoke(
@transient lazy val method = targetObject.dataType match {
case ObjectType(cls) =>
- // Looking with function name + argument classes first.
- try {
-Some(cls.getMethod(encodedFunctionName, argClasses: _*))
- } catch {
-case _: NoSuchMethodException =>
- // For some cases, e.g. arg class is Object, `getMethod` cannot find
the method.
- // We look at function name + argument length
- val m = cls.getMethods.filter { m =>
-m.getName == encodedFunctionName && m.getParameterCount ==
arguments.length
- }
- if (m.isEmpty) {
-sys.error(s"Couldn't find $encodedFunctionName on $cls")
- } else if (m.length > 1) {
-// More than one matched method signature. Exclude synthetic one,
e.g. generic one.
-val realMethods = m.filter(!_.isSynthetic)
-if (realMethods.length > 1) {
- // Ambiguous case, we don't know
[spark] branch branch-3.1 updated: [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match
This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new a4c34ba [SPARK-35288][SQL] StaticInvoke should find the method
without exact argument classes match
a4c34ba is described below
commit a4c34ba161557c8857a1b639441fe79912021b23
Author: Liang-Chi Hsieh
AuthorDate: Fri May 7 09:07:57 2021 -0700
[SPARK-35288][SQL] StaticInvoke should find the method without exact
argument classes match
### What changes were proposed in this pull request?
This patch proposes to make StaticInvoke able to find method with given
method name even the parameter types do not exactly match to argument classes.
### Why are the changes needed?
Unlike `Invoke`, `StaticInvoke` only tries to get the method with exact
argument classes. If the calling method's parameter types are not exactly
matched with the argument classes, `StaticInvoke` cannot find the method.
`StaticInvoke` should be able to find the method under the cases too.
### Does this PR introduce _any_ user-facing change?
Yes. `StaticInvoke` can find a method even the argument classes are not
exactly matched.
### How was this patch tested?
Unit test.
Closes #32413 from viirya/static-invoke.
Authored-by: Liang-Chi Hsieh
Signed-off-by: Liang-Chi Hsieh
(cherry picked from commit 33fbf5647b4a5587c78ac51339c0cbc9d70547a4)
Signed-off-by: Liang-Chi Hsieh
---
.../sql/catalyst/expressions/objects/objects.scala | 56 --
.../expressions/ObjectExpressionsSuite.scala | 34 +++--
2 files changed, 60 insertions(+), 30 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
index 53a2bfd..f78e3f5 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
@@ -141,6 +141,34 @@ trait InvokeLike extends Expression with NonSQLExpression {
}
}
}
+
+ final def findMethod(cls: Class[_], functionName: String, argClasses:
Seq[Class[_]]): Method = {
+// Looking with function name + argument classes first.
+try {
+ cls.getMethod(functionName, argClasses: _*)
+} catch {
+ case _: NoSuchMethodException =>
+// For some cases, e.g. arg class is Object, `getMethod` cannot find
the method.
+// We look at function name + argument length
+val m = cls.getMethods.filter { m =>
+ m.getName == functionName && m.getParameterCount == arguments.length
+}
+if (m.isEmpty) {
+ sys.error(s"Couldn't find $functionName on $cls")
+} else if (m.length > 1) {
+ // More than one matched method signature. Exclude synthetic one,
e.g. generic one.
+ val realMethods = m.filter(!_.isSynthetic)
+ if (realMethods.length > 1) {
+// Ambiguous case, we don't know which method to choose, just fail
it.
+sys.error(s"Found ${realMethods.length} $functionName on $cls")
+ } else {
+realMethods.head
+ }
+} else {
+ m.head
+}
+}
+ }
}
/**
@@ -232,7 +260,7 @@ case class StaticInvoke(
override def children: Seq[Expression] = arguments
lazy val argClasses = ScalaReflection.expressionJavaClasses(arguments)
- @transient lazy val method = cls.getDeclaredMethod(functionName, argClasses
: _*)
+ @transient lazy val method = findMethod(cls, functionName, argClasses)
override def eval(input: InternalRow): Any = {
invoke(null, method, arguments, input, dataType)
@@ -319,31 +347,7 @@ case class Invoke(
@transient lazy val method = targetObject.dataType match {
case ObjectType(cls) =>
- // Looking with function name + argument classes first.
- try {
-Some(cls.getMethod(encodedFunctionName, argClasses: _*))
- } catch {
-case _: NoSuchMethodException =>
- // For some cases, e.g. arg class is Object, `getMethod` cannot find
the method.
- // We look at function name + argument length
- val m = cls.getMethods.filter { m =>
-m.getName == encodedFunctionName && m.getParameterCount ==
arguments.length
- }
- if (m.isEmpty) {
-sys.error(s"Couldn't find $encodedFunctionName on $cls")
- } else if (m.length > 1) {
-// More than one matched method signature. Exclude synthetic one,
e.g. generic one.
-val realMethods = m.filter(!_.isSynthetic)
-if (realMethods.length > 1) {
- // Ambiguous case, we don't know
[spark] branch master updated (6f0ef93 -> 33fbf56)
This is an automated email from the ASF dual-hosted git repository. viirya pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6f0ef93 [SPARK-35297][CORE][DOC][MINOR] Modify the comment about the executor add 33fbf56 [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/objects/objects.scala | 56 -- .../expressions/ObjectExpressionsSuite.scala | 34 +++-- 2 files changed, 60 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2634dba -> 6f0ef93)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2634dba [SPARK-35175][BUILD] Add linter for JavaScript source files add 6f0ef93 [SPARK-35297][CORE][DOC][MINOR] Modify the comment about the executor No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/executor/Executor.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (72d3266 -> d3b92ee)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 72d3266 [SPARK-35144][SQL] Migrate to transformWithPruning for object rules add d3b92ee [SPARK-35021][SQL] Group exception messages in connector/catalog No new revisions were added by this update. Summary of changes: .../sql/connector/catalog/CatalogV2Implicits.scala | 36 +-- .../sql/connector/catalog/CatalogV2Util.scala | 5 ++- .../sql/connector/catalog/LookupCatalog.scala | 6 ++-- .../spark/sql/errors/QueryCompilationErrors.scala | 42 -- 4 files changed, 60 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-35144][SQL] Migrate to transformWithPruning for object rules
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 72d3266 [SPARK-35144][SQL] Migrate to transformWithPruning for object
rules
72d3266 is described below
commit 72d32662d470e286a639783fed8dcf6c3948
Author: Yingyi Bu
AuthorDate: Fri May 7 18:36:28 2021 +0800
[SPARK-35144][SQL] Migrate to transformWithPruning for object rules
### What changes were proposed in this pull request?
Added the following TreePattern enums:
- APPEND_COLUMNS
- DESERIALIZE_TO_OBJECT
- LAMBDA_VARIABLE
- MAP_OBJECTS
- SERIALIZE_FROM_OBJECT
- PROJECT
- TYPED_FILTER
Added tree traversal pruning to the following rules dealing with objects:
- EliminateSerialization
- CombineTypedFilters
- EliminateMapObjects
- ObjectSerializerPruning
### Why are the changes needed?
Reduce the number of tree traversals and hence improve the query
compilation latency.
### How was this patch tested?
Existing tests.
Closes #32451 from sigmod/object.
Authored-by: Yingyi Bu
Signed-off-by: Gengliang Wang
---
.../spark/sql/catalyst/expressions/objects/objects.scala | 6 +-
.../org/apache/spark/sql/catalyst/optimizer/objects.scala | 15 ++-
.../catalyst/plans/logical/basicLogicalOperators.scala| 2 ++
.../apache/spark/sql/catalyst/plans/logical/object.scala | 8
.../spark/sql/catalyst/rules/RuleIdCollection.scala | 5 +
.../apache/spark/sql/catalyst/trees/TreePatterns.scala| 7 +++
6 files changed, 37 insertions(+), 6 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
index 469c895..40378a3 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
@@ -33,7 +33,7 @@ import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.codegen._
import org.apache.spark.sql.catalyst.expressions.codegen.Block._
import org.apache.spark.sql.catalyst.trees.TernaryLike
-import org.apache.spark.sql.catalyst.trees.TreePattern.{NULL_CHECK,
TreePattern}
+import org.apache.spark.sql.catalyst.trees.TreePattern._
import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData,
GenericArrayData, MapData}
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.types._
@@ -669,6 +669,8 @@ case class LambdaVariable(
private val accessor: (InternalRow, Int) => Any =
InternalRow.getAccessor(dataType, nullable)
+ final override val nodePatterns: Seq[TreePattern] = Seq(LAMBDA_VARIABLE)
+
// Interpreted execution of `LambdaVariable` always get the 0-index element
from input row.
override def eval(input: InternalRow): Any = {
assert(input.numFields == 1,
@@ -781,6 +783,8 @@ case class MapObjects private(
override def second: Expression = lambdaFunction
override def third: Expression = inputData
+ final override val nodePatterns: Seq[TreePattern] = Seq(MAP_OBJECTS)
+
// The data with UserDefinedType are actually stored with the data type of
its sqlType.
// When we want to apply MapObjects on it, we have to use it.
lazy private val inputDataType = inputData.dataType match {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
index 97712a0..52544ff 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
@@ -24,6 +24,7 @@ import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.objects._
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules._
+import org.apache.spark.sql.catalyst.trees.TreePattern._
import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType,
UserDefinedType}
/*
@@ -35,7 +36,8 @@ import org.apache.spark.sql.types.{ArrayType, DataType,
MapType, StructType, Use
* representation of data item. For example back to back map operations.
*/
object EliminateSerialization extends Rule[LogicalPlan] {
- def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning(
+_.containsAnyPattern(DESERIALIZE_TO_OBJECT, APPEND_COLUMNS, TYPED_FILTER),
ruleId) {
case d @ DeserializeToObject(_, _, s: SerializeFromObject)
if
[spark] branch master updated (cf2c4ba -> 9aa18df)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from cf2c4ba [SPARK-35020][SQL] Group exception messages in catalyst/util add 9aa18df [SPARK-35333][SQL] Skip object null check in Invoke if possible No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/objects/objects.scala | 27 -- 1 file changed, 20 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e83910f -> cf2c4ba)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e83910f [SPARK-26164][SQL][FOLLOWUP] WriteTaskStatsTracker should know which file the row is written to add cf2c4ba [SPARK-35020][SQL] Group exception messages in catalyst/util No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/ArrayBasedMapBuilder.scala | 16 +++ .../catalyst/util/DateTimeFormatterHelper.scala| 21 +++-- .../spark/sql/catalyst/util/DateTimeUtils.scala| 7 +-- .../spark/sql/catalyst/util/IntervalUtils.scala| 3 +- .../apache/spark/sql/catalyst/util/TypeUtils.scala | 4 +- .../spark/sql/errors/QueryCompilationErrors.scala | 4 ++ .../spark/sql/errors/QueryExecutionErrors.scala| 54 +- 7 files changed, 77 insertions(+), 32 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (33c1034 -> e83910f)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 33c1034 [SPARK-34701][SQL][FOLLOW-UP] Children/innerChildren should be mutually exclusive for AnalysisOnlyCommand add e83910f [SPARK-26164][SQL][FOLLOWUP] WriteTaskStatsTracker should know which file the row is written to No new revisions were added by this update. Summary of changes: .../datasources/BasicWriteStatsTracker.scala | 2 +- .../datasources/FileFormatDataWriter.scala | 4 +- .../execution/datasources/WriteStatsTracker.scala | 3 +- .../CustomWriteTaskStatsTrackerSuite.scala | 72 ++ 4 files changed, 77 insertions(+), 4 deletions(-) create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CustomWriteTaskStatsTrackerSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (42f59ca -> 33c1034)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 42f59ca [SPARK-35133][SQL] Explain codegen works with AQE add 33c1034 [SPARK-34701][SQL][FOLLOW-UP] Children/innerChildren should be mutually exclusive for AnalysisOnlyCommand No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/plans/logical/Command.scala | 2 ++ .../apache/spark/sql/execution/command/views.scala | 5 - .../sql/execution/command/PlanResolutionSuite.scala | 21 - 3 files changed, 22 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
