This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new c730365827dc [SPARK-45588][SPARK-45640][SQL][TESTS][3.4] Fix flaky
ProtobufCatalystDataConversionSuite
c730365827dc is described below
commit c730365827dc9c73c485d35d4cd21cb25cb8c795
Author: panbingkun <[email protected]>
AuthorDate: Wed Oct 25 13:56:51 2023 +0900
[SPARK-45588][SPARK-45640][SQL][TESTS][3.4] Fix flaky
ProtobufCatalystDataConversionSuite
### What changes were proposed in this pull request?
The pr aims to fix flaky ProtobufCatalystDataConversionSuite, include:
- Fix the type check (when the random value was empty array, we didn't skip
it. Original intention is to skip default values for types.) [SPARK-45588]
- When data.get(0) is null, data.get(0).asInstanceOf[Array[Byte]].isEmpty
will be thrown java.lang.NullPointerException. [SPARK-45640]
Backport above to branch 3.4.
Master branch pr: https://github.com/apache/spark/pull/43424 &
https://github.com/apache/spark/pull/43493
### Why are the changes needed?
Fix flaky ProtobufCatalystDataConversionSuite.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass GA
- Manually test
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43520 from panbingkun/branch-3.4_SPARK-45640.
Authored-by: panbingkun <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala
b/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala
index 3e9273835e3d..904827174e56 100644
---
a/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala
+++
b/connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufCatalystDataConversionSuite.scala
@@ -136,7 +136,8 @@ class ProtobufCatalystDataConversionSuite
while (
data != null &&
(data.get(0) == defaultValue ||
- (dt == BinaryType &&
+ (dt.fields(0).dataType == BinaryType &&
+ data.get(0) != null &&
data.get(0).asInstanceOf[Array[Byte]].isEmpty)))
data = generator().asInstanceOf[Row]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
