[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1277873304 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,10 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1277815958 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,10 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1277806568 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,10 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1277762673 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,10 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1277762673 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,10 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1277762673 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,10 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1213739673 ## core/src/main/resources/error/error-classes.json: ## @@ -64,6 +64,16 @@ } } }, + "AVRO_INCORRECT_TYPE" : { +"message" : [ + "Cannot

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1213739528 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -4209,6 +4209,18 @@ object SQLConf { .booleanConf

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1212410836 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,10 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1212410513 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -3510,4 +3510,34 @@ private[sql] object QueryCompilationErrors extends

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1210709157 ## sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala: ## @@ -4199,6 +4199,18 @@ object SQLConf { .booleanConf

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1210708290 ## docs/sql-migration-guide.md: ## @@ -26,6 +26,7 @@ license: | - Since Spark 3.5, the JDBC options related to DS V2 pushdown are `true` by default. These options

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1210655713 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -3510,4 +3510,34 @@ private[sql] object QueryCompilationErrors extends

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1210539286 ## sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala: ## @@ -3510,4 +3510,34 @@ private[sql] object QueryCompilationErrors extends

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1207262937 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +118,24 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1207262643 ## connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala: ## @@ -639,6 +640,110 @@ abstract class AvroSuite } } + test("SPARK-43380: Fix

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1205900695 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -128,6 +151,26 @@ private[sql] class AvroDeserializer( case (INT,

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1205881016 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -128,6 +143,26 @@ private[sql] class AvroDeserializer( case (INT,

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1204863903 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -128,6 +143,26 @@ private[sql] class AvroDeserializer( case (INT,

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1204863903 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -128,6 +143,26 @@ private[sql] class AvroDeserializer( case (INT,

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1204863393 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -128,6 +143,26 @@ private[sql] class AvroDeserializer( case (INT,

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1202636697 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,19 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1201281025 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,19 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1201271382 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,19 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1189119562 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +119,19 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1189112485 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -204,17 +239,28 @@ private[sql] class AvroDeserializer( }

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1189110112 ## connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala: ## @@ -639,6 +640,39 @@ abstract class AvroSuite } } + test("SPARK-43380: Fix

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1187737160 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -128,6 +142,26 @@ private[sql] class AvroDeserializer( case (INT,

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1186389980 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -128,6 +142,26 @@ private[sql] class AvroDeserializer( case (INT,

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1186384497 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +118,19 @@ private[sql] class AvroDeserializer( val

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1187677039 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -128,6 +143,26 @@ private[sql] class AvroDeserializer( case (INT,

[GitHub] [spark] zeruibao commented on a diff in pull request #41052: [SPARK-43380][SQL] Fix Avro data type conversion issues to avoid producing incorrect results

zeruibao commented on code in PR #41052: URL: https://github.com/apache/spark/pull/41052#discussion_r1185721125 ## connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala: ## @@ -117,6 +118,20 @@ private[sql] class AvroDeserializer( val