Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2046598213 Thank you, @pan3793 and all. Merged to master for Apache Spark 4.0.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun closed pull request #45838: [SPARK-47706][BUILD] Bump json4s 4.0.7 URL: https://github.com/apache/spark/pull/45838 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2046557968 @LuciferYang @dongjoon-hyun The failed test is known flaky case, I re-triggered CI, thanks for your approval and help with Mima -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2046536383 Could you re-trigger the failed test pipeline, @pan3793 ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2046420830 rebased on master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1557856537 ## sql/api/pom.xml: ## @@ -56,13 +56,6 @@ org.json4s json4s-jackson_${scala.binary.version} -3.7.0-M11 Review Comment: Thanks. I merged it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556905110 ## sql/api/pom.xml: ## @@ -56,13 +56,6 @@ org.json4s json4s-jackson_${scala.binary.version} -3.7.0-M11 Review Comment: opened https://github.com/apache/spark/pull/45943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556899643 ## sql/api/pom.xml: ## @@ -56,13 +56,6 @@ org.json4s json4s-jackson_${scala.binary.version} -3.7.0-M11 Review Comment: Could you spin off this file change as a new PR, please, @pan3793 ? We can merge that before this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556900847 ## sql/api/pom.xml: ## @@ -56,13 +56,6 @@ org.json4s json4s-jackson_${scala.binary.version} -3.7.0-M11 Review Comment: Okay -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556899643 ## sql/api/pom.xml: ## @@ -56,13 +56,6 @@ org.json4s json4s-jackson_${scala.binary.version} -3.7.0-M11 Review Comment: Could you spin off this as a new PR, please, @pan3793 ? We can merge that before this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2044149193 Thank you for rebasing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2044102847 @dongjoon-hyun @cloud-fan @gengliangwang @LuciferYang I updated MimaExcludes, now it looks good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556835313 ## project/MimaExcludes.scala: ## @@ -90,7 +90,21 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.mllib.tree.model.TreeEnsembleModel#SaveLoadV1_0.readMetadata") Review Comment: this should not be an issue ``` private[tree] object TreeEnsembleModel extends Logging { object SaveLoadV1_0 { ... } ... } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556834466 ## project/MimaExcludes.scala: ## @@ -90,7 +90,11 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), Review Comment: this should not be a real issue ``` object DataType { private object JSortedObject { ... } ... } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556833527 ## project/MimaExcludes.scala: ## @@ -90,7 +90,11 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), Review Comment: it's weird ``` trait Row { ... private[sql] def jsonValue: JValue ... } abstract class MutableAggregationBuffer extends Row { ... } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556656706 ## project/MimaExcludes.scala: ## @@ -90,7 +90,21 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply"), Review Comment: I filed SPARK-47770. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2043927068 To @pan3793 , for MIMA issue, I made the following PR to fix it. - #45938 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556539422 ## project/MimaExcludes.scala: ## @@ -90,7 +90,21 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply"), Review Comment: I took a look at that briefly and found that `dev/mima` shows error for this class already. ``` $ dev/mima | grep org.apache.spark.ml.util.DefaultParamsReader Using SPARK_LOCAL_IP=localhost Using SPARK_LOCAL_IP=localhost Error instrumenting class:org.apache.spark.ml.util.DefaultParamsReader$Metadata$ Error instrumenting class:org.apache.spark.ml.util.DefaultParamsReader$Metadata Using SPARK_LOCAL_IP=localhost spark/connect/relations.proto:26:1: warning: Import spark/connect/common.proto is unused. ... ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556539422 ## project/MimaExcludes.scala: ## @@ -90,7 +90,21 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply"), Review Comment: I took a look at that briefly and found that `dev/mima` shows error for this class already. ``` $ dev/mima | grep org.apache.spark.ml.util.DefaultParamsReader Using SPARK_LOCAL_IP=localhost Using SPARK_LOCAL_IP=localhost Error instrumenting class:org.apache.spark.ml.util.DefaultParamsReader$Metadata$ Error instrumenting class:org.apache.spark.ml.util.DefaultParamsReader$Metadata Using SPARK_LOCAL_IP=localhost spark/connect/relations.proto:26:1: warning: Import spark/connect/common.proto is unused. ... ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556151836 ## project/MimaExcludes.scala: ## @@ -90,7 +90,21 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply"), Review Comment: Could you file a JIRA issue for this, @pan3793 ? It seems that we had better fix that first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1556151836 ## project/MimaExcludes.scala: ## @@ -90,7 +90,21 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply"), Review Comment: Could you file a JIRA issue for this, @pan3793 ? We had better fix that first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1555230740 ## project/MimaExcludes.scala: ## @@ -90,7 +90,21 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply"), Review Comment: I noticed that `DefaultParamsReader`'s scope is `private[ml]`, but Mima still complains, is it expected? cc @LuciferYang ``` private[ml] class DefaultParamsReader ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2041867141 @cloud-fan seems json4s does not have release note. the API binary incompatible does not affect JSON serialize/deserialize, the case of parsing event log generated by the previous Spark version is already covered by Spark UT. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
cloud-fan commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2041836659 does json4s release notes mention anything about backward compatibility? We have persistent event logs in JSON format that may need to be consumed by new version of Spark history server. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
gengliangwang commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2041235111 Since there are many connectors/libraries that depend on Spark, I am +0 on this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1549972129 ## connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufFunctionsSuite.scala: ## @@ -22,7 +22,6 @@ import java.time.Duration import scala.jdk.CollectionConverters._ import com.google.protobuf.{Any => AnyProto, BoolValue, ByteString, BytesValue, DoubleValue, DynamicMessage, FloatValue, Int32Value, Int64Value, StringValue, UInt32Value, UInt64Value} -import org.json4s.StringInput Review Comment: No, it's there. But method `JsonMethods#parse` signature changed. As this is the test code, I chose the simplest way to change it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1549948290 ## project/MimaExcludes.scala: ## @@ -90,7 +90,21 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.mllib.tree.model.TreeEnsembleModel#SaveLoadV1_0.readMetadata") Review Comment: This is because the `JValue` package is changed, it does not mean can not read the model saved by the old Spark ``` def readMetadata(metadata: JValue): Metadata ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1549942965 ## connector/protobuf/src/test/scala/org/apache/spark/sql/protobuf/ProtobufFunctionsSuite.scala: ## @@ -22,7 +22,6 @@ import java.time.Duration import scala.jdk.CollectionConverters._ import com.google.protobuf.{Any => AnyProto, BoolValue, ByteString, BytesValue, DoubleValue, DynamicMessage, FloatValue, Int32Value, Int64Value, StringValue, UInt32Value, UInt64Value} -import org.json4s.StringInput Review Comment: Is this removed or deprecated? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
dongjoon-hyun commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1549937536 ## project/MimaExcludes.scala: ## @@ -90,7 +90,21 @@ object MimaExcludes { ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.MySQLDialect#MySQLSQLQueryBuilder.this"), ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLBuilder.this"), - ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this") + ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.sql.jdbc.OracleDialect#OracleSQLQueryBuilder.this"), +// SPARK-47706: Bump json4s from 3.7.0-M11 to 4.0.7 + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6"), + ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$7"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.this"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.apply"), + ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.mllib.tree.model.TreeEnsembleModel#SaveLoadV1_0.readMetadata") Review Comment: Wow.. Everything is inevitable? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2034852723 @LuciferYang `MimaExcludes` is updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2034104674 @LuciferYang json4s classes leaked to public(maybe? at least in the Mima audit list) API, and json4s 4.x changed those classes, this requires the Spark community to evaluate if we should do such an upgrading -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
LuciferYang commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2034076896 I get errors when I manually run the mima check. Could you please update the `MimaExcludes`? ``` [error] spark-sql-api: Failed binary compatibility check against org.apache.spark:spark-sql-api_2.13:3.5.0! Found 1 potential problems (filtered 170) [error] * method unapplySeq(org.json4s.JsonAST#JValue)scala.Option in object org.apache.spark.sql.types.DataType#JSortedObject's type is different in current version, where it is (org.json4s.JValue)scala.Option instead of (org.json4s.JsonAST#JValue)scala.Option [error]filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.sql.types.DataType#JSortedObject.unapplySeq") [error] spark-sql: Failed binary compatibility check against org.apache.spark:spark-sql_2.13:3.5.0! Found 1 potential problems (filtered 861) [error] * method jsonValue()org.json4s.JsonAST#JValue in class org.apache.spark.sql.expressions.MutableAggregationBuffer has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue [error]filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.sql.expressions.MutableAggregationBuffer.jsonValue") [error] spark-mllib: Failed binary compatibility check against org.apache.spark:spark-mllib_2.13:3.5.0! Found 11 potential problems (filtered 477) [error] * method params()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue [error]filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.params") [error] * method defaultParams()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue [error]filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.defaultParams") [error] * method metadata()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue [error]filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.metadata") [error] * method getParamValue(java.lang.String)org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue [error]filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.getParamValue") [error] * method copy(java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,java.lang.String)org.apache.spark.ml.util.DefaultParamsReader#Metadata in class org.apache.spark.ml.util.DefaultParamsReader#Metadata's type is different in current version, where it is (java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JValue,org.json4s.JValue,org.json4s.JValue,java.lang.String)org.apache.spark.ml.util.DefaultParamsReader#Metadata instead of (java.lang.String,java.lang.String,Long,java.lang.String,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,org.json4s.JsonAST#JValue,java.lang.String)org.apache.spark.ml.util.DefaultParamsReader#Metadata [error]filter with: ProblemFilters.exclude[IncompatibleMethTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy") [error] * synthetic method copy$default$5()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue [error]filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$5") [error] * synthetic method copy$default$6()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue rather than org.json4s.JsonAST#JValue [error]filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.ml.util.DefaultParamsReader#Metadata.copy$default$6") [error] * synthetic method copy$default$7()org.json4s.JsonAST#JValue in class org.apache.spark.ml.util.DefaultParamsReader#Metadata has a different result type in current version, where it is org.json4s.JValue
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
LuciferYang commented on PR #45838: URL: https://github.com/apache/spark/pull/45838#issuecomment-2034039323 Previously, there were two unsuccessful upgrade attempts: - https://github.com/apache/spark/pull/32636 - https://github.com/apache/spark/pull/33630 cc @dongjoon-hyun @sarutak @JoshRosen FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
Re: [PR] [SPARK-47706][BUILD] Bump json4s 4.0.7 [spark]
pan3793 commented on code in PR #45838: URL: https://github.com/apache/spark/pull/45838#discussion_r1549070164 ## sql/api/pom.xml: ## @@ -56,13 +56,6 @@ org.json4s json4s-jackson_${scala.binary.version} -3.7.0-M11 Review Comment: redundant, it's same as the one defined in root `pom.xml` `dependencyManagement`. ## sql/api/pom.xml: ## @@ -56,13 +56,6 @@ org.json4s json4s-jackson_${scala.binary.version} -3.7.0-M11 Review Comment: redundant, it same as the one defined in root `pom.xml` `dependencyManagement`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org