[GitHub] [spark] LuciferYang commented on a diff in pull request #38635: [SPARK-41118][SQL] `to_number`/`try_to_number` should return `null` when format is `null`
LuciferYang commented on code in PR #38635: URL: https://github.com/apache/spark/pull/38635#discussion_r1023569619 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala: ## @@ -26,6 +26,62 @@ import org.apache.spark.sql.catalyst.util.ToNumberParser import org.apache.spark.sql.types.{AbstractDataType, DataType, Decimal, DecimalType, StringType} import org.apache.spark.unsafe.types.UTF8String +abstract class ToNumberBase(left: Expression, right: Expression, errorOnFail: Boolean) + extends BinaryExpression with Serializable with ImplicitCastInputTypes with NullIntolerant { + + private lazy val numberFormatter = { +val value = right.eval() +if (value != null) { + new ToNumberParser(value.toString.toUpperCase(Locale.ROOT), errorOnFail) +} else { + null +} + } + + override def dataType: DataType = if (numberFormatter != null) { +numberFormatter.parsedDecimalType + } else { +DecimalType.USER_DEFAULT + } + + override def inputTypes: Seq[DataType] = Seq(StringType, StringType) + + override def checkInputDataTypes(): TypeCheckResult = { +val inputTypeCheck = super.checkInputDataTypes() +if (inputTypeCheck.isSuccess) { + if (numberFormatter == null) { +TypeCheckResult.TypeCheckSuccess + } else if (right.foldable) { +numberFormatter.check() + } else { +TypeCheckResult.TypeCheckFailure(s"Format expression must be foldable, but got $right") Review Comment: https://github.com/apache/spark/pull/38531 is merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38635: [SPARK-41118][SQL] `to_number`/`try_to_number` should return `null` when format is `null`
LuciferYang commented on code in PR #38635: URL: https://github.com/apache/spark/pull/38635#discussion_r1022266640 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala: ## @@ -26,6 +26,62 @@ import org.apache.spark.sql.catalyst.util.ToNumberParser import org.apache.spark.sql.types.{AbstractDataType, DataType, Decimal, DecimalType, StringType} import org.apache.spark.unsafe.types.UTF8String +abstract class ToNumberBase(left: Expression, right: Expression, errorOnFail: Boolean) + extends BinaryExpression with Serializable with ImplicitCastInputTypes with NullIntolerant { + + private lazy val numberFormatter = { +val value = right.eval() +if (value != null) { + new ToNumberParser(value.toString.toUpperCase(Locale.ROOT), errorOnFail) +} else { + null +} + } + + override def dataType: DataType = if (numberFormatter != null) { +numberFormatter.parsedDecimalType + } else { +DecimalType.USER_DEFAULT + } + + override def inputTypes: Seq[DataType] = Seq(StringType, StringType) + + override def checkInputDataTypes(): TypeCheckResult = { +val inputTypeCheck = super.checkInputDataTypes() +if (inputTypeCheck.isSuccess) { + if (numberFormatter == null) { +TypeCheckResult.TypeCheckSuccess + } else if (right.foldable) { +numberFormatter.check() + } else { +TypeCheckResult.TypeCheckFailure(s"Format expression must be foldable, but got $right") Review Comment: also cc @panbingkun -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #38635: [SPARK-41118][SQL] `to_number`/`try_to_number` should return `null` when format is `null`
LuciferYang commented on code in PR #38635: URL: https://github.com/apache/spark/pull/38635#discussion_r1022266469 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala: ## @@ -26,6 +26,62 @@ import org.apache.spark.sql.catalyst.util.ToNumberParser import org.apache.spark.sql.types.{AbstractDataType, DataType, Decimal, DecimalType, StringType} import org.apache.spark.unsafe.types.UTF8String +abstract class ToNumberBase(left: Expression, right: Expression, errorOnFail: Boolean) + extends BinaryExpression with Serializable with ImplicitCastInputTypes with NullIntolerant { + + private lazy val numberFormatter = { +val value = right.eval() +if (value != null) { + new ToNumberParser(value.toString.toUpperCase(Locale.ROOT), errorOnFail) +} else { + null +} + } + + override def dataType: DataType = if (numberFormatter != null) { +numberFormatter.parsedDecimalType + } else { +DecimalType.USER_DEFAULT + } + + override def inputTypes: Seq[DataType] = Seq(StringType, StringType) + + override def checkInputDataTypes(): TypeCheckResult = { +val inputTypeCheck = super.checkInputDataTypes() +if (inputTypeCheck.isSuccess) { + if (numberFormatter == null) { +TypeCheckResult.TypeCheckSuccess + } else if (right.foldable) { +numberFormatter.check() + } else { +TypeCheckResult.TypeCheckFailure(s"Format expression must be foldable, but got $right") Review Comment: error framework should be used, and seems https://github.com/apache/spark/pull/38531 is doing similar work -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org