MaxGekk commented on code in PR #46880:
URL: https://github.com/apache/spark/pull/46880#discussion_r1756259640
##########
python/pyspark/sql/functions/builtin.py:
##########
@@ -11241,13 +11241,27 @@ def sentences(
) -> Column:
"""
Splits a string into arrays of sentences, where each sentence is an array
of words.
- The 'language' and 'country' arguments are optional, and if omitted, the
default locale is used.
+ The `language` and `country` arguments are optional,
+ When they are omitted:
+ 1.If they are both omitted, the `Locale.ROOT - locale(language='',
country='')` is used.
+ The `Locale.ROOT` is regarded as the base locale of all locales, and is
used as the
+ language/country neutral locale for the locale sensitive operations.
+ 2.If the `country` is omitted, the `locale(language, country='')` is used.
+ When they are null:
+ 1.If they are both `null`, the `Locale.US - locale(language='en',
country='US')` is used.
+ 2.If the `language` is null and the `country` is not null,
+ the `Locale.US - locale(language='en', country='US')` is used.
Review Comment:
You cannot construct a Locale only by a `country`.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:
##########
@@ -3362,49 +3387,18 @@ case class Sentences(
override def second: Expression = language
override def third: Expression = country
- override def eval(input: InternalRow): Any = {
- val string = str.eval(input)
- if (string == null) {
- null
- } else {
- val languageStr = language.eval(input).asInstanceOf[UTF8String]
- val countryStr = country.eval(input).asInstanceOf[UTF8String]
- val locale = if (languageStr != null && countryStr != null) {
- new Locale(languageStr.toString, countryStr.toString)
- } else {
- Locale.US
Review Comment:
I believe old behavior is counterintuitive from user perspective. An user
set a `language` explicitly, let's say to Chinese or Russian, but we ignore it,
and set a geographically-specific locale linked to the USA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
