cloud-fan commented on a change in pull request #31769:
URL: https://github.com/apache/spark/pull/31769#discussion_r589304987
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala
##########
@@ -379,15 +390,13 @@ final class DataFrameNaFunctions private[sql](df:
DataFrame) {
case _: String => StringType
}
- val columnEquals = df.sparkSession.sessionState.analyzer.resolver
- val projections = df.schema.fields.map { f =>
- val shouldReplace = cols.exists(colName => columnEquals(colName, f.name))
Review comment:
Previously, we use string match directly, which means
`df.na.replace("col.A", ...)` works. Now, we parse the column name as SQL
syntax, which may break `df.na.replace("col.A", ...)`.
I think the new behavior makes more sense since it makes SQL and DataFrame
API more consistent about column names. It also allows qualified column names
which is useful for joined dataframes. Let's add an item in the migration guide
to describe the behavior change.
also cc @HyukjinKwon @srowen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
