YannByron commented on a change in pull request #5201:
URL: https://github.com/apache/hudi/pull/5201#discussion_r841239286
##########
File path:
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala
##########
@@ -286,8 +319,16 @@ abstract class HoodieBaseRelation(val sqlContext:
SQLContext,
}
protected final def appendMandatoryColumns(requestedColumns: Array[String]):
Array[String] = {
- val missing = mandatoryColumns.filter(col =>
!requestedColumns.contains(col))
- requestedColumns ++ missing
+ if (dropPartitionColumnsWhenWrite) {
+ if (requestedColumns.isEmpty) {
+ mandatoryColumns.toArray
+ } else {
+ requestedColumns
Review comment:
Using `requestedColumns ++ missing` will involve other columns which
don't be needed, and this does not work well alone with the partition columns
provided. The phenomenon is that the value of an extra useless fields replaces
the values of the partition fields.
In the case, I test and notice that will return list of `InternalRow`,
inside of which the extra, useless columns will be put between the required
data columns and partition columns. I guess the first n values will be
returned, instead of return values by schema.
Anyway, the logical above works well in all cases, and don't take more
overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
