QiangCai commented on a change in pull request #3694: [CARBONDATA-3763] Fix
wrong insert result during insert stage command
URL: https://github.com/apache/carbondata/pull/3694#discussion_r405359588
##########
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
##########
@@ -767,7 +773,16 @@ object CommonLoadUtils {
internalRow.getInt(index) +
DateDirectDictionaryGenerator.cutOffDate)
}
}
- internalRow
+ if (isInsertFromStageCommand) {
+ // Insert stage command, logical plan already consist of LogicalRDD of
internalRow.
+ // When it is converted to DataFrame, spark is reusing the same
internalRow.
+ // So, need to have a copy before the last transformation.
+ // TODO: Even though copying internalRow is faster, we should avoid it
+ // by finding a better way
+ internalRow.copy()
+ } else {
+ internalRow
+ }
Review comment:
How about moving code block to line 750?
Because this change faced NullPointerException in some scenarios.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
