ajantha-bhat commented on a change in pull request #3694: [CARBONDATA-3763] Fix
wrong insert result during insert stage command
URL: https://github.com/apache/carbondata/pull/3694#discussion_r405363099
##########
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CommonLoadUtils.scala
##########
@@ -767,7 +773,16 @@ object CommonLoadUtils {
internalRow.getInt(index) +
DateDirectDictionaryGenerator.cutOffDate)
}
}
- internalRow
+ if (isInsertFromStageCommand) {
+ // Insert stage command, logical plan already consist of LogicalRDD of
internalRow.
+ // When it is converted to DataFrame, spark is reusing the same
internalRow.
+ // So, need to have a copy before the last transformation.
+ // TODO: Even though copying internalRow is faster, we should avoid it
+ // by finding a better way
+ internalRow.copy()
+ } else {
+ internalRow
+ }
Review comment:
@QiangCai: CI all test case run. when we can get NPE ?
Shall I just add null check in the same line 782 ? Becuase moving to line
750, how it can avoid NPE ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
