akashrn5 commented on a change in pull request #3856:
URL: https://github.com/apache/carbondata/pull/3856#discussion_r474500538
##########
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -439,6 +449,11 @@ case class CarbonInsertIntoCommand(databaseNameOp:
Option[String],
def insertData(loadParams: CarbonLoadParams): (Seq[Row],
LoadMetadataDetails) = {
var rows = Seq.empty[Row]
+ val loadDataFrame = if (updateModel.isDefined &&
!updateModel.get.loadAsNewSegment) {
+ Some(CommonLoadUtils.getDataFrameWithTupleID(Some(dataFrame)))
Review comment:
done
##########
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertIntoCommand.scala
##########
@@ -439,6 +449,11 @@ case class CarbonInsertIntoCommand(databaseNameOp:
Option[String],
def insertData(loadParams: CarbonLoadParams): (Seq[Row],
LoadMetadataDetails) = {
var rows = Seq.empty[Row]
+ val loadDataFrame = if (updateModel.isDefined &&
!updateModel.get.loadAsNewSegment) {
+ Some(CommonLoadUtils.getDataFrameWithTupleID(Some(dataFrame)))
Review comment:
> This InsertIntoCommand flow is not meant for update flow yet. Because
update will have an implicit column and rearrange schema and all will fail. so,
I suggest if `updateModel.get.loadAsNewSegment` is `false` throw unsupported
exception now and handle this requirement later.
>
> Also when `updateModel.get.loadAsNewSegment = true` (which is our current
cdc history data case), **this flow can be used** (as it is just a insert, no
actual update flow used). only when `updateModel.get.loadAsNewSegment = false`
we cannot use this flow.
>
> so someone might use it because of update model support. so, I suggest to
throw an exception in the beginning of this function when
`updateModel.get.loadAsNewSegment = false`
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
