pratyakshsharma commented on a change in pull request #4227:
URL: https://github.com/apache/carbondata/pull/4227#discussion_r732178451
##########
File path:
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/mutation/merge/CarbonMergeDataSetUtil.scala
##########
@@ -462,4 +474,413 @@ object CarbonMergeDataSetUtil {
columnMinMaxInBlocklet.asScala
}
}
+
+ /**
+ * This method verifies source and target schemas for the following:
+ * If additional columns are present in source schema as compared to target,
simply ignore them.
+ * If some columns are missing in source schema as compared to target
schema, exception is thrown.
+ * If data type of some column differs in source and target schemas,
exception is thrown.
+ * If source schema has multiple columns whose names differ only in case
sensitivity, exception
+ * is thrown.
+ * @param targetDs target carbondata table
+ * @param srcDs source/incoming data
+ */
+ def verifySourceAndTargetSchemas(targetDs: Dataset[Row], srcDs:
Dataset[Row]): Unit = {
+ LOGGER.info("schema enforcement is enabled. Source and target schemas will
be verified")
+ // get the source and target dataset schema
+ val sourceSchema = srcDs.schema
+ val targetSchema = targetDs.schema
+
+ targetSchema.fields.foreach(tgtField => {
+ // check if some field is missing in source schema
+ if
(!sourceSchema.fields.map(_.name.toLowerCase).contains(tgtField.name.toLowerCase))
{
Review comment:
Thank you for pointing this out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@carbondata.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
