niuge01 commented on a change in pull request #3515: [CARBONDATA-3623]: Fixed
global sort compaction failure on timestamp column
URL: https://github.com/apache/carbondata/pull/3515#discussion_r364137512
##########
File path:
integration/spark-common/src/main/scala/org/apache/carbondata/spark/load/DataLoadProcessBuilderOnSpark.scala
##########
@@ -443,23 +438,18 @@ object DataLoadProcessBuilderOnSpark {
.asScala
.map(_.getColName)
.toArray
+
+ /**
+ *
[[org.apache.spark.sql.catalyst.expressions.objects.ValidateExternalType]]
validates the
+ * datatype of column data and corresponding datatype in schema provided
to create dataframe.
+ * Since carbonScanRDD gives Long data for timestamp column and
corresponding column datatype in
+ * schema is Timestamp, this validation fails if we use createDataFrame
API which takes rdd as
+ * input. Hence, We need to give the List[Row] compatible with the schema
datatypes. So using
+ * the createDataFrame API which takes List[Row] and schema as input.
+ */
val schema = SparkTypeConverter.createSparkSchema(carbonTable, columns)
- val rdd: RDD[InternalRow] = new CarbonScanRDD[CarbonRow](
- sparkSession,
- columnProjection = new CarbonProjection(columns),
- null,
- carbonTable.getAbsoluteTableIdentifier,
- carbonTable.getTableInfo.serialize,
- carbonTable.getTableInfo,
- new CarbonInputMetrics,
- null,
- classOf[SparkDataTypeConverterImpl],
- classOf[CarbonRowReadSupport],
- splits.asJava)
- .map { row =>
- new GenericInternalRow(row.getData.asInstanceOf[Array[Any]])
- }
- SparkSQLUtil.execute(rdd, schema, sparkSession)
+ val listOfRows =
sparkSession.sqlContext.table(carbonTable.getTableName).collect().toList.asJava
Review comment:
carbonTable is the target table of load process, not the source table.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
