Zhangshunyu commented on a change in pull request #4020:
URL: https://github.com/apache/carbondata/pull/4020#discussion_r531370782
##########
File path:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/datacompaction/MajorCompactionIgnoreInMinorTest.scala
##########
@@ -186,6 +187,78 @@ class MajorCompactionIgnoreInMinorTest extends QueryTest
with BeforeAndAfterAll
}
+ def generateData(numOrders: Int = 100000): DataFrame = {
+ import sqlContext.implicits._
+ sqlContext.sparkContext.parallelize(1 to numOrders, 4)
+ .map { x => ("country" + x, x, "07/23/2015", "name" + x, "phonetype" + x,
+ "serialname" + x, x + 10000)
+ }.toDF("country", "ID", "date", "name", "phonetype", "serialname",
"salary")
+ }
+
+ test("test skip segment whose data size exceed threshold in minor
compaction") {
Review comment:
@ajantha-bhat set to 1MB means the segment size > 1MB, it should be
ignore in compaction flow. how can 4 lines data reach to 1MB? if use 4 lines
data, both will compact.
here we use huge data to exceed 1MB to test the hug segment is ignored
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
