ajantha-bhat commented on a change in pull request #3538: [CARBONDATA-3637]
Optimize insert into flow
URL: https://github.com/apache/carbondata/pull/3538#discussion_r378751755
##########
File path:
integration/spark-common-test/src/test/scala/org/apache/carbondata/integration/spark/testsuite/dataload/TestLoadDataGeneral.scala
##########
@@ -150,10 +150,14 @@ class TestLoadDataGeneral extends QueryTest with
BeforeAndAfterEach {
sql("drop table if exists load32000chardata")
sql("drop table if exists load32000chardata_dup")
sql("CREATE TABLE load32000chardata(dim1 String, dim2 String, mes1 int)
STORED AS carbondata")
- sql("CREATE TABLE load32000chardata_dup(dim1 String, dim2 String, mes1
int) STORED AS carbondata")
+ sql("CREATE TABLE load32000chardata_dup(dim1 String, dim2 String, mes1
int) STORED AS " +
+ "carbondata tblproperties('local_dictionary_enable'='false')")
sql(s"LOAD DATA LOCAL INPATH '$testdata' into table load32000chardata
OPTIONS('FILEHEADER'='dim1,dim2,mes1')")
+ // Previously converter step was checking more than 32k length and
throwing exception.
+ // Now, due to local dictionary is true. Insert will not fail.
+ // when local dictionary is false, insert will fail at write step
intercept[Exception] {
- sql("insert into load32000chardata_dup select
dim1,concat(load32000chardata.dim2,'aaaa'),mes1 from load32000chardata").show()
+ sql("insert into load32000chardata_dup select
dim1,concat(load32000chardata.dim2,load32000chardata.dim2),mes1 from
load32000chardata").show()
Review comment:
In new flow no converter step doesn't check string is more than 32k or not
now. Write step didn't fail because after compression it is less than 32k. so
I add 32k + 32k. so after compression it will still fail.
I can set bad record handle and make it same as old test case also
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
