Re: [Issue] Long string columns config for big strings not work

2018-10-12 Thread xuchuanyin
Hi, aaron: A PR has been raised for this issue https://github.com/apache/carbondata/pull/2812, please check. -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Issue] Long string columns config for big strings not work

2018-10-12 Thread xuchuanyin
Hi, arron, I go through the code and find the root cause. While writing dataframe to carbontable, we have to keep the order of the fields in dataframe the same as that in carbontable. The code lies in `NewCarbonDataLoadRDD.scala#486`. This is because we judge whether the field is a

Re: [Issue] Long string columns config for big strings not work

2018-10-11 Thread aaron
Thanks, I will have a try -- Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: [Issue] Long string columns config for big strings not work

2018-10-11 Thread xuchuanyin
Yeah, aaron, the problem may lies in the dataframe and long_string_columns. Can you try the following statement? It is from the test code in 'VarcharDataTypesBasicTestCase', which suggests you to specify the 'long_string_columns' while writing the dataframe. ```scala test("write from dataframe

Re: [Issue] Long string columns config for big strings not work

2018-10-10 Thread aaron
Hi Community, I found that if I match the table columns order and dataframe order through below way, then it works. _df .select( "market_code", "product_id", "country_code", "category_id", "company_id", "name", "company", "release_date", "price", "version",

[Issue] Long string columns config for big strings not work

2018-10-10 Thread aaron
Hi Community, I encounter a issue, the LONG_STRING_COLUMNS config for big strings not work. My env is spark2.3.2 + carbon 1.5.0 1. DDL Sql carbon.sql( s""" |CREATE TABLE IF NOT EXISTS product( |market_code STRING, |product_id LONG, |country_code STRING,