Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Jacky Li]

The example is missing in my last mail, now I have put the example in 
CARBONDATA-3087   ,
please go to the JIRA and reply if you have any comment

Regards,
Jacky



--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Jacky Li]

Hi,

I revisit this discussion again, and suggest to change the DESC FORMATTED
output to following:



The information is outline in 6 sections:
1. Table basic information
2. Index information
3. Encoding information
4. Compaction information
5. Partition information (only for partition table)
6. Dynamic information

Please check whether it contains enough information of your preference, I
will create a JIRA and PR soon. 

Regards,
Jacky



--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Jacky Li]

I think for all table property that documented in  
https://github.com/apache/carbondata/blob/master/docs/ddl-of-carbondata.md 
, 
we should write their values in the schema file. 
Because the default value for these properties may change, if they are changes, 
user will not know what is the table property that was used when writing the 
file.

Regards,
Jacky Li


> 在 2018年10月8日，上午12:20，xm_zzc <441586...@qq.com> 写道：
> 
> Hi:
>  I agree with Jacky. 
>  Currently if i use the default value of blocklet size (64mb) to create a
> table and load some data into table, and then change the default value of
> blocklet size to 128mb, it will affect the table created before, is it
> right? I think it still need to use 64mb as blocklet size for tables created
> before.
> 
> These properties either specified by user or from default value need to be
> saved when create table:
> propertyvaluedefault
> value
> |Blocklet Size   |64 MB  |64 MB 
> |
> |Table Block Size  |1024 MB|1024 MB|
> |SORT_SCOPE  |LOCAL_SORT |LOCAL_SORT |
> |CACHE_LEVEL |BLOCKLET   |BLOCK  |
> |AUTO_LOAD_MERGE|true   |false  |
> |COMPACTION_LEVEL_THRESHOLD|2,8|4,3|
> |COMPACTION_PRESERVE_SEGMENTS|0  |0  |
> |ALLOWED_COMPACTION_DAYS |0  |0  |
> |MAJOR_COMPACTION_SIZE  |3072 MB|1024 MB|
> |Local Dictionary Enabled   |false  |false  |
> 
> Hi Jacky:
>  I think we need to refactor CarbonCli module and move some common tools to
> core module, and then CarbonCli module and Spark2 module can use them,
> right?
> 
> 
> 
> 
> --
> Sent from: 
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> 

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Jacky Li]

Looking at the DESC FORMATTED command again, I still feel it is not very
clear for the table property section. 
For table properties, I think it is not very good for DESC command to print
the default value if the user does not specify when creating the table.
Because the default value in CarbonCommonConstain file may change from
version to version, I think it is better to always write the default value
to table property (in schema file) when loading the table. Then in DESC
table, we can always get the table properties from the schema file. 

So I suggest we do following:
1. categorize the properties into file level, table level, system level
2. write the file level property into data file's footer, including all file
level properties either specified by user or from default value.
3. write the table level property into schema file, including all table
level properties either specified by user or from default value.
4. DESC command should print the properties read from the schema file, which
should contain all table level properties.

Another suggestion is that besides just printing the schema and table
properties like the standard hive DESC command, we can introduce another
command to print the output from calling CarbonCli tool for more profiling
and debugging information, like writing how many files the table contains,
what is the average size of page/blocklet, min/max percentage etc. For
example, the syntax of this command can be "SUMMARY table_name" 

Regards,
Jacky



--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[xm_zzc]

Hi all:
  Got it, thanks for your suggestions, I will implement this and raise a pr.



--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Ravindra Pesala]

Yes, I agree with Liang. We no need to consider showing sql in describe
table in case of CTAS.

Regards
Ravindra

On Tue, 21 Aug 2018 at 20:47, Raghunandan S <
carbondatacontributi...@gmail.com> wrote:

> Hi,
> In opinion it is not required to show the original select sql. Also is
> there a way to get it? I don't think it can be got.
>
>
> Regards
> Raghu
>
> On Tue, 21 Aug 2018, 8:02 pm Liang Chen,  wrote:
>
> > Hi
> >
> > 1. Agree with likun's comments(4 points) :
> >
> > 2. About 'select sql' for CTAS , you can leave it. we can consider it
> > later.
> >
> > Regards
> > Liang
> >
> > Jacky Li wrote
> > > Hi ZZC,
> > >
> > > I have checked the doc in CARBONDATA-2595. I have following comments:
> > > 1. In the Table Basic Information section, it is better to print the
> > Table
> > > Path instead of "CARBON Store Path”
> > > 2. For the Table Data Size  and Index Size, can you format the output
> in
> > > GB, MB, KB, etc
> > > 3. For the Last Update Time, can you format the output in UTC time like
> > > -MM-DD hh:mm:ss
> > > 4. In table property, I think maybe some properties are missing, like
> > > block size, blocklet size, long string
> > >
> > > For implementation, I suggest to write the main logic of collecting
> these
> > > information in java so that it is easier to write tools for it. One
> tool
> > > can be this SQL command and another tool I can think of is an
> standalone
> > > java executable that  can print these information on the screen by
> > reading
> > > the given table path. (We can put this standalone tool in SDK module)
> > >
> > > Regards,
> > > Jacky
> > >
> > >
> > >> 在 2018年8月20日，上午11:20，xm_zzc <
> >
> > > 441586683@
> >
> > >> 写道：
> > >>
> > >> Hi dev:
> > >>  Now I am working on this, the new format is shown in attachment,
> please
> > >> give me some feedback.
> > >>  There is one question: if user uses CTAS to create table, do we need
> to
> > >> show the 'select sql' in the result of 'desc formatted table'? If yes,
> > >> how
> > >> to get 'select sql'? now I just can get a non-formatted sql from
> > >> 'CarbonSparkSqlParser.scala' (Jacky mentioned), for example:
> > >>
> > >> *CREATE TABLE IF NOT EXISTS test_table
> > >> STORED BY 'carbondata'
> > >> TBLPROPERTIES(
> > >> 'streaming'='false', 'sort_columns'='id,city',
> > >> 'dictionary_include'='name')
> > >> AS SELECT * from source_test ;*
> > >>
> > >> The non-formatted sql I get is :
> > >> *SELECT*fromsource_test*
> > >>
> > >> desc_formatted.txt
> > >> 
> >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t133/desc_formatted.txt
> ;
> >
> > >> desc_formatted_external.txt
> > >> 
> >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t133/desc_formatted_external.txt
> ;
> >
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Sent from:
> > >>
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> > >>
> >
> >
> >
> >
> >
> > --
> > Sent from:
> > http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >
>


-- 
Thanks & Regards,
Ravi

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Raghunandan S]

Hi,
In opinion it is not required to show the original select sql. Also is
there a way to get it? I don't think it can be got.


Regards
Raghu

On Tue, 21 Aug 2018, 8:02 pm Liang Chen,  wrote:

> Hi
>
> 1. Agree with likun's comments(4 points) :
>
> 2. About 'select sql' for CTAS , you can leave it. we can consider it
> later.
>
> Regards
> Liang
>
> Jacky Li wrote
> > Hi ZZC,
> >
> > I have checked the doc in CARBONDATA-2595. I have following comments:
> > 1. In the Table Basic Information section, it is better to print the
> Table
> > Path instead of "CARBON Store Path”
> > 2. For the Table Data Size  and Index Size, can you format the output in
> > GB, MB, KB, etc
> > 3. For the Last Update Time, can you format the output in UTC time like
> > -MM-DD hh:mm:ss
> > 4. In table property, I think maybe some properties are missing, like
> > block size, blocklet size, long string
> >
> > For implementation, I suggest to write the main logic of collecting these
> > information in java so that it is easier to write tools for it. One tool
> > can be this SQL command and another tool I can think of is an standalone
> > java executable that  can print these information on the screen by
> reading
> > the given table path. (We can put this standalone tool in SDK module)
> >
> > Regards,
> > Jacky
> >
> >
> >> 在 2018年8月20日，上午11:20，xm_zzc <
>
> > 441586683@
>
> >> 写道：
> >>
> >> Hi dev:
> >>  Now I am working on this, the new format is shown in attachment, please
> >> give me some feedback.
> >>  There is one question: if user uses CTAS to create table, do we need to
> >> show the 'select sql' in the result of 'desc formatted table'? If yes,
> >> how
> >> to get 'select sql'? now I just can get a non-formatted sql from
> >> 'CarbonSparkSqlParser.scala' (Jacky mentioned), for example:
> >>
> >> *CREATE TABLE IF NOT EXISTS test_table
> >> STORED BY 'carbondata'
> >> TBLPROPERTIES(
> >> 'streaming'='false', 'sort_columns'='id,city',
> >> 'dictionary_include'='name')
> >> AS SELECT * from source_test ;*
> >>
> >> The non-formatted sql I get is :
> >> *SELECT*fromsource_test*
> >>
> >> desc_formatted.txt
> >> 
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t133/desc_formatted.txt;
>
> >> desc_formatted_external.txt
> >> 
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t133/desc_formatted_external.txt;
>
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Sent from:
> >>
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> >>
>
>
>
>
>
> --
> Sent from:
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Liang Chen]

Hi

1. Agree with likun's comments(4 points) : 

2. About 'select sql' for CTAS , you can leave it. we can consider it later.

Regards
Liang

Jacky Li wrote
> Hi ZZC,
> 
> I have checked the doc in CARBONDATA-2595. I have following comments:
> 1. In the Table Basic Information section, it is better to print the Table
> Path instead of "CARBON Store Path”
> 2. For the Table Data Size  and Index Size, can you format the output in
> GB, MB, KB, etc
> 3. For the Last Update Time, can you format the output in UTC time like
> -MM-DD hh:mm:ss
> 4. In table property, I think maybe some properties are missing, like
> block size, blocklet size, long string
> 
> For implementation, I suggest to write the main logic of collecting these
> information in java so that it is easier to write tools for it. One tool
> can be this SQL command and another tool I can think of is an standalone
> java executable that  can print these information on the screen by reading
> the given table path. (We can put this standalone tool in SDK module)
> 
> Regards,
> Jacky
> 
> 
>> 在 2018年8月20日，上午11:20，xm_zzc <

> 441586683@

>> 写道：
>> 
>> Hi dev:
>>  Now I am working on this, the new format is shown in attachment, please
>> give me some feedback.
>>  There is one question: if user uses CTAS to create table, do we need to
>> show the 'select sql' in the result of 'desc formatted table'? If yes,
>> how
>> to get 'select sql'? now I just can get a non-formatted sql from
>> 'CarbonSparkSqlParser.scala' (Jacky mentioned), for example:
>> 
>> *CREATE TABLE IF NOT EXISTS test_table
>> STORED BY 'carbondata'
>> TBLPROPERTIES(
>> 'streaming'='false', 'sort_columns'='id,city',
>> 'dictionary_include'='name')
>> AS SELECT * from source_test ;*
>> 
>> The non-formatted sql I get is :
>> *SELECT*fromsource_test*
>> 
>> desc_formatted.txt
>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t133/desc_formatted.txt;
>>   
>> desc_formatted_external.txt
>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t133/desc_formatted_external.txt;
>>   
>> 
>> 
>> 
>> 
>> 
>> 
>> --
>> Sent from:
>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>>





--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Jacky Li]

Hi ZZC,

I have checked the doc in CARBONDATA-2595. I have following comments:
1. In the Table Basic Information section, it is better to print the Table Path 
instead of "CARBON Store Path”
2. For the Table Data Size  and Index Size, can you format the output in GB, 
MB, KB, etc
3. For the Last Update Time, can you format the output in UTC time like 
-MM-DD hh:mm:ss
4. In table property, I think maybe some properties are missing, like block 
size, blocklet size, long string

For implementation, I suggest to write the main logic of collecting these 
information in java so that it is easier to write tools for it. One tool can be 
this SQL command and another tool I can think of is an standalone java 
executable that  can print these information on the screen by reading the given 
table path. (We can put this standalone tool in SDK module)

Regards,
Jacky


> 在 2018年8月20日，上午11:20，xm_zzc <441586...@qq.com> 写道：
> 
> Hi dev:
>  Now I am working on this, the new format is shown in attachment, please
> give me some feedback.
>  There is one question: if user uses CTAS to create table, do we need to
> show the 'select sql' in the result of 'desc formatted table'? If yes, how
> to get 'select sql'? now I just can get a non-formatted sql from
> 'CarbonSparkSqlParser.scala' (Jacky mentioned), for example:
> 
> *CREATE TABLE IF NOT EXISTS test_table
> STORED BY 'carbondata'
> TBLPROPERTIES(
> 'streaming'='false', 'sort_columns'='id,city', 'dictionary_include'='name')
> AS SELECT * from source_test ;*
> 
> The non-formatted sql I get is :
> *SELECT*fromsource_test*
> 
> desc_formatted.txt
> 
>   
> desc_formatted_external.txt
> 
>   
> 
> 
> 
> 
> 
> 
> --
> Sent from: 
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> 

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Jacky Li]

Hi ZZC,

Can you create a JIRA ticket and upload the design doc, in mail list we can not 
get the attachment

Regards,
Jacky

> 在 2018年8月20日，上午11:20，xm_zzc <441586...@qq.com> 写道：
> 
> Hi dev:
>  Now I am working on this, the new format is shown in attachment, please
> give me some feedback.
>  There is one question: if user uses CTAS to create table, do we need to
> show the 'select sql' in the result of 'desc formatted table'? If yes, how
> to get 'select sql'? now I just can get a non-formatted sql from
> 'CarbonSparkSqlParser.scala' (Jacky mentioned), for example:
> 
> *CREATE TABLE IF NOT EXISTS test_table
> STORED BY 'carbondata'
> TBLPROPERTIES(
> 'streaming'='false', 'sort_columns'='id,city', 'dictionary_include'='name')
> AS SELECT * from source_test ;*
> 
> The non-formatted sql I get is :
> *SELECT*fromsource_test*
> 
> desc_formatted.txt
> 
>   
> desc_formatted_external.txt
> 
>   
> 
> 
> 
> 
> 
> 
> --
> Sent from: 
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
> 

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[xm_zzc]

Hi dev:
  Now I am working on this, the new format is shown in attachment, please
give me some feedback.
  There is one question: if user uses CTAS to create table, do we need to
show the 'select sql' in the result of 'desc formatted table'? If yes, how
to get 'select sql'? now I just can get a non-formatted sql from
'CarbonSparkSqlParser.scala' (Jacky mentioned), for example:

*CREATE TABLE IF NOT EXISTS test_table
STORED BY 'carbondata'
TBLPROPERTIES(
'streaming'='false', 'sort_columns'='id,city', 'dictionary_include'='name')
AS SELECT * from source_test ;*

The non-formatted sql I get is :
*SELECT*fromsource_test*

desc_formatted.txt

  
desc_formatted_external.txt

  






--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[xm_zzc]

please see:
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t1/desc_table_info.txt



--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[xuchuanyin]

Then what's the final output looks like?



--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Liang Chen]

Hi Ravi

Good thinking.

Because the inverted index columns by default are the same as sort_column
columns,  from the user perspective, he only need to set no_inverted_index
columns in sort_column columns,  so i proposed to display only the
no_inverted_index columns info which be set by user.

Anyway, in "data management on carbondata" need to update the info: inverted
index columns by default be enabled along with sort_columns

Regards
Liang


ravipesala wrote
> I agree with Liang's suggestion to align the information with table
> schema. And I have one suggestion related to NO_INVERTED_INDEX , instead
> of mentioning no inverted index columns better mention which are inverted
> index columns. It is very hard user to understand which are inverted index
> columns and it is useful if we change our default behaviour of selecting
> index columns if we provide this information to table describe command.
> Regards,
> Ravindra.
> 
> Sent from Mailspring (https://link.getmailspring.com/link/

> 1524723947.local-c273b69b-15c2-v1.2.1-7e7447b6@

> /0?redirect=https%3A%2F%2Fgetmailspring.com%2F=ZGV2QGNhcmJvbmRhdGEuYXBhY2hlLm9yZw%3D%3D),
> the best free email app for work
> On Apr 26 2018, at 10:34 am, manishgupta88 

> tomanishgupta18@

>  wrote:
>>
>> I agree with Liang. We can modify the complete describe formatted command
>> display and show the detailed information as suggested by Liang.
>> Liang we can make a small change in your suggestion. As we are displaying
>> the information to the user we should not include Underscore(_) in the
>> property names and in place of DICTIONARY_INCLUDE and DICTIONARY_EXCLUDE
>> we
>> can just say Dictionary columns and No Dictionary Columns.
>>
>> ## Detailed Table Properties Information
>> |Sort Columns |name,id
>> |No Inverted Index |id
>> |Dictionary Columns |name
>> |Table BlockSize |1024 MB
>> |Sort Scope |LOCAL_SORT
>> |Streaming |false
>>
>> Regards
>> Manish Gupta
>>
>>
>>
>> --
>> Sent from:
>> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>>





--
Sent from: 
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Ravindra Pesala]

I agree with Liang's suggestion to align the information with table schema. And 
I have one suggestion related to NO_INVERTED_INDEX , instead of mentioning no 
inverted index columns better mention which are inverted index columns. It is 
very hard user to understand which are inverted index columns and it is useful 
if we change our default behaviour of selecting index columns if we provide 
this information to table describe command.
Regards,
Ravindra.

Sent from Mailspring 
(https://link.getmailspring.com/link/1524723947.local-c273b69b-15c2-v1.2.1-7e744...@getmailspring.com/0?redirect=https%3A%2F%2Fgetmailspring.com%2F=ZGV2QGNhcmJvbmRhdGEuYXBhY2hlLm9yZw%3D%3D),
 the best free email app for work
On Apr 26 2018, at 10:34 am, manishgupta88  wrote:
>
> I agree with Liang. We can modify the complete describe formatted command
> display and show the detailed information as suggested by Liang.
> Liang we can make a small change in your suggestion. As we are displaying
> the information to the user we should not include Underscore(_) in the
> property names and in place of DICTIONARY_INCLUDE and DICTIONARY_EXCLUDE we
> can just say Dictionary columns and No Dictionary Columns.
>
> ## Detailed Table Properties Information
> |Sort Columns |name,id
> |No Inverted Index |id
> |Dictionary Columns |name
> |Table BlockSize |1024 MB
> |Sort Scope |LOCAL_SORT
> |Streaming |false
>
> Regards
> Manish Gupta
>
>
>
> --
> Sent from: 
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
>

Re: Change the 'comment' content for column when execute command 'desc formatted table_name'

[Venkata Gollamudi]

I agree with Liang, Better we align with create table terminology and
properties. Details of properties, user can easily get from Create table
DDL documentation.

Regards,
Ramana

On Thu, Apr 26, 2018 at 8:17 AM, Liang Chen  wrote:

> Hi
>
> Attaching my proposed "desc_table_info":
> desc_table_info.txt
>  n5.nabble.com/file/t1/desc_table_info.txt>
>
> Regards
> Liang
>
>
>
> --
> Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.
> n5.nabble.com/
>