[jira] [Created] (CARBONDATA-482) improve integration test case of AllDataTypesTestCase6

2016-12-01 Thread Rahul Kumar (JIRA) 
Rahul Kumar created CARBONDATA-482:
--

 Summary: improve integration test case of AllDataTypesTestCase6
 Key: CARBONDATA-482
 URL: https://issues.apache.org/jira/browse/CARBONDATA-482
 Project: CarbonData
  Issue Type: Test
Reporter: Rahul Kumar






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: CarbonData propose major version number increment for next version (to 1.0.0)

2016-12-01 Thread Liang Chen 
Hi

Thanks for all of your comments, will change the current master-SNAPSHOT
version to 1.0.0

Regards
Liang


Venkata Gollamudi wrote
> Hi All,
> 
> CarbonData 0.2.0 has been a good work and stable release with lot of
> defects fixed and with number of performance improvements.
> https://issues.apache.org/jira/browse/CARBONDATA-320?jql=project%20%3D%20CARBONDATA%20AND%20fixVersion%20%3D%200.2.0-incubating%20ORDER%20BY%20updated%20DESC%2C%20priority%20DESC%2C%20created%20ASC
> 
> Next version has many major and new value added features are planned,
> taking CarbonData capability to next level.
> Like
> - IUD(Insert-Update-Delete) support,
> - complete rewrite of data load flow with out Kettle,
> - Spark 2.x support,
> - Standardize CarbonInputFormat and CarbonOutputFormat,
> - alluxio(tachyon) file system support,
> - Carbon thrift format optimization for fast query,
> - Data loading performance improvement and In memory off heap sorting,
> - Query performance improvement using off heap,
> - Support Vectorized batch reader.
> 
> https://issues.apache.org/jira/browse/CARBONDATA-301?jql=project%20%3D%20CARBONDATA%20AND%20fixVersion%20%3D%200.3.0-incubating%20ORDER%20BY%20updated%20DESC%2C%20priority%20DESC%2C%20created%20ASC
> 
> I think it makes sense to change CarbonData Major version in next version
> to 1.0.0.
> Please comment and vote on this.
> 
> Thanks,
> Ramana





--
View this message in context: 
http://apache-carbondata-mailing-list-archive.1130556.n5.nabble.com/CarbonData-propose-major-version-number-increment-for-next-version-to-1-0-0-tp3131p3544.html
Sent from the Apache CarbonData Mailing List archive mailing list archive at 
Nabble.com.

[jira] [Created] (CARBONDATA-481) [SPARK2]fix late decoder and support whole stage code gen

2016-12-01 Thread QiangCai (JIRA) 
QiangCai created CARBONDATA-481:
---

 Summary: [SPARK2]fix late decoder and support whole stage code gen
 Key: CARBONDATA-481
 URL: https://issues.apache.org/jira/browse/CARBONDATA-481
 Project: CarbonData
  Issue Type: Bug
  Components: data-query
Affects Versions: 0.2.0-incubating
Reporter: QiangCai
 Fix For: 0.3.0-incubating






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

carbondata loading

2016-12-01 Thread Lu Cao 
Hi dev team,
I'm loading data from parquet file to carbondata file(DF read parquet and
save to csv then load into carbondata file). The job is blocked at "collect
at CarbonDataRDDFactory.scala:963"



*Job Id*

*Description*

*Submitted*

*Duration*

*Stages: Succeeded/Total*

*Tasks (for all stages): Succeeded/Total*

6

collect at CarbonDataRDDFactory.scala:963


2016/12/01 13:56:43

3.1 h

0/1

0/2
Completed Jobs (6)

*Job Id*

*Description*

*Submitted*

*Duration*

*Stages: Succeeded/Total*

*Tasks (for all stages): Succeeded/Total*

5

collect at GlobalDictionaryUtil.scala:800


2016/12/01 13:34:25

22 min

2/2

422/422

4

take at CarbonCsvRelation.scala:181


2016/12/01 13:34:25

0.1 s

1/1

1/1

3

saveAsTextFile at package.scala:169


2016/12/01 13:11:02

23 min

1/1

50/50

2

count at SaicSparkConvert.scala:40


2016/12/01 13:10:31

31 s

2/2

51/51

1

parquet at SaicSparkConvert.scala:35


2016/12/01 13:10:28

1 s

1/1

2/2

0

parquet at SaicSparkConvert.scala:35


2016/12/01 13:10:26

2 s

1/1

2/2


I looked into the stdout, the log are all the same warning.


WARN  01-12 13:56:46,096 - [pool-25-thread-5][partitionID:carbontest]
Cannot convert : null to Numeric type value. Value considered as null.

WARN  01-12 13:56:46,096 - [pool-25-thread-4][partitionID:carbontest]
Cannot convert : null to Numeric type value. Value considered as null.

WARN  01-12 13:56:46,096 - [pool-25-thread-1][partitionID:carbontest]
Cannot convert : null to Numeric type value. Value considered as null.

WARN  01-12 13:56:46,096 - [pool-25-thread-2][partitionID:carbontest]
Cannot convert : null to Numeric type value. Value considered as null.

WARN  01-12 13:56:46,096 - [pool-25-thread-6][partitionID:carbontest]
Cannot convert : null to Numeric type value. Value considered as null.

WARN  01-12 13:56:46,096 - [pool-25-thread-2][partitionID:carbontest]
Cannot convert : null to Numeric type value. Value considered as null.

WARN  01-12 13:56:46,096 - [pool-25-thread-1][partitionID:carbontest]
Cannot convert : null to Numeric type value. Value considered as null.


My configuration is

--master yarn-custer

--driver-memory 8g

--executor-memory 120g

--num-executors 3


Any idea for this? Is it caused by data type?


Thanks,

Lionel