date:20170516

[GitHub] carbondata issue #916: [CARBONDATA-938] Prune partitions for filter query on...

2017-05-16 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/916
  
Build Success with Spark 1.6.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2048/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #915: [CARBONDATA-946] Spark 2x tupleId support for ...

2017-05-16 Thread nareshpr

GitHub user nareshpr reopened a pull request:

https://github.com/apache/carbondata/pull/915

[CARBONDATA-946] Spark 2x tupleId support for IUD Feature



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nareshpr/incubator-carbondata 
spark2xIUDtupleId

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #915


commit 9b817c6dad626d5636f6c6e5d01550595e082f55
Author: nareshpr 
Date:   2017-05-15T07:40:08Z

Spark 2x tupleId support for IUD Feature




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata pull request #915: [CARBONDATA-946] Spark 2x tupleId support for ...

2017-05-16 Thread nareshpr

Github user nareshpr closed the pull request at:

https://github.com/apache/carbondata/pull/915


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (CARBONDATA-1056) Data_load failure using single_pass true with spark 2.1

2017-05-16 Thread Kunal Kapoor (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kunal Kapoor reassigned CARBONDATA-1056:


Assignee: Kunal Kapoor

> Data_load failure using single_pass true with spark 2.1
> ---
>
> Key: CARBONDATA-1056
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1056
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.1.0
> Environment: spark 2.1
>Reporter: Vandana Yadav
>Assignee: Kunal Kapoor
>Priority: Minor
> Attachments: 2000_UniqData.csv
>
>
> Data_load failure using single_pass true with spark 2.1
> Steps to reproduce:
> 1)Create Table:
> CREATE TABLE uniq_exclude_sp1 (CUST_ID int,CUST_NAME 
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, 
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), 
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 
> double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_EXCLUDE'='CUST_NAME,ACTIVE_EMUI_VERSION');
> 2) Load Data:
> LOAD DATA INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table 
> uniq_exclude_sp1 OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');
> 3)Result:
> Actual result on beeline:
> Error: java.lang.Exception: Dataload failed due to error while writing 
> dictionary file! (state=,code=0)
> Expected Result: data should be load successfully 
> 4)Thriftserver logs:
> 17/05/16 16:07:20 INFO SparkExecuteStatementOperation: Running query 'LOAD 
> DATA INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table 
> uniq_exclude_sp1 OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true')'
>  with 34eb7e9e-bd49-495c-af68-8f0b5e36b786
> 17/05/16 16:07:20 INFO CarbonSparkSqlParser: Parsing command: LOAD DATA 
> INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table uniq_exclude_sp1 
> OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true')
> 17/05/16 16:07:20 INFO CarbonLateDecodeRule: pool-23-thread-4 Skip 
> CarbonOptimizer
> 17/05/16 16:07:20 INFO HdfsFileLock: pool-23-thread-4 HDFS lock 
> path:hdfs://localhost:54310/opt/prestocarbonStore/default/uniq_exclude_sp1/meta.lock
> 17/05/16 16:07:20 INFO LoadTable: pool-23-thread-4 Successfully able to get 
> the table metadata file lock
> 17/05/16 16:07:20 INFO LoadTable: pool-23-thread-4 Initiating Direct Load for 
> the Table : (default.uniq_exclude_sp1)
> 17/05/16 16:07:20 AUDIT CarbonDataRDDFactory$: 
> [knoldus][hduser][Thread-137]Data load request has been received for table 
> default.uniq_exclude_sp1
> 17/05/16 16:07:20 INFO CommonUtil$: pool-23-thread-4 [Block Distribution]
> 17/05/16 16:07:20 INFO CommonUtil$: pool-23-thread-4 totalInputSpaceConsumed: 
> 376223 , defaultParallelism: 4
> 17/05/16 16:07:20 INFO CommonUtil$: pool-23-thread-4 
> mapreduce.input.fileinputformat.split.maxsize: 16777216
> 17/05/16 16:07:20 INFO FileInputFormat: Total input paths to process : 1
> 17/05/16 16:07:20 INFO DistributionUtil$: pool-23-thread-4 Executors 
> configured : 1
> 17/05/16 16:07:20 INFO DistributionUtil$: pool-23-thread-4 Total Time taken 
> to ensure the required executors : 0
> 17/05/16 16:07:20 INFO DistributionUtil$: pool-23-thread-4 Time elapsed to 
> allocate the required executors: 0
> 17/05/16 16:07:20 INFO CarbonDataRDDFactory$: pool-23-thread-4 Total Time 
> taken in block allocation: 1
> 17/05/16 16:07:20 INFO CarbonDataRDDFactory$: pool-23-thread-4 Total no of 
> blocks: 1, No.of Nodes: 1
> 17/05/16 16:07:20 INFO CarbonDataRDDFactory$: pool-23-thread-4 #Node: knoldus 
> no.of.blocks: 1
> 17/05/16 16:07:20 INFO MemoryStore: Block broadcast_2 stored as values in 
> memory (estimated size 53.7 MB, free 291.4 MB)
> 17/05/16 16:07:20 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes 
> in memory (estimated size 23.2 KB, free 291.4 MB)
> 17/05/16 16:07:20 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory 
> on 192.168.1.10:42046 (size: 23.2 KB, free: 366.2 MB)
> 17/05/16 16:07:20 INFO SparkContext: Created broadcast 2 from broadcast at 
> NewCarbonDataLoadRDD.scala:185
> 17/05/16 16:07:20 INFO SparkContext: Starting job:

[GitHub] carbondata issue #917: [CARBONDATA-1053] supported char type for hive

2017-05-16 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/917
  
Build Failed  with Spark 1.6.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2047/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (CARBONDATA-852) Less than or equal to operator(<=) does not work properly in Range Filter.

2017-05-16 Thread Vinod Rohilla (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Rohilla reassigned CARBONDATA-852:


Assignee: (was: Vinod Rohilla)

> Less than or equal to operator(<=) does not work properly in Range Filter.
> --
>
> Key: CARBONDATA-852
> URL: https://issues.apache.org/jira/browse/CARBONDATA-852
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load
>Affects Versions: 1.1.0
> Environment: Spark 2-1
>Reporter: Vinod Rohilla
>Priority: Minor
> Attachments: 2000_UniqData.csv
>
>
> Less than or equal (<=) to operator does not work properly in range filter.
> Steps to reproduces:
> 1)Create table:
> CREATE TABLE uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION 
> string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 
> bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 
> decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
> int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES 
> ("TABLE_BLOCKSIZE"= "256 MB");
> 2)Load Data in a table:
> LOAD DATA INPATH 'HDFS_URL/BabuStore/Data/uniqdata/2000_UniqData.csv' into 
> table uniqdata OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> 3: Run the Query.
> select dob from uniqdata where dob <= '1972-12-10' and dob >= '1972-12-01';
> 4:Result on beeline:
> ++--+
> |  dob   |
> ++--+
> | 1972-12-01 01:00:03.0  |
> | 1972-12-02 01:00:03.0  |
> | 1972-12-03 01:00:03.0  |
> | 1972-12-04 01:00:03.0  |
> | 1972-12-05 01:00:03.0  |
> | 1972-12-06 01:00:03.0  |
> | 1972-12-07 01:00:03.0  |
> | 1972-12-08 01:00:03.0  |
> | 1972-12-09 01:00:03.0  |
> ++--+
> Expected Result: It should include " 1972-12-10 " in the result set.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (CARBONDATA-904) ArrayIndexOutOfBoundsException

2017-05-16 Thread SWATI RAO (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SWATI RAO closed CARBONDATA-904.

Resolution: Fixed

Working fine

> ArrayIndexOutOfBoundsException 
> ---
>
> Key: CARBONDATA-904
> URL: https://issues.apache.org/jira/browse/CARBONDATA-904
> Project: CarbonData
>  Issue Type: Bug
> Environment: Spark1.6
>Reporter: SWATI RAO
>Assignee: Rahul Kumar
> Fix For: 1.1.1
>
> Attachments: Test_Data1_h1.csv, Test_Data1_h1.csv
>
>
> Or operator is not working properly.
> When we execute these query in hive it is working fine but when we execute 
> the same in carbondata it throws an exception:
> java.lang.ArrayIndexOutOfBoundsException
> HIVE:
> 0: jdbc:hive2://hadoop-master:1> create table Test_Boundary_h1 (c1_int 
> int,c2_Bigint Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string 
> string,c6_Timestamp Timestamp,c7_Datatype_Desc string) ROW FORMAT DELIMITED 
> FIELDS TERMINATED BY ',' ;
> +-+--+
> | result  |
> +-+--+
> +-+--+
> No rows selected (1.177 seconds)
> 0: jdbc:hive2://hadoop-master:1> load data local inpath 
> '/opt/Carbon/CarbonData/TestData/Data/Test_Data1_h1.csv' OVERWRITE INTO TABLE 
> Test_Boundary_h1 ;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.437 seconds)
> 0: jdbc:hive2://hadoop-master:1> select c6_Timestamp,max(c6_Timestamp) 
> from Test_Boundary_h1 where c6_Timestamp ='2017-07-01 12:07:28' or 
> c6_Timestamp ='2019-07-05 13:07:30' or c6_Timestamp = '1999-01-06 10:05:29' 
> group by c6_Timestamp ;
> +++--+
> |  c6_Timestamp  |  _c1   |
> +++--+
> | 2017-07-01 12:07:28.0  | 2017-07-01 12:07:28.0  |
> +++--+
> 1 row selected (1.637 seconds)
> CARBONDATA:
> 0: jdbc:hive2://hadoop-master:1> create table Test_Boundary (c1_int 
> int,c2_Bigint Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string 
> string,c6_Timestamp Timestamp,c7_Datatype_Desc string) STORED BY 
> 'org.apache.carbondata.format' ;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (4.48 seconds)
> 0: jdbc:hive2://hadoop-master:1> LOAD DATA INPATH 
> 'hdfs://192.168.2.145:54310/BabuStore/Data/Test_Data1.csv' INTO table 
> Test_Boundary 
> OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
>  ;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (4.445 seconds)
> 0: jdbc:hive2://hadoop-master:1> select c6_Timestamp,max(c6_Timestamp) 
> from Test_Boundary where c6_Timestamp ='2017-07-01 12:07:28' or c6_Timestamp 
> =' 2019-07-05 13:07:30' or c6_Timestamp = '1999-01-06 10:05:29' group by 
> c6_Timestamp ;
> Error: org.apache.spark.SparkException: Job aborted due to stage failure: 
> Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in 
> stage 5.0 (TID 8, hadoop-master): java.lang.RuntimeException: 
> java.util.concurrent.ExecutionException: 
> java.lang.ArrayIndexOutOfBoundsException: 0
>   at 
> org.apache.carbondata.core.scan.processor.AbstractDataBlockIterator.updateScanner(AbstractDataBlockIterator.java:136)
>   at 
> org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:50)
>   at 
> org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32)
>   at 
> org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:50)
>   at 
> org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:41)
>   at 
> org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:31)
>   at 
> org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.(ChunkRowIterator.java:41)
>   at 
> org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:79)
>   at 
> org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:204)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>

[jira] [Commented] (CARBONDATA-904) ArrayIndexOutOfBoundsException

2017-05-16 Thread SWATI RAO (JIRA)


[ 
https://issues.apache.org/jira/browse/CARBONDATA-904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012250#comment-16012250
 ] 

SWATI RAO commented on CARBONDATA-904:
--

@Rahul : Ya its working fine.

> ArrayIndexOutOfBoundsException 
> ---
>
> Key: CARBONDATA-904
> URL: https://issues.apache.org/jira/browse/CARBONDATA-904
> Project: CarbonData
>  Issue Type: Bug
> Environment: Spark1.6
>Reporter: SWATI RAO
>Assignee: Rahul Kumar
> Fix For: 1.1.1
>
> Attachments: Test_Data1_h1.csv, Test_Data1_h1.csv
>
>
> Or operator is not working properly.
> When we execute these query in hive it is working fine but when we execute 
> the same in carbondata it throws an exception:
> java.lang.ArrayIndexOutOfBoundsException
> HIVE:
> 0: jdbc:hive2://hadoop-master:1> create table Test_Boundary_h1 (c1_int 
> int,c2_Bigint Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string 
> string,c6_Timestamp Timestamp,c7_Datatype_Desc string) ROW FORMAT DELIMITED 
> FIELDS TERMINATED BY ',' ;
> +-+--+
> | result  |
> +-+--+
> +-+--+
> No rows selected (1.177 seconds)
> 0: jdbc:hive2://hadoop-master:1> load data local inpath 
> '/opt/Carbon/CarbonData/TestData/Data/Test_Data1_h1.csv' OVERWRITE INTO TABLE 
> Test_Boundary_h1 ;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (0.437 seconds)
> 0: jdbc:hive2://hadoop-master:1> select c6_Timestamp,max(c6_Timestamp) 
> from Test_Boundary_h1 where c6_Timestamp ='2017-07-01 12:07:28' or 
> c6_Timestamp ='2019-07-05 13:07:30' or c6_Timestamp = '1999-01-06 10:05:29' 
> group by c6_Timestamp ;
> +++--+
> |  c6_Timestamp  |  _c1   |
> +++--+
> | 2017-07-01 12:07:28.0  | 2017-07-01 12:07:28.0  |
> +++--+
> 1 row selected (1.637 seconds)
> CARBONDATA:
> 0: jdbc:hive2://hadoop-master:1> create table Test_Boundary (c1_int 
> int,c2_Bigint Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string 
> string,c6_Timestamp Timestamp,c7_Datatype_Desc string) STORED BY 
> 'org.apache.carbondata.format' ;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (4.48 seconds)
> 0: jdbc:hive2://hadoop-master:1> LOAD DATA INPATH 
> 'hdfs://192.168.2.145:54310/BabuStore/Data/Test_Data1.csv' INTO table 
> Test_Boundary 
> OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='')
>  ;
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (4.445 seconds)
> 0: jdbc:hive2://hadoop-master:1> select c6_Timestamp,max(c6_Timestamp) 
> from Test_Boundary where c6_Timestamp ='2017-07-01 12:07:28' or c6_Timestamp 
> =' 2019-07-05 13:07:30' or c6_Timestamp = '1999-01-06 10:05:29' group by 
> c6_Timestamp ;
> Error: org.apache.spark.SparkException: Job aborted due to stage failure: 
> Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in 
> stage 5.0 (TID 8, hadoop-master): java.lang.RuntimeException: 
> java.util.concurrent.ExecutionException: 
> java.lang.ArrayIndexOutOfBoundsException: 0
>   at 
> org.apache.carbondata.core.scan.processor.AbstractDataBlockIterator.updateScanner(AbstractDataBlockIterator.java:136)
>   at 
> org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:50)
>   at 
> org.apache.carbondata.core.scan.processor.impl.DataBlockIteratorImpl.next(DataBlockIteratorImpl.java:32)
>   at 
> org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.getBatchResult(DetailQueryResultIterator.java:50)
>   at 
> org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:41)
>   at 
> org.apache.carbondata.core.scan.result.iterator.DetailQueryResultIterator.next(DetailQueryResultIterator.java:31)
>   at 
> org.apache.carbondata.core.scan.result.iterator.ChunkRowIterator.(ChunkRowIterator.java:41)
>   at 
> org.apache.carbondata.hadoop.CarbonRecordReader.initialize(CarbonRecordReader.java:79)
>   at 
> org.apache.carbondata.spark.rdd.CarbonScanRDD.compute(CarbonScanRDD.scala:204)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>   at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>   at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>   at

[jira] [Created] (CARBONDATA-1059) While executing Select Query getting an exception : "org.apache.hadoop.hive.ql.metadata.HiveException"

2017-05-16 Thread SWATI RAO (JIRA)

SWATI RAO created CARBONDATA-1059:
-

 Summary: While executing Select Query getting an exception : 
"org.apache.hadoop.hive.ql.metadata.HiveException"
 Key: CARBONDATA-1059
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1059
 Project: CarbonData
  Issue Type: Bug
Affects Versions: 1.1.0
 Environment: SPARK1.6
Reporter: SWATI RAO
 Attachments: Test_Data1.csv, Test_Data1_h1.csv

CARBONDATA:
CREATE:
0: jdbc:hive2://192.168.2.126:1> create table Test_Boundary (c1_int 
int,c2_Bigint Bigint,c3_Decimal Decimal(38,30),c4_double double,c5_string 
string,c6_Timestamp Timestamp,c7_Datatype_Desc string) STORED BY 
'org.apache.carbondata.format'
0: jdbc:hive2://192.168.2.126:1> ;
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (0.28 seconds)
LOAD:
0: jdbc:hive2://192.168.2.126:1> LOAD DATA INPATH 
'hdfs://localhost:54311/BabuStore/TestData/Data/Test_Data1.csv' INTO table 
Test_Boundary 
OPTIONS('DELIMITER'=',','QUOTECHAR'='','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='');
+-+--+
| Result  |
+-+--+
+-+--+
No rows selected (1.912 seconds)

SELECT :
0: jdbc:hive2://192.168.2.126:1> select 
min(c1_int),max(c1_int),sum(c1_int),avg(c1_int) , count(c1_int), 
variance(c1_int) from Test_Boundary where rand(c1_int)=0.6201007799387834 or 
rand(c1_int)=0.45540022789662593
0: jdbc:hive2://192.168.2.126:1> ;
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 
0 in stage 20.0 failed 1 times, most recent failure: Lost task 0.0 in stage 
20.0 (TID 21, localhost): org.apache.hadoop.hive.ql.metadata.HiveException: 
Unable to execute method public org.apache.hadoop.hive.serde2.io.DoubleWritable 
org.apache.hadoop.hive.ql.udf.UDFRand.evaluate(org.apache.hadoop.io.LongWritable)
  on object org.apache.hadoop.hive.ql.udf.UDFRand@160c41fd of class 
org.apache.hadoop.hive.ql.udf.UDFRand with arguments {null} of size 1
at 
org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:981)
at org.apache.spark.sql.hive.HiveSimpleUDF.eval(hiveUDFs.scala:185)
at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate.eval(Unknown
 Source)
at 
org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate$$anonfun$create$2.apply(GeneratePredicate.scala:68)
at 
org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate$$anonfun$create$2.apply(GeneratePredicate.scala:68)
at 
org.apache.spark.sql.execution.Filter$$anonfun$2$$anonfun$apply$2.apply(basicOperators.scala:74)
at 
org.apache.spark.sql.execution.Filter$$anonfun$2$$anonfun$apply$2.apply(basicOperators.scala:72)
at scala.collection.Iterator$$anon$14.hasNext(Iterator.scala:390)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
at 
org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.processInputs(TungstenAggregationIterator.scala:504)
at 
org.apache.spark.sql.execution.aggregate.TungstenAggregationIterator.(TungstenAggregationIterator.scala:686)
at 
org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:95)
at 
org.apache.spark.sql.execution.aggregate.TungstenAggregate$$anonfun$doExecute$1$$anonfun$2.apply(TungstenAggregate.scala:86)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at

[jira] [Created] (CARBONDATA-1057) Incorrect result displays while using (>=) and (<=) operators in range filter

2017-05-16 Thread Vandana Yadav (JIRA)

Vandana Yadav created CARBONDATA-1057:
-

 Summary: Incorrect result displays while using (>=) and (<=) 
operators in range filter
 Key: CARBONDATA-1057
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1057
 Project: CarbonData
  Issue Type: Bug
  Components: data-query
Affects Versions: 1.1.0
 Environment: spark 2.1
Reporter: Vandana Yadav
Priority: Minor
 Attachments: 2000_UniqData.csv

Incorrect result displays while using (>=) and (<=) operators in range filter

Steps to reproduce:

1)Create Table:
CREATE TABLE uniqdata (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION string, 
DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
int) STORED BY 'org.apache.carbondata.format' TBLPROPERTIES ("TABLE_BLOCKSIZE"= 
"256 MB")

2)Load Data:
LOAD DATA INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table uniqdata 
OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1')

3)Execute Query:
select dob from uniqdata where dob>='1972-12-23' and dob <= '1972-12-31'

4) Actual Result:
In Carbondata:
+--+--+
| dob  |
+--+--+
+--+--+
No rows selected (0.259 seconds)

In Hive:
++--+
|  dob   |
++--+
| 1972-12-23 01:00:03.0  |
| 1972-12-24 01:00:03.0  |
| 1972-12-25 01:00:03.0  |
| 1972-12-26 01:00:03.0  |
| 1972-12-27 01:00:03.0  |
| 1972-12-28 01:00:03.0  |
| 1972-12-29 01:00:03.0  |
| 1972-12-30 01:00:03.0  |
++--+
8 rows selected (0.131 seconds)

5) Expected result: It should display all dates between the given range 




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CARBONDATA-1046) Single_pass_loading is throwing an error in Spark1.6 in automation

2017-05-16 Thread Liang Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liang Chen updated CARBONDATA-1046:
---
Fix Version/s: (was: 1.1.0)
   1.2.0

> Single_pass_loading is throwing an error in Spark1.6 in automation
> --
>
> Key: CARBONDATA-1046
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1046
> Project: CarbonData
>  Issue Type: Bug
> Environment: Spark1.6
>Reporter: SWATI RAO
> Fix For: 1.2.0
>
> Attachments: 7000_UniqData.csv
>
>
> Steps to Reproduce :
> Create Table :
> CREATE TABLE uniqdata_INCLUDEDICTIONARY (CUST_ID int,CUST_NAME 
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, 
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), 
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 
> double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (1.709 seconds)
> Load Query :
> LOAD DATA INPATH 
> 'hdfs://hadoop-master:54310/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
> table uniqdata_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');
> Stack Trace :
> INFO  11-05 13:54:45,047 - Running query 'LOAD DATA INPATH 
> 'hdfs://hadoop-master:54310/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
> table uniqdata_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true')'
>  with 44e92bcb-f9e1-4b2e-835e-a82eae525fe4
> INFO  11-05 13:54:45,047 - pool-31-thread-3 Query [LOAD DATA INPATH 
> 'HDFS://HADOOP-MASTER:54310/BABUSTORE/DATA/UNIQDATA/7000_UNIQDATA.CSV' INTO 
> TABLE UNIQDATA_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,DOUBLE_COLUMN1,DOUBLE_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='TRUE')]
> INFO  11-05 13:54:45,065 - pool-31-thread-3 HDFS lock 
> path:hdfs://192.168.2.145:54310/opt/olapcontent/default/uniqdata_includedictionary/meta.lock
> INFO  11-05 13:54:45,097 - Successfully able to get the table metadata file 
> lock
> INFO  11-05 13:54:45,099 - pool-31-thread-3 Initiating Direct Load for the 
> Table : (default.uniqdata_includedictionary)
> AUDIT 11-05 13:54:45,100 - [hadoop-master][hduser][Thread-150]Data load 
> request has been received for table default.uniqdata_includedictionary
> AUDIT 11-05 13:54:45,100 - [hadoop-master][hduser][Thread-150]Data is loading 
> with New Data Flow for table default.uniqdata_includedictionary
> ERROR 11-05 13:54:45,104 - Dictionary server Dictionary Server Start Failed
> java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:433)
>   at sun.nio.ch.Net.bind(Net.java:425)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
>   at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
>   at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
>   at 
> io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
>   at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
>   at 
> io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
>   at 
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
>   at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
>   at 
>

[jira] [Commented] (CARBONDATA-1046) Single_pass_loading is throwing an error in Spark1.6 in automation

2017-05-16 Thread SWATI RAO (JIRA)


[ 
https://issues.apache.org/jira/browse/CARBONDATA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16012124#comment-16012124
 ] 

SWATI RAO commented on CARBONDATA-1046:
---

Above issue was resolved but data load failure occurred now, when executing the 
query in automation while same query is running fine when we execute it 
manually.

> Single_pass_loading is throwing an error in Spark1.6 in automation
> --
>
> Key: CARBONDATA-1046
> URL: https://issues.apache.org/jira/browse/CARBONDATA-1046
> Project: CarbonData
>  Issue Type: Bug
> Environment: Spark1.6
>Reporter: SWATI RAO
> Fix For: 1.1.0
>
> Attachments: 7000_UniqData.csv
>
>
> Steps to Reproduce :
> Create Table :
> CREATE TABLE uniqdata_INCLUDEDICTIONARY (CUST_ID int,CUST_NAME 
> String,ACTIVE_EMUI_VERSION string, DOB timestamp, DOJ timestamp, 
> BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 bigint,DECIMAL_COLUMN1 decimal(30,10), 
> DECIMAL_COLUMN2 decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 
> double,INTEGER_COLUMN1 int) STORED BY 'org.apache.carbondata.format' 
> TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
> +-+--+
> | Result  |
> +-+--+
> +-+--+
> No rows selected (1.709 seconds)
> Load Query :
> LOAD DATA INPATH 
> 'hdfs://hadoop-master:54310/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
> table uniqdata_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');
> Stack Trace :
> INFO  11-05 13:54:45,047 - Running query 'LOAD DATA INPATH 
> 'hdfs://hadoop-master:54310/BabuStore/Data/uniqdata/7000_UniqData.csv' into 
> table uniqdata_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true')'
>  with 44e92bcb-f9e1-4b2e-835e-a82eae525fe4
> INFO  11-05 13:54:45,047 - pool-31-thread-3 Query [LOAD DATA INPATH 
> 'HDFS://HADOOP-MASTER:54310/BABUSTORE/DATA/UNIQDATA/7000_UNIQDATA.CSV' INTO 
> TABLE UNIQDATA_INCLUDEDICTIONARY OPTIONS('DELIMITER'=',' , 
> 'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE', 
> 'BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,DOUBLE_COLUMN1,DOUBLE_COLUMN2,INTEGER_COLUMN1','SINGLE_PASS'='TRUE')]
> INFO  11-05 13:54:45,065 - pool-31-thread-3 HDFS lock 
> path:hdfs://192.168.2.145:54310/opt/olapcontent/default/uniqdata_includedictionary/meta.lock
> INFO  11-05 13:54:45,097 - Successfully able to get the table metadata file 
> lock
> INFO  11-05 13:54:45,099 - pool-31-thread-3 Initiating Direct Load for the 
> Table : (default.uniqdata_includedictionary)
> AUDIT 11-05 13:54:45,100 - [hadoop-master][hduser][Thread-150]Data load 
> request has been received for table default.uniqdata_includedictionary
> AUDIT 11-05 13:54:45,100 - [hadoop-master][hduser][Thread-150]Data is loading 
> with New Data Flow for table default.uniqdata_includedictionary
> ERROR 11-05 13:54:45,104 - Dictionary server Dictionary Server Start Failed
> java.net.BindException: Address already in use
>   at sun.nio.ch.Net.bind0(Native Method)
>   at sun.nio.ch.Net.bind(Net.java:433)
>   at sun.nio.ch.Net.bind(Net.java:425)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
>   at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
>   at 
> io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
>   at 
> io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
>   at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
>   at 
> io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
>   at 
> io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
>   at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
>   at 
> io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
>   at 
>

[jira] [Created] (CARBONDATA-1056) Data_load failure using single_pass true with spark 2.1

2017-05-16 Thread Vandana Yadav (JIRA)

Vandana Yadav created CARBONDATA-1056:
-

 Summary: Data_load failure using single_pass true with spark 2.1
 Key: CARBONDATA-1056
 URL: https://issues.apache.org/jira/browse/CARBONDATA-1056
 Project: CarbonData
  Issue Type: Bug
  Components: data-load
Affects Versions: 1.1.0
 Environment: spark 2.1
Reporter: Vandana Yadav
Priority: Minor
 Attachments: 2000_UniqData.csv

Data_load failure using single_pass true with spark 2.1

Steps to reproduce:

1)Create Table:
CREATE TABLE uniq_exclude_sp1 (CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION 
string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
int) STORED BY 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_EXCLUDE'='CUST_NAME,ACTIVE_EMUI_VERSION');

2) Load Data:
LOAD DATA INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table 
uniq_exclude_sp1 OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true');

3)Result:
Actual result on beeline:
Error: java.lang.Exception: Dataload failed due to error while writing 
dictionary file! (state=,code=0)

Expected Result: data should be load successfully 

4)Thriftserver logs:
17/05/16 16:07:20 INFO SparkExecuteStatementOperation: Running query 'LOAD DATA 
INPATH 'hdfs://localhost:54310/2000_UniqData.csv' into table uniq_exclude_sp1 
OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true')'
 with 34eb7e9e-bd49-495c-af68-8f0b5e36b786
17/05/16 16:07:20 INFO CarbonSparkSqlParser: Parsing command: LOAD DATA INPATH 
'hdfs://localhost:54310/2000_UniqData.csv' into table uniq_exclude_sp1 
OPTIONS('DELIMITER'=',' , 
'QUOTECHAR'='"','BAD_RECORDS_ACTION'='FORCE','FILEHEADER'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1','SINGLE_Pass'='true')
17/05/16 16:07:20 INFO CarbonLateDecodeRule: pool-23-thread-4 Skip 
CarbonOptimizer
17/05/16 16:07:20 INFO HdfsFileLock: pool-23-thread-4 HDFS lock 
path:hdfs://localhost:54310/opt/prestocarbonStore/default/uniq_exclude_sp1/meta.lock
17/05/16 16:07:20 INFO LoadTable: pool-23-thread-4 Successfully able to get the 
table metadata file lock
17/05/16 16:07:20 INFO LoadTable: pool-23-thread-4 Initiating Direct Load for 
the Table : (default.uniq_exclude_sp1)
17/05/16 16:07:20 AUDIT CarbonDataRDDFactory$: 
[knoldus][hduser][Thread-137]Data load request has been received for table 
default.uniq_exclude_sp1
17/05/16 16:07:20 INFO CommonUtil$: pool-23-thread-4 [Block Distribution]
17/05/16 16:07:20 INFO CommonUtil$: pool-23-thread-4 totalInputSpaceConsumed: 
376223 , defaultParallelism: 4
17/05/16 16:07:20 INFO CommonUtil$: pool-23-thread-4 
mapreduce.input.fileinputformat.split.maxsize: 16777216
17/05/16 16:07:20 INFO FileInputFormat: Total input paths to process : 1
17/05/16 16:07:20 INFO DistributionUtil$: pool-23-thread-4 Executors configured 
: 1
17/05/16 16:07:20 INFO DistributionUtil$: pool-23-thread-4 Total Time taken to 
ensure the required executors : 0
17/05/16 16:07:20 INFO DistributionUtil$: pool-23-thread-4 Time elapsed to 
allocate the required executors: 0
17/05/16 16:07:20 INFO CarbonDataRDDFactory$: pool-23-thread-4 Total Time taken 
in block allocation: 1
17/05/16 16:07:20 INFO CarbonDataRDDFactory$: pool-23-thread-4 Total no of 
blocks: 1, No.of Nodes: 1
17/05/16 16:07:20 INFO CarbonDataRDDFactory$: pool-23-thread-4 #Node: knoldus 
no.of.blocks: 1

17/05/16 16:07:20 INFO MemoryStore: Block broadcast_2 stored as values in 
memory (estimated size 53.7 MB, free 291.4 MB)
17/05/16 16:07:20 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in 
memory (estimated size 23.2 KB, free 291.4 MB)
17/05/16 16:07:20 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 
192.168.1.10:42046 (size: 23.2 KB, free: 366.2 MB)
17/05/16 16:07:20 INFO SparkContext: Created broadcast 2 from broadcast at 
NewCarbonDataLoadRDD.scala:185
17/05/16 16:07:20 INFO SparkContext: Starting job: collect at 
CarbonDataRDDFactory.scala:630
17/05/16 16:07:20 INFO DAGScheduler: Got job 1 (collect at 
CarbonDataRDDFactory.scala:630) with 1 output partitions
17/05/16 16:07:20 INFO DAGScheduler: Final stage: ResultStage 1 (collect at 
CarbonDataRDDFactory.scala:630)
17/05/16 16:07:20 INFO DAGScheduler: Parents of final stage: List()
17/05/16 16:07:20 INFO DAGScheduler:

[jira] [Assigned] (CARBONDATA-776) Alter table support for spark 1.6

2017-05-16 Thread Manish Gupta (JIRA)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manish Gupta reassigned CARBONDATA-776:
---

Assignee: Naresh P R

> Alter table support for spark 1.6
> -
>
> Key: CARBONDATA-776
> URL: https://issues.apache.org/jira/browse/CARBONDATA-776
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Manish Gupta
>Assignee: Naresh P R
>Priority: Minor
>
> Alter feature need to be supported for spark 1.6



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] carbondata issue #877: [CARBONDATA-951] create table ddl can specify a bitma...

2017-05-16 Thread CarbonDataQA

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/877
  
Build Failed  with Spark 1.6.2, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2046/



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] carbondata issue #916: [CARBONDATA-938] Prune partitions for filter query on...

[GitHub] carbondata pull request #915: [CARBONDATA-946] Spark 2x tupleId support for ...

[GitHub] carbondata pull request #915: [CARBONDATA-946] Spark 2x tupleId support for ...

[jira] [Assigned] (CARBONDATA-1056) Data_load failure using single_pass true with spark 2.1

[GitHub] carbondata issue #917: [CARBONDATA-1053] supported char type for hive

[jira] [Assigned] (CARBONDATA-852) Less than or equal to operator(<=) does not work properly in Range Filter.

[jira] [Closed] (CARBONDATA-904) ArrayIndexOutOfBoundsException

[jira] [Commented] (CARBONDATA-904) ArrayIndexOutOfBoundsException

[jira] [Created] (CARBONDATA-1059) While executing Select Query getting an exception : "org.apache.hadoop.hive.ql.metadata.HiveException"

[jira] [Created] (CARBONDATA-1057) Incorrect result displays while using (>=) and (<=) operators in range filter

[jira] [Updated] (CARBONDATA-1046) Single_pass_loading is throwing an error in Spark1.6 in automation

[jira] [Commented] (CARBONDATA-1046) Single_pass_loading is throwing an error in Spark1.6 in automation

[jira] [Created] (CARBONDATA-1056) Data_load failure using single_pass true with spark 2.1

[jira] [Assigned] (CARBONDATA-776) Alter table support for spark 1.6

[GitHub] carbondata issue #877: [CARBONDATA-951] create table ddl can specify a bitma...

15 matches

Site Navigation

Mail list logo

Footer information