Priyal Sachdeva created CARBONDATA-660:
------------------------------------------
Summary: Bad Records Logs and Raw CSVs should get display under
segment id instead of Tasks id
Key: CARBONDATA-660
URL: https://issues.apache.org/jira/browse/CARBONDATA-660
Project: CarbonData
Issue Type: Improvement
Components: data-load
Reporter: Priyal Sachdeva
Priority: Minor
create table if not exists Badrecords_test (imei string,AMSize int) STORED BY
'org.apache.carbondata.format';
LOAD DATA INPATH 'hdfs://hacluster/CSVs/bad_records.csv' into table
Badrecords_test OPTIONS('DELIMITER'=',' ,
'QUOTECHAR'='"','BAD_RECORDS_LOGGER_ENABLE'='TRUE',
'BAD_RECORDS_ACTION'='REDIRECT','FILEHEADER'='imei,AMSize');
Bad Records Logs and raw csvs are getting display under Task ID
linux-61:/srv/OSCON/BigData/HACluster/install/hadoop/datanode #
bin/hadoop fs -ls /tmp/carbon/default/badrecords_test
drwxr-xr-x - root users 0 2017-01-18 21:08
/tmp/carbon/default/badrecords_test/0--------------------------->Task ID
0: jdbc:hive2://172.168.100.205:23040> show segments for table Badrecords_test;
+--------------------+------------------+--------------------------+--------------------------+--+
| SegmentSequenceId | Status | Load Start Time | Load
End Time |
+--------------------+------------------+--------------------------+--------------------------+--+
| 8 | Partial Success | 2017-01-18 21:12:58.018 | 2017-01-18
21:12:59.652 |
| 7 | Partial Success | 2017-01-18 21:08:07.426 | 2017-01-18
21:08:11.791 |
| 6 | Partial Success | 2017-01-18 21:07:07.645 | 2017-01-18
21:07:08.747 |
| 5 | Partial Success | 2017-01-18 19:34:16.163 | 2017-01-18
19:34:18.163 |
| 4 | Partial Success | 2017-01-18 19:34:13.669 | 2017-01-18
19:34:15.811 |
| 3 | Partial Success | 2017-01-18 19:30:18.753 | 2017-01-18
19:30:19.644 |
| 2 | Partial Success | 2017-01-18 19:30:13.508 | 2017-01-18
19:30:15.578 |
| 1 | Partial Success | 2017-01-18 19:18:54.787 | 2017-01-18
19:18:54.94 |
| 0 | Partial Success | 2017-01-18 19:18:53.741 | 2017-01-18
19:18:54.614 |
+--------------------+------------------+--------------------------+--------------------------+--+
Bad Records Logs and raw csvs are getting display under Task ID. It would be
good to have the information of bad records as per the load i.e under segment
id..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
