Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1942#discussion_r168395269
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/steps/DataConverterProcessorStepImpl.java
---
@@ -100,10 +103,17 @@ public void initialize() throws IOException {
* @return processed row.
*/
protected CarbonRowBatch processRowBatch(CarbonRowBatch rowBatch,
RowConverter localConverter) {
+ String badRecordsAction = (String)
configuration.getDataLoadProperty(BAD_RECORDS_LOGGER_ACTION);
while (rowBatch.hasNext()) {
CarbonRow convertRow = localConverter.convert(rowBatch.next());
- rowBatch.setPreviousRow(convertRow);
+ if (convertRow == null &&
(badRecordsAction.equals(LoggerAction.IGNORE.toString()) ||
+ badRecordsAction.equals(LoggerAction.REDIRECT.toString()))) {
--- End diff --
@geetikagupta16 and @QiangCai
In case of bucketing and Stream Record writer also the same exist.
Please correct there as well.
For your reference:
1. For bucketing.
org/apache/carbondata/processing/loading/steps/DataConverterProcessorWithBucketingStepImpl.java:128
2. For Stream Record Writer
org/apache/carbondata/hadoop/streaming/CarbonStreamRecordWriter.java:191
---
