Github user mohammadshahidkhan commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1942#discussion_r168394055
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/loading/steps/DataConverterProcessorStepImpl.java
---
@@ -100,10 +103,17 @@ public void initialize() throws IOException {
* @return processed row.
*/
protected CarbonRowBatch processRowBatch(CarbonRowBatch rowBatch,
RowConverter localConverter) {
+ String badRecordsAction = (String)
configuration.getDataLoadProperty(BAD_RECORDS_LOGGER_ACTION);
while (rowBatch.hasNext()) {
CarbonRow convertRow = localConverter.convert(rowBatch.next());
- rowBatch.setPreviousRow(convertRow);
+ if (convertRow == null &&
(badRecordsAction.equals(LoggerAction.IGNORE.toString()) ||
+ badRecordsAction.equals(LoggerAction.REDIRECT.toString()))) {
--- End diff --
Please remove check
(badRecordsAction.equals(LoggerAction.IGNORE.toString()) ||
badRecordsAction.equals(LoggerAction.REDIRECT.toString())
BadRecordsAction check is not required, only null check over convertRow is
enough.
The convert method return's null only in case of bad_records_action is
either IGNORE or REDIRECT.
---
