GitHub user ajantha-bhat opened a pull request:
https://github.com/apache/carbondata/pull/3003
[WIP]Avoid creating empty carbondata file when all the records are bad
record with action redirect.
**problem:** In the no_sort flow, writer will be open as there is no
blocking sort step.
So, when all the record goes as bad record with redirect in converted step.
writer is closing the empty .carbondata file.
when this empty carbondata file is queried , we get multiple issues
including NPE.
**solution:** When the file size is 0 bytes. do the following
a) If one data and one index file -- delete carbondata file and avoid index
file creation
b) If multiple data and one index file (with few data file is full of bad
recod) -- delete carbondata files, remove them from blockIndexInfoList, so
index file not will not have that info of empty carbon files
c) In case direct write to store path is enable. need to delete data file
from there and avoid writing index file with that carbondata in info.
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed? NA
- [ ] Any backward compatibility impacted? NA
- [ ] Document update required? NA
- [ ] Testing done
Added UT
- [ ] For large changes, please consider breaking it into sub-tasks under
an umbrella JIRA. NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ajantha-bhat/carbondata issue_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/3003.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3003
----
commit 7701f0488b886d8db679bb8b69aee79bc515fe4f
Author: ajantha-bhat <ajanthabhat@...>
Date: 2018-12-19T12:57:53Z
Avoid creating empty carbondata file when all the records are bad record
with action redirect.
----
---
