This is an automated email from the ASF dual-hosted git repository.
jackylk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/carbondata.git
The following commit(s) were added to refs/heads/master by this push:
new 994034b [CARBONDATA-3756] Fix stage query bug it only read the first
blocklet of each carbondata file
994034b is described below
commit 994034b0477fe2b846699421161d6c5fcd5120be
Author: QiangCai <[email protected]>
AuthorDate: Fri Mar 27 16:35:38 2020 +0800
[CARBONDATA-3756] Fix stage query bug it only read the first blocklet of
each carbondata file
Why is this PR needed?
The query of stage files only read the first blocklet of each carbondata
file.
So when the file contains multiple blocklets, the query result will be
wrong.
What changes were proposed in this PR?
The query of stage files should read the all blocklets of all carbondata
files.
Does this PR introduce any user interface change?
No
Is any new testcase added?
No
This closes #3684
---
.../org/apache/carbondata/core/util/CarbonUtil.java | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
b/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
index 41902b6..e56b8b1 100644
--- a/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
+++ b/core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java
@@ -927,14 +927,22 @@ public final class CarbonUtil {
AbstractDataFileFooterConverter footerConverter =
DataFileFooterConverterFactory.getInstance().getDataFileFooterConverter(version);
List<DataFileFooter> footers = footerConverter.getIndexInfo(indexFilePath,
null, true);
-
+ DataFileFooter blockFooter = null;
// find the footer of the input data file (tableBlockInfo)
- for (DataFileFooter footer : footers) {
- if (footer.getBlockInfo().getFilePath().equals(dataFilePath)) {
- return footer;
+ for (DataFileFooter blockletFooter : footers) {
+ if (blockletFooter.getBlockInfo().getFilePath().equals(dataFilePath)) {
+ if (blockFooter == null) {
+ blockFooter = blockletFooter;
+ } else {
+
blockFooter.getBlockletList().addAll(blockletFooter.getBlockletList());
+ }
}
}
- throw new RuntimeException("Footer not found in index file");
+ if (blockFooter == null) {
+ throw new RuntimeException("Footer not found in index file");
+ } else {
+ return blockFooter;
+ }
}
/**
