shaofengshi closed pull request #404: KYLIN-3699 Spark cubing failed when build
with empty data
URL: https://github.com/apache/kylin/pull/404
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-common/src/main/java/org/apache/kylin/common/util/HadoopUtil.java
b/core-common/src/main/java/org/apache/kylin/common/util/HadoopUtil.java
index 5d09ea7347..4afce58a23 100644
--- a/core-common/src/main/java/org/apache/kylin/common/util/HadoopUtil.java
+++ b/core-common/src/main/java/org/apache/kylin/common/util/HadoopUtil.java
@@ -256,8 +256,8 @@ public boolean accept(Path path) {
if (fileStatuses != null && fileStatuses.length > 0) {
return isSequenceFile(conf, fileStatuses[0].getPath());
+ } else {
+ return true;
}
-
- return false;
}
}
diff --git
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
index be5bdbf44b..58f0e66419 100644
---
a/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
+++
b/engine-mr/src/main/java/org/apache/kylin/engine/mr/common/CubeStatsReader.java
@@ -299,7 +299,7 @@ public double estimateLayerSize(int level) {
Map<Long, Double> cuboidSizeMap = getCuboidSizeMap();
double ret = 0;
for (Long cuboidId : layeredCuboids.get(level)) {
- ret += cuboidSizeMap.get(cuboidId);
+ ret += cuboidSizeMap.get(cuboidId) == null ? 0.0 :
cuboidSizeMap.get(cuboidId);
}
logger.info("Estimating size for layer {}, all cuboids are {}, total
size is {}", level,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
