This is an automated email from the ASF dual-hosted git repository.
shaofengshi pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kylin.git.
from 050f1c1 KYLIN-3485 Make unloading table more flexible
new 9a92f49 KYLIN-3471 Merge dictionary using Mapreduce
new 74ac1cd KYLIN-3471 Merge dictionary using Spark
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../kylin/job/constant/ExecutableConstants.java | 1 +
.../kylin/engine/mr/BatchMergeJobBuilder2.java | 30 +-
.../apache/kylin/engine/mr/JobBuilderSupport.java | 36 ++-
.../kylin/engine/mr/common/AbstractHadoopJob.java | 41 +++
.../kylin/engine/mr/common/BatchConstants.java | 1 +
.../kylin/engine/mr/steps/MergeDictionaryJob.java | 234 ++++++++++++++++
.../engine/mr/steps/MergeDictionaryMapper.java | 207 ++++++++++++++
...kerReducer.java => MergeDictionaryReducer.java} | 40 ++-
.../engine/mr/steps/UpdateDictionaryStep.java | 151 ++++++++++
.../engine/spark/SparkBatchMergeJobBuilder2.java | 39 ++-
.../apache/kylin/engine/spark/SparkExecutable.java | 7 +-
.../kylin/engine/spark/SparkMergingDictionary.java | 307 +++++++++++++++++++++
12 files changed, 1050 insertions(+), 44 deletions(-)
create mode 100644
engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/MergeDictionaryJob.java
create mode 100644
engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/MergeDictionaryMapper.java
copy
engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/{RowKeyDistributionCheckerReducer.java
=> MergeDictionaryReducer.java} (55%)
create mode 100644
engine-mr/src/main/java/org/apache/kylin/engine/mr/steps/UpdateDictionaryStep.java
create mode 100644
engine-spark/src/main/java/org/apache/kylin/engine/spark/SparkMergingDictionary.java
