[41/41] hbase git commit: HBASE-18640 Move mapreduce out of hbase-server into separate module.

2017-08-26 Thread appy 
HBASE-18640 Move mapreduce out of hbase-server into separate module.

- Moves out o.a.h.h.{mapred, mapreduce} to new hbase-mapreduce module which 
depends
  on hbase-server because of classes like *Snapshot{Input,Output}Format.java, 
WALs, replication, etc
- hbase-backup depends on it for WALPlayer and MR job stuff
- A bunch of tools needed to be pulled into hbase-mapreduce becuase of their 
dependencies on MR.
  These are: CompactionTool, LoadTestTool, PerformanceEvaluation, ExportSnapshot
  This is better place of them than hbase-server. But ideal place would be in 
separate hbase-tools module.
- There were some tests in hbase-server which were digging into these tools for 
static util funtions or
  confs. Moved these to better/easily shared place. For eg. security related 
stuff to HBaseKerberosUtils.
- Note that hbase-mapreduce has secondPartExecution tests. On my machine they 
took like 20 min, so maybe
  more on apache jenkins. That's basically equal reduction of runtime of 
hbase-server tests, which is a
  big win!

Change-Id: Ieeb7235014717ca83ee5cb13b2a27fddfa6838e8


Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/59d03410
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/59d03410
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/59d03410

Branch: refs/heads/branch-2
Commit: 59d034106f9309190066eceac794f2d9d61b2575
Parents: 95bc464
Author: Apekshit Sharma 
Authored: Sun Aug 20 14:34:16 2017 -0700
Committer: Apekshit Sharma 
Committed: Sat Aug 26 01:55:33 2017 -0700

--
 hbase-assembly/pom.xml  |4 +
 .../src/main/assembly/hadoop-two-compat.xml |1 +
 hbase-assembly/src/main/assembly/src.xml|1 +
 hbase-backup/pom.xml|   10 +
 hbase-examples/pom.xml  |4 +
 hbase-it/pom.xml|   16 +
 .../hadoop/hbase/IntegrationTestIngest.java |5 +-
 .../IntegrationTestIngestStripeCompactions.java |4 +-
 .../hbase/IntegrationTestIngestWithMOB.java |5 +-
 .../hbase/IntegrationTestRegionReplicaPerf.java |3 +-
 .../mapreduce/IntegrationTestImportTsv.java |1 -
 .../test/IntegrationTestLoadAndVerify.java  |2 +-
 hbase-mapreduce/pom.xml |  316 +++
 .../org/apache/hadoop/hbase/mapred/Driver.java  |   52 +
 .../hadoop/hbase/mapred/GroupingTableMap.java   |  157 ++
 .../hadoop/hbase/mapred/HRegionPartitioner.java |   95 +
 .../hadoop/hbase/mapred/IdentityTableMap.java   |   76 +
 .../hbase/mapred/IdentityTableReduce.java   |   61 +
 .../mapred/MultiTableSnapshotInputFormat.java   |  128 +
 .../apache/hadoop/hbase/mapred/RowCounter.java  |  121 +
 .../hadoop/hbase/mapred/TableInputFormat.java   |   90 +
 .../hbase/mapred/TableInputFormatBase.java  |  313 +++
 .../apache/hadoop/hbase/mapred/TableMap.java|   38 +
 .../hadoop/hbase/mapred/TableMapReduceUtil.java |  376 +++
 .../hadoop/hbase/mapred/TableOutputFormat.java  |  134 +
 .../hadoop/hbase/mapred/TableRecordReader.java  |  139 +
 .../hbase/mapred/TableRecordReaderImpl.java |  259 ++
 .../apache/hadoop/hbase/mapred/TableReduce.java |   38 +
 .../hbase/mapred/TableSnapshotInputFormat.java  |  166 ++
 .../apache/hadoop/hbase/mapred/TableSplit.java  |  154 +
 .../hadoop/hbase/mapred/package-info.java   |   26 +
 .../hadoop/hbase/mapreduce/CellCounter.java |  333 +++
 .../hadoop/hbase/mapreduce/CellCreator.java |  134 +
 .../hadoop/hbase/mapreduce/CopyTable.java   |  386 +++
 .../DefaultVisibilityExpressionResolver.java|  144 +
 .../apache/hadoop/hbase/mapreduce/Driver.java   |   64 +
 .../apache/hadoop/hbase/mapreduce/Export.java   |  197 ++
 .../hbase/mapreduce/GroupingTableMapper.java|  177 ++
 .../hbase/mapreduce/HFileInputFormat.java   |  174 ++
 .../hbase/mapreduce/HFileOutputFormat2.java |  902 ++
 .../hbase/mapreduce/HRegionPartitioner.java |  140 +
 .../hadoop/hbase/mapreduce/HashTable.java   |  747 +
 .../hbase/mapreduce/IdentityTableMapper.java|   67 +
 .../hbase/mapreduce/IdentityTableReducer.java   |   79 +
 .../apache/hadoop/hbase/mapreduce/Import.java   |  780 ++
 .../hadoop/hbase/mapreduce/ImportTsv.java   |  793 ++
 .../hadoop/hbase/mapreduce/JarFinder.java   |  186 ++
 .../hbase/mapreduce/KeyValueSerialization.java  |   88 +
 .../hbase/mapreduce/KeyValueSortReducer.java|   57 +
 .../mapreduce/MultiTableHFileOutputFormat.java  |  122 +
 .../hbase/mapreduce/MultiTableInputFormat.java  |  104 +
 .../mapreduce/MultiTableInputFormatBase.java|  296 ++
 .../hbase/mapreduce/MultiTableOutputFormat.java |  176 ++
 .../MultiTableSnapshotInputFormat.java  |  106 +
 .../MultiTableSnapshotInputFormatImpl.java  |  252 ++
 .../mapreduce/MultithreadedTableMapper.java |  301 ++

[41/41] hbase git commit: HBASE-18640 Move mapreduce out of hbase-server into separate module.

2017-08-25 Thread appy 
HBASE-18640 Move mapreduce out of hbase-server into separate module.

- Moves out o.a.h.h.{mapred, mapreduce} to new hbase-mapreduce module which 
depends
  on hbase-server because of classes like *Snapshot{Input,Output}Format.java, 
WALs, replication, etc
- hbase-backup depends on it for WALPlayer and MR job stuff
- A bunch of tools needed to be pulled into hbase-mapreduce becuase of their 
dependencies on MR.
  These are: CompactionTool, LoadTestTool, PerformanceEvaluation, ExportSnapshot
  This is better place of them than hbase-server. But ideal place would be in 
separate hbase-tools module.
- There were some tests in hbase-server which were digging into these tools for 
static util funtions or
  confs. Moved these to better/easily shared place. For eg. security related 
stuff to HBaseKerberosUtils.
- Note that hbase-mapreduce has secondPartExecution tests. On my machine they 
took like 20 min, so maybe
  more on apache jenkins. That's basically equal reduction of runtime of 
hbase-server tests, which is a
  big win!

Change-Id: Ieeb7235014717ca83ee5cb13b2a27fddfa6838e8


Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/664b6be0
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/664b6be0
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/664b6be0

Branch: refs/heads/master
Commit: 664b6be0ef65218328847ea501fa88cb877e6759
Parents: 8d33949
Author: Apekshit Sharma 
Authored: Sun Aug 20 14:34:16 2017 -0700
Committer: Apekshit Sharma 
Committed: Fri Aug 25 18:38:48 2017 -0700

--
 hbase-assembly/pom.xml  |4 +
 .../src/main/assembly/hadoop-two-compat.xml |1 +
 hbase-assembly/src/main/assembly/src.xml|1 +
 hbase-backup/pom.xml|   10 +
 hbase-examples/pom.xml  |4 +
 hbase-it/pom.xml|   16 +
 .../hadoop/hbase/IntegrationTestIngest.java |5 +-
 .../IntegrationTestIngestStripeCompactions.java |4 +-
 .../hbase/IntegrationTestIngestWithMOB.java |5 +-
 .../hbase/IntegrationTestRegionReplicaPerf.java |3 +-
 .../mapreduce/IntegrationTestImportTsv.java |1 -
 .../test/IntegrationTestLoadAndVerify.java  |2 +-
 hbase-mapreduce/pom.xml |  316 +++
 .../org/apache/hadoop/hbase/mapred/Driver.java  |   52 +
 .../hadoop/hbase/mapred/GroupingTableMap.java   |  157 ++
 .../hadoop/hbase/mapred/HRegionPartitioner.java |   95 +
 .../hadoop/hbase/mapred/IdentityTableMap.java   |   76 +
 .../hbase/mapred/IdentityTableReduce.java   |   61 +
 .../mapred/MultiTableSnapshotInputFormat.java   |  128 +
 .../apache/hadoop/hbase/mapred/RowCounter.java  |  121 +
 .../hadoop/hbase/mapred/TableInputFormat.java   |   90 +
 .../hbase/mapred/TableInputFormatBase.java  |  313 +++
 .../apache/hadoop/hbase/mapred/TableMap.java|   38 +
 .../hadoop/hbase/mapred/TableMapReduceUtil.java |  376 +++
 .../hadoop/hbase/mapred/TableOutputFormat.java  |  134 +
 .../hadoop/hbase/mapred/TableRecordReader.java  |  139 +
 .../hbase/mapred/TableRecordReaderImpl.java |  259 ++
 .../apache/hadoop/hbase/mapred/TableReduce.java |   38 +
 .../hbase/mapred/TableSnapshotInputFormat.java  |  166 ++
 .../apache/hadoop/hbase/mapred/TableSplit.java  |  154 +
 .../hadoop/hbase/mapred/package-info.java   |   26 +
 .../hadoop/hbase/mapreduce/CellCounter.java |  333 +++
 .../hadoop/hbase/mapreduce/CellCreator.java |  134 +
 .../hadoop/hbase/mapreduce/CopyTable.java   |  386 +++
 .../DefaultVisibilityExpressionResolver.java|  144 +
 .../apache/hadoop/hbase/mapreduce/Driver.java   |   64 +
 .../apache/hadoop/hbase/mapreduce/Export.java   |  197 ++
 .../hbase/mapreduce/GroupingTableMapper.java|  177 ++
 .../hbase/mapreduce/HFileInputFormat.java   |  174 ++
 .../hbase/mapreduce/HFileOutputFormat2.java |  902 ++
 .../hbase/mapreduce/HRegionPartitioner.java |  140 +
 .../hadoop/hbase/mapreduce/HashTable.java   |  747 +
 .../hbase/mapreduce/IdentityTableMapper.java|   67 +
 .../hbase/mapreduce/IdentityTableReducer.java   |   79 +
 .../apache/hadoop/hbase/mapreduce/Import.java   |  780 ++
 .../hadoop/hbase/mapreduce/ImportTsv.java   |  793 ++
 .../hadoop/hbase/mapreduce/JarFinder.java   |  186 ++
 .../hbase/mapreduce/KeyValueSerialization.java  |   88 +
 .../hbase/mapreduce/KeyValueSortReducer.java|   57 +
 .../mapreduce/MultiTableHFileOutputFormat.java  |  122 +
 .../hbase/mapreduce/MultiTableInputFormat.java  |  104 +
 .../mapreduce/MultiTableInputFormatBase.java|  296 ++
 .../hbase/mapreduce/MultiTableOutputFormat.java |  176 ++
 .../MultiTableSnapshotInputFormat.java  |  106 +
 .../MultiTableSnapshotInputFormatImpl.java  |  252 ++
 .../mapreduce/MultithreadedTableMapper.java |  301 ++