-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69410/
-----------------------------------------------------------
Review request for hive, Nandor Kollar and Peter Vary.
Bugs: HIVE-20330
https://issues.apache.org/jira/browse/HIVE-20330
Repository: hive-git
Description
-------
The change in this patch is that we're not just serializing and putting one
InputJobInfo into JobConf, but rather always append to a list (or create it on
the first occurrence) of InputJobInfo instances in it.
This ensures that if multiple tables serve as inputs in a job, Pig can retrieve
information for each of the tables, not just the last one added.
I've also discovered a bug in InputJobInfo.writeObject() where the
ObjectOutputStream was closed by mistake after writing partition information in
a compressed manner. Closing the compressed writer inevitably closed the OOS on
the context and prevented any other objects to be written into OOS - I had to
fix that because it prevented serializing InputJobInfo instances inside a list.
Diffs
-----
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java
8e72a1275a5cdcc2d778080fff6bb82198395f5f
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
195eaa367933990e3ef0ef879f34049c65822aee
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseInputFormat.java
8d7a8f9df9412105ec7d77fad9af0d7dd18f4323
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatInputFormat.java
ad6f3eb9f93338023863c6239d6af0449b20ff9c
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InitializeInput.java
364382d9ccf6eb9fc29689b0eb5f973f422051b4
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InputJobInfo.java
ac1dd54be821d32aa008d41514df05a41f16223c
hcatalog/core/src/test/java/org/apache/hive/hcatalog/common/TestHCatUtil.java
91aa4fa2693e0b0bd65c1667210af340619f552d
hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatLoader.java
c3bde2d2a3cbd09fb0b1ed758bf4f2b1041a23cb
hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/AbstractHCatLoaderTest.java
58981f88ef6abfbf7a4b7ffc3116c53d47e86fde
Diff: https://reviews.apache.org/r/69410/diff/1/
Testing
-------
Added (true) unit tests to verify my method of adding/retrieving InputJobInfo
instances to/from config instances.
Added (integration-like) unit tests to mock Pig calling HCatLoader for multiple
input tables, and checking the reported input sizes.
Thanks,
Adam Szita
