----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69410/#review210718 -----------------------------------------------------------
Ship it! Ship It! - Nandor Kollar On Nov. 20, 2018, 12:53 p.m., Adam Szita wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69410/ > ----------------------------------------------------------- > > (Updated Nov. 20, 2018, 12:53 p.m.) > > > Review request for hive, Nandor Kollar and Peter Vary. > > > Bugs: HIVE-20330 > https://issues.apache.org/jira/browse/HIVE-20330 > > > Repository: hive-git > > > Description > ------- > > The change in this patch is that we're not just serializing and putting one > InputJobInfo into JobConf, but rather always append to a list (or create it > on the first occurrence) of InputJobInfo instances in it. > This ensures that if multiple tables serve as inputs in a job, Pig can > retrieve information for each of the tables, not just the last one added. > > I've also discovered a bug in InputJobInfo.writeObject() where the > ObjectOutputStream was closed by mistake after writing partition information > in a compressed manner. Closing the compressed writer inevitably closed the > OOS on the context and prevented any other objects to be written into OOS - I > had to fix that because it prevented serializing InputJobInfo instances > inside a list. > > > Diffs > ----- > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java > 8e72a1275a5cdcc2d778080fff6bb82198395f5f > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java > 195eaa367933990e3ef0ef879f34049c65822aee > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseInputFormat.java > 8d7a8f9df9412105ec7d77fad9af0d7dd18f4323 > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatInputFormat.java > ad6f3eb9f93338023863c6239d6af0449b20ff9c > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InitializeInput.java > 364382d9ccf6eb9fc29689b0eb5f973f422051b4 > > hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InputJobInfo.java > ac1dd54be821d32aa008d41514df05a41f16223c > > hcatalog/core/src/test/java/org/apache/hive/hcatalog/common/TestHCatUtil.java > 91aa4fa2693e0b0bd65c1667210af340619f552d > > hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatLoader.java > c3bde2d2a3cbd09fb0b1ed758bf4f2b1041a23cb > > hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/AbstractHCatLoaderTest.java > 58981f88ef6abfbf7a4b7ffc3116c53d47e86fde > > > Diff: https://reviews.apache.org/r/69410/diff/1/ > > > Testing > ------- > > Added (true) unit tests to verify my method of adding/retrieving InputJobInfo > instances to/from config instances. > Added (integration-like) unit tests to mock Pig calling HCatLoader for > multiple input tables, and checking the reported input sizes. > > > Thanks, > > Adam Szita > >
