----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57996/#review175313 -----------------------------------------------------------
Ship it! Ship It! - Rohini Palaniswamy On May 15, 2017, 1:27 p.m., Adam Szita wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57996/ > ----------------------------------------------------------- > > (Updated May 15, 2017, 1:27 p.m.) > > > Review request for pig, Daniel Dai, liyun zhang, Rohini Palaniswamy, and > Xuefu Zhang. > > > Bugs: PIG-5186 > https://issues.apache.org/jira/browse/PIG-5186 > > > Repository: pig-git > > > Description > ------- > > Aggregate warnings were not supported in Spark mode yet (hence the e2e > Warning test case failures). I aim to enable this now. > In MR/Tez we use counters, and in Spark we rely on Accumulators (a means to > support distributed counters). > Pig has some builtin warning enums in PigWarning, and also supports custom > warnings for user defined functions. > This latter is problematic with Spark because you cannot register new > accumulators on the backend and read their values later in the driver. > > A workaround has been implemented in my patch whereas we define Map type of > Accumulators (beside the Long type we already use). One for the builtin > warnings, one for the custom ones. These are passed from driver to backend, > where the executors can create entries in the maps or increment preexisting > values. > > Also added upgrade of DummyContextUDF, this will help fix HiveUDF_7 e2e test > case on Spark. > Previously this was using org.apache.hadoop.mapred.Reporter we have to update > this to PigHadoopLogger which supports Spark too. > > > Diffs > ----- > > src/org/apache/pig/PigWarning.java fcda1145f4e7c16940a540222ac7cc5370e3db33 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigHadoopLogger.java > 255650edb519acc452812a5d67f3ac2376c278c2 > src/org/apache/pig/backend/hadoop/executionengine/spark/SparkLauncher.java > 85e3dc2f80c22b358560c716151d0ba931c78e4e > > src/org/apache/pig/backend/hadoop/executionengine/spark/running/PigInputFormatSpark.java > a22c90dd8dd9b6463f22597effd884bc0f12bb73 > src/org/apache/pig/tools/pigstats/PigStatusReporter.java > 5396535301b0e90dc5d3be2064cfe0bdf488bf6a > src/org/apache/pig/tools/pigstats/PigWarnCounter.java PRE-CREATION > src/org/apache/pig/tools/pigstats/spark/SparkCounter.java > 2411f875ec996fedb870c1b709b99e949803ed50 > src/org/apache/pig/tools/pigstats/spark/SparkCounterGroup.java > c23624dfcd2e11429fd8355497d184b155450c1f > src/org/apache/pig/tools/pigstats/spark/SparkCounters.java > 5ca077ca519ad766c8f6a23ef5b69cd02f3abe99 > src/org/apache/pig/tools/pigstats/spark/SparkJobStats.java > 4c81a414be071d5e08689fde2a5abd847a5fd0b6 > src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java > 4e3644209817eeffabcec13d8aac5179bbad1c62 > src/org/apache/pig/tools/pigstats/spark/SparkStatsUtil.java > 8f3bf6d736de08ae963152553e975e865f875ab2 > test/e2e/pig/tests/nightly.conf 2048740b6a1b73862694805cc07bdd084c66a300 > > > Diff: https://reviews.apache.org/r/57996/diff/2/ > > > Testing > ------- > > After this patch Warning E2E tests on Spark pass. > > > Thanks, > > Adam Szita > >
