> On April 17, 2013, 12:37 a.m., Alessandro Presta wrote: > > Can you add a test case with multiple edge input formats?
We have no tests in giraph-hive, and that's the only place where I added a way to set multiple edge inputs. I am not sure it makes sense to add method to InternalVertexRunner which accepts several String[], one for each edge input (and then for vertex input to). > On April 17, 2013, 12:37 a.m., Alessandro Presta wrote: > > giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java, > > line 687 > > <https://reviews.apache.org/r/10564/diff/1/?file=281712#file281712line687> > > > > This is going to print "-hiveToEdgeClass=XXX" for multiple classes. I > > think this is confusing, seems like the same option is overridden multiple > > times. Probably better to rename the option to "-hiveToEdgeClasses" and > > print the comma-separated list. I changed the logging a bit, now it will print this for each edge input: Edge input format: hiveToEdgeClass=org.apache.giraph.hive.SomeInputFormat edgeInputTable=table_name edgeInputFilter="ds='2013-01-01'" The whole HiveGiraphRunner is a bit messy, I think adding something like BenchmarkOption stuff I added some time ago could make it much cleaner, but I'll leave that for a separate issue. > On April 17, 2013, 12:37 a.m., Alessandro Presta wrote: > > giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java, > > line 86 > > <https://reviews.apache.org/r/10564/diff/1/?file=281713#file281713line86> > > > > Is there a way you can map EdgeInputFormat classes to HiveToEdge > > classes without going through the index? > > The index seems a bit low-level, and also forces Giraph to maintain the > > ordering. Can you please give some details about the design which you have in mind? - Maja ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10564/#review19285 ----------------------------------------------------------- On April 17, 2013, 4:19 a.m., Maja Kabiljo wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/10564/ > ----------------------------------------------------------- > > (Updated April 17, 2013, 4:19 a.m.) > > > Review request for giraph. > > > Description > ------- > > For now, I did this only for Edge input, once I get some feedback I'll do the > exactly same thing for vertex input. > Also, I added direct support only to HiveGiraphRunner, we can extend it later > to others as well. > > > This addresses bug GIRAPH-639. > https://issues.apache.org/jira/browse/GIRAPH-639 > > > Diffs > ----- > > giraph-core/src/main/java/org/apache/giraph/bsp/BspService.java 178c96f > giraph-core/src/main/java/org/apache/giraph/conf/GiraphClasses.java 4a0e8f7 > giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java > 7f9e38e > giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java > 6a5949e > > giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java > 8dfe546 > giraph-core/src/main/java/org/apache/giraph/io/InputFormatWithIndex.java > PRE-CREATION > > giraph-core/src/main/java/org/apache/giraph/job/GiraphConfigurationValidator.java > cc6b126 > giraph-core/src/main/java/org/apache/giraph/master/BspServiceMaster.java > d01dbb4 > giraph-core/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java > 029cb5d > giraph-core/src/main/java/org/apache/giraph/worker/BspServiceWorker.java > 037cdfc > > giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallable.java > afb636b > > giraph-core/src/main/java/org/apache/giraph/worker/EdgeInputSplitsCallableFactory.java > 4a1705b > giraph-hive/src/main/java/org/apache/giraph/hive/HiveGiraphRunner.java > 6e40b7f > > giraph-hive/src/main/java/org/apache/giraph/hive/common/GiraphHiveConstants.java > f8363b1 > giraph-hive/src/main/java/org/apache/giraph/hive/common/HiveProfiles.java > 892d443 > > giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeInputFormat.java > c482cf0 > > giraph-hive/src/main/java/org/apache/giraph/hive/input/edge/HiveEdgeReader.java > 09476cd > > Diff: https://reviews.apache.org/r/10564/diff/ > > > Testing > ------- > > mvn clean verify > Run application with two edge input tables - verified results. > > > Thanks, > > Maja Kabiljo > >
