[jira] [Commented] (GIRAPH-111) Refactor I/O to be independent of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13174085#comment-13174085 ] Ed Kohlwey commented on GIRAPH-111: --- Maybe I'm getting ahead of myself. I discovered the need for such a thing while working on my patch for GIRAPH-108, but others might have better ideas on how to address the problem. The issue comes from the desire to create a giraph-specific context that can use delegation to either hook into the existing hadoop context/reporting system or report back to a context system that is specific to the resource allocator being used on the cluster (Mesos, YARN). Furthermore, this system should be backwards-compatible with (meaning, they should inherit from) Hadoop's system in order to instantiate InputFormats, etc. This is where the problem occurs. Since Hadoop's io libraries use class-based inheritance rather than interfaces, you have to use the hack I reference in GIRAPH-108 (creating a wrapper class where every method is overrided) that uses a sort of nasty constructor with null arguments to trick the underlying superclass's code into being instantiated, when in reality you're really ignoring the superclass implementation entirely. You can check the code out in the patch - GraphMapper.HadoopContext and GraphProcess.Context. Refactor I/O to be independent of Map/Reduce Key: GIRAPH-111 URL: https://issues.apache.org/jira/browse/GIRAPH-111 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey The I/O mechanisms should probably be abstracted entirely from Map/Reduce in order to support making Giraph an independent framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-111) Refactor I/O to be independent of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173512#comment-13173512 ] Avery Ching commented on GIRAPH-111: I'm not clear on why this is necessary. Couldn't we simply call the I/O methods as Hadoop would when we're not using Hadoop? Am I missing something? Refactor I/O to be independent of Map/Reduce Key: GIRAPH-111 URL: https://issues.apache.org/jira/browse/GIRAPH-111 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey The I/O mechanisms should probably be abstracted entirely from Map/Reduce in order to support making Giraph an independent framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-111) Refactor I/O to be independent of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173537#comment-13173537 ] Jakob Homan commented on GIRAPH-111: bq. I'm not clear on why this is necessary. I agree. Hadoop's file formats, etc. are designed to be exceedingly forgiving and flexible as to the underlying storage mechanism. Can you point to where they're lacking for Mesos' case? bq. We could also copy out the relevant Hadoop I/O classes (InputFormat, OutputFormat, etc) into Giraph, rename their packages, and begin reworking them in an appropriate way to better suit Giraph. -1. Therein lies madness. Refactor I/O to be independent of Map/Reduce Key: GIRAPH-111 URL: https://issues.apache.org/jira/browse/GIRAPH-111 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey The I/O mechanisms should probably be abstracted entirely from Map/Reduce in order to support making Giraph an independent framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
