I've been looking at this and here is what I think. This looks better than what I am using now in Infovore (I might even try snapping it in) but probably isn't as good, performance-wise, as what I'd like to have.
Something I learned back in grad school is that if you have a matrix of 100 million floating point numbers you can do a lot of flops on them in the time it takes to turn them into base ten and back, especially if you're using Java and you have to puff up ASCII to UTF-16 and back. A lot of the jobs I am interested in doing involve a lot of sums and counts so storing numbers in a native representation will make a big difference. Similarly there are a lot of situations where you can treat a IRI as completely opaque (you read it in and write it out without ever looking at it, or if you do look at it you are equality testing it or pattern matching) In cases like that there is a lot to gain from making as few memory allocations copies as possible, so I don't want something that "wraps" a Jena Node but I would definitely like to get the Jena Node (maybe even lazy evaled) if I want it. IRI representation is also an interesting question. I think almost all triple stores keep a dictionary of "namespaces" (which just might be observed prefixes) and represent IRIs as a namespace pointer plus a suffix. Gzip and other compression algorithms eat some of the overhead but you get even better results if you compress prefixes first and then gzip. I did some work two years ago where I was using Pig to do pagerank-like computations on my home cluster and found I could get 20x speedups by taking the cumulative probability distribution of the IRIs and representing them with variable length codes. In some data sets I see that the "a" predicate is 10% of the predicates, for instance, and in a case like that you would ideally code it with 4 bits or so. It becomes a big pain if you want to look at the IRIs (gotta join) but often you don't need to look at the IRIs. (Since then I also discovered a scalable and stable algorithm for the cumulative probability distribution) At this point splitability is not a big concern for me, largely because I am using multiple moderate size files gzip compressed most of the time. I'm thinking a lot about how to compress the raw data for this http://basekb.com/subjectiveEye/ partially because it costs a few hundred a month to store it in AWS, but also because it costs a few hundred dollars to do a full scan of the data. It turns out the keys that come from Wikipedia are in sequential order and Hadoop doesn't mess that up if it isn't splitting, and that can be taken advantage of compress the data. Another requirement for this kind of thing is metadata about data sets. For instance, things that have gone through the reducer usually have had sorting and grouping done on them and if you have two data sets that have been sorted and grouped the same way you can merge them together with a priority queue instead of doing a reduce side join and that saves you the cost of doing the reduce. Today this is an optimization on the low-level coding I'm doing, but a piggish kind of system might change the query plan if it knew how the data is organized. (Or maybe knew that multiple copies of the data set existed packaged different ways) Similarly, namespace prefixes, information about encoding choices, types and such can all go in that metadata. I look at slides 21 and 25 and see a lot of choices I don't like. >From a reliability point of view, I find only the line-based processing acceptable. I can't accept data loss because of the syntax errors that are endemic in "in the wild" RDF data sets. I can't accept the possibility of memory blow-outs, in fact the whole reason I am using Hadoop is because my last system used to run out of memory. On the other hand I am very aware of how much it costs to create a new RIOT parser for each line, and that's the reason why it takes 25 machine*hours to process :BaseKB. I think some of these implementations involve passing data between threads; perhaps you can get the cost of that down by batching, but thread switching and coordination is expensive. For me at this point, multiple outputs are necessary because I'm using them in my sieve3 stage. (The need to recode this for API changes is the reason I haven't switched to Hadoop 2) Another thing I am thinking about is that the Map/Reduce API isn't a perfect match for most workloads and that the real future is http://tez.incubator.apache.org/ Tez is a better paradigm for jobs that have multiple ins and outs, so it might make sense to just skip the Rube Goldberg machine that is used for multiple outs in the MR API and just do things the Tez way. On Tue, Apr 1, 2014 at 9:09 AM, Andy Seaborne <[email protected]> wrote: > On 01/04/14 12:11, Rob Vesse wrote: >> Ok, I think probably what is best is if I roll up the relevant stuff into >> patches on the JIRA so you can apply them yourself. They'll be one for >> the incubator website and one for Jena >> >> I can probably also add my Powerpoint slides to the JIRA issue which >> summarise a lot of what is (and isn't there) and I'll send a write up to >> the list at some point this week. > > Great. > > Andy > >> >> Rob >> >> On 01/04/2014 11:19, "Andy Seaborne" <[email protected]> wrote: >> >>> On 01/04/14 10:25, Rob Vesse wrote: >>>> Andy >>>> >>>> I have now got all the necessary legal and management clearance from >>>> Cray to >>>> move ahead with donating the experimental Hadoop RDF work that we¹ve >>>> previously discussed privately to the Jena project. >>> >>> I know about it but not what it is I'm very interested in seeing it ... >>> >>>> So I am ready to move >>>> forwards with the IP Clearance process but I¹ve run into a slight snag >>>> in >>>> that it appears to require an ASF Member/Officer to actually formally be >>>> responsible for executing the process. >>>> >>>> I think in reality this just means that an ASF member has to check that >>>> all >>>> the boxes have been appropriately ticked and run the necessary votes >>>> once we >>>> reach that stage of the process. As you are an ASF member would you be >>>> willing to do this? >>>> >>>> I am happy to do all the leg work, I already have the IP clearance >>>> document >>>> ready to commit to the Incubator website and the initial code base >>>> ready to >>>> commit to the Experimental area. Once these initial things are >>>> committed I >>>> can start working through the remaining steps such as changing the >>>> Copyright >>>> Headers, putting together the NOTICE and LICENSE files appropriately >>>> etc. I >>>> will also go ahead and file a JIRA for tracking purposes. >>>> >>>> Before I start on this I wanted to check that you are OK with acting as >>>> the >>>> responsible person for this? >>> >>> /me off to read up on the process ... >>> >>> Yes, fine. I have incubator karma to update the IP pages. >>> >>> Full steam ahead! >>> >>>> >>>> Cheers, >>>> >>>> Rob >>> >>> In terms of the destination being the Jena project, this looks like it >>> is fully aligned to the project charter: >>> >>> """ >>> the creation and maintenance of >>> open-source software related to accessing, storing, querying, >>> publishing and reasoning with semantic web data while >>> adhering to relevant W3C and community standards >>> for distribution at no charge to the public. >>> """ >>> >>> and I (personally) am quite relaxed about any variety if there is some >>> connection to Jena even if it's quite small (e.g. there is at least one >>> import statement like "org.apache.jena....."!). We can't, in all >>> honesty, adopt random other unrelated software without at least checking >>> with other linked data projects in ASF. But at the same time, linked >>> data @ASF isn't the Hadoop-infrastructure ecosystem where each piece can >>> sustain it's own TLP. >>> >>> Andy >>> >> >> >> >> > -- Paul Houle Expert on Freebase, DBpedia, Hadoop and RDF (607) 539 6254 paul.houle on Skype [email protected]
