Pat, That really sounds great.
I should find some time (who needs sleep) to generate music logs for you as well. On Mon, Aug 19, 2013 at 8:31 AM, Pat Ferrel <[email protected]> wrote: > There are three things I could work on my free time: > > 1) test this on a bigger data set gathered from rotten tomatoes, which > only has B data (movie thumbs up) > 2) begin work on the Solr query and service integration, rather than the > current loose LucidWorks Search integration. > 3) make sure everything is set up for different item spaces in B and A. > > Planning to tackle in this order, unless someone speaks up. > > > On Aug 16, 2013, at 1:39 PM, Pat Ferrel <[email protected]> wrote: > > Works on a cluster but have only tested on the trivial test data set. > > On Aug 13, 2013, at 4:49 PM, Pat Ferrel <[email protected]> wrote: > > OK single action recs are working so output to Solr with only [B'B] and B. > > On Aug 13, 2013, at 10:52 AM, Pat Ferrel <[email protected]> wrote: > > Corrections inline > > > On Aug 13, 2013, at 10:49 AM, Pat Ferrel <[email protected]> wrote: > > > > I finally got some time to work on this and have a first cut at output > to Solr working on the github repo. It only works on 2-action input but > I'll have that cleaned up soon so it will work with one action. Solr > indexing has not been tested yet and the field names and/or types may need > tweaking. > > > > It takes the result of the previous drop: > > 1) DRMs for B (user history or B items action1) and A (user history of A > items action2) > > 2) DRMs for [B'B] using LLR, and [B'A] using cooccurrence > > > > There are two final outputs created using mapreduce but requiring 2 > in-memory hashmaps. I think this will work on a cluster (the hashmaps are > instantiated on each node) but haven't tried yet. It orders items in #2 > fields by strength of "link", which is the similarity value used in [B'B] > or [B'A]. It would be nice to order #1 by recency but there is no provision > for passing through timestamps at present so they are ordered by the > strength of preference. This is probably not useful and so can be ignored. > Ordering by recency might be useful for truncating queries by recency while > leaving the training data containing 100% of available history. > > > > 1) It joins #1 DRMs to produce a single set of docs in CSV form, which > looks like this: > > id,history_b,history_a > u1,iphone ipad,iphone ipad galaxy > > ... > > > > 2) it joins #2 DRMs to produce a single set of docs in CSV form, which > looks like this: > > id,b_b_links,b_a_links > iphone,iphone ipad,iphone ipad galaxy > > … > > > > It may work on a cluster, I haven't tried yet. As soon as someone has > some large-ish sample log files I'll give them a try. Check the sample > input files in the resources dir for format. > > > > https://github.com/pferrel/solr-recommender > > > > > > On Aug 13, 2013, at 10:17 AM, Pat Ferrel <[email protected]> wrote: > > > > When I started looking at this I was a bit skeptical. As a Search engine > Solr may be peerless, but as yet another NoSQL db? > > > > However getting further into this I see one very large benefit. It has > one feature that sets it completely apart from the typical NoSQL db. The > type of queries you do return fuzzy results--in the very best sense of that > word. The most interesting queries are based on similarity to some > exemplar. Results are returned in order of similarity strength, not ordered > by a sort field. > > > > Wherever similarity based queries are important I'll look at Solr first. > SolrJ looks like an interesting way to get Solr queries on POJOs. It's > probably at least an alternative to using docs and CSVs to import the data > from Mahout. > > > > > > > > On Aug 12, 2013, at 2:32 PM, Ted Dunning <[email protected]> wrote: > > > > Yes. That would be interesting. > > > > > > > > > > On Mon, Aug 12, 2013 at 1:25 PM, Gokhan Capan <[email protected]> wrote: > > > >> A little digression: Might a Matrix implementation backed by a Solr > index > >> and uses SolrJ for querying help at all for the Solr recommendation > >> approach? > >> > >> It supports multiple fields of String, Text, or boolean flags. > >> > >> Best > >> Gokhan > >> > >> > >> On Wed, Aug 7, 2013 at 9:42 PM, Pat Ferrel <[email protected]> > wrote: > >> > >>> Also a question about user history. > >>> > >>> I was planning to write these into separate directories so Solr could > >>> fetch them from different sources but it occurs to me that it would be > >>> better to join A and B by user ID and output a doc per user ID with > three > >>> fields, id, A item history, and B item history. Other fields could be > >> added > >>> for users metadata. > >>> > >>> Sound correct? This is what I'll do unless someone stops me. > >>> > >>> On Aug 7, 2013, at 11:25 AM, Pat Ferrel <[email protected]> wrote: > >>> > >>> Once you have a sample or example of what you think the > >>> "log file" version will look like, can you post it? It would be great > to > >>> have example lines for two actions with or without the same item IDs. > >> I'll > >>> make sure we can digest it. > >>> > >>> I thought more about the ingest part and I don't think the > one-item-space > >>> is actually a problem. It just means one item dictionary. A and B will > >> have > >>> the right content, all I have to do is make sure the right ranks are > >> input > >>> to the MM, > >>> Transpose, and RSJ. This in turn is only one extra count of the # of > >> items > >>> in A's item space. This should be a very easy change If my thinking is > >>> correct. > >>> > >>> > >>> On Aug 7, 2013, at 8:09 AM, Ted Dunning <[email protected]> wrote: > >>> > >>> On Tue, Aug 6, 2013 at 7:57 AM, Pat Ferrel <[email protected]> > wrote: > >>> > >>>> 4) To add more metadata to the Solr output will be left to the > consumer > >>>> for now. If there is a good data set to use we can illustrate how to > do > >>> it > >>>> in the project. Ted may have some data for this from musicbrainz. > >>> > >>> > >>> I am working on this issue now. > >>> > >>> The current state is that I can bring in a bunch of track names and > links > >>> to artist names and so on. This would provide the basic set of items > >>> (artists, genres, tracks and tags). > >>> > >>> There is a hitch in bringing in the data needed to generate the logs > >> since > >>> that part of MB is not Apache compatible. I am working on that issue. > >>> > >>> Technically, the data is in a massively normalized relational form > right > >>> now, but it isn't terribly hard to denormalize into a form that we > need. > >>> > >>> > >>> > >> > > > > > > > > > >
