Hi Joshua, "My question is; why would I need to setup different transform modules? Since there is no real config to do in the transform connector (all the good stuff seems to be under Task config) I'm not sure why I would need to make more than one and keep reusing it by changing the transform paeans under task?"
While the Metadata Adjuster transformer has no configuration, the model that MCF uses for transformers is just like the model it uses for other kinds of connectors. Pretend for a moment that you needed to call an external system to do content extraction, then you will see the point. Thanks, Karl On Wed, Sep 9, 2015 at 12:55 PM, Joshua Dunham <[email protected]> wrote: > Hi Karl, Rafa, > > I finally had some time to work on this and I have a scheme which > (largely) works very well but I have some question, one stumbling block, > and one comment. > > First, my environment consists of, Manifold v 2.1, MariaDB which I > imported a small CSV into for testing, and Marmotta 3.3. > > The real interesting bits are in specifying the Task. I have the mySQL > input -> metadata adjuster -> filesystem output. mySQL is setup and the > connection shows as OK and on starting the job, it does write files to the > output folder. > > Getting the list of ID's works well no issue there, and I'm not using > versioning or access tokens yet. The stumbling block has to do with setting > up the Data Query and the best use of the $URL and $DATA variables. First: > I've hijacked the $URL into ~ CONCAT("addresses/", id) AS $(URLCOLUMN) > which has the effect of creating a folder called addresses in the root of > the output folder. Inside of the addresses folder it makes numbered files > corresponding to the rowID. I can point the root folder path at the > marmotta import directory and even use the context templating feature > (setting 'addresses' into the real context name). That's really slick for > out of the box hack at integration. > > What is not apparent is how to use the metadata adjuster to interact with > the variables in the Data query. I've followed the guide and made a simple > hello, False, ${city} statement but the only bits that are written into the > file are the contents of the $DATACOLUMN variable. So, given a simple > address book in a database with columns, id, street, city, region, country, > post code, latitude, longitude ... how should I approach making such a data > query? My real use cases will be much much more complicated so I'm > wondering if you have some explanation of how I should want to use that > field and maybe a small SQL snippet example with those columns? :) My end > goal is to have a column called out and then use the metadata adjuster to > simply prepend each column's value with a string. So if the city is 'New > York' it would write out city:New_York or the like. > > ===== > > The comment was in regards to a bit of sample data which could ship with > the source. It would be very educational if there was a complex but real > configuration of ManifoldCF that links to a sqlite3 file as input and maybe > the same one input db but a different table as output? > > ===== > > My question is; why would I need to setup different transform modules? > Since there is no real config to do in the transform connector (all the > good stuff seems to be under Task config) I'm not sure why I would need to > make more than one and keep reusing it by changing the transform paeans > under task? > > > Thank you! > > J > > > > On 5 July 2015 at 17:27, Karl Wright <[email protected]> wrote: > > Hi Joshua, > > > > My take: > > > > --> (A) How I define the data to grab, whether some SQL statement or the > > like. <-- > > > > Have a look at the user documentation here: > > > https://manifoldcf.apache.org/release/release-1.9/en_US/end-user-documentation.html#jdbcrepository > > > > It should be pretty clear how you define what you are looking for. > > > > --> (B) How to use this data as individual variables which I can arrange > > into a linked data relationship (ManifoldCF mapping module?) <-- > > > > Rafa's previous reply about the RepositoryDocument is appropriate. > > Basically, an output connector will be handed one of those objects for > every > > MCF "document". The javadoc for it is here: > > > > > https://manifoldcf.apache.org/release/trunk/api/framework/org/apache/manifoldcf/agents/interfaces/RepositoryDocument.html > > > > --> (C) How difficult would it be to connect to Marmotta's webservice(s). > > I'm not familiar with the exact mechanism, but I saw ManifoldCF has > > support for elasticsearch so maybe I could put something together that > > talks to Marmotta..<-- > > > > You can readily write your own output connector. There's a book, in > fact, > > describing how to do that. See: > > > > https://github.com/DaddyWri/manifoldcfinaction/tree/master/pdfs > > > > ... and read Chapter 9. > > > > Thanks, > > Karl > > > > > > On Sun, Jul 5, 2015 at 11:53 AM, Joshua Dunham <[email protected]> > > wrote: > >> > >> That sounds promising. Would you recommend ManifoldCF for this? If so, > >> do you know of any resources which I can use to get up to speed with > >> using it in this way? > >> > >> -J > >> > >>> On 4 July 2015 at 21:48, <[email protected]> wrote: > >>> Hi Joshua, > >>> > >>> The ManifoldCF unit logic in terms of indexing is the Repository > >>> Document > >>> which, simplifying a lot, model a document composed by content plus > >>> metadata > >>> (key-value). It should be relative easy to tripifly that structure and > >>> push > >>> it to Marmotta using SPARQL update queries or Marmotta’s java client > for > >>> adding resources. > >>> The Generic Database connector uses a set of queries for crawling the > >>> database. You should have to use that queries to get you data. I’m not > >>> completely sure if each record result is converted directly to a > >>> Repository > >>> Document, that is something that I would need to check. > >>> > >>> Hope that helps, > >>> Cheers, Rafa > >>> > >>> > >>> > >>> > >>> On Sun, Jul 5, 2015 at 2:56 AM, Joshua Dunham <[email protected] > > > >>> wrote: > >>>> > >>>> Hi ManifoldCF Users (and Devs) > >>>> > >>>> I'm wondering if ManifoldCF can work in my use case. I have some > >>>> random mySQL and Oracle DB's that I would like to connect to and > >>>> extract certain known bits of info, format them each a certain way and > >>>> then store the info in Apache Marmotta [1]. Marmotta is an RDF triple > >>>> store for linked data so I would need to parse and store the mySQL and > >>>> Oracle DB's info into a linked format, which is no problem for me to > >>>> create the relationships etc, I just need something that would let me > >>>> specifically do this. > >>>> > >>>> From what I've read, ManifoldCF can connect to mySQL and Oracle > >>>> (via non-distributed libraries), and store the results out in several > >>>> target data stores. What isn't clear is > >>>> (A) How I define the data to grab, whether some SQL statement or the > >>>> like. > >>>> (B) How to use this data as individual variables which I can arrange > >>>> into a linked data relationship (ManifoldCF mapping module?) > >>>> (C) How difficult would it be to connect to Marmotta's webservice(s). > >>>> I'm not familiar with the exact mechanism, but I saw ManifoldCF has > >>>> support for elasticsearch so maybe I could put something together that > >>>> talks to Marmotta.. > >>>> > >>>> Would this be possible? If so, could someone point me in the right > >>>> direction? > >>>> > >>>> Thanks! > >>>> -Joshua > >>>> > >>>> > >>>> [1] - http://marmotta.apache.org/index.html > >
