Hi Micha and others,
Of course we have to share the workload. I'd rather help develop something
that is already taking shape (because I'm still not very familiar with the
API). Let's see what people think about MAGE-TAB etc, but hopefully I can
carry on where you leave off. That's not a 100% commitment, but I hope I'll
have the time/ability to do this.
Just one thought, isn't another option to cannibalise the migration code to
make a command-line bulk importer? I guess that's no good for sites who want
the users to submit through the web interface...
Another thought: my BASE 1 loaders were bolted on top of BASE and did all the
sample creation etc through HTTP requests to the BASE 1 GUI with a Perl LWP
robot. These could be modified and bolted on to BASE 2, although the approach
is very sensitive to GUI changes, and just doesn't feel like a long-term
solution to me... (the motivation to do it this way with BASE 1 was to take
the path of least resistance and avoid learning the PHP API).
cheers,
Bob.
Micha Bayer writes:
> That sounds like a hell of a lot more work than what I have done so far....
> :-)
>
> My first shot at this has been quite basic and it only creates new
> RawBioAssays and a single Experiment that these belong to. It could of
> course be extended to do more stuff, but my time allowance for this is
> limited and I don't think I'll be able to spend a lot more time on this than
> I already have.
>
> I think the upshot of all this discussion has been:
>
> 1. people need a batch importer (desperately, in fact), and
> 2. a plugin is much better and will (hopefully) be possible
>
> With that in mind, I'll go away and look into turning my stuff into a plugin
> now, and when I have got the basic version of that working (i.e. file
> upload, RawBioAssay and Experiment creation only, as above) I'll make the
> code available to the community somehow and someone else can pick it up from
> there if they want to make it fancier. I hope that's okay -- even with no
> extra effort from someone else it'll still mean that a hell of a lot of
> effort can be saved.
>
> Perhaps the dev team can point me in the right direction initially. My
> colleague Iain here is just working on a file importer plugin that handles
> Excel files and his code starts off like this:
>
>
> public class ExcelFileImporter extends AbstractPlugin implements
> InteractivePlugin, AutoDetectingImporter
> {
> private static final Set<GuiContext> guiContexts =
> Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
> GuiContext.Type.ITEM));
>
>
> So am I right in thinking that if I change this to the following...
>
>
> public class ZipFileImporter extends AbstractPlugin implements
> InteractivePlugin, AutoDetectingImporter
> {
> private static final Set<GuiContext> guiContexts =
> Collections.singleton(new GuiContext(Item.EXPERIMENT,
> GuiContext.Type.ITEM));
>
>
> ....then I'll get an import tab created in the Experiment list view which
> will let me import the zip file using my plugin?
>
> We're planning to make the Excel plugin available for everyone too BTW, once
> it's finished.
>
> Cheers
> Micha
>
> ==================================
> Dr Micha M Bayer
> Bioinformatics Specialist
> Genetics Programme
> The Scottish Crop Research Institute
> Invergowrie
> Dundee
> DD2 5DA
> Scotland, UK
> Telephone +44(0)1382 562731 ext. 2309
> Fax +44(0)1382 562426
> http://www.scri.sari.ac.uk/MichaBayer.htm
> =============================
>
>
> > -----Original Message-----
> > From: [EMAIL PROTECTED] [mailto:basedb-users-
> > [EMAIL PROTECTED] On Behalf Of Bob MacCallum
> > Sent: 29 November 2006 18:36
> > To: BASE ML
> > Subject: Re: [base] Offline files
> >
> >
> > It looks like I let the cat out of the bag on this one...
> >
> > Some discussion among the community would be good to establish what we
> > want
> > from a bulk loader. For example, I would like all objects created and
> > linked,
> > from samples (with annotations) right up to raw bioassays (grouped into an
> > experiment as Micha does, ideally). Obviously to do this you need to
> > specify
> > the relationships between all the samples, annotations, dyes, hybs, data
> > files, but that could easily end up being as much work as manually loading
> > the
> > data ;-)
> >
> > Therefore you have to make compromises; the main one being that you treat
> > every sample/extract/etc in exactly the same way (same protocols, same
> > amounts, same labels, same array design). This would probably work for
> > most
> > people - but let's discuss it anyway...
> >
> > One issue with the bulk loader I wrote for BASE 1 is that if your
> > experiment
> > involves a dye swap for one biological replicate, the bulk loader will
> > label
> > each extract with both dyes for *all* replicates (leaving you with unused
> > labelled extracts after linking to the hybs).
> >
> > One quite low cost way to define your experiment for a bulk loader is
> > Tab2MAGE
> > (soon to be MAGE-TAB, see
> > http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it
> > feels
> > strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2
> > rather than importing it. The input files for our bulk loader are
> > specified
> > here:
> > http://wiki.vectorbase.org/vectorbase/index.php/MicroarrayExperiments
> > (look for the example file links) but note that the BASE-specific
> > parameters
> > (protocols, raw file parsers, array designs, etc) are configured by the
> > user
> > in an interactive process before the upload. Our bulk loader is also
> > 2-channel only...
> >
> > I have a feeling that the Tab2MAGE parser could be adapted into a BASE 2
> > bulk
> > loader somehow, although I think all Tab2MAGE code is Perl.
> >
> > I think MAGE-TAB is something we could get the average bench scientist to
> > provide (with a little help MGEDifying the annotations). So far every
> > experiment I have loaded has come with some kind of spreadsheet (each one
> > in a
> > different format of course), so the leap to MAGE-TAB is not so great.
> >
> > Ok, enough from me...
> >
> >
> >
> >
> > Micha Bayer writes:
> > >
> > >
> > > > -----Original Message-----
> > > > From: [EMAIL PROTECTED] [mailto:basedb-
> > users-
> > > > [EMAIL PROTECTED] On Behalf Of Nicklas Nordborg
> > > > Sent: 29 November 2006 14:40
> > > > To: BASE ML
> > > > Subject: Re: [base] Offline files
> > > >
> > > > Jari Häkkinen wrote:
> > > > > Hi all,
> > > > >
> > > > > Just a thought, I think the best choice to trigger a large import
> > like
> > > > > this is to create a plug-in instead of adding extra tabs to the
> > file
> > > > > browser. The plug-in should start by requesting the zip
> > file/directory
> > > > > to import. If the zip file is not already uploaded the user will
> > have an
> > > > > opportunity to upload it. After choosing the zip file/directory the
> > > > > plug-in could start its import and creation of necessary items.
> > > > >
> > > > > The above is fairly straightforward already today and the new code
> > would
> > > > > start at import/creation of necessary items.
> > > > >
> > > > > This will yield very little (or no) changes to the core and the
> > plug-in
> > > > > should probably be started from the experiment listing page. This
> > will
> > > > > of course change the experiment list jsp (a import tab is needed).
> > > >
> > > > No change is needed to the jsp. The new plugin will be detected and
> > an
> > > > "Import" button will appear.
> > > >
> > > > /Nicklas
> > >
> > > So what interface would I have to implement for this to happen, if I
> > were to turn my code into a plugin?
> > >
> > > If it's not too much work I might consider doing this, that would be
> > much better.
> > >
> > > Cheers
> > >
> > > Micha
> > >
> > > ==================================
> > > Dr Micha M Bayer
> > > Bioinformatics Specialist
> > > Genetics Programme
> > > The Scottish Crop Research Institute
> > > Invergowrie
> > > Dundee
> > > DD2 5DA
> > > Scotland, UK
> > > Telephone +44(0)1382 562731 ext. 2309
> > > Fax +44(0)1382 562426
> > > http://www.scri.sari.ac.uk/MichaBayer.htm
> > > =============================
> > >
> > >
> > >
> > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> > _ _
> > >
> > > DISCLAIMER:
> > >
> > > This email is from the Scottish Crop Research Institute, but the views
> > > expressed by the sender are not necessarily the views of SCRI and its
> > > subsidiaries. This email and any files transmitted with it are
> > confidential
> > > to the intended recipient at the e-mail address to which it has been
> > > addressed. It may not be disclosed or used by any other than that
> > addressee.
> > > If you are not the intended recipient you are requested to preserve
> > this
> > > confidentiality and you must not use, disclose, copy, print or rely on
> > this
> > > e-mail in any way. Please notify [EMAIL PROTECTED] quoting the
> > > name of the sender and delete the email from your system.
> > >
> > > Although SCRI has taken reasonable precautions to ensure no viruses are
> > > present in this email, neither the Institute nor the sender accepts any
> > > responsibility for any viruses, and it is your responsibility to scan
> > the email
> > > and the attachments (if any).
> > >
> > >
> > > -----------------------------------------------------------------------
> > --
> > > Take Surveys. Earn Cash. Influence the Future of IT
> > > Join SourceForge.net's Techsay panel and you'll get the chance to share
> > your
> > > opinions on IT & business topics through brief surveys - and earn cash
> > >
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > > _______________________________________________
> > > The BASE general discussion mailing list
> > > basedb-users@lists.sourceforge.net
> > > unsubscribe: send a mail with subject "unsubscribe" to
> > > [EMAIL PROTECTED]
> >
> > --
> > Bob MacCallum | VectorBase Developer | Kafatos/Christophides Groups |
> > Division of Cell and Molecular Biology | Imperial College London |
> > Phone +442075941945 | Email [EMAIL PROTECTED]
> >
> > -------------------------------------------------------------------------
> > Take Surveys. Earn Cash. Influence the Future of IT
> > Join SourceForge.net's Techsay panel and you'll get the chance to share
> > your
> > opinions on IT & business topics through brief surveys - and earn cash
> > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> > _______________________________________________
> > The BASE general discussion mailing list
> > basedb-users@lists.sourceforge.net
> > unsubscribe: send a mail with subject "unsubscribe" to
> > [EMAIL PROTECTED]
>
>
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys - and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> _______________________________________________
> The BASE general discussion mailing list
> basedb-users@lists.sourceforge.net
> unsubscribe: send a mail with subject "unsubscribe" to
> [EMAIL PROTECTED]
--
Bob MacCallum | VectorBase Developer | Kafatos/Christophides Groups |
Division of Cell and Molecular Biology | Imperial College London |
Phone +442075941945 | Email [EMAIL PROTECTED]
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
The BASE general discussion mailing list
basedb-users@lists.sourceforge.net
unsubscribe: send a mail with subject "unsubscribe" to
[EMAIL PROTECTED]
