Re: [base] Offline files
Yes, there is a problem with spammers of trac environments. We have changed the ticket reporting system to require authentication (details on the baseplugins wiki). For BASE and Proteios this has fixed the problem of ticket spam (at least for now), so hopefully the ticketing system will stay clean. Jari Micha Bayer wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Jari Häkkinen Sent: 30 November 2006 15:42 To: BASE ML Subject: Re: [base] Offline files There is a possibility to get access to the subversion repository and trac environment just for this kind of stuff. Check out http://baseplugins.thep.lu.se Trac environment allows for discussions about the importer through the ticketing system. I just looked at the ticketing system on the plugins page and it looks like it's been majorly vandalized by spammers. Is this email based? Someone might want to take a look at this. Cheers Micha _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ DISCLAIMER: This email is from the Scottish Crop Research Institute, but the views expressed by the sender are not necessarily the views of SCRI and its subsidiaries. This email and any files transmitted with it are confidential to the intended recipient at the e-mail address to which it has been addressed. It may not be disclosed or used by any other than that addressee. If you are not the intended recipient you are requested to preserve this confidentiality and you must not use, disclose, copy, print or rely on this e-mail in any way. Please notify [EMAIL PROTECTED] quoting the name of the sender and delete the email from your system. Although SCRI has taken reasonable precautions to ensure no viruses are present in this email, neither the Institute nor the sender accepts any responsibility for any viruses, and it is your responsibility to scan the email and the attachments (if any). - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED] -- Jari Häkkinen, PhD Complex Systems Divisionmailto:[EMAIL PROTECTED] Department of Theoretical Physics phone: +46 (0)46 2229347 Lund University fax: +46 (0)46 2229686 Sölvegatan 14a, SE-223 62 Lund, Sweden http://www.thep.lu.se - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
That sounds like a hell of a lot more work than what I have done so far :-)
My first shot at this has been quite basic and it only creates new RawBioAssays
and a single Experiment that these belong to. It could of course be extended to
do more stuff, but my time allowance for this is limited and I don't think I'll
be able to spend a lot more time on this than I already have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and
2. a plugin is much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a plugin
now, and when I have got the basic version of that working (i.e. file upload,
RawBioAssay and Experiment creation only, as above) I'll make the code
available to the community somehow and someone else can pick it up from there
if they want to make it fancier. I hope that's okay -- even with no extra
effort from someone else it'll still mean that a hell of a lot of effort can be
saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which will
let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW, once
it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want
from a bulk loader. For example, I would like all objects created and
linked,
from samples (with annotations) right up to raw bioassays (grouped into an
experiment as Micha does, ideally). Obviously to do this you need to
specify
the relationships between all the samples, annotations, dyes, hybs, data
files, but that could easily end up being as much work as manually loading
the
data ;-)
Therefore you have to make compromises; the main one being that you treat
every sample/extract/etc in exactly the same way (same protocols, same
amounts, same labels, same array design). This would probably work for
most
people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment
involves a dye swap for one biological replicate, the bulk loader will
label
each extract with both dyes for *all* replicates (leaving you with unused
labelled extracts after linking to the hybs).
One quite low cost way to define your experiment for a bulk loader is
Tab2MAGE
(soon to be MAGE-TAB, see
http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it
feels
strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2
rather than importing it. The input files for our bulk loader are
specified
here:
http://wiki.vectorbase.org/vectorbase/index.php/MicroarrayExperiments
(look for the example file links) but note that the BASE-specific
parameters
(protocols, raw file parsers, array designs, etc) are configured by the
user
in an interactive process before the upload. Our bulk loader is also
2-channel only...
I have a feeling that the Tab2MAGE parser could be adapted into a BASE 2
bulk
loader somehow, although I think all Tab2MAGE code is Perl.
I think MAGE-TAB is something we could get the average bench scientist to
provide (with a little help MGEDifying the annotations). So far every
experiment I have loaded has come with some kind of spreadsheet (each one
in a
different format of course), so the leap to MAGE-TAB is not so great.
Ok, enough from me...
Micha Bayer writes:
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-
users-
[EMAIL PROTECTED] On Behalf Of Nicklas Nordborg
Sent: 29 November
Re: [base] Offline files
Hi,
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter {
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
That code is correct but to see an import tab you will have to specify that
the plugin is an import plugin .For instance:
public MainType getMainType()
{
return Plugin.MainType.IMPORT;
}
Please, see the plugin examples in base2 source code for more information on
this.
We're planning to make the Excel plugin available for everyone too BTW,
once it's finished.
Cool, this will be good for the community.
Dominic
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Micha Bayer
Sent: 30 November 2006 09:52
To: BASE ML
Subject: Re: [base] Offline files
That sounds like a hell of a lot more work than what I have done so far
:-)
My first shot at this has been quite basic and it only creates new
RawBioAssays and a single Experiment that these belong to. It could of
course be extended to do more stuff, but my time allowance for this is
limited and I don't think I'll be able to spend a lot more time on this than
I already have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and 2. a plugin is
much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a plugin
now, and when I have got the basic version of that working (i.e. file
upload, RawBioAssay and Experiment creation only, as above) I'll make the
code available to the community somehow and someone else can pick it up from
there if they want to make it fancier. I hope that's okay -- even with no
extra effort from someone else it'll still mean that a hell of a lot of
effort can be saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter {
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter {
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW, once
it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want from a bulk loader. For example, I would like all objects
created and linked, from samples (with annotations) right up to raw
bioassays (grouped into an experiment as Micha does, ideally).
Obviously to do this you need to specify the relationships between all
the samples, annotations, dyes, hybs, data files, but that could
easily end up being as much work as manually loading the data ;-)
Therefore you have to make compromises; the main one being that you
treat every sample/extract/etc in exactly the same way (same
protocols, same amounts, same labels, same array design). This would
probably work for most people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment involves a dye swap for one biological replicate, the bulk
loader will label each extract with both dyes for *all* replicates
(leaving you with unused labelled extracts after linking to the hybs).
One quite low cost way to define your experiment for a bulk loader is
Tab2MAGE (soon to be MAGE-TAB, see
http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it
feels strange to say this because I have been *exporting* Tab2MAGE
from BASE 1.2 rather than importing it. The input files for our bulk
Re: [base] Offline files
Micha Bayer wrote:
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the
zip file using my plugin?
If you want the plugin to appear on the list page you should use,
GuiContext.Type.List. The GuiContext.Type.ITEM makes it appear in the
view page for a single experiment.
/Nicklas
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
The BASE general discussion mailing list
basedb-users@lists.sourceforge.net
unsubscribe: send a mail with subject unsubscribe to
[EMAIL PROTECTED]
Re: [base] Offline files
Hi Micha and others,
Of course we have to share the workload. I'd rather help develop something
that is already taking shape (because I'm still not very familiar with the
API). Let's see what people think about MAGE-TAB etc, but hopefully I can
carry on where you leave off. That's not a 100% commitment, but I hope I'll
have the time/ability to do this.
Just one thought, isn't another option to cannibalise the migration code to
make a command-line bulk importer? I guess that's no good for sites who want
the users to submit through the web interface...
Another thought: my BASE 1 loaders were bolted on top of BASE and did all the
sample creation etc through HTTP requests to the BASE 1 GUI with a Perl LWP
robot. These could be modified and bolted on to BASE 2, although the approach
is very sensitive to GUI changes, and just doesn't feel like a long-term
solution to me... (the motivation to do it this way with BASE 1 was to take
the path of least resistance and avoid learning the PHP API).
cheers,
Bob.
Micha Bayer writes:
That sounds like a hell of a lot more work than what I have done so far
:-)
My first shot at this has been quite basic and it only creates new
RawBioAssays and a single Experiment that these belong to. It could of
course be extended to do more stuff, but my time allowance for this is
limited and I don't think I'll be able to spend a lot more time on this than
I already have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and
2. a plugin is much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a plugin
now, and when I have got the basic version of that working (i.e. file
upload, RawBioAssay and Experiment creation only, as above) I'll make the
code available to the community somehow and someone else can pick it up from
there if they want to make it fancier. I hope that's okay -- even with no
extra effort from someone else it'll still mean that a hell of a lot of
effort can be saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW, once
it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want
from a bulk loader. For example, I would like all objects created and
linked,
from samples (with annotations) right up to raw bioassays (grouped into an
experiment as Micha does, ideally). Obviously to do this you need to
specify
the relationships between all the samples, annotations, dyes, hybs, data
files, but that could easily end up being as much work as manually loading
the
data ;-)
Therefore you have to make compromises; the main one being that you treat
every sample/extract/etc in exactly the same way (same protocols, same
amounts, same labels, same array design). This would probably work for
most
people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment
involves a dye swap for one biological replicate, the bulk loader will
label
each extract with both dyes for *all* replicates (leaving you with unused
labelled extracts after linking to the hybs).
One quite low cost way to define your experiment for a bulk loader is
Tab2MAGE
Re: [base] Offline files
There is a possibility to get access to the subversion repository and
trac environment just for this kind of stuff. Check out
http://baseplugins.thep.lu.se Trac environment allows for discussions
about the importer through the ticketing system.
Jari
Micha Bayer wrote:
That sounds like a hell of a lot more work than what I have done so far
:-)
My first shot at this has been quite basic and it only creates new
RawBioAssays and a single Experiment that these belong to. It could of course
be extended to do more stuff, but my time allowance for this is limited and I
don't think I'll be able to spend a lot more time on this than I already
have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and
2. a plugin is much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a plugin
now, and when I have got the basic version of that working (i.e. file upload,
RawBioAssay and Experiment creation only, as above) I'll make the code
available to the community somehow and someone else can pick it up from there
if they want to make it fancier. I hope that's okay -- even with no extra
effort from someone else it'll still mean that a hell of a lot of effort can
be saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW, once
it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want
from a bulk loader. For example, I would like all objects created and
linked,
from samples (with annotations) right up to raw bioassays (grouped into an
experiment as Micha does, ideally). Obviously to do this you need to
specify
the relationships between all the samples, annotations, dyes, hybs, data
files, but that could easily end up being as much work as manually loading
the
data ;-)
Therefore you have to make compromises; the main one being that you treat
every sample/extract/etc in exactly the same way (same protocols, same
amounts, same labels, same array design). This would probably work for
most
people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment
involves a dye swap for one biological replicate, the bulk loader will
label
each extract with both dyes for *all* replicates (leaving you with unused
labelled extracts after linking to the hybs).
One quite low cost way to define your experiment for a bulk loader is
Tab2MAGE
(soon to be MAGE-TAB, see
http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it
feels
strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2
rather than importing it. The input files for our bulk loader are
specified
here:
http://wiki.vectorbase.org/vectorbase/index.php/MicroarrayExperiments
(look for the example file links) but note that the BASE-specific
parameters
(protocols, raw file parsers, array designs, etc) are configured by the
user
in an interactive process before the upload. Our bulk loader is also
2-channel only...
I have a feeling that the Tab2MAGE parser could be adapted into a BASE 2
bulk
loader somehow, although I think all Tab2MAGE code is Perl.
I think MAGE-TAB is something we could get the average bench scientist to
provide (with a little help MGEDifying the annotations). So far every
experiment I have loaded has come
Re: [base] Offline files
A command line bulk importer ... well, we are just starting to look at
adding remote services to BASE. Web services or something similar might
be useful in this context.
Jari
Bob MacCallum wrote:
Hi Micha and others,
Of course we have to share the workload. I'd rather help develop something
that is already taking shape (because I'm still not very familiar with the
API). Let's see what people think about MAGE-TAB etc, but hopefully I can
carry on where you leave off. That's not a 100% commitment, but I hope I'll
have the time/ability to do this.
Just one thought, isn't another option to cannibalise the migration code to
make a command-line bulk importer? I guess that's no good for sites who want
the users to submit through the web interface...
Another thought: my BASE 1 loaders were bolted on top of BASE and did all the
sample creation etc through HTTP requests to the BASE 1 GUI with a Perl LWP
robot. These could be modified and bolted on to BASE 2, although the approach
is very sensitive to GUI changes, and just doesn't feel like a long-term
solution to me... (the motivation to do it this way with BASE 1 was to take
the path of least resistance and avoid learning the PHP API).
cheers,
Bob.
Micha Bayer writes:
That sounds like a hell of a lot more work than what I have done so
far :-)
My first shot at this has been quite basic and it only creates new
RawBioAssays and a single Experiment that these belong to. It could of course
be extended to do more stuff, but my time allowance for this is limited and I
don't think I'll be able to spend a lot more time on this than I already
have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and
2. a plugin is much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a
plugin now, and when I have got the basic version of that working (i.e. file
upload, RawBioAssay and Experiment creation only, as above) I'll make the
code available to the community somehow and someone else can pick it up from
there if they want to make it fancier. I hope that's okay -- even with no
extra effort from someone else it'll still mean that a hell of a lot of
effort can be saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW,
once it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want
from a bulk loader. For example, I would like all objects created and
linked,
from samples (with annotations) right up to raw bioassays (grouped into
an
experiment as Micha does, ideally). Obviously to do this you need to
specify
the relationships between all the samples, annotations, dyes, hybs, data
files, but that could easily end up being as much work as manually
loading
the
data ;-)
Therefore you have to make compromises; the main one being that you treat
every sample/extract/etc in exactly the same way (same protocols, same
amounts, same labels, same array design). This would probably work for
most
people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment
involves
Re: [base] Offline files
= -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Bob MacCallum Sent: 29 November 2006 18:36 To: BASE ML Subject: Re: [base] Offline files It looks like I let the cat out of the bag on this one... Some discussion among the community would be good to establish what we want from a bulk loader. For example, I would like all objects created and linked, from samples (with annotations) right up to raw bioassays (grouped into an experiment as Micha does, ideally). Obviously to do this you need to specify the relationships between all the samples, annotations, dyes, hybs, data files, but that could easily end up being as much work as manually loading the data ;-) Therefore you have to make compromises; the main one being that you treat every sample/extract/etc in exactly the same way (same protocols, same amounts, same labels, same array design). This would probably work for most people - but let's discuss it anyway... One issue with the bulk loader I wrote for BASE 1 is that if your experiment involves a dye swap for one biological replicate, the bulk loader will label each extract with both dyes for *all* replicates (leaving you with unused labelled extracts after linking to the hybs). One quite low cost way to define your experiment for a bulk loader is Tab2MAGE (soon to be MAGE-TAB, see http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it feels strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2 rather than importing it. The input files for our bulk loader are specified here: http://wiki.vectorbase.org/vectorbase/index.php/MicroarrayExperiments (look for the example file links) but note that the BASE-specific parameters (protocols, raw file parsers, array designs, etc) are configured by the user in an interactive process before the upload. Our bulk loader is also 2-channel only... I have a feeling that the Tab2MAGE parser could be adapted into a BASE 2 bulk loader somehow, although I think all Tab2MAGE code is Perl. I think MAGE-TAB is something we could get the average bench scientist to provide (with a little help MGEDifying the annotations). So far every experiment I have loaded has come with some kind of spreadsheet (each one in a different format of course), so the leap to MAGE-TAB is not so great. Ok, enough from me... Micha Bayer writes: -Original Message- From: [EMAIL PROTECTED] [mailto:basedb- users- [EMAIL PROTECTED] On Behalf Of Nicklas Nordborg Sent: 29 November 2006 14:40 To: BASE ML Subject: Re: [base] Offline files Jari Häkkinen wrote: Hi all, Just a thought, I think the best choice to trigger a large import like this is to create a plug-in instead of adding extra tabs to the file browser. The plug-in should start by requesting the zip file/directory to import. If the zip file is not already uploaded the user will have an opportunity to upload it. After choosing the zip file/directory the plug-in could start its import and creation of necessary items. The above is fairly straightforward already today and the new code would start at import/creation of necessary items. This will yield very little (or no) changes to the core and the plug-in should probably be started from the experiment listing page. This will of course change the experiment list jsp (a import tab is needed). No change is needed to the jsp. The new plugin will be detected and an Import button will appear. /Nicklas So what interface would I have to implement for this to happen, if I were to turn my code into a plugin? If it's not too much work I might consider doing this, that would be much better. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ DISCLAIMER: This email is from the Scottish Crop Research Institute, but the views expressed by the sender are not necessarily the views of SCRI and its subsidiaries. This email and any files transmitted with it are confidential to the intended recipient at the e-mail
Re: [base] Offline files
Nicklas Nordborg writes: Giovanni Coppola wrote: Hi everybody, I have two questions: 1) is there a way to download images or raw data files in batches? No. yes but there is a zip file unpacker (in BASE 2) I haven't used it, but doesn't this cut down on the GUI clicking if you want to upload a lot of files? At some point, someone will come up with a bulk loader which will batch/bulk/mass upload files *and* create/annotate/link the data objects (I'm not volunteering, yet...) 2) while I was trying that (to download more than one file at once), in the 'Files and Directories' tree, I accidentally moved Offline some files, and now I don't seem able to download them anymore. Is there a way to restore them in the Primary position? They have been deleted from the server. You have to upload them again. /Nicklas - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED] -- Bob MacCallum | VectorBase Developer | Kafatos/Christophides Groups | Division of Cell and Molecular Biology | Imperial College London | Phone +442075941945 | Email [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
The zip up-loader is meant to be used for uploading bulks of files, and is working well. For retrieving files from the server there is no batch support. Jari Bob MacCallum wrote: Nicklas Nordborg writes: Giovanni Coppola wrote: Hi everybody, I have two questions: 1) is there a way to download images or raw data files in batches? No. yes but there is a zip file unpacker (in BASE 2) I haven't used it, but doesn't this cut down on the GUI clicking if you want to upload a lot of files? At some point, someone will come up with a bulk loader which will batch/bulk/mass upload files *and* create/annotate/link the data objects (I'm not volunteering, yet...) 2) while I was trying that (to download more than one file at once), in the 'Files and Directories' tree, I accidentally moved Offline some files, and now I don't seem able to download them anymore. Is there a way to restore them in the Primary position? They have been deleted from the server. You have to upload them again. /Nicklas - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED] -- Jari Häkkinen, PhD Complex Systems Divisionmailto:[EMAIL PROTECTED] Department of Theoretical Physics phone: +46 (0)46 2229347 Lund University fax: +46 (0)46 2229686 Sölvegatan 14a, SE-223 62 Lund, Sweden http://www.thep.lu.se - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
What does mean to move offline ? Is it the same as: Delete THEN Empty trash ? -- Emmanuel COURCELLE[EMAIL PROTECTED] L.I.P.M. (UMR CNRS-INRA 2594/441) tel (33) 5-61-28-54-50 B.P.52627 - 31326 CASTANET TOLOSAN Cedex - FRANCE -- - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Emmanuel Courcelle wrote: What does mean to move offline ? Is it the same as: Delete THEN Empty trash ? No it is not. A file exists in two places. As an entry in the database and as a file on the disk. The Move offline feature is used when you want to delete the file on disk but keep the entry in the database. The main reason for this may be that your quota is running out and you don't need the file anymore. A file that is offline may be re-uploaded if you need it again. If you delete a file and empty the trashcan both the entry in the database and the file on the disk will be deleted. /Nicklas - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
-Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Bob MacCallum Sent: 29 November 2006 10:15 To: BASE ML Subject: Re: [base] Offline files [...] At some point, someone will come up with a bulk loader which will batch/bulk/mass upload files *and* create/annotate/link the data objects (I'm not volunteering, yet...) I have just finished writing one. The user uploads a zip file, a new directory is created in the user's raw data directory (with the same name as the zip file) and the files are then unzipped in there. They are then imported one by one as new RawBioAssay objects and at the end a new Experiment is created that has all these hanging off it. At the moment it only works for non-Affy data though. To do this I had to hack the BASE source (plugin wasn't possible as far as I can tell), and I am always reluctant to do that sort of thing because of versioning/upgrade issues (i.e. having to repeat this every time a new version is out). I have modified the JSP page that controls the menus and inserted a new item there, and added a couple of JSP pages myself that provide the UI for all this. I did some more detailed testing yesterday and things worked fine but it's all still pretty raw and probably full of bugs. I am happy to share the code with others though, or even make it a full contribution to BASE if people are happy with it (perhaps after it has been gone over by some of the BASE team). I also think the exception handling needs more work probably. cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ DISCLAIMER: This email is from the Scottish Crop Research Institute, but the views expressed by the sender are not necessarily the views of SCRI and its subsidiaries. This email and any files transmitted with it are confidential to the intended recipient at the e-mail address to which it has been addressed. It may not be disclosed or used by any other than that addressee. If you are not the intended recipient you are requested to preserve this confidentiality and you must not use, disclose, copy, print or rely on this e-mail in any way. Please notify [EMAIL PROTECTED] quoting the name of the sender and delete the email from your system. Although SCRI has taken reasonable precautions to ensure no viruses are present in this email, neither the Institute nor the sender accepts any responsibility for any viruses, and it is your responsibility to scan the email and the attachments (if any). - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Hi Micha, i would love to see how you accomplished this. after having put in over 100 RawBioAssays plus affiliated biosources, samples, extracts, labeled extracts, hybridizations, and scans by hand.. i am strongly motivated to program an automated solution. -keith Micha Bayer wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Bob MacCallum Sent: 29 November 2006 10:15 To: BASE ML Subject: Re: [base] Offline files [...] At some point, someone will come up with a bulk loader which will batch/bulk/mass upload files *and* create/annotate/link the data objects (I'm not volunteering, yet...) I have just finished writing one. The user uploads a zip file, a new directory is created in the user's raw data directory (with the same name as the zip file) and the files are then unzipped in there. They are then imported one by one as new RawBioAssay objects and at the end a new Experiment is created that has all these hanging off it. At the moment it only works for non-Affy data though. To do this I had to hack the BASE source (plugin wasn't possible as far as I can tell), and I am always reluctant to do that sort of thing because of versioning/upgrade issues (i.e. having to repeat this every time a new version is out). I have modified the JSP page that controls the menus and inserted a new item there, and added a couple of JSP pages myself that provide the UI for all this. I did some more detailed testing yesterday and things worked fine but it's all still pretty raw and probably full of bugs. I am happy to share the code with others though, or even make it a full contribution to BASE if people are happy with it (perhaps after it has been gone over by some of the BASE team). I also think the exception handling needs more work probably. cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ DISCLAIMER: This email is from the Scottish Crop Research Institute, but the views expressed by the sender are not necessarily the views of SCRI and its subsidiaries. This email and any files transmitted with it are confidential to the intended recipient at the e-mail address to which it has been addressed. It may not be disclosed or used by any other than that addressee. If you are not the intended recipient you are requested to preserve this confidentiality and you must not use, disclose, copy, print or rely on this e-mail in any way. Please notify [EMAIL PROTECTED] quoting the name of the sender and delete the email from your system. Although SCRI has taken reasonable precautions to ensure no viruses are present in this email, neither the Institute nor the sender accepts any responsibility for any viruses, and it is your responsibility to scan the email and the attachments (if any). - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED] -- Keith Ching, Ph.D. Bioinformatics, Laboratory of Gene Regulation Ludwig Institute for Cancer Research 9500 Gilman Drive, #0653 CMM East, Room 3020 La Jolla, CA 92093-0653 [EMAIL PROTECTED] 858-822-5767 - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Micha Bayer Hi Micha, I have just finished writing one. The user uploads a zip file, a new directory is created in the user's raw data directory (with the same name as the zip file) and the files are then unzipped in there. They are then imported one by one as new RawBioAssay objects and at the end a new Experiment is created that has all these hanging off it. At the moment it only works for non-Affy data though. To do this I had to hack the BASE source (plugin wasn't possible as far as I can tell), and I am always reluctant to do that sort of thing because of versioning/upgrade issues (i.e. having to repeat this every time a new version is out). I have modified the JSP page that controls the menus and inserted a new item there, and added a couple of JSP pages myself that provide the UI for all this. I did some more detailed testing yesterday and things worked fine but it's all still pretty raw and probably full of bugs. I am happy to share the code with others though, or even make it a full contribution to BASE if people are happy with it (perhaps after it has been gone over by some of the BASE team). I also think the exception handling needs more work probably. Base 2 has plans to have tab2mage import (importing experiments and associated files) see Ticket #338 on base2 trac http://base.thep.lu.se/ticket/338 system, and your zip file loader would certaily help to in importing experiments from repositories and doing other interesting stuff. We have this on our do list and would certainly appreciate if you can share the code with us to help in this process. Also, do you have plans to extend the functionality to non-affy data too? Thanks and hope to read from you. Dominic - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Hi all, Just a thought, I think the best choice to trigger a large import like this is to create a plug-in instead of adding extra tabs to the file browser. The plug-in should start by requesting the zip file/directory to import. If the zip file is not already uploaded the user will have an opportunity to upload it. After choosing the zip file/directory the plug-in could start its import and creation of necessary items. The above is fairly straightforward already today and the new code would start at import/creation of necessary items. This will yield very little (or no) changes to the core and the plug-in should probably be started from the experiment listing page. This will of course change the experiment list jsp (a import tab is needed). In my naive view this will only affect a jsp and a new plug-in is needed. Jari dominic oyeniran wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Micha Bayer Hi Micha, I have just finished writing one. The user uploads a zip file, a new directory is created in the user's raw data directory (with the same name as the zip file) and the files are then unzipped in there. They are then imported one by one as new RawBioAssay objects and at the end a new Experiment is created that has all these hanging off it. At the moment it only works for non-Affy data though. To do this I had to hack the BASE source (plugin wasn't possible as far as I can tell), and I am always reluctant to do that sort of thing because of versioning/upgrade issues (i.e. having to repeat this every time a new version is out). I have modified the JSP page that controls the menus and inserted a new item there, and added a couple of JSP pages myself that provide the UI for all this. I did some more detailed testing yesterday and things worked fine but it's all still pretty raw and probably full of bugs. I am happy to share the code with others though, or even make it a full contribution to BASE if people are happy with it (perhaps after it has been gone over by some of the BASE team). I also think the exception handling needs more work probably. Base 2 has plans to have tab2mage import (importing experiments and associated files) see Ticket #338 on base2 trac http://base.thep.lu.se/ticket/338 system, and your zip file loader would certaily help to in importing experiments from repositories and doing other interesting stuff. We have this on our do list and would certainly appreciate if you can share the code with us to help in this process. Also, do you have plans to extend the functionality to non-affy data too? Thanks and hope to read from you. Dominic - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED] -- Jari Häkkinen, PhD Complex Systems Divisionmailto:[EMAIL PROTECTED] Department of Theoretical Physics phone: +46 (0)46 2229347 Lund University fax: +46 (0)46 2229686 Sölvegatan 14a, SE-223 62 Lund, Sweden http://www.thep.lu.se - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Hello everybody, thank you for you replies. I'll transform my questions in suggestions: 1) data batch downloading would be great, since this is a pretty basic task, many experiments have 30+ arrays, and right now the user needs two clicks for each. 2) it is extremely misleading, to have a delete button and a trash can, but also something that permanently deletes your files, under a *Move* menu... named *Offline*... It would make more sense to me to let the user choose (under Delete) whether to keep the entry or not when deleting a file. Thanks again for this nice tool Giovanni On Nov 29, 2006, at 6:14 AM, Jari Häkkinen wrote: Hi all, Just a thought, I think the best choice to trigger a large import like this is to create a plug-in instead of adding extra tabs to the file browser. The plug-in should start by requesting the zip file/directory to import. If the zip file is not already uploaded the user will have an opportunity to upload it. After choosing the zip file/directory the plug-in could start its import and creation of necessary items. The above is fairly straightforward already today and the new code would start at import/creation of necessary items. This will yield very little (or no) changes to the core and the plug-in should probably be started from the experiment listing page. This will of course change the experiment list jsp (a import tab is needed). In my naive view this will only affect a jsp and a new plug-in is needed. Jari dominic oyeniran wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Micha Bayer Hi Micha, I have just finished writing one. The user uploads a zip file, a new directory is created in the user's raw data directory (with the same name as the zip file) and the files are then unzipped in there. They are then imported one by one as new RawBioAssay objects and at the end a new Experiment is created that has all these hanging off it. At the moment it only works for non-Affy data though. To do this I had to hack the BASE source (plugin wasn't possible as far as I can tell), and I am always reluctant to do that sort of thing because of versioning/upgrade issues (i.e. having to repeat this every time a new version is out). I have modified the JSP page that controls the menus and inserted a new item there, and added a couple of JSP pages myself that provide the UI for all this. I did some more detailed testing yesterday and things worked fine but it's all still pretty raw and probably full of bugs. I am happy to share the code with others though, or even make it a full contribution to BASE if people are happy with it (perhaps after it has been gone over by some of the BASE team). I also think the exception handling needs more work probably. Base 2 has plans to have tab2mage import (importing experiments and associated files) see Ticket #338 on base2 trac http://base.thep.lu.se/ticket/338 system, and your zip file loader would certaily help to in importing experiments from repositories and doing other interesting stuff. We have this on our do list and would certainly appreciate if you can share the code with us to help in this process. Also, do you have plans to extend the functionality to non-affy data too? Thanks and hope to read from you. Dominic - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php? page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED] -- Jari Häkkinen, PhD Complex Systems Divisionmailto:[EMAIL PROTECTED] Department of Theoretical Physics phone: +46 (0)46 2229347 Lund University fax: +46 (0)46 2229686 Sölvegatan 14a, SE-223 62 Lund, Sweden http://www.thep.lu.se -- --- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php? page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the
Re: [base] Offline files
Hi Jari, That sounds perfectly reasonable when you put it like this, but I was put off going down that route by a post to the list from Nicklas (http://www.mail-archive.com/basedb-users@lists.sourceforge.net/msg00228.html) where he says that it is best to not use a plugin to do this because of interface issues (whatever they are). Personally, I think a plugin would be far better than what I have done -- if I was to persist with my solution I would have to modify the BASE source every time a new version comes out. I also think that batch import *is* a high priority thing for most users, whether they have a pile of legacy data or not. Most biologists I know would probably rather spend a solid period doing labwork, and then process a whole bunch of experiments in a oner. I also think it is a nonsense to have a human doing boring, repetitive stuff -- computers are there to do that. Why waste your time on something that a for loop can do. So perhaps the dev team should reconsider this. I am just talking to my boss about code ownership issues but I can probably offer up my stuff one way or another, and people can then either cannibalise it and turn it into a plugin, or adopt it (or a modified version) into the core etc etc -- happy with any solution that makes others benefit from my blood sweat and tears shed . :-) Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Jari Häkkinen Sent: 29 November 2006 14:14 To: BASE ML Subject: Re: [base] Offline files Hi all, Just a thought, I think the best choice to trigger a large import like this is to create a plug-in instead of adding extra tabs to the file browser. The plug-in should start by requesting the zip file/directory to import. If the zip file is not already uploaded the user will have an opportunity to upload it. After choosing the zip file/directory the plug-in could start its import and creation of necessary items. The above is fairly straightforward already today and the new code would start at import/creation of necessary items. This will yield very little (or no) changes to the core and the plug-in should probably be started from the experiment listing page. This will of course change the experiment list jsp (a import tab is needed). In my naive view this will only affect a jsp and a new plug-in is needed. Jari dominic oyeniran wrote: -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Micha Bayer Hi Micha, I have just finished writing one. The user uploads a zip file, a new directory is created in the user's raw data directory (with the same name as the zip file) and the files are then unzipped in there. They are then imported one by one as new RawBioAssay objects and at the end a new Experiment is created that has all these hanging off it. At the moment it only works for non-Affy data though. To do this I had to hack the BASE source (plugin wasn't possible as far as I can tell), and I am always reluctant to do that sort of thing because of versioning/upgrade issues (i.e. having to repeat this every time a new version is out). I have modified the JSP page that controls the menus and inserted a new item there, and added a couple of JSP pages myself that provide the UI for all this. I did some more detailed testing yesterday and things worked fine but it's all still pretty raw and probably full of bugs. I am happy to share the code with others though, or even make it a full contribution to BASE if people are happy with it (perhaps after it has been gone over by some of the BASE team). I also think the exception handling needs more work probably. Base 2 has plans to have tab2mage import (importing experiments and associated files) see Ticket #338 on base2 trac http://base.thep.lu.se/ticket/338 system, and your zip file loader would certaily help to in importing experiments from repositories and doing other interesting stuff. We have this on our do list and would certainly appreciate if you can share the code with us to help in this process. Also, do you have plans to extend the functionality to non-affy data too? Thanks and hope to read from you. Dominic - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com
Re: [base] Offline files
Jari Häkkinen wrote: Hi all, Just a thought, I think the best choice to trigger a large import like this is to create a plug-in instead of adding extra tabs to the file browser. The plug-in should start by requesting the zip file/directory to import. If the zip file is not already uploaded the user will have an opportunity to upload it. After choosing the zip file/directory the plug-in could start its import and creation of necessary items. The above is fairly straightforward already today and the new code would start at import/creation of necessary items. This will yield very little (or no) changes to the core and the plug-in should probably be started from the experiment listing page. This will of course change the experiment list jsp (a import tab is needed). No change is needed to the jsp. The new plugin will be detected and an Import button will appear. /Nicklas - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Ah, even better ;-) Nicklas Nordborg wrote: Jari Häkkinen wrote: Hi all, Just a thought, I think the best choice to trigger a large import like this is to create a plug-in instead of adding extra tabs to the file browser. The plug-in should start by requesting the zip file/directory to import. If the zip file is not already uploaded the user will have an opportunity to upload it. After choosing the zip file/directory the plug-in could start its import and creation of necessary items. The above is fairly straightforward already today and the new code would start at import/creation of necessary items. This will yield very little (or no) changes to the core and the plug-in should probably be started from the experiment listing page. This will of course change the experiment list jsp (a import tab is needed). No change is needed to the jsp. The new plugin will be detected and an Import button will appear. /Nicklas - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED] -- Jari Häkkinen, PhD Complex Systems Divisionmailto:[EMAIL PROTECTED] Department of Theoretical Physics phone: +46 (0)46 2229347 Lund University fax: +46 (0)46 2229686 Sölvegatan 14a, SE-223 62 Lund, Sweden http://www.thep.lu.se - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Micha Bayer wrote: I am just talking to my boss about code ownership issues but I can probably offer up my stuff one way or another, and people can then either cannibalise it and turn it into a plugin, or adopt it (or a modified version) into the core etc etc -- happy with any solution that makes others benefit from my blood sweat and tears shed . :-) Please note that the GPL license BASE is released under requires that work based on it is released as GPL unless you keep it private. In other words if you share your code with others, you are required to release it as GPL as well. /Nicklas - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Micha Bayer wrote: Hi Jari, That sounds perfectly reasonable when you put it like this, but I was put off going down that route by a post to the list from Nicklas http://www.mail-archive.com/basedb-users@lists.sourceforge.net/msg00228.html) where he says that it is best to not use a plugin to do this because of interface issues (whatever they are). That message was referring to how it was done in Base 1, where a lot of information needed to be entered in the web interface. The batch import described earlier in this thread probably doesn't need any advanced user interface since it seems (from the description below) to get all the information it needs from a single zip file. I have just finished writing one. The user uploads a zip file, a new directory is created in the user's raw data directory (with the same name as the zip file) and the files are then unzipped in there. They are then imported one by one as new RawBioAssay objects and at the end a new Experiment is created that has all these hanging off it. /Nicklas - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Micha Bayer wrote: Hi Jari, That sounds perfectly reasonable when you put it like this, but I was put off going down that route by a post to the list from Nicklas (http://www.mail-archive.com/basedb-users@lists.sourceforge.net/msg00228.html) where he says that it is best to not use a plugin to do this because of interface issues (whatever they are). I think Nicklas was thinking about very interactive plug-ins. Personally, I think a plugin would be far better than what I have done -- if I was to persist with my solution I would have to modify the BASE source every time a new version comes out. Our belief is that basically no modification to BASE is needed if one creates a import plug-in that creates all needed items automatically (using information from files of course). Trouble starts when communication is needed from the user after plug-in start. The plug-in system allows for communication just before a plug-in is started (at least was design like this, I suppose one can tweak user interaction with the plug-in but thats out of this topic). I also think that batch import *is* a high priority thing for most users, whether they have a pile of legacy data or not. Most biologists I know would probably rather spend a solid period doing labwork, and then process a whole bunch of experiments in a oner. I also think it is a nonsense to have a human doing boring, repetitive stuff -- computers are there to do that. Why waste your time on something that a for loop can do. So perhaps the dev team should reconsider this. I agree, I think it is an important issue as well. However, we think that we work on many other important issues already. We are mainly working on core stuff and hoping that others contribute with things that can be written as plug-ins. You are already doing this, and unfortunately you were pushed in the wrong direction. I am just talking to my boss about code ownership issues but I can probably offer up my stuff one way or another, and people can then either cannibalise it and turn it into a plugin, or adopt it (or a modified version) into the core etc etc -- happy with any solution that makes others benefit from my blood sweat and tears shed . :-) Dominic would probably benefit from your code and through him the code will probably end up in the BASE repository. Contributions are welcome and we consider all of them for inclusion the the BASE code base. Cheers Micha Cheers, Jari - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
-Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Nicklas Nordborg Sent: 29 November 2006 14:40 To: BASE ML Subject: Re: [base] Offline files Jari Häkkinen wrote: Hi all, Just a thought, I think the best choice to trigger a large import like this is to create a plug-in instead of adding extra tabs to the file browser. The plug-in should start by requesting the zip file/directory to import. If the zip file is not already uploaded the user will have an opportunity to upload it. After choosing the zip file/directory the plug-in could start its import and creation of necessary items. The above is fairly straightforward already today and the new code would start at import/creation of necessary items. This will yield very little (or no) changes to the core and the plug-in should probably be started from the experiment listing page. This will of course change the experiment list jsp (a import tab is needed). No change is needed to the jsp. The new plugin will be detected and an Import button will appear. /Nicklas So what interface would I have to implement for this to happen, if I were to turn my code into a plugin? If it's not too much work I might consider doing this, that would be much better. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ DISCLAIMER: This email is from the Scottish Crop Research Institute, but the views expressed by the sender are not necessarily the views of SCRI and its subsidiaries. This email and any files transmitted with it are confidential to the intended recipient at the e-mail address to which it has been addressed. It may not be disclosed or used by any other than that addressee. If you are not the intended recipient you are requested to preserve this confidentiality and you must not use, disclose, copy, print or rely on this e-mail in any way. Please notify [EMAIL PROTECTED] quoting the name of the sender and delete the email from your system. Although SCRI has taken reasonable precautions to ensure no viruses are present in this email, neither the Institute nor the sender accepts any responsibility for any viruses, and it is your responsibility to scan the email and the attachments (if any). - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
It looks like I let the cat out of the bag on this one... Some discussion among the community would be good to establish what we want from a bulk loader. For example, I would like all objects created and linked, from samples (with annotations) right up to raw bioassays (grouped into an experiment as Micha does, ideally). Obviously to do this you need to specify the relationships between all the samples, annotations, dyes, hybs, data files, but that could easily end up being as much work as manually loading the data ;-) Therefore you have to make compromises; the main one being that you treat every sample/extract/etc in exactly the same way (same protocols, same amounts, same labels, same array design). This would probably work for most people - but let's discuss it anyway... One issue with the bulk loader I wrote for BASE 1 is that if your experiment involves a dye swap for one biological replicate, the bulk loader will label each extract with both dyes for *all* replicates (leaving you with unused labelled extracts after linking to the hybs). One quite low cost way to define your experiment for a bulk loader is Tab2MAGE (soon to be MAGE-TAB, see http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it feels strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2 rather than importing it. The input files for our bulk loader are specified here: http://wiki.vectorbase.org/vectorbase/index.php/MicroarrayExperiments (look for the example file links) but note that the BASE-specific parameters (protocols, raw file parsers, array designs, etc) are configured by the user in an interactive process before the upload. Our bulk loader is also 2-channel only... I have a feeling that the Tab2MAGE parser could be adapted into a BASE 2 bulk loader somehow, although I think all Tab2MAGE code is Perl. I think MAGE-TAB is something we could get the average bench scientist to provide (with a little help MGEDifying the annotations). So far every experiment I have loaded has come with some kind of spreadsheet (each one in a different format of course), so the leap to MAGE-TAB is not so great. Ok, enough from me... Micha Bayer writes: -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Nicklas Nordborg Sent: 29 November 2006 14:40 To: BASE ML Subject: Re: [base] Offline files Jari Häkkinen wrote: Hi all, Just a thought, I think the best choice to trigger a large import like this is to create a plug-in instead of adding extra tabs to the file browser. The plug-in should start by requesting the zip file/directory to import. If the zip file is not already uploaded the user will have an opportunity to upload it. After choosing the zip file/directory the plug-in could start its import and creation of necessary items. The above is fairly straightforward already today and the new code would start at import/creation of necessary items. This will yield very little (or no) changes to the core and the plug-in should probably be started from the experiment listing page. This will of course change the experiment list jsp (a import tab is needed). No change is needed to the jsp. The new plugin will be detected and an Import button will appear. /Nicklas So what interface would I have to implement for this to happen, if I were to turn my code into a plugin? If it's not too much work I might consider doing this, that would be much better. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ DISCLAIMER: This email is from the Scottish Crop Research Institute, but the views expressed by the sender are not necessarily the views of SCRI and its subsidiaries. This email and any files transmitted with it are confidential to the intended recipient at the e-mail address to which it has been addressed. It may not be disclosed or used by any other than that addressee. If you are not the intended recipient you are requested to preserve this confidentiality and you must not use, disclose, copy, print or rely on this e-mail in any way. Please notify [EMAIL PROTECTED] quoting the name of the sender and delete the email from your system. Although SCRI has taken reasonable precautions to ensure no viruses are present in this email, neither the Institute nor the sender accepts any responsibility for any viruses, and it is your responsibility to scan the email and the attachments (if any
Re: [base] Offline files
I think a bulk importer is essential. We have found that all users enter data in bulk. They set aside some time to enter a set of data, usually from the same experiment. Having the user enter each chip individually leads to user frustration and clerical errors. We created a bulk uploader which takes a annotation file ( one line per chip). Each line contains all the information necessary to create everything from sample to rawbioassay. Unfortunately the uploader is useless for BASE2. It would be great to have a plugin that was independent of BASE version. Anup Parikh, Shaulsky Lab Baylor College of Medicine - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
Giovanni Coppola wrote: Hi everybody, I have two questions: 1) is there a way to download images or raw data files in batches? No. 2) while I was trying that (to download more than one file at once), in the 'Files and Directories' tree, I accidentally moved Offline some files, and now I don't seem able to download them anymore. Is there a way to restore them in the Primary position? They have been deleted from the server. You have to upload them again. /Nicklas - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
