[base] JEP extra value calculator again...
Hi all. Sorry to bother you again with this point, but I still does not understand everything... I tried to define a new type, called float colored: it is a float, colored with min value as 0, max as 0, mid as 5 I want to add 2 values to my data: -2 * (ch(1) / ch(2)) -3 * (ch(1) / ch(2)) (Probably stupid for the analysis, but I just want to check the functionality) So, I tried the JEP extra value calculator, specifying float coloured as a data type, and 2 * (ch(1) / ch(2)) as the expression. Then, I give a look to the data, and it works: the numbers are preceded with a little coloured square, the colour depends on the value. Great, indeed. So, I tried a second run, this time with 3 * (ch(1) / ch(2)) as expression. It got the following error message: The bioassay set already has extra values of the type: Extra value type[id=2824; name=float colored] This sounds very strange to me: If float colored is an Extra Value Type, (this sounds to me as some data type, it's not right ?) why should it be applied ONLY one time ? Something else, which is probably a bug. When I go to Administrate/Types/Extra value types, I click on any extra type declared (for instance for editing the type), I get the following: Base version2.1 (build #2847; schema #20) Web server Apache Tomcat/5.5.20 Database Dialectorg.hibernate.dialect.PostgreSQLDialect JDBC Driver org.postgresql.Driver (version 8.1) Java runtimeJava(TM) 2 Runtime Environment, Standard Edition (1.5.0_06-b05), Sun Microsystems Inc. http://java.sun.com/ Operating systemLinux i386 2.6.16-2-em64t-p4-smp Memory Total: 155.3 MB Free: 30.9 MB Max: 986.1 MB Browser Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.8.1) Gecko/20061010 Firefox/2.0 Error message /usr/local/apache-tomcat-5.5.20/work/Catalina/localhost/base2/org/apache/jsp/admin/extravaluetypes/view_005fextravaluetype_jsp.java (Permission denied) Stack trace ...at java.io.FileOutputStream.open(Native Method) ...at java.io.FileOutputStream.(Unknown Source) ...at java.io.FileOutputStream.(Unknown Source) ...at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:140) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:295) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:276) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:264) ...at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:563) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:305) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672) ...at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463) ...at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398) ...at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301) ...at org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:688) ...at org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:658) ...at org.apache.jsp.admin.extravaluetypes.index_jsp._jspService(index_jsp.java:278) ...at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at net.sf.basedb.clients.web.servlet.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:71) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) ...at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178) ...at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126) ...at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105) ...at
Re: [base] JEP extra value calculator again...
Emmanuel Courcelle wrote: Hi all. Sorry to bother you again with this point, but I still does not understand everything... I tried to define a new type, called float colored: it is a float, colored with min value as 0, max as 0, mid as 5 I want to add 2 values to my data: -2 * (ch(1) / ch(2)) -3 * (ch(1) / ch(2)) (Probably stupid for the analysis, but I just want to check the functionality) So, I tried the JEP extra value calculator, specifying float coloured as a data type, and 2 * (ch(1) / ch(2)) as the expression. Then, I give a look to the data, and it works: the numbers are preceded with a little coloured square, the colour depends on the value. Great, indeed. So, I tried a second run, this time with 3 * (ch(1) / ch(2)) as expression. It got the following error message: The bioassay set already has extra values of the type: Extra value type[id=2824; name=float colored] You can only calculate one extra value of each type per bioassayset. This sounds very strange to me: If float colored is an Extra Value Type, (this sounds to me as some data type, it's not right ?) why should it be applied ONLY one time ? Because the extra value type itself can be used in further calculations, filters, by plugins, etc. You then refer to it by the External ID of the extra value type. If there are more than one value for each type, it is not possible to know which one to choose. Something else, which is probably a bug. When I go to Administrate/Types/Extra value types, I click on any extra type declared (for instance for editing the type), I get the following: It looks like some internal Tomcat error. Try removing everything from the /usr/local/apache-tomcat-5.5.20/work/Catalina/localhost/base2/ directory. /Nicklas Base version 2.1 (build #2847; schema #20) Web serverApache Tomcat/5.5.20 Database Dialect org.hibernate.dialect.PostgreSQLDialect JDBC Driver org.postgresql.Driver (version 8.1) Java runtime Java(TM) 2 Runtime Environment, Standard Edition (1.5.0_06-b05), Sun Microsystems Inc. http://java.sun.com/ Operating system Linux i386 2.6.16-2-em64t-p4-smp MemoryTotal: 155.3 MB Free: 30.9 MB Max: 986.1 MB Browser Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.8.1) Gecko/20061010 Firefox/2.0 Error message /usr/local/apache-tomcat-5.5.20/work/Catalina/localhost/base2/org/apache/jsp/admin/extravaluetypes/view_005fextravaluetype_jsp.java (Permission denied) Stack trace ...at java.io.FileOutputStream.open(Native Method) ...at java.io.FileOutputStream.(Unknown Source) ...at java.io.FileOutputStream.(Unknown Source) ...at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:140) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:295) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:276) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:264) ...at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:563) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:305) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:672) ...at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:463) ...at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:398) ...at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301) ...at org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:688) ...at org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:658) ...at org.apache.jsp.admin.extravaluetypes.index_jsp._jspService(index_jsp.java:278) ...at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:334) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) ...at
Re: [base] Offline files
That sounds like a hell of a lot more work than what I have done so far :-)
My first shot at this has been quite basic and it only creates new RawBioAssays
and a single Experiment that these belong to. It could of course be extended to
do more stuff, but my time allowance for this is limited and I don't think I'll
be able to spend a lot more time on this than I already have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and
2. a plugin is much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a plugin
now, and when I have got the basic version of that working (i.e. file upload,
RawBioAssay and Experiment creation only, as above) I'll make the code
available to the community somehow and someone else can pick it up from there
if they want to make it fancier. I hope that's okay -- even with no extra
effort from someone else it'll still mean that a hell of a lot of effort can be
saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which will
let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW, once
it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want
from a bulk loader. For example, I would like all objects created and
linked,
from samples (with annotations) right up to raw bioassays (grouped into an
experiment as Micha does, ideally). Obviously to do this you need to
specify
the relationships between all the samples, annotations, dyes, hybs, data
files, but that could easily end up being as much work as manually loading
the
data ;-)
Therefore you have to make compromises; the main one being that you treat
every sample/extract/etc in exactly the same way (same protocols, same
amounts, same labels, same array design). This would probably work for
most
people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment
involves a dye swap for one biological replicate, the bulk loader will
label
each extract with both dyes for *all* replicates (leaving you with unused
labelled extracts after linking to the hybs).
One quite low cost way to define your experiment for a bulk loader is
Tab2MAGE
(soon to be MAGE-TAB, see
http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it
feels
strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2
rather than importing it. The input files for our bulk loader are
specified
here:
http://wiki.vectorbase.org/vectorbase/index.php/MicroarrayExperiments
(look for the example file links) but note that the BASE-specific
parameters
(protocols, raw file parsers, array designs, etc) are configured by the
user
in an interactive process before the upload. Our bulk loader is also
2-channel only...
I have a feeling that the Tab2MAGE parser could be adapted into a BASE 2
bulk
loader somehow, although I think all Tab2MAGE code is Perl.
I think MAGE-TAB is something we could get the average bench scientist to
provide (with a little help MGEDifying the annotations). So far every
experiment I have loaded has come with some kind of spreadsheet (each one
in a
different format of course), so the leap to MAGE-TAB is not so great.
Ok, enough from me...
Micha Bayer writes:
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-
users-
[EMAIL PROTECTED] On Behalf Of Nicklas Nordborg
Sent: 29 November
Re: [base] JEP extra value calculator again...
Hi Emmanuel, The Catalina/work directory caches copies of your compiled JSP pages. It looks from the stacktrace like the JSP compiler can't write to the work directory because you don't have permissions to it -- check what they are set to. Cheers Micha == Dr Micha M Bayer Bioinformatics Specialist Genetics Programme The Scottish Crop Research Institute Invergowrie Dundee DD2 5DA Scotland, UK Telephone +44(0)1382 562731 ext. 2309 Fax +44(0)1382 562426 http://www.scri.sari.ac.uk/MichaBayer.htm = -Original Message- From: [EMAIL PROTECTED] [mailto:basedb-users- [EMAIL PROTECTED] On Behalf Of Emmanuel Courcelle Sent: 30 November 2006 09:23 To: BASE ML Subject: [base] JEP extra value calculator again... Hi all. Sorry to bother you again with this point, but I still does not understand everything... I tried to define a new type, called float colored: it is a float, colored with min value as 0, max as 0, mid as 5 I want to add 2 values to my data: -2 * (ch(1) / ch(2)) -3 * (ch(1) / ch(2)) (Probably stupid for the analysis, but I just want to check the functionality) So, I tried the JEP extra value calculator, specifying float coloured as a data type, and 2 * (ch(1) / ch(2)) as the expression. Then, I give a look to the data, and it works: the numbers are preceded with a little coloured square, the colour depends on the value. Great, indeed. So, I tried a second run, this time with 3 * (ch(1) / ch(2)) as expression. It got the following error message: The bioassay set already has extra values of the type: Extra value type[id=2824; name=float colored] This sounds very strange to me: If float colored is an Extra Value Type, (this sounds to me as some data type, it's not right ?) why should it be applied ONLY one time ? Something else, which is probably a bug. When I go to Administrate/Types/Extra value types, I click on any extra type declared (for instance for editing the type), I get the following: Base version 2.1 (build #2847; schema #20) Web serverApache Tomcat/5.5.20 Database Dialect org.hibernate.dialect.PostgreSQLDialect JDBC Driver org.postgresql.Driver (version 8.1) Java runtime Java(TM) 2 Runtime Environment, Standard Edition (1.5.0_06-b05), Sun Microsystems Inc. http://java.sun.com/ Operating system Linux i386 2.6.16-2-em64t-p4-smp MemoryTotal: 155.3 MB Free: 30.9 MB Max: 986.1 MB Browser Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.8.1) Gecko/20061010 Firefox/2.0 Error message /usr/local/apache-tomcat- 5.5.20/work/Catalina/localhost/base2/org/apache/jsp/admin/extravaluetype s/ view_005fextravaluetype_jsp.java (Permission denied) Stack trace ...at java.io.FileOutputStream.open(Native Method) ...at java.io.FileOutputStream.(Unknown Source) ...at java.io.FileOutputStream.(Unknown Source) ...at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:140) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:295) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:276) ...at org.apache.jasper.compiler.Compiler.compile(Compiler.java:264) ...at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.ja va :563) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja va :305) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica ti onFilterChain.java:252) ...at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt er Chain.java:173) ...at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatc he r.java:672) ...at org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicatio nD ispatcher.java:463) ...at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDisp at cher.java:398) ...at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispat ch er.java:301) ...at org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java :6 88) ...at org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:6 58 ) ...at org.apache.jsp.admin.extravaluetypes.index_jsp._jspService(index_jsp.jav a: 278) ...at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.ja va :334) ...at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:314) ...at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:264) ...at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) ...at
Re: [base] Offline files
Hi,
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter {
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
That code is correct but to see an import tab you will have to specify that
the plugin is an import plugin .For instance:
public MainType getMainType()
{
return Plugin.MainType.IMPORT;
}
Please, see the plugin examples in base2 source code for more information on
this.
We're planning to make the Excel plugin available for everyone too BTW,
once it's finished.
Cool, this will be good for the community.
Dominic
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Micha Bayer
Sent: 30 November 2006 09:52
To: BASE ML
Subject: Re: [base] Offline files
That sounds like a hell of a lot more work than what I have done so far
:-)
My first shot at this has been quite basic and it only creates new
RawBioAssays and a single Experiment that these belong to. It could of
course be extended to do more stuff, but my time allowance for this is
limited and I don't think I'll be able to spend a lot more time on this than
I already have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and 2. a plugin is
much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a plugin
now, and when I have got the basic version of that working (i.e. file
upload, RawBioAssay and Experiment creation only, as above) I'll make the
code available to the community somehow and someone else can pick it up from
there if they want to make it fancier. I hope that's okay -- even with no
extra effort from someone else it'll still mean that a hell of a lot of
effort can be saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter {
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter {
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW, once
it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want from a bulk loader. For example, I would like all objects
created and linked, from samples (with annotations) right up to raw
bioassays (grouped into an experiment as Micha does, ideally).
Obviously to do this you need to specify the relationships between all
the samples, annotations, dyes, hybs, data files, but that could
easily end up being as much work as manually loading the data ;-)
Therefore you have to make compromises; the main one being that you
treat every sample/extract/etc in exactly the same way (same
protocols, same amounts, same labels, same array design). This would
probably work for most people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment involves a dye swap for one biological replicate, the bulk
loader will label each extract with both dyes for *all* replicates
(leaving you with unused labelled extracts after linking to the hybs).
One quite low cost way to define your experiment for a bulk loader is
Tab2MAGE (soon to be MAGE-TAB, see
http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it
feels strange to say this because I have been *exporting* Tab2MAGE
from BASE 1.2 rather than importing it. The input files for our bulk
Re: [base] Offline files
Micha Bayer wrote:
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the
zip file using my plugin?
If you want the plugin to appear on the list page you should use,
GuiContext.Type.List. The GuiContext.Type.ITEM makes it appear in the
view page for a single experiment.
/Nicklas
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
The BASE general discussion mailing list
basedb-users@lists.sourceforge.net
unsubscribe: send a mail with subject unsubscribe to
[EMAIL PROTECTED]
Re: [base] Offline files
Hi Micha and others,
Of course we have to share the workload. I'd rather help develop something
that is already taking shape (because I'm still not very familiar with the
API). Let's see what people think about MAGE-TAB etc, but hopefully I can
carry on where you leave off. That's not a 100% commitment, but I hope I'll
have the time/ability to do this.
Just one thought, isn't another option to cannibalise the migration code to
make a command-line bulk importer? I guess that's no good for sites who want
the users to submit through the web interface...
Another thought: my BASE 1 loaders were bolted on top of BASE and did all the
sample creation etc through HTTP requests to the BASE 1 GUI with a Perl LWP
robot. These could be modified and bolted on to BASE 2, although the approach
is very sensitive to GUI changes, and just doesn't feel like a long-term
solution to me... (the motivation to do it this way with BASE 1 was to take
the path of least resistance and avoid learning the PHP API).
cheers,
Bob.
Micha Bayer writes:
That sounds like a hell of a lot more work than what I have done so far
:-)
My first shot at this has been quite basic and it only creates new
RawBioAssays and a single Experiment that these belong to. It could of
course be extended to do more stuff, but my time allowance for this is
limited and I don't think I'll be able to spend a lot more time on this than
I already have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and
2. a plugin is much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a plugin
now, and when I have got the basic version of that working (i.e. file
upload, RawBioAssay and Experiment creation only, as above) I'll make the
code available to the community somehow and someone else can pick it up from
there if they want to make it fancier. I hope that's okay -- even with no
extra effort from someone else it'll still mean that a hell of a lot of
effort can be saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW, once
it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want
from a bulk loader. For example, I would like all objects created and
linked,
from samples (with annotations) right up to raw bioassays (grouped into an
experiment as Micha does, ideally). Obviously to do this you need to
specify
the relationships between all the samples, annotations, dyes, hybs, data
files, but that could easily end up being as much work as manually loading
the
data ;-)
Therefore you have to make compromises; the main one being that you treat
every sample/extract/etc in exactly the same way (same protocols, same
amounts, same labels, same array design). This would probably work for
most
people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment
involves a dye swap for one biological replicate, the bulk loader will
label
each extract with both dyes for *all* replicates (leaving you with unused
labelled extracts after linking to the hybs).
One quite low cost way to define your experiment for a bulk loader is
Tab2MAGE
Re: [base] JEP extra value calculator again...
Micha Bayer a écrit : Hi Emmanuel, The Catalina/work directory caches copies of your compiled JSP pages. It looks from the stacktrace like the JSP compiler can't write to the work directory because you don't have permissions to it -- check what they are set to. Right, Micha, some files and directories did in fact belong to root The pb is fixed now Thanks for your help ! -- Emmanuel COURCELLE[EMAIL PROTECTED] L.I.P.M. (UMR CNRS-INRA 2594/441) tel (33) 5-61-28-54-50 B.P.52627 - 31326 CASTANET TOLOSAN Cedex - FRANCE -- - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ The BASE general discussion mailing list basedb-users@lists.sourceforge.net unsubscribe: send a mail with subject unsubscribe to [EMAIL PROTECTED]
Re: [base] Offline files
There is a possibility to get access to the subversion repository and
trac environment just for this kind of stuff. Check out
http://baseplugins.thep.lu.se Trac environment allows for discussions
about the importer through the ticketing system.
Jari
Micha Bayer wrote:
That sounds like a hell of a lot more work than what I have done so far
:-)
My first shot at this has been quite basic and it only creates new
RawBioAssays and a single Experiment that these belong to. It could of course
be extended to do more stuff, but my time allowance for this is limited and I
don't think I'll be able to spend a lot more time on this than I already
have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and
2. a plugin is much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a plugin
now, and when I have got the basic version of that working (i.e. file upload,
RawBioAssay and Experiment creation only, as above) I'll make the code
available to the community somehow and someone else can pick it up from there
if they want to make it fancier. I hope that's okay -- even with no extra
effort from someone else it'll still mean that a hell of a lot of effort can
be saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW, once
it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want
from a bulk loader. For example, I would like all objects created and
linked,
from samples (with annotations) right up to raw bioassays (grouped into an
experiment as Micha does, ideally). Obviously to do this you need to
specify
the relationships between all the samples, annotations, dyes, hybs, data
files, but that could easily end up being as much work as manually loading
the
data ;-)
Therefore you have to make compromises; the main one being that you treat
every sample/extract/etc in exactly the same way (same protocols, same
amounts, same labels, same array design). This would probably work for
most
people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment
involves a dye swap for one biological replicate, the bulk loader will
label
each extract with both dyes for *all* replicates (leaving you with unused
labelled extracts after linking to the hybs).
One quite low cost way to define your experiment for a bulk loader is
Tab2MAGE
(soon to be MAGE-TAB, see
http://www.biomedcentral.com/1471-2105/7/489/abstract ) - although it
feels
strange to say this because I have been *exporting* Tab2MAGE from BASE 1.2
rather than importing it. The input files for our bulk loader are
specified
here:
http://wiki.vectorbase.org/vectorbase/index.php/MicroarrayExperiments
(look for the example file links) but note that the BASE-specific
parameters
(protocols, raw file parsers, array designs, etc) are configured by the
user
in an interactive process before the upload. Our bulk loader is also
2-channel only...
I have a feeling that the Tab2MAGE parser could be adapted into a BASE 2
bulk
loader somehow, although I think all Tab2MAGE code is Perl.
I think MAGE-TAB is something we could get the average bench scientist to
provide (with a little help MGEDifying the annotations). So far every
experiment I have loaded has come with
Re: [base] Offline files
A command line bulk importer ... well, we are just starting to look at
adding remote services to BASE. Web services or something similar might
be useful in this context.
Jari
Bob MacCallum wrote:
Hi Micha and others,
Of course we have to share the workload. I'd rather help develop something
that is already taking shape (because I'm still not very familiar with the
API). Let's see what people think about MAGE-TAB etc, but hopefully I can
carry on where you leave off. That's not a 100% commitment, but I hope I'll
have the time/ability to do this.
Just one thought, isn't another option to cannibalise the migration code to
make a command-line bulk importer? I guess that's no good for sites who want
the users to submit through the web interface...
Another thought: my BASE 1 loaders were bolted on top of BASE and did all the
sample creation etc through HTTP requests to the BASE 1 GUI with a Perl LWP
robot. These could be modified and bolted on to BASE 2, although the approach
is very sensitive to GUI changes, and just doesn't feel like a long-term
solution to me... (the motivation to do it this way with BASE 1 was to take
the path of least resistance and avoid learning the PHP API).
cheers,
Bob.
Micha Bayer writes:
That sounds like a hell of a lot more work than what I have done so
far :-)
My first shot at this has been quite basic and it only creates new
RawBioAssays and a single Experiment that these belong to. It could of course
be extended to do more stuff, but my time allowance for this is limited and I
don't think I'll be able to spend a lot more time on this than I already
have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and
2. a plugin is much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a
plugin now, and when I have got the basic version of that working (i.e. file
upload, RawBioAssay and Experiment creation only, as above) I'll make the
code available to the community somehow and someone else can pick it up from
there if they want to make it fancier. I hope that's okay -- even with no
extra effort from someone else it'll still mean that a hell of a lot of
effort can be saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that handles
Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view which
will let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW,
once it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
=
-Original Message-
From: [EMAIL PROTECTED] [mailto:basedb-users-
[EMAIL PROTECTED] On Behalf Of Bob MacCallum
Sent: 29 November 2006 18:36
To: BASE ML
Subject: Re: [base] Offline files
It looks like I let the cat out of the bag on this one...
Some discussion among the community would be good to establish what we
want
from a bulk loader. For example, I would like all objects created and
linked,
from samples (with annotations) right up to raw bioassays (grouped into
an
experiment as Micha does, ideally). Obviously to do this you need to
specify
the relationships between all the samples, annotations, dyes, hybs, data
files, but that could easily end up being as much work as manually
loading
the
data ;-)
Therefore you have to make compromises; the main one being that you treat
every sample/extract/etc in exactly the same way (same protocols, same
amounts, same labels, same array design). This would probably work for
most
people - but let's discuss it anyway...
One issue with the bulk loader I wrote for BASE 1 is that if your
experiment
involves a
Re: [base] Offline files
Philippe Rocca-Serra writes:
Hi Everyone,
As Dominic pointed out, we are working anyway on an import into BASE2
using tab2mage format in the short time (see ticket 338), and once
MAGE-TAB specifications are stable move on to support this format too.
excellent!
There is a MAGE-TAB jamboree happening in Philadelphia next week, so I
will count on Helen to update us on the outcome of this meeting so we
can possibly expedite the process.
will there be Java code to parse MAGE-TAB?
I guess that would speed up the development of the BASE importer.
The tab2mage import will prove no doubt extremely useful to exchange
data and to get data from repositories as pointed out
So everything is in the pipeline and we jsut need a little bit a
patience, days have only 24 hours
cheers
Philippe
--
Philippe Rocca-Serra (Ph.D)
Nutri/Toxicogenomics Internal Coordinator
EBI- The European Bioinformatics Institute
EMBL Outstation Hinxton
Wellcome Trust Genome Campus
CB10 1SD Cambridge, UK
email: [EMAIL PROTECTED]
Tel: +44 (0) 1223 492 553
Fax: +44 (0) 1223 494 468
http://www.ebi.ac.uk/microarray/Projects/tox-nutri/index.html
Helen Parkinson wrote:
Hi,
this is an interesting problem and one we have been dealing with at
ArrayExpress as well. For really large experiments the only thing that
that has worked for submissions is a spreadsheet - currently we use
tab2mage for this purpose. *tab2mage*.sourceforge.net/
You might think about an importer that takes a spreadsheet, or partial
spreadsheet plus files in an archive as input, e.g. MAGETAB, this would
allow people to pull in other people's data downloaded from public
repositories, and allow biologists to format up their own data easily.
And as spreadsheet export from BASE is planned the import would also be
nice, round tripping is also a good test of format and application
cheers
Helen
Jari Häkkinen wrote:
There is a possibility to get access to the subversion repository and
trac environment just for this kind of stuff. Check out
http://baseplugins.thep.lu.se Trac environment allows for discussions
about the importer through the ticketing system.
Jari
Micha Bayer wrote:
That sounds like a hell of a lot more work than what I have done so
far :-)
My first shot at this has been quite basic and it only creates new
RawBioAssays and a single Experiment that these belong to. It could of
course be extended to do more stuff, but my time allowance for this is
limited and I don't think I'll be able to spend a lot more time on this
than I already have.
I think the upshot of all this discussion has been:
1. people need a batch importer (desperately, in fact), and
2. a plugin is much better and will (hopefully) be possible
With that in mind, I'll go away and look into turning my stuff into a
plugin now, and when I have got the basic version of that working (i.e.
file upload, RawBioAssay and Experiment creation only, as above) I'll
make the code available to the community somehow and someone else can
pick it up from there if they want to make it fancier. I hope that's
okay -- even with no extra effort from someone else it'll still mean
that a hell of a lot of effort can be saved.
Perhaps the dev team can point me in the right direction initially. My
colleague Iain here is just working on a file importer plugin that
handles Excel files and his code starts off like this:
public class ExcelFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.RAWBIOASSAY,
GuiContext.Type.ITEM));
So am I right in thinking that if I change this to the following...
public class ZipFileImporter extends AbstractPlugin implements
InteractivePlugin, AutoDetectingImporter
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.EXPERIMENT,
GuiContext.Type.ITEM));
then I'll get an import tab created in the Experiment list view
which will let me import the zip file using my plugin?
We're planning to make the Excel plugin available for everyone too BTW,
once it's finished.
Cheers
Micha
==
Dr Micha M Bayer
Bioinformatics Specialist
Genetics Programme
The Scottish Crop Research Institute
Invergowrie
Dundee
DD2 5DA
Scotland, UK
Telephone +44(0)1382 562731 ext. 2309
Fax +44(0)1382 562426
http://www.scri.sari.ac.uk/MichaBayer.htm
