[basedb-devel] MageTab export plugin
Hi Nicklas, I guess the MageTab export plugin is ready for submission. I followed instructions for plugin developers (http://baseplugins.thep.lu.se/wiki/PluginSubmitReq), so all the documentation bits should be in place. How could I send you the bundle? Would you provide a repository dir to park the source? The plugin works with BASE2, but we'll start looking at BASE3 soon, and as part of preparations for migration I will port the plugin to BASE3. Pawel -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
Re: [basedb-devel] Mage-Tab export plugin
Great. Yes, I am planning to implement that as well. Pawel On 12/09/11 11:48, Gildas Le Corguillé wrote: Bonjour, This is good news if you can develope this plugin and I have enought datas on Base2 to test it :P My contribution on the former plugin Tab2Mage was to add a feature to manage pooling of samples (or extracts) and the possibility to have several Files[raw] per RawBioAssay (ex : one cy3 and one cy5) because we used nimblegen multi-color arrays. Are you plan to manage that feature ? Gildas Le 01/09/2011 14:43, Pawel Sztromwasser a écrit : Hallo, I haven't been to the milestones page for a while, and I didn't know that work on BASE 3 was in such an advanced state. Great job guys! Support for sequencing data is excellent news for us. Anyway, before it is released and before we are ready to migrate, we will certainly have experiments to export. We could keep using the Tab2Mage exporter this year, and the conversion script later, but I think I'd prefer to start working on the plugin now when we have time, develop against BASE 2 where we would have immediate use for it, and port to BASE 3 later after we migrate. It will not be a big change I hope. Gildas, I have actually started developing the plugin. If you're willing to contribute, I could send you the jar/code when it is more-or-less ready for testing, so we could debug it on our sites simultanously and make it bullet-proof;) Helen, IMO it is better with one format than two, so I think removing support for Tab2Mage (and providing conversion tools) is a move forward. One thing I was wondering, if there are any plans for an XML-based format for description of MA experiments? MAGE-ML is a bit too complex I guess, but an XML Schema equivalent to the current MageTab format would be favored by programmers and for programmatic access, in comparison with the tab-delimited files. Best wishes, Pawel Nicklas Nordborg wrote: On 2011-09-01 09:15, Gildas Le Corguillé wrote: Hello, Our group still interesting in this kind of exporter. It was the main reason why we decided to use Base. We are in a public lab so we like to share :P Last year, I repaired the existing Tab2Mage-exporter module for Base. So if you need help, I can spend some time on that. But I'm not necessarily a good developer ^^' I think that a Mage-Tab exporter has to be developed by the BASE community. We can provide space and user accounts on our subversion+trac (baseplugins.thep.lu.se) if someone is interested to start on this. Unfortunately we have no time to help with the actual development. Of course, we'll try to answer questions and provide the information that is necessary. As a starting point, we try to collect and document the major API changes in http://base.thep.lu.se/ticket/1603 My guess is that #1153 and #1597 are the major changes that cause problems for the current Tab2Mage exporter. /Nicklas -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free Love Thy Logs t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free Love Thy Logs t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel -- Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/ ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel -- Doing More with Less: The Next Generation Virtual Desktop What are the key obstacles that have prevented many mid-market businesses from deploying virtual desktops? How do next-generation virtual desktops provide companies an easier-to-deploy, easier-to-manage and more affordable virtual desktop model.http
Re: [basedb-devel] Mage-Tab export plugin
Hallo, I haven't been to the milestones page for a while, and I didn't know that work on BASE 3 was in such an advanced state. Great job guys! Support for sequencing data is excellent news for us. Anyway, before it is released and before we are ready to migrate, we will certainly have experiments to export. We could keep using the Tab2Mage exporter this year, and the conversion script later, but I think I'd prefer to start working on the plugin now when we have time, develop against BASE 2 where we would have immediate use for it, and port to BASE 3 later after we migrate. It will not be a big change I hope. Gildas, I have actually started developing the plugin. If you're willing to contribute, I could send you the jar/code when it is more-or-less ready for testing, so we could debug it on our sites simultanously and make it bullet-proof;) Helen, IMO it is better with one format than two, so I think removing support for Tab2Mage (and providing conversion tools) is a move forward. One thing I was wondering, if there are any plans for an XML-based format for description of MA experiments? MAGE-ML is a bit too complex I guess, but an XML Schema equivalent to the current MageTab format would be favored by programmers and for programmatic access, in comparison with the tab-delimited files. Best wishes, Pawel Nicklas Nordborg wrote: On 2011-09-01 09:15, Gildas Le Corguillé wrote: Hello, Our group still interesting in this kind of exporter. It was the main reason why we decided to use Base. We are in a public lab so we like to share :P Last year, I repaired the existing Tab2Mage-exporter module for Base. So if you need help, I can spend some time on that. But I'm not necessarily a good developer ^^' I think that a Mage-Tab exporter has to be developed by the BASE community. We can provide space and user accounts on our subversion+trac (baseplugins.thep.lu.se) if someone is interested to start on this. Unfortunately we have no time to help with the actual development. Of course, we'll try to answer questions and provide the information that is necessary. As a starting point, we try to collect and document the major API changes in http://base.thep.lu.se/ticket/1603 My guess is that #1153 and #1597 are the major changes that cause problems for the current Tab2Mage exporter. /Nicklas -- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free Love Thy Logs t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel attachment: pawel_sztromwasser.vcf-- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free Love Thy Logs t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
[basedb-devel] Mage-Tab export plugin
Hello BASE developers, By the end of the year ArrayExpress is going to remove a possibility of importing experiments using the Tab2Mage format. The only supported format will be Mage-Tab. We have been extensively using Tab2Mage and the BASE plugin that exports experiments in this format. I was wondering if anyone else is developing or planning to develop a plugin that would support Mage-Tab? Regards, Pawel Sztromwasser attachment: pawel_sztromwasser.vcf-- Special Offer -- Download ArcSight Logger for FREE! Finally, a world-class log management solution at an even better price-free! And you'll get a free Love Thy Logs t-shirt when you download Logger. Secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsisghtdev2dev___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
[basedb-devel] Truncated files
In BASE it was a Tomcat problem. Hope it helps: http://www.mail-archive.com/basedb-users@lists.sourceforge.net/msg01838.html Pawel attachment: pawel_sztromwasser.vcf-- Get a FREE DOWNLOAD! and learn more about uberSVN rich system, user administration capabilities and model configuration. Take the hassle out of deploying and managing Subversion and the tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
Re: [basedb-devel] Programmatic login using a token
Hi Nicklas, Thanks for the suggestions. Maybe I am missing something but wouldn't it be possible to send the username and password to the STS as part of the BASE login? Or, use an empty password string? I should have explained the use case better. It does not relate to the manual login to BASE using the web interface where one types in a username and password. For this we have already implemented and set up an authentication plugin that sends the username and password typed in by a user to the STS service for verification (just like you suggest). Now I am planning to enable programmatic access to BASE (via Web services) from another system that shares identity management (STS) with BASE. The idea is that the user that is logged in to any of these, has obtained a token and is able to access his/her resources in the other system without any further authentication, using only the token (single sign-on). In this federated authentication system one application will be able to automatically integrate resources available to a user in several distinct systems. Since the common authentication mechanism will be based on tokens, I don't expect an application to provide a username/password pair anytime when it needs to access resources hosted in a different system. Currently all the login methods in BASE are username/password oriented, but with the minor change I suggest, it could be easily extended to cover token-like objects as well. At least in programmatic access scenarios, but I can't see a need for any other. I am not sure if I got your suggestion with empty password right, so correct me if I am wrong. The token is not a simple string (it is actually a little XML document), and passing it as a username (with empty password) to the authenticator plugin would not be a nice solution. In any case, I am not sure that supplying a TokenAuthenticator object as a parameter to the login method is a good idea, since it would be very easy to to provide an implementation that just accept anything. I was thinking similarly, but how does it differ from supplying a login method in an authenticator plugin? That method could accept anything as well. Access to SessionControl.login(Token, TokenAuthenticator) operation in only from the code deployed together with BASE server (in my case in implementation of Web services deployed within BASE), so an environment that a BASE administrator should have full control over. The same as external authentication plugins. Pawel Nicklas Nordborg wrote: On 2011-07-26 22:47, Pawel Sztromwasser wrote: Hello BASE team! Extending a little on our recent developments regarding external authentication in BASE [1], I am working now on a Web service interface to BASE that will authenticate users based on secure tokens issued by a trusted STS. The STS authenticates users with username and password, and issues a crypted token confirming their credentials. The token is valid for a limited time and can be used for authentication in applications and services that trust the STS. The STS token does not contain a password, only a user id, so with current implementation of SessionControl I was unable to log in. I added one method to the SessionControl class (see attached diff), included a TokenAuthenticator interface (attached), and everything worked nice. The extra operation allows to log in using a single token object and a validator that can verify the token. The operation contains substantial parts of SessionControl.verifyUserExternal(), which can be extracted to a separate method. The TokenAuthenticator interface and original Authenticator interface are also quite similar, so they could maybe share a common ancestor. I know that the change is in the heart of BASE, but it is little and would provide additional way of programmatic interaction with BASE. If they pass the tests, could you consider including the changes to the BASE codebase? Thanks for the suggestions. Maybe I am missing something but wouldn't it be possible to send the username and password to the STS as part of the BASE login? Or, use an empty password string? In any case, I am not sure that supplying a TokenAuthenticator object as a parameter to the login method is a good idea, since it would be very easy to to provide an implementation that just accept anything. There are also some plans to change the authentication as part of BASE 3 development (http://base.thep.lu.se/ticket/1599). There is not much information since we haven't been thinking very much about this and at the moment I am not sure if we will have time to do it before releasing BASE 3. It depends a bit on when some of the other BASE 3 features are needed in production. I'll keep a link to this thread so that we don't forget it. /Nicklas -- Got Input? Slashdot Needs You. Take our quick survey online. Come on, we don't ask
Re: [basedb-devel] Programmatic login using a token
See my short replies inline. Nicklas Nordborg wrote: On 2011-08-01 10:51, Pawel Sztromwasser wrote: Hi Nicklas, Thanks for the suggestions. Maybe I am missing something but wouldn't it be possible to send the username and password to the STS as part of the BASE login? Or, use an empty password string? I should have explained the use case better. It does not relate to the manual login to BASE using the web interface where one types in a username and password. For this we have already implemented and set up an authentication plugin that sends the username and password typed in by a user to the STS service for verification (just like you suggest). Now I am planning to enable programmatic access to BASE (via Web services) from another system that shares identity management (STS) with BASE. The idea is that the user that is logged in to any of these, has obtained a token and is able to access his/her resources in the other system without any further authentication, using only the token (single sign-on). In this federated authentication system one application will be able to automatically integrate resources available to a user in several distinct systems. Since the common authentication mechanism will be based on tokens, I don't expect an application to provide a username/password pair anytime when it needs to access resources hosted in a different system. Currently all the login methods in BASE are username/password oriented, but with the minor change I suggest, it could be easily extended to cover token-like objects as well. At least in programmatic access scenarios, but I can't see a need for any other. Ok, I understand that this is a bit different than the usual login procedure. But I think there is also already another possible way to solve this. It could be done in a similar way that we are doing with the job agents. They use a master user account with a known username+password that is used to login to BASE. The account should have the Act as another user permissions assigned to a role (eg. just as the Job agent role that is pre-installed in BASE). Then the SessionControl.impersonateLogin() can be used to login as any user without having to provide a password. You'll have to find the internal id of the user but that is more or less the same thing that is done in the diff for SessionControl that you posted in the original post. Ahhh, clever. And no change to BASE required. It looks promising, I will give it a try. In any case, I am not sure that supplying a TokenAuthenticator object as a parameter to the login method is a good idea, since it would be very easy to to provide an implementation that just accept anything. I was thinking similarly, but how does it differ from supplying a login method in an authenticator plugin? That method could accept anything as well. Access to SessionControl.login(Token, TokenAuthenticator) operation in only from the code deployed together with BASE server (in my case in implementation of Web services deployed within BASE), so an environment that a BASE administrator should have full control over. The same as external authentication plugins. The difference is that the administrator sets up the authenticator to use in the configuration files and it is not possible for the calling code to change this. Ok, some mean code could probably get around this, so it is good to have control over the server environment. Despite this, I think the public BASE api shouldn't provide this kind of flexibility. I would prefer a solution where the administrator can setup exactly how the authentication should happen. Fair enough. /Nicklas -- Got Input? Slashdot Needs You. Take our quick survey online. Come on, we don't ask for help often. Plus, you'll get a chance to win $100 to spend on ThinkGeek. http://p.sf.net/sfu/slashdot-survey ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel attachment: pawel_sztromwasser.vcf-- Got Input? Slashdot Needs You. Take our quick survey online. Come on, we don't ask for help often. Plus, you'll get a chance to win $100 to spend on ThinkGeek. http://p.sf.net/sfu/slashdot-survey ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
Re: [basedb-devel] External authentication
Sorry, just noticed that you mentioned the ...basedb.util.JarClassLoader, not a regular Java class loader as I initially thought. On 01/02/11 13:17, Pawel Sztromwasser wrote: Thanks for a quick reply. I really wanted to avoid selecting jars and so on since BASE has over 50 of them, and maven downloads over 60 for my plugin. Lots of manual work that needs to be done every time BASE is upgraded. I will try to change the Application class as you suggested. I have already located the place which needs a fix, but was unsure of playing so close to the core of the application. What about using the net.sf.basedb.util.JarClassLoader, as regular plugins do? I will post a patch when ready and tested. Cheers, Pawel On 01/02/11 13:03, Nicklas Nordborg wrote: On 2011-02-01 11:29, Pawel Sztromwasser wrote:s Hi, I am working on an external authentication plugin for BASE that will use an STS Web service to authenticate users. The plugin uses a bunch of external jars for WS communication, but with different versions then BASE. I was hoping for the same classloading mechanism as for regular plugins (separate classloaders), but it looks like external authentication plugins uses the default classloader of the webapp. Could this be changed? How else could I install my plugin, without messing with BASE's jars? That part of BASE is not very pluginified. I am not aware of any real case that uses external authentication so it would be very interesting to see if you succeed. It's quite common that there are dependencies to different versions of 3rd-party JAR files, but it usually works if the latest one is used. But of course, it may not work if the API has changed in an incompatible way. If you can't get it to work by clever selection of JAR files, then you might have to modify the Application.getAuthenticator() method so that it uses a different class loader. It shouldn't be too difficult. It could for example pick up a JAR path from the base.config file and then use the net.sf.basedb.util.JarClassLoader.getInstance(path) to create a class loader. Dependencies need to be listed in the MANIFEST.MF in the same way as for plug-ins. There is a short notice about this at http://base.thep.lu.se/chrome/site/latest/html/developerdoc/plugin_developer/plugin_developer.organize.html and http://base.thep.lu.se/chrome/site/latest/api/net/sf/basedb/util/JarClassLoader.html As always, patches are welcome :) /Nicklas -- Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! Finally, a world-class log management solution at an even better price-free! Download using promo code Free_Logger_4_Dev2Dev. Offer expires February 28th, so secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsight-sfd2d ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel -- Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! Finally, a world-class log management solution at an even better price-free! Download using promo code Free_Logger_4_Dev2Dev. Offer expires February 28th, so secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsight-sfd2d ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel -- Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)! Finally, a world-class log management solution at an even better price-free! Download using promo code Free_Logger_4_Dev2Dev. Offer expires February 28th, so secure your free ArcSight Logger TODAY! http://p.sf.net/sfu/arcsight-sfd2d ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
[basedb-devel] External id for protocol
Hello, We have been exporting some experiments lately to ArrayExpress and noticed that protocols need to have their own ArrayExpress IDs. Currently we have to add them manually to export file generated in BASE (Tab2MageExporter), but it would be fairly easy to have it done automatically by the plugin. If there only was a place to store it in BASE2... Would it be possible to have it as externalId (like BioMaterials have)? Or maybe it is better to enable annotations for protocols? This way it would be possible to add even more programaticaly accessible information to protocols. How do you think: which of these two is easier to implement? And which one makes more sense? If any of these is straight-forward enough, I will probably go for a hack like additional column in Protocols table, using description field or similar, but I don't really like hack solutions. Thus, if you are also interested in implementing such a functionality, I am willing to contribute some code/time. All the best Pawel -- The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your production scanning environment may not be a perfect world - but thanks to Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700 Series Scanner you'll get full speed at 300 dpi even with all image processing features enabled. http://p.sf.net/sfu/kodak-com ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
Re: [basedb-devel] Batch scan importer
I am not sure, I didn't consider image as a 'real type' of item. More like attribute of a scan. Maybe because there is no Image submenu:) I agree that looking for already created images with specific file is much better idea, then what I have implemented. Allowing users to specify image format, description and preview flag is great, but I would prefer to have it in the scan spreadsheet as additional columns (image property columns, as with file in rba import). Mainly because image has no life on its own and also because there is actually no image list where one could run batch import. OK, there is a list of images for every scan, but image imports for each and every scan would be laborious. What do you think of it? Pawel Nicklas Nordborg wrote: Thanks for the code. I am not sure that I like that it creates a new image every time. I think it should at least check if there is already an image with the specified file. Is there a need for setting the other image properties (preview, format, description)? If so, it might be better to have a specific importer for images. /Nicklas Pawel Sztromwasser wrote: Hello BASE team, We really enjoy the set of batch uploading plugins and were doing a little training for our users in Bergen. Everything seemed great except that they couldn't link freshly created scans with image files. I read in the example spreadsheet that it is not supported and was very surprised since similar 'file attaching' works very nice for raw bioassays. I sat down, did some copy pasting from RawBioAssayImporter and after little edits it worked (source attached). I hope that you can make some use of it. All the best, Pawel -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
[basedb-devel] Batch scan importer
Hello BASE team,
We really enjoy the set of batch uploading plugins and were doing a
little training for our users in Bergen. Everything seemed great except
that they couldn't link freshly created scans with image files. I read
in the example spreadsheet that it is not supported and was very
surprised since similar 'file attaching' works very nice for raw
bioassays. I sat down, did some copy pasting from RawBioAssayImporter
and after little edits it worked (source attached).
I hope that you can make some use of it.
All the best,
Pawel
/**
$Id: ScanImporter.java 4551 2008-09-29 07:47:24Z nicklas $
Copyright (C) 2008 Nicklas Nordborg
This file is part of BASE - BioArray Software Environment.
Available at http://base.thep.lu.se/
BASE is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 3
of the License, or (at your option) any later version.
BASE is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with BASE. If not, see http://www.gnu.org/licenses/.
*/
package net.sf.basedb.plugins.batchimport;
import java.util.Collections;
import java.util.List;
import java.util.Set;
import net.sf.basedb.core.DbControl;
import net.sf.basedb.core.File;
import net.sf.basedb.core.Hardware;
import net.sf.basedb.core.Hybridization;
import net.sf.basedb.core.Image;
import net.sf.basedb.core.Item;
import net.sf.basedb.core.ItemQuery;
import net.sf.basedb.core.Path;
import net.sf.basedb.core.PathParameterType;
import net.sf.basedb.core.PluginParameter;
import net.sf.basedb.core.Protocol;
import net.sf.basedb.core.ProtocolType;
import net.sf.basedb.core.Scan;
import net.sf.basedb.core.SystemItems;
import net.sf.basedb.core.Version;
import net.sf.basedb.core.plugin.About;
import net.sf.basedb.core.plugin.AboutImpl;
import net.sf.basedb.core.plugin.GuiContext;
import net.sf.basedb.util.parser.FlatFileParser;
import net.sf.basedb.util.parser.Mapper;
/**
Plug-in for importing scan items in a batch. The plug-in can create new
items and updated existing items.
@author nicklas
@version 2.8
@base.modified $Date: 2008-09-29 09:47:24 +0200 (Mon, 29 Sep 2008) $
*/
public class ScanImporterNMC
extends AbstractItemImporterScan
{
private static final SetGuiContext guiContexts =
Collections.singleton(new GuiContext(Item.SCAN, GuiContext.Type.LIST));
private static final About about =
new AboutImpl
(
Scan importer with images,
Imports and updates scans in a batch (linking with images).,
Version.getMajor() + . + Version.getMinor() + . + Version.getMaintenance(),
2008, Base 2 development team,
null,
null,
http://base.thep.lu.se;
);
protected static final PluginParameterString hardwareColumnMapping = new PluginParameterString(
hardwareColumnMapping,
Scanner,
Mapping that picks the name or ID of the array slide scanner from the data columns. +
The plug-in will first try to find a scanner with the given name. If none is found and +
the value is numeric it will try to load by internal ID. +
Example: \\Scanner\\,
optionalColumnMapping
);
protected static final PluginParameterString hybridizationColumnMapping = new PluginParameterString(
hybridizationColumnMapping,
Hybridization,
Mapping that picks the name or ID of the hybridization from the data columns. +
The plug-in will first try to find a hybridization with the given name. If none is found and +
the value is numeric it will try to load by internal ID. +
Example: \\Hybridization\\,
optionalColumnMapping
);
// extra parameter needed for linking with image files
protected static final PluginParameterString fileColumnMapping = new PluginParameterString(
fileColumnMapping,
File,
Mapping that picks the path of a image file from the columns. +
The path can be an absolute path (starting with /) or relative +
to the 'Data directory' parameter. +
Example: \\File\\,
optionalColumnMapping
);
public ScanImporterNMC()
{}
/*
From the Plugin interface
*/
@Override
public About getAbout()
{
return about;
}
//
/*
From the InteractivePlugin interface
*/
@Override
public SetGuiContext getGuiContexts()
{
return guiContexts;
}
//
/*
From the AbstractItemImporter class
*/
private Mapper nameMapper;
private Mapper descriptionMapper;
private Mapper protocolMapper;
private Mapper hardwareMapper;
private Mapper hybridizationMapper;
//extra mappers for linking with image file
private
Re: [basedb-devel] Batch scan importer
I have just noticed that the filename I sent has different name then the class inside. I used ScanImporterNMC name to distinguish from the original plugin and then renamed the file before sending. Just rename it back and all will work. Pawel Pawel Sztromwasser wrote: Hello BASE team, We really enjoy the set of batch uploading plugins and were doing a little training for our users in Bergen. Everything seemed great except that they couldn't link freshly created scans with image files. I read in the example spreadsheet that it is not supported and was very surprised since similar 'file attaching' works very nice for raw bioassays. I sat down, did some copy pasting from RawBioAssayImporter and after little edits it worked (source attached). I hope that you can make some use of it. All the best, Pawel -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel -- Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
Re: [basedb-devel] Illumina Bead Summary Files
We experienced that as well. I believe that I have checked these probes and they were controls, so they can be ignored. Pawel Nicklas Nordborg wrote: David Waring wrote: I have recently installed the Illumina Plugins and have imported features for several Array Designs. We modified our config to generate the Bead Summary files and I have imported the data. Oddly have found that the Bead Summary files contain some rows with illumina codes that are not in the BGX files. For example the datasets from HumanRef-8 arrays contain 324 probes that are not found in any of the various versions of BGX files are available from the Illumina web site. If I run these same datasets through BeadStudio these these probes are not found in the BeadStudio output. Has anyone else run into this situation? Yes, we have seen this also. Just select 'Probe not found=skip' when importing the data to BASE and everything should work. /Nicklas -- Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel -- Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
[basedb-devel] Performance/delete issues on Postgres
Hello, I found something interesting regarding BASE2 performance on Postgres (we have 8.3 but I suppose it is valid for previous versions as well). Recently I was unable to delete ArrayDesign (Illumina Human 8Ref, no items connected to it, features imported), because database query was endless. I stopped it after 3 days and started similar on development installation (much, much smaller). It took almost 20h to complete. We started digging and found (among other similar) this: http://archives.postgresql.org/pgsql-admin/2006-02/msg00238.php In short, postgres does not create indexes on foreign keys by default. That has huge impact on performance when updating/deleting items from table that is referenced by other big table. In case above Features (that are deleted with ArrayDesign) are referenced by all RawData* tables. I created indexes on feature_id column in all of these tables and delete took less than a minute. That reminds me of another case where performance for postgres could be improved. When testing variuos kinds of jobs I noticed that concurent jobs creating root bioassayset took surprisingly long and did not scale well. That was due to sequential lookup in RawData* table on rawbioassay_id column. Creating index on that column improved performance by more than 50% (see: http://picasaweb.google.com/pawel.sztromwasser/Other#5257654388791782770). I know that indexes have drawbacks as well, but in cases above they are a huge advantage. Could these indexes be incorporated in updateDb/indexes scripts? If you know about any other possible places in BASE database where similar things may occur, let me know and I will test them. Best regards, Pawel - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel
Re: [basedb-devel] Webservices transport error: 411 Error: Length Required
Totally agree. That makes it much more flexible. Please, feel free to
change the code.
Pawel
Nicklas Nordborg wrote:
Thanks for the code. It was something like this that I had in mind. The
one thing I don't like is to pass the type as an 'int' parameter in the
constructor and having a switch statement that creates the actual object
to use. This makes it impossible for others to use their own
implementations of 'RPCServiceClientCreator'. Wouldn't it be better to
pass a 'RPCServiceClientCreator' directly to the constructor?
Eg.
protected AbstractRPCClient(
String url, String service,
RPCServiceClientCreator serviceCreator)
{
...
}
/Nicklas
Pawel Sztromwasser wrote:
Hi Nicklas,
I had some time and implemented the 'ServiceFactory' functionality.
Gzipped java files are attached. To summarize changes:
1) three new classes/interfaces were created (I put them in
net.sf.basedb.ws.client.added package):
a) RPCServiceClientCreator - interface containing RPCServiceClient
getService(String serviceUrl) method for creating RPCServiceClient objects
b) DefaultRPCServiceClientCreator - factory class implementing
interface above. It creates RPCServiceClient objects in the way they
were created in AbstractRPCClient.getService() method.
c) NoChunkRPCServiceClientCreator - the same as above but with CHUNK
option set to false.
2) AbstractRPCClient class uses own RPCServiceClientCreator object to
create objects in getService() method. Creator object is initialized in
constructor (type of creator is an argument of additional constructor).
The creator type can be chosen between values supported by
AbstractRPCClient, available as public static constants (currently only
DEFAULT_RPC_SERVICE_CLIENT and NO_CHUNK_RPC_SERVICE_CLIENT). If no
creator type is supplied, default one is used.
3) all the subclasses of AbstractRPCClient have additional constructor
with last argument being type of RPCServiceClientCreator which refers to
constructor in superclass. This makes possible to choose creator type
when programming BASE2 webservices clients.
I attached diffs.txt file which contains diff output on
net/sf/basedb/ws/client/ folders for my version and 2.7.1. Just thought,
it might be helpful.
I hope the code will be useful.
Best wishes,
Pawel
Nicklas Nordborg wrote:
Pawel Sztromwasser wrote:
Hello BASE developers,
We have been recently trying to enable retrieving some data from
BASE2 installation directly into our 'home made' analysis tool. We
use webservices for that and have to say that you are doing a great
job.
Thanks.
We are really looking forward to use them more extensively.
Recently, when I started testing the tool not only within the local
network I noticed that the webservice client does not switch off
chunking. It results in attached exception cascade. Chunking is on by
default in axis2 and some servers (i.e. our gateway to BASE which
apparently uses HTTP1.0 and not 1.1) have problems with it. The
remedy is to use
options.setProperty(HTTPConstants.CHUNKED, false);
in net.sf.basedb.ws.client.AbstractRPCClient.getService(). I don't
know what are the drawbacks of it, but that was the only way I could
run the services in our current setup (BASE2.6 accessible by
proxypass from exposed webserver). Could this option be set off by
default or configurable somewhere? Or maybe there is some other
workaround?
I am afraid that I am not very familiar with all possible options that
is available in the Axis API:s. There seems to be a lot of them and
their Javadoc is not very informative about what many of the options
are used for. The CHUNKED option seems to be related to the way HTTP
requests are made, but I have to admit that this is the first time I
have heard of it. This means that I probably know even less than you
about this.
I think that, in the future, there may be more requests about being
able to set other options. So, it may be better to think ahead and try
to figure out some way to be able to do this as easy as possible. With
the current BASE API the only workaround that doesn't involve changing
the BASE code is to subclass all AbstractRPCClient subclasses and
override the 'getService()' method. This is not very elegent since
there are several such classes and the new 'getService()' will be
added to each one of them.
A better approach may be to create a 'ServiceFactory' interface which
has a 'createService()' method. The default implementation would then
work as the AbstractRPCClient.getService() does today. The last step
is to figure out a way to switch to another 'ServiceFactory'
implementation either by having a configuration file or by being able
to specify it programmatically.
I have opened a ticket for this issue
http://base.thep.lu.se/ticket/1057. It falls under the 'contributions
welcome' category since right now we are forced to prioritize other
Re: [basedb-devel] Webservices transport error: 411 Error: Length Required
Hi Nicklas, I had some time and implemented the 'ServiceFactory' functionality. Gzipped java files are attached. To summarize changes: 1) three new classes/interfaces were created (I put them in net.sf.basedb.ws.client.added package): a) RPCServiceClientCreator - interface containing RPCServiceClient getService(String serviceUrl) method for creating RPCServiceClient objects b) DefaultRPCServiceClientCreator - factory class implementing interface above. It creates RPCServiceClient objects in the way they were created in AbstractRPCClient.getService() method. c) NoChunkRPCServiceClientCreator - the same as above but with CHUNK option set to false. 2) AbstractRPCClient class uses own RPCServiceClientCreator object to create objects in getService() method. Creator object is initialized in constructor (type of creator is an argument of additional constructor). The creator type can be chosen between values supported by AbstractRPCClient, available as public static constants (currently only DEFAULT_RPC_SERVICE_CLIENT and NO_CHUNK_RPC_SERVICE_CLIENT). If no creator type is supplied, default one is used. 3) all the subclasses of AbstractRPCClient have additional constructor with last argument being type of RPCServiceClientCreator which refers to constructor in superclass. This makes possible to choose creator type when programming BASE2 webservices clients. I attached diffs.txt file which contains diff output on net/sf/basedb/ws/client/ folders for my version and 2.7.1. Just thought, it might be helpful. I hope the code will be useful. Best wishes, Pawel Nicklas Nordborg wrote: Pawel Sztromwasser wrote: Hello BASE developers, We have been recently trying to enable retrieving some data from BASE2 installation directly into our 'home made' analysis tool. We use webservices for that and have to say that you are doing a great job. Thanks. We are really looking forward to use them more extensively. Recently, when I started testing the tool not only within the local network I noticed that the webservice client does not switch off chunking. It results in attached exception cascade. Chunking is on by default in axis2 and some servers (i.e. our gateway to BASE which apparently uses HTTP1.0 and not 1.1) have problems with it. The remedy is to use options.setProperty(HTTPConstants.CHUNKED, false); in net.sf.basedb.ws.client.AbstractRPCClient.getService(). I don't know what are the drawbacks of it, but that was the only way I could run the services in our current setup (BASE2.6 accessible by proxypass from exposed webserver). Could this option be set off by default or configurable somewhere? Or maybe there is some other workaround? I am afraid that I am not very familiar with all possible options that is available in the Axis API:s. There seems to be a lot of them and their Javadoc is not very informative about what many of the options are used for. The CHUNKED option seems to be related to the way HTTP requests are made, but I have to admit that this is the first time I have heard of it. This means that I probably know even less than you about this. I think that, in the future, there may be more requests about being able to set other options. So, it may be better to think ahead and try to figure out some way to be able to do this as easy as possible. With the current BASE API the only workaround that doesn't involve changing the BASE code is to subclass all AbstractRPCClient subclasses and override the 'getService()' method. This is not very elegent since there are several such classes and the new 'getService()' will be added to each one of them. A better approach may be to create a 'ServiceFactory' interface which has a 'createService()' method. The default implementation would then work as the AbstractRPCClient.getService() does today. The last step is to figure out a way to switch to another 'ServiceFactory' implementation either by having a configuration file or by being able to specify it programmatically. I have opened a ticket for this issue http://base.thep.lu.se/ticket/1057. It falls under the 'contributions welcome' category since right now we are forced to prioritize other things. /Nicklas - Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source. http://sourceforge.net/services/buy/index.php ___ basedb-devel mailing list basedb-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/basedb-devel new_client_package.tar.gz Description: GNU Zip compressed data diff net/sf/basedb/ws/client/AbstractRPCClient.java /Home/bccs/pawels/base2/base-2.7.1-src/src/webservices/client/java/net/sf/basedb/ws/client/AbstractRPCClient.java 32,35d31 import net.sf.basedb.ws.client.added.DefaultRPCServiceClientCreator; import
