Re: [galaxy-dev] How does one run Javascript or HTML as a tool? Part 2

[rbrown1422]

Good morning Bjorn,

I accidentally sent the email before I finished.   Thanks for the response to 
q1-4.
The following questions are part of my request.


5.  In data_config_type.xml you only use the   record which -- manages 
internal or external displays or both?

6.  I added a class in the binary.py  file with the name of my new datatype.  

7.  I added a tool.mydatatype section and file with my xml to input and then 
output my datatype by specifying  format=mydatatype not 
format=binary.mydatatype   Plus I added this into Tools_conf.xml  Are there 
other xml specific items I need in my tool xml for an internal visualization

8.  How does one get the visualization icon to appear in the bottom left of the 
output file history?Is there something I need to add to the tool xml?


9. By clicking the visualization icon. what does it call  so i can add specific 
code to display my datatype internal "visualization?" (preferred) within galaxy?



thanks
Bob


- Original Message -From: Bjrn Grning 
To: rbrown1...@comcast.net, Eric Rasche 
Cc: galaxy-dev Sent: Sat, 12 Dec 
2015 14:53:56 - (UTC)Subject: Re: [galaxy-dev] How does one run Javascript 
or HTML as a tool?

Hi Bob,

> Thank you Eric,> > But I want to be clear after spending 3 days with help 
> from others trying to get this to work.> > 1. Config/plugin/visualization is 
> where code goes to have a Galaxy internal module display a user defined data 
> type -- mine is binary

Yes.

> 2. Datatype/data_applciations is where code goes for external displays that 
> use get and post reside. 

Yes.

> 3. which if any of 1or2 above use the Tools/Visualization directory. Is that 
> for the return of a Get or Post? 

This is a tool. A completely different concept. Tools are entities thatcan run 
one a cluster and can run separated from the user interface.

> 4. I create my own datatype in data_config_type.xml with a  tag

Yes.

> 5.

5 is missing :)

Cheers,Bjoern

> - Original Message -From: Eric Rasche To: 
> rbrown1422@comcast.netCc: galaxy-dev Sent: Tue, 
> 08 Dec 2015 02:34:47 - (UTC)Subject: Re: [galaxy-dev] How does one run 
> Javascript or HTML as a tool?> > > > > Hi Bob,> > > Yes, galaxy supports 
> output formats of HTML+JS, though the tool producing them may need to be 
> whitelisted (grep for 'whitelist' in your galaxy.ini) for display to work 
> properly.> > If you wish to write your own visualization plugin for a 
> specific datatype, there are assorted pieces of documentation on the wiki 
> that might help you (though I usually just look at one of the other existing 
> visualization plugins and copy+paste).> > > Cheers,> Eric> > 2015-12-07 20:08 
> GMT-06:00 :> > Good evening Galaxy team,> > Is it 
> possible to run JavaScript or HTML as a tool or a display module? > > I see 
> support for URLs, e.g. UCSC Genome Browser, but I would like to process a 
> specific datatype to allow it to be viewed in Galaxy -- but the application 
> is a combination of HTML and JavaScript. > > Does Galaxy support this? > > 
> Thank you,> Bob> ___> 
> > Please keep all replies on the list by using "reply all"> > in your mail 
> client. To manage your subscriptions to this> > and other Galaxy lists, 
> please use the interface at:> > https://lists.galaxyproject.org/> > To search 
> Galaxy mailing lists use the unified search at:> > 
> http://galaxyproject.org/search/mailinglists/> > > > 
> ___> Please keep all 
> replies on the list by using "reply all"> in your mail client. To manage your 
> subscriptions to this> and other Galaxy lists, please use the interface at:> 
> https://lists.galaxyproject.org/> > To search Galaxy mailing lists use the 
> unified search at:> http://galaxyproject.org/search/mailinglists/> 
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] How does one run Javascript or HTML as a tool? the complete message ignore previous

[Carl Eberhard]

Hi, Bob

There are a number of ways to visualize data in Galaxy, so let's settle on
some terminology first for the three major ways:
* *display applications*: these are definitions of external websites that
can fetch galaxy datasets and display them in their visualization
applications. Examples are the UCSC genome browser, GBrowse, and IGV.
* *tools*: although the tool framework generally produces 'raw' data like
tabular or binary files, it can also produce the html (and js even) that
can display data for visualization. An example would be 'Boxplot of quality
statistics'.
* *visualization plugins*: these are python/mako/js files that are meant to
be more interactive visualizations in order to explore data

It sounds to me like you'd like to use the visualization plugin system, so:
* since they're for external code or websites, *let's take display
applications off the table entirely*.
* since they can be a viable alternative to the plugin system, so *I'd keep
tools as another option but keep our focus on the plugin system*

It also sounds like you have two elements you'd like to incorporate:
* a *datatype*
* a *plugin* to display datasets of that datatype

*The datatype*
I'm more familiar with the plugin part of this process, but for the
datatype it sounds like you want to subclass the binary datatype and add it
to your galaxy installation. I'd start here (I imagine you've already seen
it, tho):
* https://wiki.galaxyproject.org/Admin/Datatypes
* https://wiki.galaxyproject.org/Admin/Datatypes/Adding%20Datatypes

Note: to see if your datatype was loaded successfully, with your server
running - you can go to: /api/datatypes/mapping. You should be able to see
your datatype listed at the first level of the json map, generally
beginning with 'galaxy.datatypes.' and the python module name you added it
to (like a python import namespace): e.g. galaxy.datatypes.binary.H5 or
galaxy.datatypes.binary.mydatatype

To simplify:
1. Add a definition to your datatypes_conf file
2. Add a sniffer for your datatype
3. Add a subclass/class for your datatype

*The plugin*
For the plugin, I also imagine you've seen this, but it's a (hopefully)
good place to start:
https://wiki.galaxyproject.org/VisualizationsRegistry

*The filesystem layout*
Here's a simplified process for creating an outline for a visualization
plugin project:
1. in the filesystem, start at */config/plugins/visualizations*
2. *think of an id for your plugin*. This can be any (filesystem
permissible) file name and is only used as an id by the registry - the
users never see it. It should be unique from any other plugins in that
directory. (e.g. myplugin)
3. *create a main directory using your id*. (e.g. /config/plugins/visualizations/myplugin)
4. inside that directory, *create three more directories: config,
templates, static*. This is where the plugin configuration, the mako
templates, and any (optional) javascript or static files are kept
respectively.
5. *the configuration file should use the same id* you used above for the
directory: e.g. config/myplugin.xml. Most people copy and rename a simple
config file like the one in
config/plugins/visualizations/scatterplot/config/scatterplot.xml. We'll
change the datatype it applies to later, but there's more at:
https://wiki.galaxyproject.org/VisualizationsRegistry#The_visualization_configuration_file
and https://wiki.galaxyproject.org/VisualizationsRegistry/Configuration
6. *a mako template file should go into the templates directory*: e.g.
templates/myplugin.mako. This template file is the entry point for your
visualization and is loaded first. You don't have to do any major coding
here and can instead just launch javascript to render the visualization.
https://wiki.galaxyproject.org/VisualizationsRegistry#Creating_the_code_and_markup_for_your_visualization

At this point, your config/plugins/visualizations/myplugin directory should
look like one of the two trees displayed here:
https://wiki.galaxyproject.org/VisualizationsRegistry#Configuring_your_visualization_plugin_in_the_.60visualization_plugins_directory.60

*The configuration*
Now, we'll configure the plugin by editing the
config/plugins/visualizations/myplugin/config/myplugin.xml file. Let's
assume you've copied the scatterplot config file (
https://github.com/galaxyproject/galaxy/blob/dev/config/plugins/visualizations/scatterplot/config/scatterplot.xml
):

1. change the name displayed to what you'd like users to see in the dataset
visualizations dropdown menu:
l

2. change the configuration for your visualization to test for your
datatype. For example, if you have mydatatype, change:


HistoryDatasetAssociation

*binary.MyDatatype*
dataset_id



The above is basically saying, if an object is a) a
HistoryDatasetAssociation (a dataset in a history) and b) it's a subclass
or instance of binary.MyDatatype, then put a link:
* in the dataset visualization dropdown menu
* that will start my visualization
* pass the encoded id of the 

Re: [galaxy-dev] How does one run Javascript or HTML as a tool? the complete message ignore previous

[rbrown1422]

Thank you Carl for the excellent write-up.  It could serve as a visualization 
plugin tutorial.
However, I am still not able to get the visualization icon to appear, see the 
attachment 2.  
I feel if I get the icon to show it will all come together.
A couple questions followed by file dumps and startup info
1. my   /api/datatypes/mapping dump   it shows registry instead of binary
near the top
"galaxy.datatypes.genetics.RexpBase": {"galaxy.datatypes.images.Html": true, 
"galaxy.datatypes.data.Data": true, "galaxy.datatypes.data.Text": true, 
"galaxy.datatypes.genetics.RexpBase": true}, 
"galaxy.datatypes.registry.MDA_zip": {"galaxy.datatypes.registry.Binary": true, 
"galaxy.datatypes.data.Data": true, "galaxy.datatypes.binary.Binary": true, 
"galaxy.datatypes.registry.MDA_zip": true},
near the bottom in lower case
"galaxy.datatypes.registry.Text", "mda_zip": 
"galaxy.datatypes.registry.MDA_zip", "chips": "galaxy.datatypes.registry.Text", 
"gg": "galaxy.datatypes.genetics.GenomeGraphs", "syco": 
"galaxy.datatypes.registry.Text", 
 
2. I also added a snapshot of the visualization/tool/ sub-directories.   It 
shows my extension because I have a format=type in the tool.xml output file 
defition but shows  file type 'binary' at the bottom of the file history.   Is 
this correct, or should it be my extension type there?
Comment -- It appears the only link between the file extension and the 
visualization plugin is via the test line in the /conf/toolxml file.  Is that 
correct?
 
My xml in my visualize directory /tool/config/tool.xml (formatting is 
messed up here) attachment 1



  
 
   
HistoryDatasetAssociation
binary.MDA_zip
 dataset_id

  
  
   dataset_id
  
  MDAheatmap.mako

 
My templates/tool.mako file (hello world)





   

 

 A Small Hello

 

   



   Hi

   This is very minimal "hello world" HTML document.





startup info seems successful
galaxy.web.base.pluginframework INFO 2015-12-14 13:20:09,631 
VisualizationsRegistry, loaded plugin: MDAheatmap
x.x.10.42 - - [14/Dec/2015:14:11:50 -0400] "PUT 
/api/histories/a799d38679e985db/contents/datasets/fb85969571388350 HTTP/1.1" 
200 - "http://xx.xx.10.4:8082/; "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"
no skipping error and no other apparent errors   in startup
 
here is my Datatypes.conf.xml 2 additions

and sniffer
   
 
my binary.py addition
class MDA_zip( Binary ):
   """Class describing an MDA Heatmap binary zip file"""
   file_ext = "MDA_zip"
   def set_peek( self, dataset, is_multi_byte=False ):
   if not dataset.dataset.purged:
   dataset.peek = "Binary MDA_zip sequence file"
   dataset.blurb = nice_size( dataset.get_size() )
   else:
   dataset.peek = 'file MDA_zip does not exist'
   dataset.blurb = 'file MDA_zip purged from disk'
   def display_peek( self, dataset ):
   try:
   return dataset.peek
   except:
   return "Binary MDA_zip heatmap file (%s)" % ( nice_size( 
dataset.get_size() ) )
 
Binary.register_sniffable_binary_format("MDA_zip","MDA_zip",MDA_zip)
 

my dataset.py addition
class MDA_zipDataProvider( base.DataProvider ):
def __init__( self, dataset, **kwards ):
   self.dataset = json.loads( "".join( dataset.readlines() ))
   """
   def __init__( self, dataset, **kwards ):
   raise NotImplementedError()
   super( MDA_zipDataProvider, self ).__init__( dataset, **kwargs )
My tool in tools dir



 Display HeatMap generated by MDAnderson HeatMap 
tools
 MDAheatmap.py $input $out_file 
 
   
 
 
   
 

 
 


- Original Message -From: Carl Eberhard To: 
rbrown1422@comcast.netCc: Eric Rasche , galaxy-dev 
Sent: Mon, 14 Dec 2015 16:40:34 - 
(UTC)Subject: Re: [galaxy-dev] How does one run Javascript or HTML as a tool? 
the complete message ignore previous

Hi, Bob

There are a number of ways to visualize data in Galaxy, so let's settle on some 
terminology first for the three major ways:
* display applications: these are definitions of external websites that can 
fetch galaxy datasets and display them in their visualization applications. 
Examples are the UCSC genome browser, GBrowse, and IGV.
* tools: although the tool framework generally produces 'raw' data like tabular 
or binary files, it can also produce the html (and js even) that can display 
data for visualization. An example would be 'Boxplot of quality statistics'. 
* visualization plugins: these are python/mako/js files that are meant to be 
more interactive visualizations in order to explore data

It sounds to me like you'd like to use the visualization plugin