Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Peter van Heusden]

Yes that was my conclusion, thanks Peter.

As a result I switched to a new table named (shapeit2_ref) and a new tool
name (data_manager_shapeit2_reference) and from planemo side upload seemed
to succeed but the tool is in an error state. I resorted to manually
editing the tool-data table.

Data managers are a bit of an ugly duckling part of Galaxy - not receiving
as much love as other kinds of tools.

Peter


On Tue, 18 Sep 2018 at 14:22 Peter Cock  wrote:

> Hi Peter vH,
>
> I wonder if adding columns is not supported - because some tools
> would define the table without the extra column? You might have
> to define a new data table instead (with largely the same content).
> But to be clear, I am not an expert in this area of Galaxy.
>
> However, it does seem the IUC could lay out some *.loc file best
> practises for tool authors. There is currently a TODO entry here:
>
>
> https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/tool_xml.html#data-managers
>
> Peter C.
>
> On Mon, Sep 17, 2018 at 10:56 PM Peter van Heusden 
> wrote:
>
>> A bit more tracing leads me to suspect that what is happening is that a
>> merge is being called here:
>>
>>
>> https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/data/__init__.py#L120
>>
>> from:
>>
>>
>> https://github.com/galaxyproject/galaxy/blob/dev/lib/tool_shed/tools/data_table_manager.py#L88
>>
>> i.e. the toolshed maintains an in-memory copy of tool-data tables and if
>> the list of columns in an incoming tool-data table is not the same as the
>> in-memory version the error I saw happens.
>>
>> Which leaves me with the question: how is one meant to update the column
>> list of a tool data table?
>>
>> Peter
>>
>>
>>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Peter Cock]

Hi Peter vH,

I wonder if adding columns is not supported - because some tools
would define the table without the extra column? You might have
to define a new data table instead (with largely the same content).
But to be clear, I am not an expert in this area of Galaxy.

However, it does seem the IUC could lay out some *.loc file best
practises for tool authors. There is currently a TODO entry here:

https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/tool_xml.html#data-managers

Peter C.

On Mon, Sep 17, 2018 at 10:56 PM Peter van Heusden  wrote:

> A bit more tracing leads me to suspect that what is happening is that a
> merge is being called here:
>
>
> https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/data/__init__.py#L120
>
> from:
>
>
> https://github.com/galaxyproject/galaxy/blob/dev/lib/tool_shed/tools/data_table_manager.py#L88
>
> i.e. the toolshed maintains an in-memory copy of tool-data tables and if
> the list of columns in an incoming tool-data table is not the same as the
> in-memory version the error I saw happens.
>
> Which leaves me with the question: how is one meant to update the column
> list of a tool data table?
>
> Peter
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Peter van Heusden]

A bit more tracing leads me to suspect that what is happening is that a
merge is being called here:

https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/data/__init__.py#L120

from:

https://github.com/galaxyproject/galaxy/blob/dev/lib/tool_shed/tools/data_table_manager.py#L88

i.e. the toolshed maintains an in-memory copy of tool-data tables and if
the list of columns in an incoming tool-data table is not the same as the
in-memory version the error I saw happens.

Which leaves me with the question: how is one meant to update the column
list of a tool data table?

Peter

On Mon, 17 Sep 2018 at 13:23 Peter van Heusden  wrote:

> Thanks - which is probably because of the previous failures with
> `shed_update` - which relate back to the topic of my original email. I
> suspect that failure from `planemo shed_update` has led to this "not
> installable" state. And I don't know how to solve the problem from
> `shed_update` which *looks* like it is coming from a clash between previous
> and current definitions of the tool-data table.
>
> Peter
>
> On Mon, 17 Sep 2018 at 12:58 Nicola Soranzo 
> wrote:
>
>> Hi Peter,
>> if you open the page for data_manager_shapeit_reference in the test
>> ToolShed you will see that revision 5 and 6 are not installable, they both
>> have "*This revision can be installed:* False" . If you instead select
>> revision 3 it is installable.
>>
>> Cheers,
>> Nicola
>>
>>
>> On 17/09/18 11:51, Peter van Heusden wrote:
>>
>> I'm following up on my previous email. I somehow managed to get `planemo
>> shed_update` to work but now I seem to have an inconsistent view of the
>> Test toolshed. This is how it looks from the Admin panel of my Galaxy
>> server:
>>
>> [image: https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png]
>> Notice that it shows 3 installable revisions for the data manager, last
>> updated 13th of September. And this is how it looks from the Test toolshed
>> website:
>>
>> [image: https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png]
>>
>>
>> Notice that it has 6 installabale revisions for the data manager. Is my
>> Galaxy server perhaps caching something?
>>
>> Peter
>>
>>
>> On Sat, 15 Sep 2018 at 09:57 Peter van Heusden  wrote:
>>
>>> Hi there
>>>
>>> I'm working on a data manager to keep track of the reference data
>>> collection used by the SHAPEIT2 tool. I had to add a column to the data
>>> manager but when I try and do a `planemo shed_update` I get the error:
>>>
>>> Unexpected HTTP status code: 500: {"err_msg": "Error attempting to parse
>>> file tool_data_table_conf.xml.sample: Merging tabular data tables with non
>>> matching columns is not allowed: shapeit_ref:{'path': 2,
>>> 'reference_prefix': 3, 'name': 1, 'value': 0, 'map_prefix': 4} !=
>>> shapeit_ref:{'name': 1, 'value': 0, 'sample_prefix': 5, 'path': 2,
>>> 'reference_prefix': 3, 'map_prefix': 4}"}
>>>
>>> This is for data_manager_shapeit_reference in testtoolshed, which is on
>>> github at
>>>
>>> https://github.com/pvanheus/data_manager_shapeit_reference
>>>
>>> As far as I can tell the columns do accord. Is there any way to see
>>> which files are being merged to help try and debug this error?
>>>
>>> Thanks!
>>> Peter
>>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/
>>
>>
>>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Peter van Heusden]

Thanks - which is probably because of the previous failures with
`shed_update` - which relate back to the topic of my original email. I
suspect that failure from `planemo shed_update` has led to this "not
installable" state. And I don't know how to solve the problem from
`shed_update` which *looks* like it is coming from a clash between previous
and current definitions of the tool-data table.

Peter

On Mon, 17 Sep 2018 at 12:58 Nicola Soranzo 
wrote:

> Hi Peter,
> if you open the page for data_manager_shapeit_reference in the test
> ToolShed you will see that revision 5 and 6 are not installable, they both
> have "*This revision can be installed:* False" . If you instead select
> revision 3 it is installable.
>
> Cheers,
> Nicola
>
>
> On 17/09/18 11:51, Peter van Heusden wrote:
>
> I'm following up on my previous email. I somehow managed to get `planemo
> shed_update` to work but now I seem to have an inconsistent view of the
> Test toolshed. This is how it looks from the Admin panel of my Galaxy
> server:
>
> [image: https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png]
> Notice that it shows 3 installable revisions for the data manager, last
> updated 13th of September. And this is how it looks from the Test toolshed
> website:
>
> [image: https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png]
>
>
> Notice that it has 6 installabale revisions for the data manager. Is my
> Galaxy server perhaps caching something?
>
> Peter
>
>
> On Sat, 15 Sep 2018 at 09:57 Peter van Heusden  wrote:
>
>> Hi there
>>
>> I'm working on a data manager to keep track of the reference data
>> collection used by the SHAPEIT2 tool. I had to add a column to the data
>> manager but when I try and do a `planemo shed_update` I get the error:
>>
>> Unexpected HTTP status code: 500: {"err_msg": "Error attempting to parse
>> file tool_data_table_conf.xml.sample: Merging tabular data tables with non
>> matching columns is not allowed: shapeit_ref:{'path': 2,
>> 'reference_prefix': 3, 'name': 1, 'value': 0, 'map_prefix': 4} !=
>> shapeit_ref:{'name': 1, 'value': 0, 'sample_prefix': 5, 'path': 2,
>> 'reference_prefix': 3, 'map_prefix': 4}"}
>>
>> This is for data_manager_shapeit_reference in testtoolshed, which is on
>> github at
>>
>> https://github.com/pvanheus/data_manager_shapeit_reference
>>
>> As far as I can tell the columns do accord. Is there any way to see which
>> files are being merged to help try and debug this error?
>>
>> Thanks!
>> Peter
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Nicola Soranzo]

Hi Peter,
if you open the page for data_manager_shapeit_reference in the test 
ToolShed you will see that revision 5 and 6 are not installable, they 
both have "*This revision can be installed:* False" . If you instead 
select revision 3 it is installable.


Cheers,
Nicola

On 17/09/18 11:51, Peter van Heusden wrote:
I'm following up on my previous email. I somehow managed to get 
`planemo shed_update` to work but now I seem to have an inconsistent 
view of the Test toolshed. This is how it looks from the Admin panel 
of my Galaxy server:


https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png
Notice that it shows 3 installable revisions for the data manager, 
last updated 13th of September. And this is how it looks from the Test 
toolshed website:


https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png


Notice that it has 6 installabale revisions for the data manager. Is 
my Galaxy server perhaps caching something?


Peter


On Sat, 15 Sep 2018 at 09:57 Peter van Heusden > wrote:


Hi there

I'm working on a data manager to keep track of the reference data
collection used by the SHAPEIT2 tool. I had to add a column to the
data manager but when I try and do a `planemo shed_update` I get
the error:

Unexpected HTTP status code: 500: {"err_msg": "Error attempting to
parse file tool_data_table_conf.xml.sample: Merging tabular data
tables with non matching columns is not allowed:
shapeit_ref:{'path': 2, 'reference_prefix': 3, 'name': 1, 'value':
0, 'map_prefix': 4} != shapeit_ref:{'name': 1, 'value': 0,
'sample_prefix': 5, 'path': 2, 'reference_prefix': 3,
'map_prefix': 4}"}

This is for data_manager_shapeit_reference in testtoolshed, which
is on github at

https://github.com/pvanheus/data_manager_shapeit_reference

As far as I can tell the columns do accord. Is there any way to
see which files are being merged to help try and debug this error?

Thanks!
Peter



___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] "Merging tabular data tables with non-matching columns is not allowed"

[Peter van Heusden]

I'm following up on my previous email. I somehow managed to get `planemo
shed_update` to work but now I seem to have an inconsistent view of the
Test toolshed. This is how it looks from the Admin panel of my Galaxy
server:

[image: https://files.gitter.im/galaxyproject/Lobby/k2qH/image.png]
Notice that it shows 3 installable revisions for the data manager, last
updated 13th of September. And this is how it looks from the Test toolshed
website:

[image: https://files.gitter.im/galaxyproject/Lobby/ymY9/image.png]


Notice that it has 6 installabale revisions for the data manager. Is my
Galaxy server perhaps caching something?

Peter


On Sat, 15 Sep 2018 at 09:57 Peter van Heusden  wrote:

> Hi there
>
> I'm working on a data manager to keep track of the reference data
> collection used by the SHAPEIT2 tool. I had to add a column to the data
> manager but when I try and do a `planemo shed_update` I get the error:
>
> Unexpected HTTP status code: 500: {"err_msg": "Error attempting to parse
> file tool_data_table_conf.xml.sample: Merging tabular data tables with non
> matching columns is not allowed: shapeit_ref:{'path': 2,
> 'reference_prefix': 3, 'name': 1, 'value': 0, 'map_prefix': 4} !=
> shapeit_ref:{'name': 1, 'value': 0, 'sample_prefix': 5, 'path': 2,
> 'reference_prefix': 3, 'map_prefix': 4}"}
>
> This is for data_manager_shapeit_reference in testtoolshed, which is on
> github at
>
> https://github.com/pvanheus/data_manager_shapeit_reference
>
> As far as I can tell the columns do accord. Is there any way to see which
> files are being merged to help try and debug this error?
>
> Thanks!
> Peter
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/