Ahh I see, tricky.
So I think what is killing you is that you are pulling each row of the
table individually over the network. Ideally you should be able to do
something like the following:
f.root.table.cols.my_col[:,n,:]
using numpy-esque multidimensional slicing. However, this fails when I
just tested it. So instead, I would just pull over the full column and
slice using numpy in memory.
my_col = f.root.table.cols.my_col[:]
my_selection = my_col[:,n,:]
We should open a ticket so that the top method works (though I think there
might already be one).
I hope this helps!
On Sun, Jul 15, 2012 at 4:27 PM, Juan Manuel Vázquez Tovar <
jmv.to...@gmail.com> wrote:
> The column I´m requesting the data from has multidimensional cells, so
> each time I request data from the table, I need to get a specific row for
> all the multidimensional cells in the column. I hope this clarifies a bit.
> I have at the office a Linux workstation, but it is part of a computing
> cluster where all the users have access, so the files are in a folder of
> the cluster, not in my hard drive.
>
> Thank you,
> Juanma
>
> 2012/7/15 Anthony Scopatz <scop...@gmail.com>
>
>> Rereading the original post, I am a little confused are your trying to
>> read the whole table, just a couple of rows that meet some condition, or
>> just one whole column, or one part of the column.
>>
>> To request the whole table without looping over each row in Python, index
>> every element:
>>
>> f.root.table[:]
>>
>>
>> To just get certain rows, use where().
>>
>> To get a single column, use the cols namespace:
>>
>> f.root.table.cols.my_column[:]
>>
>>
>> Why is this file elsewhere on the network?
>>
>> Be Well
>> Anthony
>>
>> On Sun, Jul 15, 2012 at 4:08 PM, Juan Manuel Vázquez Tovar <
>> jmv.to...@gmail.com> wrote:
>>
>>> Hello Anthony,
>>>
>>> I have to loop over the whole set of rows. Does the where method has any
>>> advantages in that case?
>>>
>>> Thank you,
>>> Juanma
>>>
>>> 2012/7/15 Anthony Scopatz <scop...@gmail.com>
>>>
>>>> Hello Juan,
>>>>
>>>> Try using the where() method [1], It has a lot of nice features under
>>>> the covers.
>>>>
>>>> Be Well
>>>> Anthony
>>>>
>>>> 1.
>>>> http://pytables.github.com/usersguide/libref.html?highlight=where#tables.Table.where
>>>>
>>>> On Sun, Jul 15, 2012 at 4:01 PM, Juan Manuel Vázquez Tovar <
>>>> jmv.to...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I have been using pytables for a few moths. The main structure of my
>>>>> files has a four column table, two of which have multidimensional cells,
>>>>> (56,1) and (133,6) respectively. The previous structure had more columns
>>>>> instead of storing the 56x1 array into the same cell. The largest file has
>>>>> almost three million rows in the table.
>>>>> I usually request data from the table looping through the entire table
>>>>> and getting for each row one specific row of the 133x6 2d array.
>>>>> Currently, each of the requests can take from 15 sec up to 10 minutes,
>>>>> I believe that depending on the status of the office network.
>>>>> Could you please advice about how to improve the reading time?
>>>>> I have tried to compress the data with zlib, but it takes more or less
>>>>> the same time.
>>>>>
>>>>> Thanks in advance,
>>>>>
>>>>> Juan Manuel
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Live Security Virtual Conference
>>>>> Exclusive live event will cover all the ways today's security and
>>>>> threat landscape has changed and how IT managers can respond.
>>>>> Discussions
>>>>> will include endpoint security, mobile security and the latest in
>>>>> malware
>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>> _______________________________________________
>>>>> Pytables-users mailing list
>>>>> Pytables-users@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>>
>>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Pytables-users mailing list
>>>> Pytables-users@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>>
>>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Pytables-users mailing list
>>> Pytables-users@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Pytables-users mailing list
>> Pytables-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/pytables-users
>>
>>
>
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Pytables-users mailing list
> Pytables-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/pytables-users
>
>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
