YannickR wrote:
>
>
> YannickR wrote:
>>
>>
>> Yannick Richard wrote:
>>>
>>>
>>> Thomas Vandahl-2 wrote:
>>>>
>>>> Yannick Richard wrote:
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> I am currently working on a Torque project that will handle database
>>>>> synchronization.
>>>>>
>>>>> The problem we have is an Out of Memory exception while selecting a
>>>>> big
>>>>> bunch of data from the database.
>>>>>
>>>>>
>>>>>
>>>>> Here is the command we are using :
>>>>>
>>>>> List ObjectsFromDB = ObjectPeer.doSelect(criteria, connection);
>>>>>
>>>>>
>>>>>
>>>>> I saw the LargeSelect class you worked on but cannot find any Java
>>>>> example that could help me go forward.
>>>>>
>>>>> Could you help me point to an example or help me understand how to
>>>>> integrate LargeSelect ?
>>>>
>>>> Just a few hints, I don't have a complete example at hand:
>>>>
>>>> LargeSelect ls = new LargeSelect(criteria, pageSize,
>>>> memoryPageLimit,
>>>> ObjectPeer.class.getName());
>>>>
>>>> where the pageSize defines how many records to get with one call and
>>>> the
>>>> memoryPageLimit defines how many of these pages to "read ahead".
>>>>
>>>> With this object you can now loop through the pages and LargeSelect
>>>> will
>>>> load the necessary data as needed, (pageSize * memoryPageLimit) records
>>>> at a time. Like:
>>>>
>>>> while (ls.getNextResultsAvailable())
>>>> {
>>>> List ObjectsFromDB = ls.getNextResults();
>>>> // do what is necessary
>>>> }
>>>>
>>>> See the JavaDoc at
>>>> http://db.apache.org/torque/releases/torque-3.3/runtime/apidocs/index.html
>>>> for more information.
>>>>
>>>> Bye, Thomas.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: [EMAIL PROTECTED]
>>>> For additional commands, e-mail: [EMAIL PROTECTED]
>>>>
>>>>
>>>>
>>>
>>> With your explanations I succeeded in running LargeSelect. It is now
>>> working while reading 10 000 records with a PageSize of 1000 and a
>>> MemoryPageLimit of 20. I don't know if it is normal, but when I use a
>>> MemoryPageLimit of 5 (5x1000 records), it is reading 1 - 5000 of 10000
>>> again and again and never get out of the loop...
>>>
>>> Next, I tried to use PageSize of 5000/MemoryPageLimit of 50 in order to
>>> read less than 250 000 records, I had following exception : Exception in
>>> thread "Thread-2" java.lang.OutOfMemoryError: Java heap space
>>>
>>> I then modified the Eclipse shortcut arguments to better manage Heap
>>> memory, etc :
>>> -vmargs -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled
>>> -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M -Xms128M -Xmx1024M
>>>
>>> Unfortunately, it jammed on first call of .getNextResults() for a
>>> night...
>>> When I debug it I can see that the Thread is sleeping in
>>> getResults(start, size) for all night on following command: while
>>> (((start + size - 1) > currentlyFilledTo) && !queryCompleted)
>>>
>>> Is this a memory problem or I did something wrong ?
>>>
>>> Regards,
>>> Yannick Richard
>>>
>>>
>>>
>>
>
> Never mind, I found the solution :jumping:
> Readind Torque documentation, I tought MSSQL was not supported with Limit
> and Offset features used by LargeSelect so I installed TORQUE-84 patch
> from https://issues.apache.org/jira/browse/TORQUE-84 and this was causing
> getNextResultsAvailable() to not function correctly...
>
> I was also misunderstanding the use of PageSize and MemoryPageLimit. Total
> batch size in memory is not only the number of records in a page but
> PageSize * MemoryPageLimit...
>
> So without that patch, that is not completed anyway, and some time spent
> on RTFM... Everything is fine now ! Thanks for your help !
>
>
--
View this message in context:
http://www.nabble.com/LargeSelect-example---tf4605414.html#a13427019
Sent from the Apache DB - Torque Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
