> Please download the complete test set and let it run and
>
I neither have a Mac nor do I have 50 GB of memory
I had a REPRODUCIBLE scenario where this problem occurs
> Out of 1200 or so runs it was only this single run that produced memory
> bloating
>
see if you can reproduce the memory problem
>
Can you explain the problem in more detail? What exactly happens when you
run which command with what arguments? What are you expecting to happen
instead and why?
it finished in 7 hours 1200 individual ooRexx processes
What does "1200 individual ooRexx processes" mean? Are you starting you
program with 1200 different sets of arguments? Sequentially or in parallel?
Which one of the programs shows the issue? Is it always the same one?
why is the interpreter not warning me when I overwrite an object with a
> string?
>
You're not overwriting an object with a string, you're changing a variable
from referring to one object to referring to another one. That's totally
normal .. similar to coding a = 1; a = 2;
On Tue, Jun 27, 2017 at 10:26 PM, P.O. Jonsson <oor...@jonases.se> wrote:
> "maybe it is just bad programming“
>
> I guess I had it coming…
>
> Thanks Erich for your advice, I will consider it all, but my intention
> with this report was another one; for the first time I had a REPRODUCIBLE
> scenario where this problem occurs. Out of 1200 or so runs it was only this
> single run that produced memory bloating so my assumption was that is was
> not ONLY :-) bad programming.
>
> Please download the complete test set and let it run and see if you can
> reproduce the memory problem I have. If so it is easy for you to just
> improve the code and see where the problem goes away. I have a feeling I am
> stuck at
>
> a = a~StableSort
>
> For quite some time, maybe because of unfavorable data. But I can´t tell
> for sure.
>
> PS I had the program run again overnight, it finished in 7 hours 1200
> individual ooRexx processes with no problem. In another run I am now at 53
> GB in a single process running at 100% CPU for 10 hours.
>
> Question on Mutable Buffers (there is a lot of *NEW* there): I understand
> I need to ~append or ~insert for the MB but why is the interpreter not
> warning me when I overwrite an object with a string? Why is that not an
> error? Is there a reason why it should be allowed to destroy an object like
> I did?
>
> Hälsningar/Regards/Grüsse,
> P.O. Jonsson
> oor...@jonases.se
>
>
>
>
> Am 27.06.2017 um 17:15 schrieb Erich Steinböck <erich.steinbo...@gmail.com
> >:
>
> maybe it is just bad programming
>
> Hi P.O.,
> I had a look at Split_data and as far as I can see there are a lot of
> things which can be improved.
>
> 1)
>
> You may want to re-read how to work with a MutableBuffer. E. g.
>
> tempMB = .mutablebuffer~new('')
> do while ..
> tempMB = qfileIn~linein
>
> Initializing a variable with a MutableBuffer instance, and afterwards
> assigning it a String (linein() resturns a String) doesn't make sense.
>
> I can see quite a few instances of this, e. g.
>
> TranslatedMB = .mutablebuffer~new('')
> do while ..
> DO i=1 TO i_End
> DO j=1 TO j_End
>
> TranslatedMB = TranslatedMB TranslateWordMB
>
> Again, the final TranslatedMB assignment is not what the ..MB ending of
> the variables suggest.
>
> 2)
>
> You might move invariant stuff (here: LeftWordsMB~Word(i) || '-') in an
> inner loop outside the loop, e.g.
>
> DO j=1 TO j_End
> TranslateWordMB = LeftWordsMB~Word(i) || '-' ||
> RightWordsMB~Word(j)
>
>
> 3)
>
> Consider using use a single startsWith() instead of the code between
> lines 448 and 485
>
> 4)
>
> IF TranslatedMB~WordPos(TranslateWordMB) > 0 THEN
> ..
> ELSE
> DO
> TranslatedMB = TranslatedMB TranslateWordMB
>
> Instead of building a long string of all things seen before, and checking
> with wordPos(), you might instead put all things seen into a Set and
> check with hasIndex()
>
> 5)
>
> Generally, using Arrays may be more efficient if you can save the Stem.0
> handling
> But then, using the proper type of Collection and appropriate algorithm
> may help much more
> To give suggestions for that, I'd need more detail would on what exactly
> you would like to achieve
>
> On Tue, Jun 27, 2017 at 7:55 AM, P.O. Jonsson <oor...@jonases.se> wrote:
>
>> Dear developers,
>>
>> I have had the memory bloating problem again, this time I reached 48 GB
>> (the maximum for one CPU in my machine) and the process only ended after
>> some 13 CPU hours with 100% CPU the whole time.
>>
>>
>>
>>
>> From the logging info I could confirm that the program was stuck
>> somewhere here most of the time, here are the rough steps
>>
>> Language pairs detected in C routine -> External call, no memory bloating
>> Data processing finished after 2107 Seconds 00:58:12
>> Splitting finished after 49487 Seconds 14:42:59 *-> Routine Split_data*
>> Sorting finished after 16527 Seconds 19:18:27 *-> Routine Sort_data*
>> Processing of Data file finished after 68123 Seconds
>> Writing the Logfile TR_DE-EN-eu_logfile.txt 26 Jun 2017 19:18:28
>>
>> I have enclosed the Routines in question.
>>
>> In my dropbox I have stored the complete program with some test data to
>> replicate the processing, the problem is reproducible. Just put the folder
>> somewhere, move there and perform the command indicated.
>>
>> https://www.dropbox.com/sh/vettlcb4f8ae3cw/AACWIQivo_F2KhhytJ6izkbFa?dl=0
>>
>> I run Open Object Rexx Version 5.0.0, Build date: May 20 2017, Addressing
>> mode: 64
>> Hardware Mac Pro with dual-CPU Xeon Processors running Mac OS Sierra
>> 10.12.5
>>
>> PS as I was making the screenshot the process finished nicely, no crash
>> or anything and the memory was released. So maybe it is just bad
>> programming, but at least you can confirm that then :-)
>>
>>
>>
>>
>> Hälsningar/Regards/Grüsse,
>> P.O. Jonsson
>> oor...@jonases.se
>>
>>
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Oorexx-devel mailing list
>> Oorexx-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>>
>>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot______
> _________________________________________
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Oorexx-devel mailing list
> Oorexx-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/oorexx-devel
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Oorexx-devel mailing list
Oorexx-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/oorexx-devel
