Re: [NTG-context] Variable instance identifiers exhausted

[Hans Hagen]

On 9/21/2018 5:04 PM, Fabrice L wrote:
Le 19 septembre 2018 à 15:47:07, Fabrice L (fabrice.al...@gmail.com 
) a écrit:




Le 19 septembre 2018 à 15:28:54, Alan Braslau 
(braslau.l...@comcast.net 
(mailto:braslau.l...@comcast.net 
)) a écrit:


> On Tue, 18 Sep 2018 19:21:31 -0700   
> Fabrice L wrote:   
>   
> > The problem is that the process take hours of compilations, and   
> > tests are not easy to do. Anyway, I will enrich my « minimal » code to my   
> > « full » code in order to find the difference… and the problem. I will   
> > update the thread as soon as can.   
>   
> My over 700k object graphics example, processed several times (repeatedly, over several pages with modifications) took less than one minute to process. In fact, each graphic took almost as long to render from the pdf as it did for metapost/lua to process.   
> 
> Alan 

I do not know what your example is (but I’m curious: can I see this 
somewhere ?).


In my case, the long time needed is to check if a new object does not 
intersect with any other object: so when I try to find the maximum 
size of the 1000th object at a given position, I have to check with 
the 999 others… I do this with « intersectiontimes ». And so on for 
the 10001th… I have a course try to find solutions to gain time, like 
using the coordinate of a new object: when the coordinate of a new 
object is chosen, I do not need to check potential intersections with 
objects far away… but surprisingly (to me) this take more time than to 
blindly check with every object.


This is an algorithmic problem more than a ConTeXt/Metapost one… but I 
you have suggestions, I’m sure I can learn !


Below is minimal code. (Just to be clear, this code work, even for 20K 
objects.)

Thanks.

\starttext
  \startMPpage
  path Obj[] , ObjTemp ;
  pair ObjPos , Inter ;
  numeric NObj ;
  % Change herte "NObj" to a big numer (says 20K)
  NObj := 100;
  numeric Danger, MaxSize ;

  % Draw an initial object of random shape and random size ;
  Obj[1]:= fullsquare randomized .001in
  scaled (uniformdeviate(.5)*1in)
  shifted (uniformdeviate(5)*1in,uniformdeviate(5)*1in) ;
  Obj[1] := curved ( Obj[1] );
  fill Obj[1] withcolor cyan randomized (.2,1.5);
  draw Obj[1] withcolor black randomized (.2,1.5);

  for i=2 upto NObj :
  Danger := 0;
% Choose a position for the object :
  ObjPos := (uniformdeviate(5)*1in,uniformdeviate(5)*1in) ;
  % Choose a form :
  Obj[i] := fullsquare randomized .01in ;
  Obj[i] := curved ( Obj[i] );
  % Choose a scale so the object does not interact with any other object:
    for j=1 upto 50:
       MaxSize := (j*.01in) ;
    ObjTemp := Obj[i] scaled MaxSize shifted ObjPos ;
       %draw ObjTemp withcolor magenta ;
       % Check if this temporary object intersect with any other object :
       for k=1 upto (i-1):
        string trace ;
         Inter := Obj[k] intersectiontimes ObjTemp ;
         if (xpart Inter) <> -1 : Danger := 1; fi ;
         if Danger=1: MaxSize := ((j-1)*.01in) ; fi;
         exitif Danger = 1;
       endfor;
       exitif Danger = 1;
  endfor ;
  Obj[i] := Obj[i] scaled MaxSize shifted ObjPos ;

  fill Obj[i] withcolor cyan randomized (.2,1.5);
  draw Obj[i] withcolor black randomized (.2,1.5);
  endfor;
  \stopMPpage
\stoptext




So after some hours of computation, this code above breaks for me if the 
number of objects,  NObj in my code, is too large. Of course, perhaps an 
error occurs at some point and is not caused directly by the number of 
objects, because according to what has been said in this thread this 
should not be is not a problem. I put below the log. If somebody has a 
hint in which direction I should look… thanks !


Fabrice.

mtx-context     | run 1: luatex 
--fmt="/Users/fabricel/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en" 
--jobname="0030-0018-minimal2" 
--lua="/Users/fabricel/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en.lui" 
--no-parse-first-line --c:currentrun=1 
--c:fulljobname="./0030-0018-minimal2.tex" 
--c:input="./0030-0018-minimal2.tex" --c:kindofrun=1 --c:maxnofruns=1 
--c:runs="1" "cont-yes.mkiv"


This is LuaTeX, Version 1.08.0 (TeX Live 2018)
  system commands enabled.
open source     > level 1, order 1, name 
'/Users/fabricel/context/tex/texmf-context/tex/context/base/mkiv/cont-yes.mkiv'

system          >
system          > ConTeXt  ver: 2018.05.17 18:32 MKIV beta  fmt: 
2018.5.18  int: english/english

system          >
system          > 'cont-new.mkiv' loaded
open source     > level 2, order 2, name 
'/Users/fabricel/context/tex/texmf-context/tex/context/base/mkiv/cont-new.mkiv'

system          > beware: some patches loaded from cont-new.mkiv
close source    > level 2, order 2, name 
'/Users/fabricel/context/tex/texmf-context/tex/context/base/mkiv/cont-new.mkiv'
system          > files 

Re: [NTG-context] Variable instance identifiers exhausted

[Fabrice L]

Le 19 septembre 2018 à 15:47:07, Fabrice L (fabrice.al...@gmail.com) a
écrit:




Le 19 septembre 2018 à 15:28:54, Alan Braslau (braslau.l...@comcast.net
(mailto:braslau.l...@comcast.net)) a écrit:

> On Tue, 18 Sep 2018 19:21:31 -0700
> Fabrice L wrote:
>
> > The problem is that the process take hours of compilations, and
> > tests are not easy to do. Anyway, I will enrich my « minimal » code to
my
> > « full » code in order to find the difference… and the problem. I will
> > update the thread as soon as can.
>
> My over 700k object graphics example, processed several times
(repeatedly, over several pages with modifications) took less than one
minute to process. In fact, each graphic took almost as long to render from
the pdf as it did for metapost/lua to process.
>
> Alan

I do not know what your example is (but I’m curious: can I see this
somewhere ?).

In my case, the long time needed is to check if a new object does not
intersect with any other object: so when I try to find the maximum size of
the 1000th object at a given position, I have to check with the 999 others…
I do this with « intersectiontimes ». And so on for the 10001th… I have a
course try to find solutions to gain time, like using the coordinate of a
new object: when the coordinate of a new object is chosen, I do not need to
check potential intersections with objects far away… but surprisingly (to
me) this take more time than to blindly check with every object.

This is an algorithmic problem more than a ConTeXt/Metapost one… but I you
have suggestions, I’m sure I can learn !

Below is minimal code. (Just to be clear, this code work, even for 20K
objects.)
Thanks.

\starttext
  \startMPpage
  path Obj[] , ObjTemp ;
  pair ObjPos , Inter ;
  numeric NObj ;
  % Change herte "NObj" to a big numer (says 20K)
  NObj := 100;
  numeric Danger, MaxSize ;

  % Draw an initial object of random shape and random size ;
  Obj[1]:= fullsquare randomized .001in
  scaled (uniformdeviate(.5)*1in)
  shifted (uniformdeviate(5)*1in,uniformdeviate(5)*1in) ;
  Obj[1] := curved ( Obj[1] );
  fill Obj[1] withcolor cyan randomized (.2,1.5);
  draw Obj[1] withcolor black randomized (.2,1.5);

  for i=2 upto NObj :
  Danger := 0;
% Choose a position for the object :
  ObjPos := (uniformdeviate(5)*1in,uniformdeviate(5)*1in) ;
  % Choose a form :
  Obj[i] := fullsquare randomized .01in ;
  Obj[i] := curved ( Obj[i] );
  % Choose a scale so the object does not interact with any other object:
for j=1 upto 50:
   MaxSize := (j*.01in) ;
ObjTemp := Obj[i] scaled MaxSize shifted ObjPos ;
   %draw ObjTemp withcolor magenta ;
   % Check if this temporary object intersect with any other object :
   for k=1 upto (i-1):
string trace ;
 Inter := Obj[k] intersectiontimes ObjTemp ;
 if (xpart Inter) <> -1 : Danger := 1; fi ;
 if Danger=1: MaxSize := ((j-1)*.01in) ; fi;
 exitif Danger = 1;
   endfor;
   exitif Danger = 1;
  endfor ;
  Obj[i] := Obj[i] scaled MaxSize shifted ObjPos ;

  fill Obj[i] withcolor cyan randomized (.2,1.5);
  draw Obj[i] withcolor black randomized (.2,1.5);
  endfor;
  \stopMPpage
\stoptext



So after some hours of computation, this code above breaks for me if the
number of objects,  NObj in my code, is too large. Of course, perhaps an
error occurs at some point and is not caused directly by the number of
objects, because according to what has been said in this thread this should
not be is not a problem. I put below the log. If somebody has a hint in
which direction I should look… thanks !

Fabrice.

mtx-context | run 1: luatex
--fmt="/Users/fabricel/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en"
--jobname="0030-0018-minimal2"
--lua="/Users/fabricel/context/tex/texmf-cache/luatex-cache/context/5fe67e0bfe781ce0dde776fb1556f32e/formats/luatex/cont-en.lui"
--no-parse-first-line --c:currentrun=1
--c:fulljobname="./0030-0018-minimal2.tex"
--c:input="./0030-0018-minimal2.tex" --c:kindofrun=1 --c:maxnofruns=1
--c:runs="1" "cont-yes.mkiv"

This is LuaTeX, Version 1.08.0 (TeX Live 2018)
 system commands enabled.
open source > level 1, order 1, name
'/Users/fabricel/context/tex/texmf-context/tex/context/base/mkiv/cont-yes.mkiv'
system  >
system  > ConTeXt  ver: 2018.05.17 18:32 MKIV beta  fmt: 2018.5.18
 int: english/english
system  >
system  > 'cont-new.mkiv' loaded
open source > level 2, order 2, name
'/Users/fabricel/context/tex/texmf-context/tex/context/base/mkiv/cont-new.mkiv'
system  > beware: some patches loaded from cont-new.mkiv
close source> level 2, order 2, name
'/Users/fabricel/context/tex/texmf-context/tex/context/base/mkiv/cont-new.mkiv'
system  > files > jobname '0030-0018-minimal2', input
'./0030-0018-minimal2', result '0030-0018-minimal2'
fonts   > latin modern fonts are not preloaded
languages   > language 'en' is active
open source > level 

Re: [NTG-context] Variable instance identifiers exhausted

[Hans Hagen]

On 9/19/2018 10:45 PM, Alan Braslau wrote:

Metapost does not have a very efficient mechanism for keeping objects in memory 
and does not have a very efficient or real array mechanism, rather it uses 
hashed lists.


fwiw, keeping objects as such in memory is quite ok, hashing is also ok, 
but indeed arrays are not really arrays but linked lists into the hash


(which actually you can speed up a lot)

so, here is a teaser for Alan (no lua magic here, just knowledge of how 
mp hashes):


\dontleavehmode \testfeatureonce{1}{
\startMPcode
numeric foo[];

def set_foo(expr c, s) =
foo[c]:= s ;
enddef ;
def get_foo(expr c) =
foo[c]
enddef ;

for i=1 upto 1 :
set_foo(i,i) ;
endfor ;

draw textext("set one dimensional");
\stopMPcode
} \elapsedtime \blank

\dontleavehmode \testfeatureonce{1}{
\startMPcode
numeric n ;
for i=1 upto 1 :
n := get_foo(i) ;
endfor ;

draw textext("get one dimensional");
\stopMPcode
} \elapsedtime \blank


\dontleavehmode \testfeatureonce{1}{
\startMPcode
numeric foo[][][][];

def set_foo(expr c, s) =
foo[c div 1000][c div 100][c div 10][c] := s ;
enddef ;
def get_foo(expr c) =
foo[c div 1000][c div 100][c div 10][c]
enddef ;

for i=1 upto 1 :
set_foo(i,i) ;
endfor ;

numeric n ;
for i=1 upto 1 :
n := get_foo(i) ;
endfor ;

draw textext("set four dimensional" );
\stopMPcode
} \elapsedtime \blank

\dontleavehmode \testfeatureonce{1}{
\startMPcode
numeric n ;
for i=1 upto 1 :
n := get_foo(i) ;
endfor ;

draw textext("get four dimensional");
\stopMPcode
} \elapsedtime \blank


Hans

-
  Hans Hagen | PRAGMA ADE
  Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
   tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-
___
If your question is of interest to others as well, please add an entry to the 
Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___

Re: [NTG-context] Variable instance identifiers exhausted

[Alan Braslau]

On Tue, 18 Sep 2018 19:21:31 -0700
Fabrice L  wrote:

> The problem is that the process take hours of compilations, and
> tests are not easy to do. Anyway, I will enrich my « minimal » code to my
> « full » code in order to find the difference… and the problem. I will
> update the thread as soon as can.

My over 700k object graphics example, processed several times (repeatedly, over 
several pages with modifications) took less than one minute to process. In 
fact, each graphic took almost as long to render from the pdf as it did for 
metapost/lua to process.

Alan
___
If your question is of interest to others as well, please add an entry to the 
Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___

Re: [NTG-context] Variable instance identifiers exhausted

[Alan Braslau]

On Wed, 19 Sep 2018 09:28:36 +0200
Hans Hagen  wrote:

> On 9/19/2018 4:24 AM, Fabrice L wrote:
> 
> > I do not think this a loop problem, because the same code runs ok for 15 
> > 000 objects, but give an error for 18 000 objects. I’m working to build 
> > a minimal example.
> So what do you do with the objects? At the ctx meeting we've seen 
> examples of graphics using many more than 15K that processed fast and 
> also (maybe surprisingly) could be previewed quite ok. So, indeed a 
> minimal example would help.

No, it was more than 700k objects!

Alan ;-)
___
If your question is of interest to others as well, please add an entry to the 
Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___

Re: [NTG-context] Variable instance identifiers exhausted

[Hans Hagen]

On 9/19/2018 4:24 AM, Fabrice L wrote:

I do not think this a loop problem, because the same code runs ok for 15 
000 objects, but give an error for 18 000 objects. I’m working to build 
a minimal example.
So what do you do with the objects? At the ctx meeting we've seen 
examples of graphics using many more than 15K that processed fast and 
also (maybe surprisingly) could be previewed quite ok. So, indeed a 
minimal example would help.


Hans

-
  Hans Hagen | PRAGMA ADE
  Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
   tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-
___
If your question is of interest to others as well, please add an entry to the 
Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___

Re: [NTG-context] Variable instance identifiers exhausted

[Fabrice L]

Le 18 septembre 2018 à 03:11:40, Taco Hoekwater (t...@elvenkind.com) a
écrit:

> On 17 Sep 2018, at 21:17, Fabrice L  wrote:
>
> Dear List,
>
> I’m running into a problem in Metapost (using Context of course!)
> creating (I think) too many objects; I got this error message:
>
> variable instance identifiers exhausted

The max value for that is 2^31, or about 2 billion (and some change).
Are you sure you do not have an eternal loop in your source?

The number of objects I have is clearly less than 2^31, so my problem is
elsewhere then.

I do not think this a loop problem, because the same code runs ok for 15
000 objects, but give an error for 18 000 objects. I’m working to build a
minimal example.

Thanks a lot for the help ! Much appreciated.

Fabrice.



Best wishes,
Taco





___
If your question is of interest to others as well, please add an entry to
the Wiki!

maillist : ntg-context@ntg.nl /
http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___
___
If your question is of interest to others as well, please add an entry to the 
Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___

Re: [NTG-context] Variable instance identifiers exhausted

[Fabrice L]

Le 18 septembre 2018 à 01:15:36, luigi scarso (luigi.sca...@gmail.com) a
écrit:



On Mon, Sep 17, 2018 at 9:18 PM Fabrice L  wrote:

> Dear List,
>
> I’m running into a problem in Metapost (using Context of course!)
> creating (I think) too many objects; I got this error message:
>
> variable instance identifiers exhausted
>
> I have tried without success to find information and a solution.
> Does somebody has a hint to solve this ?
>

Do you have  minimal example ?

I have worked on a minimal example today with no success ! What I mean is
that I have rebuilt the same algorithm from scratch in order to show
something minimal to the list, and I do not seem to encounter the problem
anymore ! The problem is that the process take hours of compilations, and
tests are not easy to do. Anyway, I will enrich my « minimal » code to my
« full » code in order to find the difference… and the problem. I will
update the thread as soon as can.

Thanks a lot for helping.

Fabrice.



--
luigi
___
If your question is of interest to others as well, please add an entry to
the Wiki!

maillist : ntg-context@ntg.nl /
http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___
___
If your question is of interest to others as well, please add an entry to the 
Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___

Re: [NTG-context] Variable instance identifiers exhausted

[Taco Hoekwater]

> On 17 Sep 2018, at 21:17, Fabrice L  wrote:
> 
> Dear List,
> 
> I’m running into a problem in Metapost (using Context of course!)
> creating (I think) too many objects; I got this error message:
> 
> variable instance identifiers exhausted

The max value for that is 2^31, or about 2 billion (and some change).
Are you sure you do not have an eternal loop in your source?

Best wishes,
Taco





___
If your question is of interest to others as well, please add an entry to the 
Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___

Re: [NTG-context] Variable instance identifiers exhausted

[luigi scarso]

On Mon, Sep 17, 2018 at 9:18 PM Fabrice L  wrote:

> Dear List,
>
> I’m running into a problem in Metapost (using Context of course!)
> creating (I think) too many objects; I got this error message:
>
> variable instance identifiers exhausted
>
> I have tried without success to find information and a solution.
> Does somebody has a hint to solve this ?
>

Do you have  minimal example ?

-- 
luigi
___
If your question is of interest to others as well, please add an entry to the 
Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___

[NTG-context] Variable instance identifiers exhausted

[Fabrice L]

Dear List,

I’m running into a problem in Metapost (using Context of course!)
creating (I think) too many objects; I got this error message:

variable instance identifiers exhausted

I have tried without success to find information and a solution.
Does somebody has a hint to solve this ?

Thanks, Fabrice.
___
If your question is of interest to others as well, please add an entry to the 
Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___