Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 13.04.2017 18:06 schrieb "Ryan Joseph" :
>
>
> > On Apr 13, 2017, at 10:29 PM, Sven Barth via fpc-pascal <
fpc-pascal@lists.freepascal.org> wrote:
> >
> > SetLength *does* use a reallocate for this, but since it doesn't give
you any guarantee for its success as its the task of the memory manager to
deal with this, it's easier to assume that the array is indeed copied.
> >
>
> So the real point here is that ReAllocMem will copy and allocate a new
block if the old one can’t be resized, therefore this really isn’t about
SetLength as much as the memory manager itself.

Correct. And since the memory manager is changeable and might implement
whatever behavior it wants nothing fixed can be done for this.

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 13, 2017, at 10:29 PM, Sven Barth via fpc-pascal 
>  wrote:
> 
> SetLength *does* use a reallocate for this, but since it doesn't give you any 
> guarantee for its success as its the task of the memory manager to deal with 
> this, it's easier to assume that the array is indeed copied.
> 

So the real point here is that ReAllocMem will copy and allocate a new block if 
the old one can’t be resized, therefore this really isn’t about SetLength as 
much as the memory manager itself.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 13.04.2017 14:47 schrieb "Ryan Joseph" :
>
>
> > On Apr 13, 2017, at 7:08 PM, Mattias Gaertner 
wrote:
> >
> >> as I understood from (http://wiki.freepascal.org/Dynamic_array<
http://wiki.freepascal.org/Dynamic_array>), SetLength will create a copy of
the array and free the memory of the shorter array. In this case, a lot of
memory operations and copy operations are performed thus degrading the
performances of the code.
> >
> > Correct.
>
> Why is it copying the array and freeing instead of resizing the existing
block (realloc)? That sounds crazy but I don’t know how memory managers
work.

SetLength *does* use a reallocate for this, but since it doesn't give you
any guarantee for its success as its the task of the memory manager to deal
with this, it's easier to assume that the array is indeed copied.

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Mattias Gaertner]

On Thu, 13 Apr 2017 19:15:13 +0700
Ryan Joseph  wrote:

> > On Apr 13, 2017, at 7:08 PM, Mattias Gaertner  
> > wrote:
> >   
> >> as I understood from 
> >> (http://wiki.freepascal.org/Dynamic_array),
> >>  SetLength will create a copy of the array and free the memory of the 
> >> shorter array. In this case, a lot of memory operations and copy 
> >> operations are performed thus degrading the performances of the code.  
> > 
> > Correct.  
> 
> Why is it copying the array and freeing instead of resizing the existing 
> block (realloc)? That sounds crazy but I don’t know how memory managers work.

It does not free the elements.
It frees the array.

Mattias
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 13, 2017, at 7:08 PM, Mattias Gaertner  
> wrote:
> 
>> as I understood from 
>> (http://wiki.freepascal.org/Dynamic_array),
>>  SetLength will create a copy of the array and free the memory of the 
>> shorter array. In this case, a lot of memory operations and copy operations 
>> are performed thus degrading the performances of the code.
> 
> Correct.

Why is it copying the array and freeing instead of resizing the existing block 
(realloc)? That sounds crazy but I don’t know how memory managers work.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 13.04.2017 13:25 schrieb "MARCOU Gilles" :
>
> Regarding this code:
>
>> SetLength(Array,Length(Array)+1);
>> Array[High(Array)] := …
>
>
> as I understood from (http://wiki.freepascal.org/Dynamic_array),
SetLength will create a copy of the array and free the memory of the
shorter array. In this case, a lot of memory operations and copy operations
are performed thus degrading the performances of the code. Then it would be
unwise to use such strategy: better allocate the size of the array
correctly at the beginning of the run, or resize the array by block, to
decrease the frequency of such operation:
>
>> SetLength(Array,Length(Array)+N);
>
>
> Can somebody confirm (or invalidate) this?

SetLength() uses reallocation for the memory. So it *might* work without
copying though that is not guaranteed.
But in general it's recommended anyway not to do "SetLength(..., ... + 1)"
loops as that will lead to fragmentation of the heap.

Regards
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Mattias Gaertner]

On Thu, 13 Apr 2017 13:17:37 +0200
MARCOU Gilles  wrote:

> Regarding this code:
> 
> > SetLength(Array,Length(Array)+1);
> > Array[High(Array)] := …  
> 
> as I understood from (http://wiki.freepascal.org/Dynamic_array 
> ), SetLength will create a copy of 
> the array and free the memory of the shorter array. In this case, a lot of 
> memory operations and copy operations are performed thus degrading the 
> performances of the code.

Correct.

> Then it would be unwise to use such strategy: better allocate the size
> of the array correctly at the beginning of the run, or resize the
> array by block, to decrease the frequency of such operation:

The usual solution is to grow exponentially:

SetLength(Array,Length(Array)+Max(4,length(Array) div 4))

Mattias
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[MARCOU Gilles]

Regarding this code:

> SetLength(Array,Length(Array)+1);
> Array[High(Array)] := …

as I understood from (http://wiki.freepascal.org/Dynamic_array 
), SetLength will create a copy of 
the array and free the memory of the shorter array. In this case, a lot of 
memory operations and copy operations are performed thus degrading the 
performances of the code. Then it would be unwise to use such strategy: better 
allocate the size of the array correctly at the beginning of the run, or resize 
the array by block, to decrease the frequency of such operation:

> SetLength(Array,Length(Array)+N);

Can somebody confirm (or invalidate) this?

Ciao,
Gilles Marcou___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Benito van der Zander]

Hi Ryan,

In the real world is anyone actually going to type that out every time? I’d bet 
most of us have an array class we use and ignore dynamic arrays all together 
because we all need basic operations like “add” and “remove”.


I, for one, have a huge list of array utility functions: 
http://hg.benibela.de/bbutils/file/tip/bbutils.inc


I use arrays rather than classes, because latter are not reference counted

Bye,
BeniBela



On 04/12/2017 04:30 PM, Ryan Joseph wrote:

On Apr 12, 2017, at 9:24 PM, Jürgen Hestermann  
wrote:

SetLength(Array,Length(Array)+1);

You will get an additional element (filled with zeros).
Then you can set

Array[High(Array)] := whateveryouwant;

Well sure, but this kind of code is exactly what programmers abstract and reuse 
so it’s a perfect candidate for a function.

SetLength(Array,Length(Array)+1);
Array[High(Array)] := …

In the real world is anyone actually going to type that out every time? I’d bet 
most of us have an array class we use and ignore dynamic arrays all together 
because we all need basic operations like “add” and “remove”.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal


___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-12 um 16:30 schrieb Ryan Joseph:
>> Array[High(Array)] := whateveryouwant;

> SetLength(Array,Length(Array)+1);
> Array[High(Array)] := …
> In the real world is anyone actually going to type that out every time?

Yes, I do this. Typing is not much work for me.
My main focus is on *readable* code (effort for writing is of low priority).
And the 2 lines show very clear what realy happens.

I can also omit the second line and set a value later
while with an Add-function I have to specify a value
for the new array element even if I just want to extend the array only.
I have finer control about what happens.

> I’d bet most of us have an array class we use and ignore dynamic arrays all 
together because we all need basic operations like “add” and “remove”.

I never use classes.
I like dynamic arrays very much.


___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 12, 2017, at 9:24 PM, Jürgen Hestermann  
> wrote:
> 
> SetLength(Array,Length(Array)+1);
> 
> You will get an additional element (filled with zeros).
> Then you can set
> 
> Array[High(Array)] := whateveryouwant;

Well sure, but this kind of code is exactly what programmers abstract and reuse 
so it’s a perfect candidate for a function. 

SetLength(Array,Length(Array)+1);
Array[High(Array)] := …

In the real world is anyone actually going to type that out every time? I’d bet 
most of us have an array class we use and ignore dynamic arrays all together 
because we all need basic operations like “add” and “remove”.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Michael Van Canneyt]

On Wed, 12 Apr 2017, Ryan Joseph wrote:




On Apr 12, 2017, at 9:25 PM, Michael Van Canneyt  wrote:

Adding a pop/push requires compiler magic, and could be implemented; but the 
question is whether it is worth it, given the plethora of other
options at your disposal.


Why magic?  It seems silly the operator += doesn’t exist or even a basic
operation set like “add”, “remove”, “insert” that every array
implementation in every language has.  Aren’t those just function around
the existing dynamic array implementation in the compiler?


That is why you need "the magic". The compiler needs to check the type, call
some helper routines etc.

Michael.___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 12.04.2017 16:51 schrieb "Ryan Joseph" :
>
>
> > On Apr 12, 2017, at 9:25 PM, Michael Van Canneyt 
wrote:
> >
> > Adding a pop/push requires compiler magic, and could be implemented;
but the question is whether it is worth it, given the plethora of other
> > options at your disposal.
>
> Why magic? It seems silly the operator += doesn’t exist or even a basic
operation set like “add”, “remove”, “insert” that every array
implementation in every language has. Aren’t those just function around the
existing dynamic array implementation in the compiler?

Delete() is already supported in trunk, Insert() and the +-operator (or
Concat()) are on the near term ToDo list (mainly out of Delphi
compatibility, though I definitely agree with their existence).

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-12 um 16:36 schrieb Ryan Joseph:
> Why magic? It seems silly the operator += doesn’t exist

Well, are we using Pascal or C? ;-)


> or even a basic operation set like “add”, “remove”, “insert” that every array 
implementation in every language has.
> Aren’t those just function around the existing dynamic array implementation 
in the compiler?

I aggree that these functions could be added.
I always create such functions myself but I need to
respect the type of my arrays and rewrite these functions
for each new type.
If they were available as standard routines it would avoid these rewrites.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 12, 2017, at 9:25 PM, Michael Van Canneyt  
> wrote:
> 
> Adding a pop/push requires compiler magic, and could be implemented; but the 
> question is whether it is worth it, given the plethora of other
> options at your disposal.

Why magic? It seems silly the operator += doesn’t exist or even a basic 
operation set like “add”, “remove”, “insert” that every array implementation in 
every language has. Aren’t those just function around the existing dynamic 
array implementation in the compiler?

I have no idea if it’s worth it though. I personally don’t use dynamic arrays 
directly because they’re too bare bones so adding them at all wasn’t worth it 
for me. However, if the API was more robust I would start using them instead of 
my own class/record wrappers.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 12, 2017, at 9:43 PM, Jürgen Hestermann  
> wrote:
> 
> "clear" is not very clear to me ;-)
> IMO SetLength(array,0) is clearing too.

SetLength resizes memory so there’s that distinction. The fact there is an API 
is what’s important because there’s at least a reference point.  

> 
> The function you want should be called "Fill".
> For example a function
> 
> Array.Fill(DefaultValue);

Good idea. Any defined function is preferable.

> 
> But what happens to a multi-dimensional (dynamic) array?
> For example, how to handle this case:
> 
> Type MyType = record
>  A : Integer;
>  B : array of char;
>  C : array of Float;
>  end; // of record
> ArrayA = array of MyType;
> ArrayB = array of ArrayA.
> 
> What would be the DefaultValue for
> 
> ArrayB.Fill(DefaultValue);

That’s one of the reasons we won’t be seeing this function any time soon. ;)

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Michael Van Canneyt]

On Wed, 12 Apr 2017, Jürgen Hestermann wrote:


Am 2017-04-12 um 16:17 schrieb Ryan Joseph:
> SetLength resizes the array in memory and “clear” means setting all values 
to default() for that type.


"clear" is not very clear to me ;-)
IMO SetLength(array,0) is clearing too.

The function you want should be called "Fill".
For example a function

Array.Fill(DefaultValue);

But what happens to a multi-dimensional (dynamic) array?
For example, how to handle this case:

Type MyType = record
  A : Integer;
  B : array of char;
  C : array of Float;
  end; // of record
 ArrayA = array of MyType;
 ArrayB = array of ArrayA.

What would be the DefaultValue for

ArrayB.Fill(DefaultValue);

?


I would think Nil is the only option here,  ArrayB is an array of arrays

Michael.___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-12 um 16:17 schrieb Ryan Joseph:
> SetLength resizes the array in memory and “clear” means setting all values to 
default() for that type.

"clear" is not very clear to me ;-)
IMO SetLength(array,0) is clearing too.

The function you want should be called "Fill".
For example a function

Array.Fill(DefaultValue);

But what happens to a multi-dimensional (dynamic) array?
For example, how to handle this case:

Type MyType = record
  A : Integer;
  B : array of char;
  C : array of Float;
  end; // of record
 ArrayA = array of MyType;
 ArrayB = array of ArrayA.

What would be the DefaultValue for

ArrayB.Fill(DefaultValue);

?
I don't see how this can be achieved in a solid and easy to unterstand 
Array.Fill routine
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Michael Van Canneyt]

On Wed, 12 Apr 2017, Ryan Joseph wrote:




On Apr 12, 2017, at 8:05 PM, nore...@z505.com wrote:

Since fillchar is not a general way to clear an item it almost seems like 
pascal needs a way to clear things generically, such as Array.clear
But I guess this would be reinventing OOP, or Ruby where everything is an 
object and can have .Something tacked on to it.


I agree. Dynamic arrays only advantage is they are managed by the compiler but 
they’re missing obvious functionality like adding/removing elements so they’re 
usually pretty useless. Why was the functionality of appending an element to 
the end never added? That’s one the most basic operations for arrays.


Because dynamic arrays are implemented after Delphi's implementation.

Only recently the new array constructors were implemented.
Copy exists.

Adding a pop/push requires compiler magic, and could be implemented; 
but the question is whether it is worth it, given the plethora of other

options at your disposal.


Michael.___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-12 um 15:46 schrieb Ryan Joseph:
> I agree. Dynamic arrays only advantage is they are managed by the compiler 
but they’re missing obvious functionality like adding/removing elements so they’re 
usually pretty useless.
> Why was the functionality of appending an element to the end never added? 
That’s one the most basic operations for arrays.

It exists:

SetLength(Array,Length(Array)+1);

You will get an additional element (filled with zeros).
Then you can set

Array[High(Array)] := whateveryouwant;
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 12, 2017, at 8:05 PM, nore...@z505.com wrote:
> 
> Since fillchar is not a general way to clear an item it almost seems like 
> pascal needs a way to clear things generically, such as Array.clear
> But I guess this would be reinventing OOP, or Ruby where everything is an 
> object and can have .Something tacked on to it.

I agree. Dynamic arrays only advantage is they are managed by the compiler but 
they’re missing obvious functionality like adding/removing elements so they’re 
usually pretty useless. Why was the functionality of appending an element to 
the end never added? That’s one the most basic operations for arrays.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 12, 2017, at 9:07 PM, Jürgen Hestermann  
> wrote:
> 
> What exactly does it do when "clearing" a dynamic array?
> There are multiple ways to "clear" it.

I see your point but there could be a good API to make this clear. SetLength 
resizes the array in memory and “clear” means setting all values to default() 
for that type. Not a difficult distinction to remember imo. The whole API is 
incomplete though so it doesn’t even make sense to add this unless your going 
to overhaul everything and make a complete list of operations.


Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-12 um 15:05 schrieb nore...@z505.com:
> Since fillchar is not a general way to clear an item

Define "clear". There are many things that can be named "clearing" (see below).


> I've never been a fan of things like fillchar as it seems like a reinvention 
of something more like Clear() and mistakes can easily be made when you use 
complex function like this:
> fillchar(something, something * something, 0)
> instead of a more simpler:
> Array.Clear
> or
> Clear(array)

It's true that the wording "char" is not ideal.
Instead you can use FillByte ( 
http://www.freepascal.org/docs-html/rtl/system/fillbyte.html )
which does the same (it has just another name).

FillByte does what it says: It fills a given memory area with bytes (which you 
can define).
Because of the nature of certain types it can damage i.e. managed types
so you have to know what you do (you have to know about the memory usage of the 
used type).

If you have a static array things are completely different as for dynamic 
arrays.
If you have a static array the memory it occupies is static (never changes) and
in one continuous block.
Dynamic arrays are completely different although their syntax is (nearly) the 
same.
So what should Array.Clear do in case of a static and in case of a dynamic 
array?

Clearing a static array can only mean filling it with bytes.
But do they have to be zero in all cases?
Array of integer may require different bytes than array of char.
In the end you as the programmer have to decide what is appropiate.
Array.Clear would not be able to guess what you intend.

Clearing a dynamic array can mean to free all memory it occupies
or fill its used memory with bytes.
Again you have to decide what to do
and which bytes should be used (if you fill everything).


> To me, fillchar doesn't actually say anything sensible about what the code is 
doing...

Well, if you use FillByte (instead of FillChar) then nothing can be clearer. It 
just fills memory with a chosen byte.


> "Clear" or "Default" (I think delphi is implementing something like this in 
later versions) says exactly what the code is doing.

What exactly does it do when "clearing" a dynamic array?
There are multiple ways to "clear" it.


> FillChar is some obscure old 1970's/80's pascal function that really doesn't 
tell what you are doing, unless you are an old pascal programmer who knows it from 
long ago.

It's true that those who grow up with it know that FillChar is actually a 
Fillbyte.
The first versions of FillChar just excepted the char type as the fill byte.


> I'm skeptical of these old ways of doing things.. Imagine a newcomer to pascal who did 
not program in the 1980's... He sees "fillchar" in the program...
> Does that code even tell this learner of pascal what is happening? What's a 
"char" have to do with an array that may not have anything to do with chars?

That applies to all Pascal functions.
You need to lookup what they actually do.
In case of FillChar/FillByte it is quite easy to unterstand.

Much more problematic are things that the compiler does
"under the hood" which is often not well documented.
If you don't know how the compiler handles dynamic arrays
you have a real problem as you cannot predict performance
nor do you know exactly how to handle them.
But that has nothing to do with FillChar...

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[noreply]

On 2017-04-01 02:39, Sven Barth via fpc-pascal wrote:

Am 01.04.2017 05:42 schrieb "Ryan Joseph"
:


As far as the compiler is concerned what’s the difference between

clearing an array using a for-loop vs. FillChar? It seems like
iterating the array would be slower but what does FillChar do exactly
and is it faster? The primary concern here is that the memory
originally allocated (using SetLength right?) remains in the same
location.


var
  list: array of integer;

SetLength(list, 10);

for i := 0 to high(list) do
  list[i] := 0;

FillChar(list[0], Length(list) * sizeof(integer), 0);


It totally depends on the type. In case of primitive types like
integers there is indeed only the performance difference (though if
you know that the element size is four FillDWord could be even faster,
depending on the implementation in the RTL).
If you have managed types however or class instances the result of the
FillChar (or equivalent) would be memory leaks.

Plase also note that after a SetLength the new elements are already 0
(or equivalent).


Since fillchar is not a general way to clear an item it almost seems 
like pascal needs a way to clear things generically, such as Array.clear
But I guess this would be reinventing OOP, or Ruby where everything is 
an object and can have .Something tacked on to it.


I've never been a fan of things like fillchar as it seems like a 
reinvention of something more like Clear() and mistakes can easily be 
made when you use complex function like this:


fillchar(something, something * something, 0)

instead of a more simpler:

Array.Clear
or
Clear(array)

You could, put the array inside a class/object I suppose and make a 
clear method... or use an old pascal object with a clear method on the 
stack so no free/create is required.


All this seems to be maybe either reinventing GoLang or Ruby though, or 
possibly python.


To me, fillchar doesn't actually say anything sensible about what the 
code is doing... "Clear" or "Default" (I think delphi is implementing 
something like this in later versions) says exactly what the code is 
doing. FillChar is some obscure old 1970's/80's pascal function that 
really doesn't tell what you are doing, unless you are an old pascal 
programmer who knows it from long ago. I'm skeptical of these old ways 
of doing things.. Imagine a newcomer to pascal who did not program in 
the 1980's... He sees "fillchar" in the program... Does that code even 
tell this learner of pascal what is happening? What's a "char" have to 
do with an array that may not have anything to do with chars?


Just my opinion and, possibly rant. Apologies.  Rant not directed at 
anyone in particular, just I'm skeptical of this whole fillchar thing... 
Even though I use it myself. Usually in a clear() wrapper of my own, to 
make the code more clear... Or a setdefault() or default() like function

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jonas Maebe]

On 05/04/17 04:49, Ryan Joseph wrote:

Yeah after all this talk, I’m going to use array[0..0, 0..0, 0..0] and allocate 
the memory myself to avoid overhead and confusion. Thanks for explaining 
everything to me but this time going low level makes the most sense.


Then you lose the ability to turn on range checking in your program, 
since any access with an index greater than 0 will cause an error.



Jonas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 4, 2017, at 11:50 PM, Jürgen Hestermann  
> wrote:
> 
> I am trying to show the memory allocation for the 10x10 array as a "graphic":
> 
> arr --> arr[0],arr[1],arr[2],arr[3],arr[4],arr[5],arr[6],arr[7],arr[8],arr[9]
>  |  |  |  |  |  |  |  | |  |
>  V  V  V  V  V  V  V  V V  V
>arr[0,0],   .  .  .  .  .  . .arr[9,0],
>arr[0,1],   .  .  .  .  .  . .arr[9,1],
>arr[0,2],   .  .  .  .  .  . .arr[9,2],
> arr[0,3], arr[9,3],
> arr[0,4], arr[9,4],
> arr[0,5], arr[9,5],
> ......
> 
> arr is a single pointer (that points to arr[0]).
> arr[0] to arr[9] are (10) pointers located in a continuous memory block each 
> pointing to
> arr[0,0],
> arr[1,0],
> arr[3,0], and so on...
> arr[0,0] is a single integer.
> arr[0,0] to arr[0,9] are (10) integers located in a continuous memory block.
> The same applies for
> arr[1,0] to arr[1,9],
> arr[2,0] to arr[2,9],
> arr[3,0] to arr[3,9], and so on.

Thanks that helps. Indeed this is not what I need and I’m not even taking 
advantage of the resizable elements so I better not use dynamic arrays for my 
matrix.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 4, 2017, at 10:10 PM, Sven Barth via fpc-pascal 
>  wrote:
> 
> While your statement regarding allocation might be true you must not
> forget that a dynamic array consists of a meta data block (length,
> reference count) that is located directly in front of the data block. So
> even if the memory blocks would be allocated consecutively then there'd
> still be the meta data blocks inbetween.
> 
> If you already know that your dynamic arrays only have a specific size
> (for matrices used in games that should usually be the case) then you're
> better off with using a static array:
> 
> === code begin ===
> 
> type
>  TMatrix = array[0..2, 0..2, 0..2] of LongInt;
> 
> === code end ===
> 
> There you can use FillChar() as much as you want as that is indeed a
> single memory block containing 9 LongInt elements.

Yeah after all this talk, I’m going to use array[0..0, 0..0, 0..0] and allocate 
the memory myself to avoid overhead and confusion. Thanks for explaining 
everything to me but this time going low level makes the most sense.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-04 um 16:54 schrieb Ryan Joseph:
>> var arr : array of array of Integer;
>> begin
>> SetLength(arr, 10, 10);
> “then the first array stores a pointer to each sub array.”
> Could you illustrate this is code?
> I don’t think I’m understanding this exactly like it’s represented in memory.
> There’s only one “sub array” in this 2x2 array so how does that look in 
memory?

Why 2x2?
SetLength(arr,10,10) creates a 10x10 array.

I am trying to show the memory allocation for the 10x10 array as a "graphic":

arr --> arr[0],arr[1],arr[2],arr[3],arr[4],arr[5],arr[6],arr[7],arr[8],arr[9]
  |  |  |  |  |  |  |  | |  |
  V  V  V  V  V  V  V  V V  V
arr[0,0],   .  .  .  .  .  . .arr[9,0],
arr[0,1],   .  .  .  .  .  . .arr[9,1],
arr[0,2],   .  .  .  .  .  . .arr[9,2],
arr[0,3], arr[9,3],
arr[0,4], arr[9,4],
arr[0,5], arr[9,5],
......

arr is a single pointer (that points to arr[0]).
arr[0] to arr[9] are (10) pointers located in a continuous memory block each 
pointing to
arr[0,0],
arr[1,0],
arr[3,0], and so on...
arr[0,0] is a single integer.
arr[0,0] to arr[0,9] are (10) integers located in a continuous memory block.
The same applies for
arr[1,0] to arr[1,9],
arr[2,0] to arr[2,9],
arr[3,0] to arr[3,9], and so on.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

On 04.04.2017 16:27, Ryan Joseph wrote:
>>> Does SetLength on a single level dynamic array not even allocate a 
>>> continuous block of memory?
>>
>> Yes, it does (as explained in all the other mails).
>> A (dynamic) array of integer will be allocated as a single block by 
>> SetLength.
>> So if you only have one level of a dynamic array as in
>>
>> var MyArray : array of integer;
>>
>> then SetLength(MyArray,1000) will allocate a single block of 1000 integers.
>> But with
>>
>> var MyArray : array of array of integer;
>>
>> SetLength(MyArray,1000);
>>
>> will allocate a single block of 1000 pointers (to an array of integer each).
>> Then, SetLength(MyArray[0],1000) will allocate one (!) block of 1000 
>> integers.
>> SetLength(MyArray[1],1000) will allocate another block of 1000 integers and 
>> so on….
> 
> I was allocating using SetLength(MyArray, 3, 3, 3) for a 3x3x3 matrix for 
> example. Maybe it depends on the memory manager and the state of heap but if 
> you called early in the programs execution it should be allocate all that 
> memory in one block I would think. 

Addendum: If you look at fpc_dynarray_setlength() again then you'll see
at line 289 that it's calling itself recursively for nested arrays.

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

On 04.04.2017 16:27, Ryan Joseph wrote:
>>> Does SetLength on a single level dynamic array not even allocate a 
>>> continuous block of memory?
>>
>> Yes, it does (as explained in all the other mails).
>> A (dynamic) array of integer will be allocated as a single block by 
>> SetLength.
>> So if you only have one level of a dynamic array as in
>>
>> var MyArray : array of integer;
>>
>> then SetLength(MyArray,1000) will allocate a single block of 1000 integers.
>> But with
>>
>> var MyArray : array of array of integer;
>>
>> SetLength(MyArray,1000);
>>
>> will allocate a single block of 1000 pointers (to an array of integer each).
>> Then, SetLength(MyArray[0],1000) will allocate one (!) block of 1000 
>> integers.
>> SetLength(MyArray[1],1000) will allocate another block of 1000 integers and 
>> so on….
> 
> I was allocating using SetLength(MyArray, 3, 3, 3) for a 3x3x3 matrix for 
> example. Maybe it depends on the memory manager and the state of heap but if 
> you called early in the programs execution it should be allocate all that 
> memory in one block I would think. 

While your statement regarding allocation might be true you must not
forget that a dynamic array consists of a meta data block (length,
reference count) that is located directly in front of the data block. So
even if the memory blocks would be allocated consecutively then there'd
still be the meta data blocks inbetween.

If you already know that your dynamic arrays only have a specific size
(for matrices used in games that should usually be the case) then you're
better off with using a static array:

=== code begin ===

type
  TMatrix = array[0..2, 0..2, 0..2] of LongInt;

=== code end ===

There you can use FillChar() as much as you want as that is indeed a
single memory block containing 9 LongInt elements.

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Mattias Gaertner]

On Tue, 4 Apr 2017 16:46:02 +0200
Sven Barth via fpc-pascal  wrote:

>[...]
> SetLength() allocates a single block of memory,

To avoid misunderstanding: SetLength(a,dim1,dim2) allocates one block
for the dim1 array and then for each element another block.

Mattias
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 4, 2017, at 10:07 PM, Jürgen Hestermann  
> wrote:
> 
> Am 2017-04-04 um 15:40 schrieb Ryan Joseph:
> > I’m glad I asked because of arrays of pointers is bad news for performance.
> 
> I don't think that you will notice a performance issue with dynamic arrays 
> (though it highly depends on the sizes and levels you use...)

As I understand it accessing continuous blocks of memory in tight loops 
increases the chance that it will be in cache L1/L2 cache. It’s probably 
trivial for my use but I’m making some game code which is highly performance 
sensitive and I’m trying to learn some best practices when writing data types. 
I watched a couple presentations from a compiler engineer from Google (works on 
clang and LLVM) that explained how this worked and it’s something worth 
learning and paying attention to imo.

> 
> > Does SetLength on a single level dynamic array not even allocate a 
> > continuous block of memory?
> 
> Yes, it does (as explained in all the other mails).
> A (dynamic) array of integer will be allocated as a single block by SetLength.
> So if you only have one level of a dynamic array as in
> 
> var MyArray : array of integer;
> 
> then SetLength(MyArray,1000) will allocate a single block of 1000 integers.
> But with
> 
> var MyArray : array of array of integer;
> 
> SetLength(MyArray,1000);
> 
> will allocate a single block of 1000 pointers (to an array of integer each).
> Then, SetLength(MyArray[0],1000) will allocate one (!) block of 1000 integers.
> SetLength(MyArray[1],1000) will allocate another block of 1000 integers and 
> so on….

I was allocating using SetLength(MyArray, 3, 3, 3) for a 3x3x3 matrix for 
example. Maybe it depends on the memory manager and the state of heap but if 
you called early in the programs execution it should be allocate all that 
memory in one block I would think. 

Another bottle neck of not using arrays of pointers is you need to perform some 
math if you want to know which element 2,2,2 is. I’m probably getting in over 
my head now. :)


Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

On 04.04.2017 16:54, Ryan Joseph wrote:
> 
>> On Apr 4, 2017, at 9:46 PM, Sven Barth via fpc-pascal 
>>  wrote:
>>
>> SetLength() allocates a single block of memory, cause array access is
>> ordinary pointer arithmetic. However if you have an array of array then
>> the first array stores a pointer to each sub array.
>>
>> E.g. the following would be valid, too:
>>
>> === code begin ===
>>
>> var
>>  arr: array of array of Integer;
>> begin
>>  SetLength(arr, 10, 10);
> 
> “then the first array stores a pointer to each sub array.”
> 
> Could you illustrate this is code? I don’t think I’m understanding this 
> exactly like it’s represented in memory. There’s only one “sub array” in this 
> 2x2 array so how does that look in memory?

Let's look at a smaller array, let's say 3 x 4, then it would look like
this:

arr = @arr[0], @arr[1], @arr[2]

arr[0] = 0, 0, 0, 0
arr[1] = 0, 0, 0, 0
arr[2] = 0, 0, 0, 0

Essentially you'd have four arrays, namely the outer array arr which
basically contains pointers to the other arrays, and the inner arrays
which are each three separate four element arrays. So all four of these
arrays could reside at completely different locations of the heap.

Is this clearer?

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

On 04.04.2017 15:40, Ryan Joseph wrote:
> 
>> On Apr 4, 2017, at 7:17 PM, Sven Barth via fpc-pascal 
>>  wrote:
>>
>> If you want continuous memory areas you need to use static arrays or develop 
>> your own dynamic data structure that uses array properties.
>>
>>
> 
> I’m glad I asked because of arrays of pointers is bad news for performance. 
> Does SetLength on a single level dynamic array not even allocate a continuous 
> block of memory? I could use GetMem and array[0..0] but it seems like dynamic 
> arrays should do basically that anyways if they’re not nested.

You could use something along these lines instead:

=== code begin ===

program tarrtest;

{$mode objfpc}
{$modeswitch advancedrecords}

type
  generic TTwoDimArray = record
  private
fData: array of T;
{ Note: Length1 and Length2 are not initialized by default, but you
could use trunk's management operators for that }
fLength1,
fLength2: LongInt;
function GetElement(Index1, Index2: LongInt): T; inline;
procedure SetElement(Index1, Index2: LongInt; aValue: T); inline;
  public
{ using SetLength() would lead to us needing to use
  "System.SetLength()" for the array which in turn would complain
  about usage of the static symtable; that's a problem that yet
  needs to be solved inside generics, for non-generics that would
  work however }
procedure AdjustLength(aLength1, aLength2: LongInt);
property Length1: LongInt read fLength1;
property Length2: LongInt read fLength2;
property Element[Index1, Index2: LongInt]: T read GetElement write
SetElement; default;
  end;

{ TTwoDimArray }

function TTwoDimArray.GetElement(Index1, Index2: LongInt): T;
begin
  { ToDo: Length check }
  Result := fData[Index1 * fLength1 + Index2];
end;

procedure TTwoDimArray.SetElement(Index1, Index2: LongInt; aValue: T);
begin
  { ToDo: Length check }
  fData[Index1 * fLength1 + Index2] := aValue;
end;

procedure TTwoDimArray.AdjustLength(aLength1, aLength2: LongInt);
begin
  SetLength(fData, aLength1 * aLength2);
  fLength1 := aLength1;
  fLength2 := aLength2;
end;

var
  arr: specialize TTwoDimArray;
  i: LongInt;
begin
  arr.AdjustLength(10, 5);
  i := arr[3, 2];
  arr[7, 1] := i;
end.

=== code end ===

Regards,
Sven

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 4, 2017, at 9:46 PM, Sven Barth via fpc-pascal 
>  wrote:
> 
> SetLength() allocates a single block of memory, cause array access is
> ordinary pointer arithmetic. However if you have an array of array then
> the first array stores a pointer to each sub array.
> 
> E.g. the following would be valid, too:
> 
> === code begin ===
> 
> var
>  arr: array of array of Integer;
> begin
>  SetLength(arr, 10, 10);

“then the first array stores a pointer to each sub array.”

Could you illustrate this is code? I don’t think I’m understanding this exactly 
like it’s represented in memory. There’s only one “sub array” in this 2x2 array 
so how does that look in memory?

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

On 04.04.2017 15:40, Ryan Joseph wrote:
> 
>> On Apr 4, 2017, at 7:17 PM, Sven Barth via fpc-pascal 
>>  wrote:
>>
>> If you want continuous memory areas you need to use static arrays or develop 
>> your own dynamic data structure that uses array properties.
>>
>>
> 
> I’m glad I asked because of arrays of pointers is bad news for performance. 
> Does SetLength on a single level dynamic array not even allocate a continuous 
> block of memory? I could use GetMem and array[0..0] but it seems like dynamic 
> arrays should do basically that anyways if they’re not nested.

SetLength() allocates a single block of memory, cause array access is
ordinary pointer arithmetic. However if you have an array of array then
the first array stores a pointer to each sub array.

E.g. the following would be valid, too:

=== code begin ===

var
  arr: array of array of Integer;
begin
  SetLength(arr, 10, 10);
  SetLength(arr[3], 5);
  arr[6] := Nil;
  SetLength(arr[8], 15);
end.

=== code end ===

Regards,
Sven

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-04 um 15:40 schrieb Ryan Joseph:
> I’m glad I asked because of arrays of pointers is bad news for performance.

I don't think that you will notice a performance issue with dynamic arrays 
(though it highly depends on the sizes and levels you use...)

> Does SetLength on a single level dynamic array not even allocate a continuous 
block of memory?

Yes, it does (as explained in all the other mails).
A (dynamic) array of integer will be allocated as a single block by SetLength.
So if you only have one level of a dynamic array as in

var MyArray : array of integer;

then SetLength(MyArray,1000) will allocate a single block of 1000 integers.
But with

var MyArray : array of array of integer;

SetLength(MyArray,1000);

will allocate a single block of 1000 pointers (to an array of integer each).
Then, SetLength(MyArray[0],1000) will allocate one (!) block of 1000 integers.
SetLength(MyArray[1],1000) will allocate another block of 1000 integers and so 
on

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 4, 2017, at 7:17 PM, Sven Barth via fpc-pascal 
>  wrote:
> 
> If you want continuous memory areas you need to use static arrays or develop 
> your own dynamic data structure that uses array properties.
> 
> 

I’m glad I asked because of arrays of pointers is bad news for performance. 
Does SetLength on a single level dynamic array not even allocate a continuous 
block of memory? I could use GetMem and array[0..0] but it seems like dynamic 
arrays should do basically that anyways if they’re not nested.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-04 um 15:18 schrieb Jürgen Hestermann:
> var MyArray : array of array of integer;
>
> you can do:
>
> SetLength(MyArray,3);
> SetLength(MyArray[0],2);
> SetLength(MyArray[1],3);
> SetLength(MyArray[2],4);
>
> So MyArray[0] points to an array of 2 integers,
> MyArray[1] points to an array of 3 integers and
> MyArray[2] points to an array of 4 integers.

The syntax of dynamic and static arrays are the same
although they should differ (because the involved pointers).
Therefore it is confusing and misleading.

For dynamic arrays
MyArray[0] should be MyArray^[0] and
MyArray[0,1] should be MyArray^[0]^[1]
and so on...

This would make it possible to distinguish between the pointer
(MyArray) and the data it points to (MyArray^).
Also MyArray^[0]^[1] would be a pointer while
MyArray^[0]^[1]^ would be the (static) array of integers.

But as you seldom access the pointers themself (they
are managed by the compiler) and because this automatic
derefencing of pointers for managed types has been defined
decades ago we have to live with this now...

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-04 um 13:55 schrieb Ryan Joseph:
> Doesn’t iterating the array default the purpose of FillChar?
> The goal was the most efficient way clear the array with zero’s.
> Even if the array if nested 3 levels deep (anArray[x][y][z])
> it should (I hope) be a contiguous block of memory that was allocated 
(correct me if I wrong please).

No. If you have a dynamic array of multiple levels it
does not generate a continuous block for all levels.
Only the last level (if it is an array of integer or other non-managed type) is 
just that.
A (dynamic) array of array if an array of pointers.
Only for static arrays all levels are a continuous block.

The first level of a multiple level (dynamic) array is an array of pointers 
(array of array).
So you have a continuous block of pointers on the first level.

Each (!) second level array also is an array of pointers (array of array).

Only the last level is an array of integer (or whatever).

Therefore, for a 3x3x3 array you have 3 pointers on the first level.
Each of these pointers (MyArray[x]) again points to an array of 3 pointers.
And on the last level (MyArray[x,y]) each pointer points to an array of integer.
So you have 3x3=9 continuous blocks of 3 integers which can be located anywhere 
in memory.

BTW:
Each level of nested dynamic arrays can have individuell size.
On a 2-dimensional array

var MyArray : array of array of integer;

you can do:

SetLength(MyArray,3);
SetLength(MyArray[0],2);
SetLength(MyArray[1],3);
SetLength(MyArray[2],4);

So MyArray[0] points to an array of 2 integers,
MyArray[1] points to an array of 3 integers and
MyArray[2] points to an array of 4 integers.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 04.04.2017 13:55 schrieb "Ryan Joseph" :
>
>
> > On Apr 4, 2017, at 4:58 PM, Howard Page-Clark via fpc-pascal <
fpc-pascal@lists.freepascal.org> wrote:
> >
> > You can always use FillChar and its kin on specific 'nested' arrays
like this
> >
> > type
> >  TIntArray = array of Integer;
> >  TIntIntArray = array of TIntArray;
> >  TIntIntIntArray = array of TIntIntArray;
> >
> >  procedure FillArray(const anArray: TIntIntIntArray; aValue: DWord);
> >  var
> >x, y: integer;
> >  begin
> >for x:=0 to High(anArray) do
> >  for y:=0 to High(anArray[x]) do
> >FillDWord(anArray[x][y][0], Length(anArray[x][y]), aValue);
> >  end;
>
> Doesn’t iterating the array default the purpose of FillChar? The goal was
the most efficient way clear the array with zero’s. Even if the array if
nested 3 levels deep (anArray[x][y][z]) it should (I hope) be a contiguous
block of memory that was allocated (correct me if I wrong please).
>

As I wrote that isn't the case. Each dynamic array is allocated
independently (and thus you could also resize each subelement
independently).

If you want continuous memory areas you need to use static arrays or
develop your own dynamic data structure that uses array properties.

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 4, 2017, at 4:58 PM, Howard Page-Clark via fpc-pascal 
>  wrote:
> 
> You can always use FillChar and its kin on specific 'nested' arrays like this
> 
> type
>  TIntArray = array of Integer;
>  TIntIntArray = array of TIntArray;
>  TIntIntIntArray = array of TIntIntArray;
> 
>  procedure FillArray(const anArray: TIntIntIntArray; aValue: DWord);
>  var
>x, y: integer;
>  begin
>for x:=0 to High(anArray) do
>  for y:=0 to High(anArray[x]) do
>FillDWord(anArray[x][y][0], Length(anArray[x][y]), aValue);
>  end;

Doesn’t iterating the array default the purpose of FillChar? The goal was the 
most efficient way clear the array with zero’s. Even if the array if nested 3 
levels deep (anArray[x][y][z]) it should (I hope) be a contiguous block of 
memory that was allocated (correct me if I wrong please). 

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 04.04.2017 12:52 schrieb "Mark Morgan Lloyd" <
markmll.fpc-pas...@telemetry.co.uk>:
>
> On 02/04/17 10:00, Jonas Maebe wrote:
>
>> Allocating new memory via setlength also clears the memory (+ the
>> overhead of allocating the memory).
>
>
> Jonas, is it still the case that if SetLength() results in existing data
being moved that the original- which might be e.g. an unencrypted password-
isn't cleared?

The reallocation is delegated to the memory manager, thus SetLength() can
not know what is really done with the data area.

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-04 um 12:51 schrieb Mark Morgan Lloyd:
> On 02/04/17 10:00, Jonas Maebe wrote:
>> Allocating new memory via setlength also clears the memory (+ the
>> overhead of allocating the memory).
> Jonas, is it still the case that if SetLength() results in existing data 
being moved that the original- which might be e.g. an unencrypted password- isn't 
cleared?

Only formerly not existing elements are overwritten via fillchar.
Already existing data is not changed of course (otherwise you would
loose them).
If you have a (dynamic) array of 5 elements and you extend it
to 7 elements via Setlength(MyArray,7) then only the last 2 (new) elements
are cleared (because otherwise they would contain garbage).

This is (and must be) independend from moving data or just extending the 
existing array.
If the existing elements are moved (because the allocated memory cannot be 
extended)
then they are not cleared (but moved).
Only formerly not existing elements are filled with zeros and
they are never moved because they did not exist before the SetLength command.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Mark Morgan Lloyd]

On 02/04/17 10:00, Jonas Maebe wrote:


Allocating new memory via setlength also clears the memory (+ the
overhead of allocating the memory).


Jonas, is it still the case that if SetLength() results in existing data 
being moved that the original- which might be e.g. an unencrypted 
password- isn't cleared?


--
Mark Morgan Lloyd
markMLl .AT. telemetry.co .DOT. uk

[Opinions above are the author's, not those of his employers or colleagues]
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Howard Page-Clark via fpc-pascal]

On 04/04/17 05:25, Ryan Joseph wrote:

Is it possible use FillChar on a multidimensional arrays?

arr: array of array of array of integer.
SetLength(arr, 3, 3, 3);
FillChar(arr[0], (3*3*3)*sizeof(integer), false);

I’m just getting crashes.


You can always use FillChar and its kin on specific 'nested' arrays like 
this


type
  TIntArray = array of Integer;
  TIntIntArray = array of TIntArray;
  TIntIntIntArray = array of TIntIntArray;

  procedure FillArray(const anArray: TIntIntIntArray; aValue: DWord);
  var
x, y: integer;
  begin
for x:=0 to High(anArray) do
  for y:=0 to High(anArray[x]) do
FillDWord(anArray[x][y][0], Length(anArray[x][y]), aValue);
  end;
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 04.04.2017 06:55 schrieb "Ryan Joseph" :
>
>
> > On Apr 2, 2017, at 11:02 PM, Sven Barth via fpc-pascal <
fpc-pascal@lists.freepascal.org> wrote:
> >
> > can be easily seen by looking at the implementation of SetLength() in
> > $fpcdir/rtl/inc/dynarr.inc, fpc_dynarray_setlength().
> > Especially since line 220 (at least in the current revision) contains a
> > call to FillChar(), the whole ordeal can only be *more* complex than
> > FillChar().
>
> That’s good to know about SetLength.
>
> Is it possible use FillChar on a multidimensional arrays?
>
> arr: array of array of array of integer.
> SetLength(arr, 3, 3, 3);
> FillChar(arr[0], (3*3*3)*sizeof(integer), false);
>
> I’m just getting crashes.

That does only work with static arrays as dynamic arrays are in reality
managed pointers, thus you override the pointers themselves with your
FillChar. Not to mention that you're overwriting the memory behind the
outermost array as that only has a size of Length(arr) * SizeOf(Pointer).

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 2, 2017, at 11:02 PM, Sven Barth via fpc-pascal 
>  wrote:
> 
> can be easily seen by looking at the implementation of SetLength() in
> $fpcdir/rtl/inc/dynarr.inc, fpc_dynarray_setlength().
> Especially since line 220 (at least in the current revision) contains a
> call to FillChar(), the whole ordeal can only be *more* complex than
> FillChar().

That’s good to know about SetLength.

Is it possible use FillChar on a multidimensional arrays?

arr: array of array of array of integer.
SetLength(arr, 3, 3, 3);
FillChar(arr[0], (3*3*3)*sizeof(integer), false);

I’m just getting crashes.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

On 02.04.2017 11:22, Ryan Joseph wrote:
> 
>> On Apr 1, 2017, at 9:25 PM, Jürgen Hestermann  
>> wrote:
>>
>> If you just need to reuse the same array and only need to zero its elements
>> then of course fillchar would be the fastest approach (it saves the memory
>> reallocation step).
> 
> Why is this “of course”? What’s the implementation of FillChar exactly? For 
> all I know it has to iterate over a range of bytes and this could be slower 
> than allocation of memory. Maybe some of the compiler people know these 
> details.

The question is not about the implementation of FillChar. It's about
reallocating a dynamic array as to reallocate it you'd need to do

=== code begin ===

SetLength(arr, 0);
SetLength(arr, newlen);

=== code end ===

And that this is more complex than

=== code begin ===

FillChar(arr, SizeOf(arr[0]) * Length(arr), 0);

=== code end ===

can be easily seen by looking at the implementation of SetLength() in
$fpcdir/rtl/inc/dynarr.inc, fpc_dynarray_setlength().
Especially since line 220 (at least in the current revision) contains a
call to FillChar(), the whole ordeal can only be *more* complex than
FillChar().

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jonas Maebe]

On 02/04/17 11:22, Ryan Joseph wrote:

On Apr 1, 2017, at 9:25 PM, Jürgen Hestermann  wrote:

If you just need to reuse the same array and only need to zero its elements
then of course fillchar would be the fastest approach (it saves the memory
reallocation step).

Why is this “of course”? What’s the implementation of FillChar exactly? For all 
I know it has to iterate over a range of bytes and this could be slower than 
allocation of memory. Maybe some of the compiler people know these details.


Allocating new memory via setlength also clears the memory (+ the 
overhead of allocating the memory).



Jonas
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 1, 2017, at 9:25 PM, Jürgen Hestermann  
> wrote:
> 
> If you just need to reuse the same array and only need to zero its elements
> then of course fillchar would be the fastest approach (it saves the memory
> reallocation step).

Why is this “of course”? What’s the implementation of FillChar exactly? For all 
I know it has to iterate over a range of bytes and this could be slower than 
allocation of memory. Maybe some of the compiler people know these details.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[wkitty42]

On 04/01/2017 02:33 PM, Jürgen Hestermann wrote:

Am 2017-04-01 um 19:42 schrieb wkitt...@windstream.net:

consider this: when using a string var and writing to a binary file...
you reuse the string var for each value written...
if you don't clear the string var between fills then the binary will
contain "garbage" in the unused positions of the string var...
you can see this "garbage" when you view the binary file in hex mode...
the "garbage" being older longer string data than the current string data in

the var...

this var may be a global throwaway string var...


This can only happen for older string types where the actual length
can differ from the reserved memory (maximum string length).
With managed strings, this should never be possible as it would write
only the bytes of the current length.


some situations call for fixed length records... this same "garbage" effect will 
be seen in them...



I also don't see how this applies to an array of integers.


it may not... the subject doesn't differentiate... it only says "array" so i 
provided an example to your query about why this clearing may be desired... i 
just happened to use an array of char that we loving know as string[255]...



--
 NOTE: No off-list assistance is given without prior approval.
   *Please keep mailing list traffic on the list* unless
   private contact is specifically requested and granted.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-01 um 19:42 schrieb wkitt...@windstream.net:
> consider this: when using a string var and writing to a binary file...
> you reuse the string var for each value written...
> if you don't clear the string var between fills then the binary will
> contain "garbage" in the unused positions of the string var...
> you can see this "garbage" when you view the binary file in hex mode...
> the "garbage" being older longer string data than the current string 
data in the var...

> this var may be a global throwaway string var...

This can only happen for older string types where the actual length
can differ from the reserved memory (maximum string length).
With managed strings, this should never be possible as it would write
only the bytes of the current length.

I also don't see how this applies to an array of integers.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[wkitty42]

On 04/01/2017 07:31 AM, Jürgen Hestermann wrote:

I am wondering what the purpose of filling all
array elements with zero's could be.
If I want to discard all elements I would simply delete
the whole array (setlength(MyArray,0) ).


consider this: when using a string var and writing to a binary file... you reuse 
the string var for each value written... if you don't clear the string var 
between fills then the binary will contain "garbage" in the unused positions of 
the string var... you can see this "garbage" when you view the binary file in 
hex mode... the "garbage" being older longer string data than the current string 
data in the var... this var may be a global throwaway string var...


we've seen the above since back in the TP3 days, at least... in some cases, 
we've been able to extract enough "garbage" to break security on a system... 
especially when there was enough left to suss out the leading part...


--
 NOTE: No off-list assistance is given without prior approval.
   *Please keep mailing list traffic on the list* unless
   private contact is specifically requested and granted.
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

Am 2017-04-01 um 15:09 schrieb Ryan Joseph:
> Because the array is being iterated and I need to know which values 
are set.
> Basically I have a dynamic array I grow to a certain size and this 
process happens in a loop.
> The options are to allocate/free the array every cycle or clear 
memory and allocate the array once.


If you just need to reuse the same array and only need to zero its elements
then of course fillchar would be the fastest approach (it saves the memory
reallocation step).

But this works only for standard array element types (like integer etc.).
Managed types have to be reset element by element.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 1, 2017, at 6:31 PM, Jürgen Hestermann  
> wrote:
> 
> I am wondering what the purpose of filling all
> array elements with zero's could be.
> If I want to discard all elements I would simply delete
> the whole array (setlength(MyArray,0) ).

Because the array is being iterated and I need to know which values are set. 
Basically I have a dynamic array I grow to a certain size and this process 
happens in a loop. The options are to allocate/free the array every cycle or 
clear memory and allocate the array once.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 01.04.2017 13:31 schrieb "Jürgen Hestermann" :
>
> I am wondering what the purpose of filling all
> array elements with zero's could be.
> If I want to discard all elements I would simply delete
> the whole array (setlength(MyArray,0) ).
>
> But when should it be useful to keep all elements and
> just overwrite them all with zero's (which is also very time consuming)?
> If the exisiting values are no longer valid then why not
> simply delete these elements?
> It's the fastest way to get rid of them.
> If elements were zero before overwriting them with zero's
> I cannot even distinguish between original and overwritten elements.
>

If you wouldn't change the size of the array then merely doing FillChar (if
not working with managed types) is more performant then freeing the array
and allocating it again (which will also zero out the elements).
Normally the performance difference isn't *that* important, but sometimes
that difference can make or break something.

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Jürgen Hestermann]

I am wondering what the purpose of filling all
array elements with zero's could be.
If I want to discard all elements I would simply delete
the whole array (setlength(MyArray,0) ).

But when should it be useful to keep all elements and
just overwrite them all with zero's (which is also very time consuming)?
If the exisiting values are no longer valid then why not
simply delete these elements?
It's the fastest way to get rid of them.
If elements were zero before overwriting them with zero's
I cannot even distinguish between original and overwritten elements.

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 01.04.2017 10:35 schrieb "Ryan Joseph" :
>
>
> > On Apr 1, 2017, at 2:50 PM, Ryan Joseph 
wrote:
> >
> > Yeah, I was concerned with just compiler types or weakly retained
classes where I’m just keeping the reference.
>
> Another question. Is it more efficient/faster to reallocate a new array
of the same size or call FillChar on the existing array?

I think that FillChar should be faster as reallocating would need setting
the array to Nil and recreating it (SetLength with the same length won't
touch the existing elements) thus loosing any gain of reallocate and then
there'd still be the internal FillChar of the array.

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 1, 2017, at 2:50 PM, Ryan Joseph  wrote:
> 
> Yeah, I was concerned with just compiler types or weakly retained classes 
> where I’m just keeping the reference.

Another question. Is it more efficient/faster to reallocate a new array of the 
same size or call FillChar on the existing array?

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Ryan Joseph]

> On Apr 1, 2017, at 2:39 PM, Sven Barth via fpc-pascal 
>  wrote:
> 
> It totally depends on the type. In case of primitive types like integers 
> there is indeed only the performance difference (though if you know that the 
> element size is four FillDWord could be even faster, depending on the 
> implementation in the RTL).
> If you have managed types however or class instances the result of the 
> FillChar (or equivalent) would be memory leaks.
> 
> 

Yeah, I was concerned with just compiler types or weakly retained classes where 
I’m just keeping the reference.

Thanks.

Regards,
Ryan Joseph

___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Re: [fpc-pascal] Array clearing

[Sven Barth via fpc-pascal]

Am 01.04.2017 05:42 schrieb "Ryan Joseph" :
>
> As far as the compiler is concerned what’s the difference between
clearing an array using a for-loop vs. FillChar? It seems like iterating
the array would be slower but what does FillChar do exactly and is it
faster? The primary concern here is that the memory originally allocated
(using SetLength right?) remains in the same location.
>
> var
>   list: array of integer;
>
> SetLength(list, 10);
>
> for i := 0 to high(list) do
>   list[i] := 0;
>
> FillChar(list[0], Length(list) * sizeof(integer), 0);

It totally depends on the type. In case of primitive types like integers
there is indeed only the performance difference (though if you know that
the element size is four FillDWord could be even faster, depending on the
implementation in the RTL).
If you have managed types however or class instances the result of the
FillChar (or equivalent) would be memory leaks.

Plase also note that after a SetLength the new elements are already 0 (or
equivalent).

Regards,
Sven
___
fpc-pascal maillist  -  fpc-pascal@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal