Re: Reading and writing large arrays to disk

[Kirk Brooks via 4D_Tech]

Miyako,
Nice explanation.

More or less - where do you think the inflection point is where simple text
concatenation becomes less efficient than adding the text to a BLOB?

On Sun, Jan 5, 2020 at 8:20 PM Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> concatenation of text is intuitive, but inefficient.
>
> every time you add text to another, a new buffer is created and the whole
> text is copied there.
> as the text gets larger, the allocation of memory and copying of data will
> become slower.
>
> variable to blob is fast as it simply packs the variable.
> I imagine you mainly use basic ASCII, compression would be really
> effective.
> I would not spend much time looking for the best compression algorithm or
> level.
> 200,000 elements / 2MB is too small to notice any difference.
>
> 2020/01/06 13:09、Mitchell Shiller via 4D_Tech <4d_tech@lists.4d.com
> >のメール:
>
> Obviously using the reverse commands to read from disk.
> Option one seems by far the slowest.
>
> Anyone know if 2 or 3 is faster?
> Does the COMPRESS BLOB help? The resultant file is about 2 MB in size
> without compression.
> If yes to COMPRESS BLOB , Any différence from native 4D vs. GZIP
> compression?
>
>
>
> **
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **



-- 
Kirk Brooks
San Francisco, CA
===

What can be said, can be said clearly,
and what you can’t say, you should shut up about

*Wittgenstein and the Computer *
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Reading and writing large arrays to disk

[Keisuke Miyako via 4D_Tech]

concatenation of text is intuitive, but inefficient.

every time you add text to another, a new buffer is created and the whole text 
is copied there.
as the text gets larger, the allocation of memory and copying of data will 
become slower.

variable to blob is fast as it simply packs the variable.
I imagine you mainly use basic ASCII, compression would be really effective.
I would not spend much time looking for the best compression algorithm or level.
200,000 elements / 2MB is too small to notice any difference.

2020/01/06 13:09、Mitchell Shiller via 4D_Tech 
<4d_tech@lists.4d.com>のメール:

Obviously using the reverse commands to read from disk.
Option one seems by far the slowest.

Anyone know if 2 or 3 is faster?
Does the COMPRESS BLOB help? The resultant file is about 2 MB in size without 
compression.
If yes to COMPRESS BLOB , Any différence from native 4D vs. GZIP compression?



**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Reading and writing large arrays to disk

[Mitchell Shiller via 4D_Tech]

Hi,

I have large string arrays (about 200 k elements).
I need to write and read them to disk.
Speed is the most important criteria.

Options
1) Create a text variable (loop with CR delimiter) and then TEXT TO DOCUMENT. 
2) VARIABLE TO BLOB, COMPRESS BLOB, BLOB TO DOCUMENT
3) OB SET ARRAY, JSON STRINGIFY, TEXT TO DOCUMENT 

Obviously using the reverse commands to read from disk.
Option one seems by far the slowest.

Anyone know if 2 or 3 is faster?
Does the COMPRESS BLOB help? The resultant file is about 2 MB in size without 
compression.
If yes to COMPRESS BLOB , Any différence from native 4D vs. GZIP compression?

Thanks as always.

Mitch



Sent from my iPad
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: gitignore for projects

[John DeSoi via 4D_Tech]

Hi Jim,

> On Jan 5, 2020, at 7:31 AM, Jim Crate via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
>> In this model, I'm only making structure and form changes in the 17 master 
>> branch. Methods can be changed in both branches.
> 
> Are there any problems with functionalities can’t be used in v18 project mode 
> that don’t have replacements in v17.3 (not R-series)? I’ve seen mentions of 
> users/groups changes in particular but I haven’t investigated the details yet.

Yes, there are a number of differences not only between 17.3 and 18 project 
mode, but between 18.4db and 18 project mode. For example, the picture library 
does not exist in project mode. Method comments are gone in project mode. I 
minimally use users/groups, but that is different also.

One problem I reported is the 4D structure catalog exported from the v18.4db is 
modified just by opening it in 18 project mode. Seems to have something to do 
with tips or comments associated with fields. For now I have to remember to 
discard catalog.4DCatalog changes before I commit branch 18 changes.

John DeSoi, Ph.D.


**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: gitignore for projects

[John DeSoi via 4D_Tech]

Hi Tom,

> On Jan 4, 2020, at 10:13 PM, Tom Benedict  wrote:
> 
> How large is your structure? I ask, because our structure has around a 
> million lines of code and probably 15,000 forms and when I’ve tried exporting 
> under 17r5 it takes many minutes.

Roughly 150,000 lines of source code and probably about 300 or so forms.

A lot has happened between 17r5 and 18, so it might be worth trying again. 
Also, the Export structure command gives you a lot of control over how 
exporting is done. It would be worth it for you to test various aspects 
independently and see what the bottleneck is and report back to 4D. 

At the end of this message I have included the export script I'm using. When 
the temporary 17 copy is converted to 18, this method is executed with PROCESS 
4D TAGS to export the structure for the master branch version control.


> Also, my attempt at setting up a git repository ran into similar performance 
> issues, only hours instead of minutes. I lost interest at that point. 

Is this with git directly from the command line or using a GUI tool? I have 
seen many performance and other issues using a GUI tool that go away when I 
execute the same operations from the command line.


> 
>> You can continue to make version 17 changes, export, and then merge into 
>> your 18 branch. At some point in the future, all your conversion issues will 
>> be resolved and you can promote the 18 branch as master and deploy with 4D 
>> 18.
> 
> This sounds almost like magic. When you “merge” how do you know what code 
> takes priority? Do you have to examine each method and form? 

Merging is why you use git for this task. It automatically identifies any 
conflicting changes between the branches and allows you to decide what to keep. 
I'm doing maintenance/bug fixes in 17 and new features in version 18. There are 
rarely any merge conflicts. If you modify the same methods in both branches, 
you'll spend a lot more time dealing with merge conflicts.


> BTW, did you have prior experience with git? I’m asking because my past 
> attempts to get a conceptual understanding of git and version control have 
> had limited success. I found the concepts and terminology overwhelming and 
> couldn't map the git workflow to my 4D workflow at all. I’m looking forward 
> to more posts like yours which provides ‘just enough’ git to get started.


I have been using git with 4D since 2014 and with other environments for much 
longer. With 4D it has been mostly for version management and change tracking. 
4D 18 changes things and makes development like the rest of the world; you can 
manage multiple branches and integrate work from multiple (remote) developers.

John DeSoi, Ph.D.





**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: gitignore for projects

[Jim Crate via 4D_Tech]

On Jan 4, 2020, at 6:42 PM, John DeSoi via 4D_Tech <4d_tech@lists.4d.com> wrote:
> 
> Yes. Switch back to the 17 "master" branch. Verify there are no uncommitted 
> changes in the Project folder (git status called from 4D). Move the Project 
> folder to the trash. Run the 17 to 18 script again and generate a new Project 
> folder with the current source. Commit as the new "master" branch. Merge (or 
> fast forward) the master branch into the 18 branch to keep it up to date with 
> the changes in 4D 17.

This is an interesting workflow, I’ll probably copy this to start prepping for 
v18.

> In this model, I'm only making structure and form changes in the 17 master 
> branch. Methods can be changed in both branches.

Are there any problems with functionalities can’t be used in v18 project mode 
that don’t have replacements in v17.3 (not R-series)? I’ve seen mentions of 
users/groups changes in particular but I haven’t investigated the details yet.

Jim Crate

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**