Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development

[Kirk Brooks via 4D_Tech]

Hi JPR,
i'm glad the 'Bat Signal' still works.

On Fri, Oct 11, 2019 at 1:34 AM JPR via 4D_Tech <4d_tech@lists.4d.com>
wrote:

> Do you like Fairy Tales?
>
I thought that was the demo...


> > So what you are saying is that if you are deploying 4D Server in a VM
> there is not need to try and use preemptive mode. You get no benefit from
> doing that. It’s a waste of time?
> Wrong. It will be a benefit in terms of speed, even not as large as you
> dream, because the system will optimise the use of each possible core. And
> a benefit in term of smoothness of execution, because no process will
> freeze the others.
>
OK - but assuming I'm running on a VM isn't that speed and usability
benefit coming from moving the processing into a Worker?


> > The whole benefit of preemptive mode is to allow 4D to assign a process
> to a core.
> Wrong. What I explained is that 4D does NOT assign a process to a core,
> but assigns a process to a THREAD, and the System assigns cores to Threads.

And once more I display my sketchy understanding of how the hardware works.


> > But in the case of a VM the actual, physical cores are managed by the VM.
> Wrong. The VM talks to the System, and the System manages cores. In fact,
> just imagine that a machine can run several VM at the same time.
>
Ibid.

If you have been at the 2019 Tour, remember J_OtherObjects example, where I
> show that if you use cooperative processes, with one executing an
> excruciating method, then the Forms on use are kind of frozen. Then, by
> switching to preemptive mode, the same awfully blocking method doesn't
> impact on interface while running.
>
True (in fact I just pulled it out to look at again). But we are running
that on our machines where the _threads_ can be assigned by the _system_ to
various cores and I can see that in the activity viewer. In fact running on
'bare metal' shows a lot of variation in the activity of the various cores.

I went rummaging around in the knowledgebase to see if I could find what
I'm remembering. I think it was one of the keynotes from the '16 Summit and
I couldn't find it. But I did find Laurent Ensault's session on the
subject: https://kb.4d.com/assetid=77547

I rewatched it and ran the example database. The demo spawns 4 cooperative
and 4 preemptive processes then runs an intensive method 1mil times in
each.

I made a small change to his demo - I added the start and end ticks to the
process record in addition to the duration. Why? Because I wanted to see
the total time spent on processing the 'job'. I do this by taking the end
of the last process to finish and subtracting the start of the first
process to begin. I also moved the demo db to v17.3.

As luck would have it I have an AWS instance with v17.3 installed so I am
able to run the demo locally and on AWS.
On my laptop:

Cooperative 1: 376 ticks
Cooperative 2: 376 ticks
Cooperative 3: 375 ticks
Cooperative 4: 367 ticks

Total run time:  395 ticks


Preemp 1: 118 ticks
Preemp 2: 119 ticks
Preemp 3: 117 ticks
Preemp 4: 119 ticks

Total run time:  119 ticks  //  that's what we want to see!


On AWS:

Cooperative 1: 740 ticks
Cooperative 2: 706 ticks
Cooperative 3: 793 ticks
Cooperative 4: 731 ticks

Total run time:  821 ticks


Preemp 1: 881 ticks
Preemp 2: 824 ticks
Preemp 3: 857 ticks
Preemp 4: 848 ticks

Total run time:  881 ticks

Your first thought might be, "hold on, why is the total run time almost the
same as each process' run time?" The very first line of the method sets a
variable with the tick count. Because Laurent included the IDLE command in
the demo method those methods yield the processor back to 4D on each
iteration. (The video talks about the details on this.) So Coop1 starts and
then yields back to 4D which starts Coop2 and so on. In contrast Preemptive
processes simply start and run without having to 'yield back' to 4D.

So what we see is that each them gets started pretty much at the same time
and complete pretty close together as well. This probably wouldn't be the
case if they weren't doing exactly the same work and if 4D were actually
working instead of doing a demo. Total run time, as I said, is the tick
count for the last process to finish minus the tick count of the first one
to start. Or the total amount of time spent processing by all 4 processes.

This illustrates the point I recalled from back then: when you run on a VM
the advantage of the the OS to distribute threads to the actual cores is
abstracted away. They may be distributed to virtual cores but that doesn't
impact the total processing time. In the case of AWS apparently this VM
overhead actually causes it to run slower but this is a small sample set. I
have no idea why and frankly don't have the expertise to figure it out. But
it's there.

FYI - the AWS instance is an m5ad.xlarge: 4 cores, 16gb RAM. It sorta
sucks. I'd love to hear from anyone actually running 4D server on AWS with
good performance.

-- 
Kirk Brooks
San Francisco, CA
===

What can be 

Re: Using Github to Manage Distributed Development

[Narinder Chandi via 4D_Tech]

I hope that what we're suggesting is that the plan is to (finally) dump the
binary format entirely and move to a collection of text files, whether they
be XML, YAML, JSON, etc.

Yes, the Project mode currently in Beta seems designed to achieve exactly that. 
In fact maybe you missed JPR's reply as it was in another thread:

I said:
But, I still need to fully understand exactly why 4D is not eliminating the 
interpreted structure binary .4DB file completely at this time? Maybe 4D plan 
to do so eventually? Was this question asked at the last tour? If we are given 
the Project feature then we should be all in and have no option but to embrace 
it and source code repos such as Git, since, pretty much every other 
programming language is doing it this way.

JPR responded:
AFAIK, if you use the Project mode, you don't need any binary, for everything 
is in text files (.json, .xml, .txt, .4DProject, .css, . 4DCatalog , .4dm, 
.4DForm, etc.) The .4DB is not necessary.

Regards,
 
Narinder Chandi,
ToolBox Systems Ltd.
 
I am available for new consulting opportunities…
http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html
-- 



**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Using Github to Manage Distributed Development

[Mike Kerner via 4D_Tech]

I hope that what we're suggesting is that the plan is to (finally) dump the
binary format entirely and move to a collection of text files, whether they
be XML, YAML, JSON, etc.

On Thu, Oct 10, 2019 at 6:53 PM Cannon Smith via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> I can’t think of any reason one would want to go back, either. Splitting
> the structure into files is something we’ve wanted for _so_ long and opens
> so many opportunities.
>
> --
> Cannon.Smith
> Synergy Farm Solutions Inc.
> Aetna, AB Canada
> 
> 
>
>
> > On Oct 10, 2019, at 4:38 PM, Tim Nevels via 4D_Tech <
> 4d_tech@lists.4d.com> wrote:
> >
> >> Yes, you're correct, currently the move from binary to Project format
> is one way, whether this will become bi-directional I don't know. It is
> still possible to compile back to a single binary though so the step to go
> from Project to interpreted structure binary should surely be possible??
> >
> > This was asked to JPR at the last 4D World Tour show and his response
> was “why would you want to do that?” And then he proceeded to explain that
> to implement a feature like that would require many hours of engineering
> time, and QA time and then it would have to be maintained into the future.
> >
> > So he said are no plans for a “project folder back to a single .4DB
> file” feature. Only way to have the happen is to convince 4D it is a needed
> and necessary feature.
>
> **
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **



-- 
On the first day, God created the heavens and the Earth
On the second day, God created the oceans.
On the third day, God put the animals on hold for a few hours,
   and did a little diving.
And God said, "This is good."
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development

[Narinder Chandi via 4D_Tech]

JPR,

Hi. Ah, that's great, thanks for confirming that. I should really find some 
time to experiment with Project mode then ;)

Regards,
 
Narinder Chandi,
ToolBox Systems Ltd.
 
I am available for new consulting opportunities…
http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html
-- 

-Original Message-
From: 4D_Tech <4d_tech-boun...@lists.4d.com> on behalf of 4D Tech Mailing List 
<4d_tech@lists.4d.com>
Reply-To: 4D Tech Mailing List <4d_tech@lists.4d.com>
Date: Friday, 11 October 2019 at 09:33
To: 4D Tech Mailing List <4d_tech@lists.4d.com>
Cc: JPR 
Subject: Re: 4D Server Hosting - The Future ? & Using Github to Manage 
Distributed Development

AFAIK, if you use the Project mode, you don't need any binary, for 
everything is in text files (.json, .xml, .txt, .4DProject, .css, . 4DCatalog , 
.4dm, .4DForm, etc.) The .4DB is not necessary.


**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development

[JPR via 4D_Tech]

[JPR]

Hi Guys,

Once upon a time, in a magic computer, a lovely virtual princess was waiting in 
a cooperative process for a strong and handsome virtual prince to awake her... 
Unfortunately, the prince was a cooperative one and he was waiting for the door 
to open, and the key was held by the princess... 

Do you like Fairy Tales?

Here is another one, from Goethe: Once upon a time, an old sorcerer departed 
his workshop, leaving his apprentice with programs to execute. Tired of waiting 
tasks waiting in queue to access one of the 4 CPUs of his computer, the 
apprentice enchanted a virtual machine to do the work, using preemptive 
programming in which he was not fully trained. The apprentice soon realised 
that he couldn't control the tasks because he did not know the magic required 
to do so. The apprentice began to split the CPU in two with a magic mouse, but 
each of the pieces becomes a whole CPU that was splitting again and again in 
the virtual space, until the power supply became red hot!  When all seemed 
lost, the old sorcerer returned and quickly reduced the number of preemptive 
tasks to a more reasonable number! 

Do you believe it? If you have been assisting to the training day I gave during 
the 2019 Tour, you remember that we have discussed about that, and we have seen 
examples on the practical use of preemptive workers, the use of the Signal 
object, how to handle this typical Catch-22 situation, etc. We have also seen 
that the preemptive multithreading was useful to gain speed only in some very 
very specific cases (Remember the demo K_PreemptiveVsCooperative)

Sorry for those of you who still believe in him, but Santa doesn't exist. 
Virtual cores are NOT actual cores. And even if you get zillions of cores, 
there is still only ONE bus to access to memory. So, don't dream, multiplying 
the number of cores will NOT multiply the speed of every database operations. I 
mean, it will not be slower, but (cores * 8) ≠ (speed * 8). And this is even 
much more true in case of hyper threading. And obviously even much much moe 
true in case of a VM.

So let's try to be crystal clear:

> So what you are saying is that if you are deploying 4D Server in a VM there 
> is not need to try and use preemptive mode. You get no benefit from doing 
> that. It’s a waste of time?
Wrong. It will be a benefit in terms of speed, even not as large as you dream, 
because the system will optimise the use of each possible core. And a benefit 
in term of smoothness of execution, because no process will freeze the others.

> The whole benefit of preemptive mode is to allow 4D to assign a process to a 
> core.
Wrong. What I explained is that 4D does NOT assign a process to a core, but 
assigns a process to a THREAD, and the System assigns cores to Threads. The 
same Thread can be switched from core to core by the System, you don't even 
have to know.

> But in the case of a VM the actual, physical cores are managed by the VM.
Wrong. The VM talks to the System, and the System manages cores. In fact, just 
imagine that a machine can run several VM at the same time.

If you have been at the 2019 Tour, remember J_OtherObjects example, where I 
show that if you use cooperative processes, with one executing an excruciating 
method, then the Forms on use are kind of frozen. Then, by switching to 
preemptive mode, the same awfully blocking method doesn't impact on interface 
while running. 

So please, don't mix Processes, Threads, and Cores. Preemptive programming is a 
real plus from many points of view, including it helps 4D to optimise it's own 
internal optimisation processing. And also, remember that Preemptive 
programming is like food, it's necessary, but stay reasonable...

Regarding the other question:
> Yes, you're correct, currently the move from binary to Project format is one 
> way, whether this will become bi-directional I don't know.

> This was asked to JPR at the last 4D World Tour show and his response was 
> “why would you want to do that?” And then he proceeded to explain that to 
> implement a feature like that would require many hours of engineering time, 
> and QA time and then it would have to be maintained into the future. 
> So he said are no plans for a “project folder back to a single .4DB file” 
> feature. Only way to have the happen is to convince 4D it is a needed and 
> necessary feature. 
Thank you, Tim, it's exactly what I said (plus the accent) and it's still the 
current status.

> But, I still need to fully understand exactly why 4D is not eliminating the 
> interpreted structure binary .4DB file completely at this time? Maybe 4D plan 
> to do so eventually? Was this question asked at the last tour? If we are 
> given the Project feature then we should be all in and have no option but to 
> embrace it and source code repos such as Git, since, pretty much every other 
> programming language is doing it this way.
AFAIK, if you use the Project mode, you don't need any binary, for 

Re: Using Github to Manage Distributed Development

[Kirk Brooks via 4D_Tech]

Narinder & Tom,
You can move the structure, including relations, out of Projects via the
Export Structure to XML command and build a new database based on that.
However this only exports the actual structure. Importing the methods from
the Project folder is no big deal. The deal is getting the forms back. No
way to do that. I doubt they are going to invest the resources to build the
equivalent of Convert form to dynamic to go the other way.

But that is certainly something you could post on the forum as a feature
request.

On Thu, Oct 10, 2019 at 1:56 PM Narinder Chandi via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Tom,
>
> Yes, you're correct, currently the move from binary to Project format is
> one way, whether this will become bi-directional I don't know. It is still
> possible to compile back to a single binary though so the step to go from
> Project to interpreted structure binary should surely be possible??
>
> Yes, Branching is a core concept of working with Git.
>
> I think what each of your developers need to do is to Checkout a Git repo
> Feature Branch of Master and export their changes into that. Then, they
> Push that Feature Branch up to Master and make a Pull request into Master.
> Once approved then the Feature Branch is merged into Master. Other
> developers would do the same. Then, each developer Pulls Master to update
> their local Checked out copy of Master. They can Diff that against their
> previous Feature Branch which will give them a list of all the changes that
> the _other_ developers have made and merged into Master and synch their
> structure up.
>
> I think that's it above! The ultimate benefit of all that is that your
> devs are only ever moving around Diffs, not the whole structure export.
>
> Regards,
>
> Narinder Chandi,
> ToolBox Systems Ltd.
>
> I am available for new consulting opportunities…
> http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html
> --
>
> -Original Message-
> From: Tom Benedict 
> Date: Thursday, 10 October 2019 at 21:22
> To: 4D Tech Mailing List <4d_tech@lists.4d.com>
> Cc: Narinder Chandi <4dtechmailingl...@toolbox.uk.com>
> Subject: Re: Using Github to Manage Distributed Development
>
> Thanks for your response Narinder,
>
> >ditch the 4D binary based structure file and use v17's 4D Project
> format
>
> Indeed, that is an attractive option, although it requires 17R5 or
> higher. Once it’s out of beta I may consider it. I’ve heard that once a
> structure is moved to Project Mode, there’s no way to get it out of Project
> Mode. I don’t really know what that means, but that’s one of the reasons I
> haven’t tried it yet.
>
> >Some variation of Gitflow has been used at every client I've worked
> for over the last few years (in a PHP context). I think what you describe
> below loosely maps to Gitflow so take a look at that and see if this is the
> case. Briefly, each of your developers creates their own Feature branch
> from the Master branch on Git, work on it and then merge back into Master
> (there's a bit more to the process but I'm just trying to simplify).
>
> This is great guidance. I wasn’t sure whether each developer would be
> a separate branch or not.
>
> Right now my biggest challenge is logistical. The structure export
> consists of over 55,000 files. Running that every few days and uploading to
> Git takes hours. There’s gotta be a more efficient way.
>
> I’ll check out Gitflow and see whether it can help me.
>
> Thanks again,
>
> Tom Benedict
>
> > On Oct 10, 2019, at 13:12, Narinder Chandi via 4D_Tech <
> 4d_tech@lists.4d.com> wrote:
> >
> > Tom,
> >
> > What version of 4D are you using?
> >
> > It seems to me that the way forward for team development (or solo
> for that matter) will be to ditch the 4D binary based structure file and
> use v17's 4D Project format stored that in a Git repo, even though it's
> still Beta it should be finalised soon enough. IMO, this is going to be a
> revolutionary change for managing 4D code and supporting assets.
> >
> > The most popular Git branching model is Gitflow:
> > https://nvie.com/posts/a-successful-git-branching-model/
> >
> > Some variation of Gitflow has been used at every client I've worked
> for over the last few years (in a PHP context). I think what you describe
> below loosely maps to Gitflow so take a look at that and see if this is the
> case. Briefly, each of your developers creates their own Feature branch
> from the Master branch on Git, work on it and then merge back into Master
> (there's a bit more to the process but I'm just tryi

Re: Using Github to Manage Distributed Development

[Cannon Smith via 4D_Tech]

I can’t think of any reason one would want to go back, either. Splitting the 
structure into files is something we’ve wanted for _so_ long and opens so many 
opportunities.

--
Cannon.Smith
Synergy Farm Solutions Inc.
Aetna, AB Canada




> On Oct 10, 2019, at 4:38 PM, Tim Nevels via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
>> Yes, you're correct, currently the move from binary to Project format is one 
>> way, whether this will become bi-directional I don't know. It is still 
>> possible to compile back to a single binary though so the step to go from 
>> Project to interpreted structure binary should surely be possible??
> 
> This was asked to JPR at the last 4D World Tour show and his response was 
> “why would you want to do that?” And then he proceeded to explain that to 
> implement a feature like that would require many hours of engineering time, 
> and QA time and then it would have to be maintained into the future. 
> 
> So he said are no plans for a “project folder back to a single .4DB file” 
> feature. Only way to have the happen is to convince 4D it is a needed and 
> necessary feature. 

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Using Github to Manage Distributed Development

[Narinder Chandi via 4D_Tech]

Well, JPR's as always makes a good response! But, I still need to fully 
understand exactly why 4D is not eliminating the interpreted structure binary 
.4DB file completely at this time? Maybe 4D plan to do so eventually? Was this 
question asked at the last tour? If we are given the Project feature then we 
should be all in and have no option but to embrace it and source code repos 
such as Git, since, pretty much every other programming language is doing it 
this way.

Regards,
 
Narinder Chandi,
ToolBox Systems Ltd.
 
I am available for new consulting opportunities…
http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html
-- 

-Original Message-
From: 4D_Tech <4d_tech-boun...@lists.4d.com> on behalf of 4D Tech Mailing List 
<4d_tech@lists.4d.com>
Reply-To: 4D Tech Mailing List <4d_tech@lists.4d.com>
Date: Thursday, 10 October 2019 at 23:38
To: 4D Tech Mailing List <4d_tech@lists.4d.com>
Cc: Tim Nevels 
Subject: Re: Using Github to Manage Distributed Development

On Oct 10, 2019, at 4:44 PM, Narinder Chandi wrote:

> Yes, you're correct, currently the move from binary to Project format is 
one way, whether this will become bi-directional I don't know. It is still 
possible to compile back to a single binary though so the step to go from 
Project to interpreted structure binary should surely be possible??

This was asked to JPR at the last 4D World Tour show and his response was 
“why would you want to do that?” And then he proceeded to explain that to 
implement a feature like that would require many hours of engineering time, and 
QA time and then it would have to be maintained into the future. 

So he said are no plans for a “project folder back to a single .4DB file” 
feature. Only way to have the happen is to convince 4D it is a needed and 
necessary feature. 

Tim

*
Tim Nevels
Innovative Solutions
785-749-3444
timnev...@mac.com
*

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**


**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Using Github to Manage Distributed Development

[Tim Nevels via 4D_Tech]

On Oct 10, 2019, at 4:44 PM, Narinder Chandi wrote:

> Yes, you're correct, currently the move from binary to Project format is one 
> way, whether this will become bi-directional I don't know. It is still 
> possible to compile back to a single binary though so the step to go from 
> Project to interpreted structure binary should surely be possible??

This was asked to JPR at the last 4D World Tour show and his response was “why 
would you want to do that?” And then he proceeded to explain that to implement 
a feature like that would require many hours of engineering time, and QA time 
and then it would have to be maintained into the future. 

So he said are no plans for a “project folder back to a single .4DB file” 
feature. Only way to have the happen is to convince 4D it is a needed and 
necessary feature. 

Tim

*
Tim Nevels
Innovative Solutions
785-749-3444
timnev...@mac.com
*

**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Using Github to Manage Distributed Development

[Narinder Chandi via 4D_Tech]

Tom,

Yes, you're correct, currently the move from binary to Project format is one 
way, whether this will become bi-directional I don't know. It is still possible 
to compile back to a single binary though so the step to go from Project to 
interpreted structure binary should surely be possible??

Yes, Branching is a core concept of working with Git.

I think what each of your developers need to do is to Checkout a Git repo 
Feature Branch of Master and export their changes into that. Then, they Push 
that Feature Branch up to Master and make a Pull request into Master. Once 
approved then the Feature Branch is merged into Master. Other developers would 
do the same. Then, each developer Pulls Master to update their local Checked 
out copy of Master. They can Diff that against their previous Feature Branch 
which will give them a list of all the changes that the _other_ developers have 
made and merged into Master and synch their structure up.

I think that's it above! The ultimate benefit of all that is that your devs are 
only ever moving around Diffs, not the whole structure export.

Regards,
 
Narinder Chandi,
ToolBox Systems Ltd.
 
I am available for new consulting opportunities…
http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html
-- 

-Original Message-
From: Tom Benedict 
Date: Thursday, 10 October 2019 at 21:22
To: 4D Tech Mailing List <4d_tech@lists.4d.com>
Cc: Narinder Chandi <4dtechmailingl...@toolbox.uk.com>
Subject: Re: Using Github to Manage Distributed Development

Thanks for your response Narinder,

>ditch the 4D binary based structure file and use v17's 4D Project format

Indeed, that is an attractive option, although it requires 17R5 or higher. 
Once it’s out of beta I may consider it. I’ve heard that once a structure is 
moved to Project Mode, there’s no way to get it out of Project Mode. I don’t 
really know what that means, but that’s one of the reasons I haven’t tried it 
yet.

>Some variation of Gitflow has been used at every client I've worked for 
over the last few years (in a PHP context). I think what you describe below 
loosely maps to Gitflow so take a look at that and see if this is the case. 
Briefly, each of your developers creates their own Feature branch from the 
Master branch on Git, work on it and then merge back into Master (there's a bit 
more to the process but I'm just trying to simplify).

This is great guidance. I wasn’t sure whether each developer would be a 
separate branch or not.

Right now my biggest challenge is logistical. The structure export consists 
of over 55,000 files. Running that every few days and uploading to Git takes 
hours. There’s gotta be a more efficient way.

I’ll check out Gitflow and see whether it can help me.

Thanks again,

Tom Benedict

> On Oct 10, 2019, at 13:12, Narinder Chandi via 4D_Tech 
<4d_tech@lists.4d.com> wrote:
> 
> Tom,
> 
> What version of 4D are you using?
> 
> It seems to me that the way forward for team development (or solo for 
that matter) will be to ditch the 4D binary based structure file and use v17's 
4D Project format stored that in a Git repo, even though it's still Beta it 
should be finalised soon enough. IMO, this is going to be a revolutionary 
change for managing 4D code and supporting assets.
> 
> The most popular Git branching model is Gitflow:
> https://nvie.com/posts/a-successful-git-branching-model/
> 
> Some variation of Gitflow has been used at every client I've worked for 
over the last few years (in a PHP context). I think what you describe below 
loosely maps to Gitflow so take a look at that and see if this is the case. 
Briefly, each of your developers creates their own Feature branch from the 
Master branch on Git, work on it and then merge back into Master (there's a bit 
more to the process but I'm just trying to simplify).
> 
> Regards,
> 
> Narinder Chandi,
> ToolBox Systems Ltd.
> 
> I am available for new consulting opportunities…
> http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html
> -- 
> 
> -Original Message-
> From: 4D_Tech <4d_tech-boun...@lists.4d.com> on behalf of 4D Tech Mailing 
List <4d_tech@lists.4d.com>
> Reply-To: 4D Tech Mailing List <4d_tech@lists.4d.com>
> Date: Thursday, 10 October 2019 at 18:42
> To: 4D Tech Mailing List <4d_tech@lists.4d.com>
> Cc: Tom Benedict 
> Subject: Using Github to Manage Distributed Development
> 
>In my current environment, each developer works on a separate copy of 
a “Master" 4D structure. Periodically each developer runs the "Export structure 
file” command to export everything (methods, forms etc) to local disk. Then 
they comp

Re: Using Github to Manage Distributed Development

[Tom Benedict via 4D_Tech]

Thanks for your response Narinder,

>ditch the 4D binary based structure file and use v17's 4D Project format

Indeed, that is an attractive option, although it requires 17R5 or higher. Once 
it’s out of beta I may consider it. I’ve heard that once a structure is moved 
to Project Mode, there’s no way to get it out of Project Mode. I don’t really 
know what that means, but that’s one of the reasons I haven’t tried it yet.

>Some variation of Gitflow has been used at every client I've worked for over 
>the last few years (in a PHP context). I think what you describe below loosely 
>maps to Gitflow so take a look at that and see if this is the case. Briefly, 
>each of your developers creates their own Feature branch from the Master 
>branch on Git, work on it and then merge back into Master (there's a bit more 
>to the process but I'm just trying to simplify).

This is great guidance. I wasn’t sure whether each developer would be a 
separate branch or not.

Right now my biggest challenge is logistical. The structure export consists of 
over 55,000 files. Running that every few days and uploading to Git takes 
hours. There’s gotta be a more efficient way.

I’ll check out Gitflow and see whether it can help me.

Thanks again,

Tom Benedict

> On Oct 10, 2019, at 13:12, Narinder Chandi via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> Tom,
> 
> What version of 4D are you using?
> 
> It seems to me that the way forward for team development (or solo for that 
> matter) will be to ditch the 4D binary based structure file and use v17's 4D 
> Project format stored that in a Git repo, even though it's still Beta it 
> should be finalised soon enough. IMO, this is going to be a revolutionary 
> change for managing 4D code and supporting assets.
> 
> The most popular Git branching model is Gitflow:
> https://nvie.com/posts/a-successful-git-branching-model/
> 
> Some variation of Gitflow has been used at every client I've worked for over 
> the last few years (in a PHP context). I think what you describe below 
> loosely maps to Gitflow so take a look at that and see if this is the case. 
> Briefly, each of your developers creates their own Feature branch from the 
> Master branch on Git, work on it and then merge back into Master (there's a 
> bit more to the process but I'm just trying to simplify).
> 
> Regards,
> 
> Narinder Chandi,
> ToolBox Systems Ltd.
> 
> I am available for new consulting opportunities…
> http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html
> -- 
> 
> -Original Message-
> From: 4D_Tech <4d_tech-boun...@lists.4d.com> on behalf of 4D Tech Mailing 
> List <4d_tech@lists.4d.com>
> Reply-To: 4D Tech Mailing List <4d_tech@lists.4d.com>
> Date: Thursday, 10 October 2019 at 18:42
> To: 4D Tech Mailing List <4d_tech@lists.4d.com>
> Cc: Tom Benedict 
> Subject: Using Github to Manage Distributed Development
> 
>In my current environment, each developer works on a separate copy of a 
> “Master" 4D structure. Periodically each developer runs the "Export structure 
> file” command to export everything (methods, forms etc) to local disk. Then 
> they compare those files using a diff tool (Examdiff in our case) against a 
> folder full of files exported from the "master” copy of the 4D structure. The 
> diff tool shows us what methods/forms/etc are different. The developer then 
> uses that list as a guide and, with both the local 4D structure open and the 
> “master” 4D structure open, copy/pastes or ‘moves’ the edits from local to 
> master.
> 
>We would like to move this process to Github. I’ve tried, but have not 
> been able to map the Git concepts to my existing process. Is anyone using 
> Github with 4D in this way? If so, could you describe your workflow?
> 
>BTW, I’ve read the Tech Note about 4D on Github, but it looks like it is 
> focused on using Project Mod. I also read Thomas Maul’s post 
> <https://forums.4d.com/Post/EN/30825681/1/3056#30825682> about his 
> component which will commit methods to Github. I’m also aware of Lutz 
> Epperlein’s use of VC Framework. None of those matches my current workflow.
> 
>Thanks for any help.
> 
>Tom Benedict
>**
>4D Internet Users Group (4D iNUG)
>Archive:  http://lists.4d.com/archives.html
>Options: https://lists.4d.com/mailman/options/4d_tech
>Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
>**
> 
> 
> **
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.

Re: Using Github to Manage Distributed Development

[Narinder Chandi via 4D_Tech]

Tom,

What version of 4D are you using?

It seems to me that the way forward for team development (or solo for that 
matter) will be to ditch the 4D binary based structure file and use v17's 4D 
Project format stored that in a Git repo, even though it's still Beta it should 
be finalised soon enough. IMO, this is going to be a revolutionary change for 
managing 4D code and supporting assets.

The most popular Git branching model is Gitflow:
https://nvie.com/posts/a-successful-git-branching-model/

Some variation of Gitflow has been used at every client I've worked for over 
the last few years (in a PHP context). I think what you describe below loosely 
maps to Gitflow so take a look at that and see if this is the case. Briefly, 
each of your developers creates their own Feature branch from the Master branch 
on Git, work on it and then merge back into Master (there's a bit more to the 
process but I'm just trying to simplify).

Regards,
 
Narinder Chandi,
ToolBox Systems Ltd.
 
I am available for new consulting opportunities…
http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html
-- 

-Original Message-
From: 4D_Tech <4d_tech-boun...@lists.4d.com> on behalf of 4D Tech Mailing List 
<4d_tech@lists.4d.com>
Reply-To: 4D Tech Mailing List <4d_tech@lists.4d.com>
Date: Thursday, 10 October 2019 at 18:42
To: 4D Tech Mailing List <4d_tech@lists.4d.com>
Cc: Tom Benedict 
Subject: Using Github to Manage Distributed Development

In my current environment, each developer works on a separate copy of a 
“Master" 4D structure. Periodically each developer runs the "Export structure 
file” command to export everything (methods, forms etc) to local disk. Then 
they compare those files using a diff tool (Examdiff in our case) against a 
folder full of files exported from the "master” copy of the 4D structure. The 
diff tool shows us what methods/forms/etc are different. The developer then 
uses that list as a guide and, with both the local 4D structure open and the 
“master” 4D structure open, copy/pastes or ‘moves’ the edits from local to 
master.

We would like to move this process to Github. I’ve tried, but have not been 
able to map the Git concepts to my existing process. Is anyone using Github 
with 4D in this way? If so, could you describe your workflow?

BTW, I’ve read the Tech Note about 4D on Github, but it looks like it is 
focused on using Project Mod. I also read Thomas Maul’s post 
<https://forums.4d.com/Post/EN/30825681/1/3056#30825682> about his 
component which will commit methods to Github. I’m also aware of Lutz 
Epperlein’s use of VC Framework. None of those matches my current workflow.

Thanks for any help.

Tom Benedict
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**


**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Using Github to Manage Distributed Development

[Tom Benedict via 4D_Tech]

In my current environment, each developer works on a separate copy of a 
“Master" 4D structure. Periodically each developer runs the "Export structure 
file” command to export everything (methods, forms etc) to local disk. Then 
they compare those files using a diff tool (Examdiff in our case) against a 
folder full of files exported from the "master” copy of the 4D structure. The 
diff tool shows us what methods/forms/etc are different. The developer then 
uses that list as a guide and, with both the local 4D structure open and the 
“master” 4D structure open, copy/pastes or ‘moves’ the edits from local to 
master.

We would like to move this process to Github. I’ve tried, but have not been 
able to map the Git concepts to my existing process. Is anyone using Github 
with 4D in this way? If so, could you describe your workflow?

BTW, I’ve read the Tech Note about 4D on Github, but it looks like it is 
focused on using Project Mod. I also read Thomas Maul’s post 
 about his 
component which will commit methods to Github. I’m also aware of Lutz 
Epperlein’s use of VC Framework. None of those matches my current workflow.

Thanks for any help.

Tom Benedict
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**