Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development
Hi JPR, i'm glad the 'Bat Signal' still works. On Fri, Oct 11, 2019 at 1:34 AM JPR via 4D_Tech <4d_tech@lists.4d.com> wrote: > Do you like Fairy Tales? > I thought that was the demo... > > So what you are saying is that if you are deploying 4D Server in a VM > there is not need to try and use preemptive mode. You get no benefit from > doing that. It’s a waste of time? > Wrong. It will be a benefit in terms of speed, even not as large as you > dream, because the system will optimise the use of each possible core. And > a benefit in term of smoothness of execution, because no process will > freeze the others. > OK - but assuming I'm running on a VM isn't that speed and usability benefit coming from moving the processing into a Worker? > > The whole benefit of preemptive mode is to allow 4D to assign a process > to a core. > Wrong. What I explained is that 4D does NOT assign a process to a core, > but assigns a process to a THREAD, and the System assigns cores to Threads. And once more I display my sketchy understanding of how the hardware works. > > But in the case of a VM the actual, physical cores are managed by the VM. > Wrong. The VM talks to the System, and the System manages cores. In fact, > just imagine that a machine can run several VM at the same time. > Ibid. If you have been at the 2019 Tour, remember J_OtherObjects example, where I > show that if you use cooperative processes, with one executing an > excruciating method, then the Forms on use are kind of frozen. Then, by > switching to preemptive mode, the same awfully blocking method doesn't > impact on interface while running. > True (in fact I just pulled it out to look at again). But we are running that on our machines where the _threads_ can be assigned by the _system_ to various cores and I can see that in the activity viewer. In fact running on 'bare metal' shows a lot of variation in the activity of the various cores. I went rummaging around in the knowledgebase to see if I could find what I'm remembering. I think it was one of the keynotes from the '16 Summit and I couldn't find it. But I did find Laurent Ensault's session on the subject: https://kb.4d.com/assetid=77547 I rewatched it and ran the example database. The demo spawns 4 cooperative and 4 preemptive processes then runs an intensive method 1mil times in each. I made a small change to his demo - I added the start and end ticks to the process record in addition to the duration. Why? Because I wanted to see the total time spent on processing the 'job'. I do this by taking the end of the last process to finish and subtracting the start of the first process to begin. I also moved the demo db to v17.3. As luck would have it I have an AWS instance with v17.3 installed so I am able to run the demo locally and on AWS. On my laptop: Cooperative 1: 376 ticks Cooperative 2: 376 ticks Cooperative 3: 375 ticks Cooperative 4: 367 ticks Total run time: 395 ticks Preemp 1: 118 ticks Preemp 2: 119 ticks Preemp 3: 117 ticks Preemp 4: 119 ticks Total run time: 119 ticks // that's what we want to see! On AWS: Cooperative 1: 740 ticks Cooperative 2: 706 ticks Cooperative 3: 793 ticks Cooperative 4: 731 ticks Total run time: 821 ticks Preemp 1: 881 ticks Preemp 2: 824 ticks Preemp 3: 857 ticks Preemp 4: 848 ticks Total run time: 881 ticks Your first thought might be, "hold on, why is the total run time almost the same as each process' run time?" The very first line of the method sets a variable with the tick count. Because Laurent included the IDLE command in the demo method those methods yield the processor back to 4D on each iteration. (The video talks about the details on this.) So Coop1 starts and then yields back to 4D which starts Coop2 and so on. In contrast Preemptive processes simply start and run without having to 'yield back' to 4D. So what we see is that each them gets started pretty much at the same time and complete pretty close together as well. This probably wouldn't be the case if they weren't doing exactly the same work and if 4D were actually working instead of doing a demo. Total run time, as I said, is the tick count for the last process to finish minus the tick count of the first one to start. Or the total amount of time spent processing by all 4 processes. This illustrates the point I recalled from back then: when you run on a VM the advantage of the the OS to distribute threads to the actual cores is abstracted away. They may be distributed to virtual cores but that doesn't impact the total processing time. In the case of AWS apparently this VM overhead actually causes it to run slower but this is a small sample set. I have no idea why and frankly don't have the expertise to figure it out. But it's there. FYI - the AWS instance is an m5ad.xlarge: 4 cores, 16gb RAM. It sorta sucks. I'd love to hear from anyone actually running 4D server on AWS with good performance. -- Kirk Brooks San Francisco, CA === What can be
Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development
JPR, Hi. Ah, that's great, thanks for confirming that. I should really find some time to experiment with Project mode then ;) Regards, Narinder Chandi, ToolBox Systems Ltd. I am available for new consulting opportunities… http://4d.1045681.n5.nabble.com/ANN-4D-Developer-Available-td5765443.html -- -Original Message- From: 4D_Tech <4d_tech-boun...@lists.4d.com> on behalf of 4D Tech Mailing List <4d_tech@lists.4d.com> Reply-To: 4D Tech Mailing List <4d_tech@lists.4d.com> Date: Friday, 11 October 2019 at 09:33 To: 4D Tech Mailing List <4d_tech@lists.4d.com> Cc: JPR Subject: Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development AFAIK, if you use the Project mode, you don't need any binary, for everything is in text files (.json, .xml, .txt, .4DProject, .css, . 4DCatalog , .4dm, .4DForm, etc.) The .4DB is not necessary. ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: 4D Server Hosting - The Future ? & Using Github to Manage Distributed Development
[JPR] Hi Guys, Once upon a time, in a magic computer, a lovely virtual princess was waiting in a cooperative process for a strong and handsome virtual prince to awake her... Unfortunately, the prince was a cooperative one and he was waiting for the door to open, and the key was held by the princess... Do you like Fairy Tales? Here is another one, from Goethe: Once upon a time, an old sorcerer departed his workshop, leaving his apprentice with programs to execute. Tired of waiting tasks waiting in queue to access one of the 4 CPUs of his computer, the apprentice enchanted a virtual machine to do the work, using preemptive programming in which he was not fully trained. The apprentice soon realised that he couldn't control the tasks because he did not know the magic required to do so. The apprentice began to split the CPU in two with a magic mouse, but each of the pieces becomes a whole CPU that was splitting again and again in the virtual space, until the power supply became red hot! When all seemed lost, the old sorcerer returned and quickly reduced the number of preemptive tasks to a more reasonable number! Do you believe it? If you have been assisting to the training day I gave during the 2019 Tour, you remember that we have discussed about that, and we have seen examples on the practical use of preemptive workers, the use of the Signal object, how to handle this typical Catch-22 situation, etc. We have also seen that the preemptive multithreading was useful to gain speed only in some very very specific cases (Remember the demo K_PreemptiveVsCooperative) Sorry for those of you who still believe in him, but Santa doesn't exist. Virtual cores are NOT actual cores. And even if you get zillions of cores, there is still only ONE bus to access to memory. So, don't dream, multiplying the number of cores will NOT multiply the speed of every database operations. I mean, it will not be slower, but (cores * 8) ≠ (speed * 8). And this is even much more true in case of hyper threading. And obviously even much much moe true in case of a VM. So let's try to be crystal clear: > So what you are saying is that if you are deploying 4D Server in a VM there > is not need to try and use preemptive mode. You get no benefit from doing > that. It’s a waste of time? Wrong. It will be a benefit in terms of speed, even not as large as you dream, because the system will optimise the use of each possible core. And a benefit in term of smoothness of execution, because no process will freeze the others. > The whole benefit of preemptive mode is to allow 4D to assign a process to a > core. Wrong. What I explained is that 4D does NOT assign a process to a core, but assigns a process to a THREAD, and the System assigns cores to Threads. The same Thread can be switched from core to core by the System, you don't even have to know. > But in the case of a VM the actual, physical cores are managed by the VM. Wrong. The VM talks to the System, and the System manages cores. In fact, just imagine that a machine can run several VM at the same time. If you have been at the 2019 Tour, remember J_OtherObjects example, where I show that if you use cooperative processes, with one executing an excruciating method, then the Forms on use are kind of frozen. Then, by switching to preemptive mode, the same awfully blocking method doesn't impact on interface while running. So please, don't mix Processes, Threads, and Cores. Preemptive programming is a real plus from many points of view, including it helps 4D to optimise it's own internal optimisation processing. And also, remember that Preemptive programming is like food, it's necessary, but stay reasonable... Regarding the other question: > Yes, you're correct, currently the move from binary to Project format is one > way, whether this will become bi-directional I don't know. > This was asked to JPR at the last 4D World Tour show and his response was > “why would you want to do that?” And then he proceeded to explain that to > implement a feature like that would require many hours of engineering time, > and QA time and then it would have to be maintained into the future. > So he said are no plans for a “project folder back to a single .4DB file” > feature. Only way to have the happen is to convince 4D it is a needed and > necessary feature. Thank you, Tim, it's exactly what I said (plus the accent) and it's still the current status. > But, I still need to fully understand exactly why 4D is not eliminating the > interpreted structure binary .4DB file completely at this time? Maybe 4D plan > to do so eventually? Was this question asked at the last tour? If we are > given the Project feature then we should be all in and have no option but to > embrace it and source code repos such as Git, since, pretty much every other > programming language is doing it this way. AFAIK, if you use the Project mode, you don't need any binary, for
