Re: Preemptive mode flaw

[David Adams via 4D_Tech]

For anyone following along or reading this on the archives, I wanted to add
an update here from the 4D Forums:

http://forums.4d.fr/Post/EN/19391591/1/19391592

Thomas MAULRe: Provide developer control when a preemptive worker receives
non-preemptive code

> For me this is ON ERR CALL, which should be executed, but seems not.

as written, it was expected tht ON ERR CALL detects an illegal usage inside
a preemptive process and not quietly abort. It was filed as ACI0096789.
Already fixed in our development branch, modification currently in testing,
supposed to be available soon in 16.x nightly build branch.

So the answer is: always use ON ERR CALL in every process to detect
unexpected failures (out of memory, disk error - or using non thread safe
commands in a preemptive process).


So, not a design flaw - just a bug that's being addressed.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

Jeff and David, thanks for the update. I don't know why we don't have
direct access to TAOW, but I'm glad to hear that it works well & is
available to the bulk of 4D developers.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Samson via 4D_Tech]

I can second this. The TAOW system is well monitored and responsive.

 > I have to say that the Taow system works really well, in my experience.


-- 
D Samson
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[Jeffrey Kain via 4D_Tech]

Weird that a market large enough for its own branch office doesn't use the same 
tools as the rest of the world. I have to say that the Taow system works really 
well, in my experience. If I had to email someone and then have no further 
contact about an issue until it showed up in a release note, I probably 
wouldn't bother either.

> On May 9, 2017, at 8:25 PM, Wayne Stewart via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> Tim & David,
> 
> The protocol for ANZ is to email techsupp...@4d.net.au with bug details.
> 
> I then submit them.

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[Wayne Stewart via 4D_Tech]

Tim & David,

The protocol for ANZ is to email techsupp...@4d.net.au with bug details.

I then submit them.


Regards,

Wayne


[image: --]
Wayne Stewart
[image: http://]about.me/waynestewart



On 10 May 2017 at 10:20, David Adams via 4D_Tech <4d_tech@lists.4d.com>
wrote:

> > If you're a partner then the preferred way to submit a bug is through
> taow.4d.com which provides you a line of communication
> > specific to the issue reported, you can therefore request updates via
> this line of communication.
>
> It's not available to Partners in Australia.
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
>
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

> If you're a partner then the preferred way to submit a bug is through
taow.4d.com which provides you a line of communication
> specific to the issue reported, you can therefore request updates via
this line of communication.

It's not available to Partners in Australia.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

RE: Preemptive mode flaw

[Timothy Penner via 4D_Tech]

Hi David,

If you're a partner then the preferred way to submit a bug is through 
taow.4d.com which provides you a line of communication specific to the issue 
reported, you can therefore request updates via this line of communication.

If the issue is a show stopper or otherwise critical in nature then you need to 
communicate its importance to your account/sales rep for them to prioritize the 
TAOW case.

It looks like you are a partner but It doesn’t look like you have ever created 
a case on TAOW - please try using taow.4d.com next time you have a bug to 
report and I truly hope your experience improves.

-Tim



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

> I may have missed something while reviewing this thread (there was a lot
of text)
My fault. Mea culpa.

> but I am very curious about the following:

> Just to get back to the original report, I'm talking about a *bug* in the
*current version* that makes these features *unreliable* for use in the
very situation they were supposedly designed for.

> Did you file a bug report?  If not, please do so that our engineers can
be made aware promptly of the flaws
> you have uncovered!

Nope. I didn't file a bug report. It's not easy to do down here, and it's
impossible to see your own bug reports in the southern hemisphere. I did go
through the effort (sample database, etc.) some months back on a more
serious bug and it's received no attention. The 4D bug reporting system is
so unrewarding it actively discourages submitting reports. Also, it's just
plain hard to do.  I've found stacks of 4D bugs lately but they don't get
reported to 4D. I find a work-around, give up, use another tool, etc. I
move on. I do report things here and/or on the forums to help out other
developers via the archives. I've been noisy lately, but that will stop
again later this month and none of my reports are likely to make it out
publicly in any form. There's just no motivation to do it. Unless there's
feedback and bug fixes, why would I take the time? (Trust me, I talk to a
lot of serious 4D programmers regularly and this is a pretty widespread
attitude.)

None of this is on you Tim! And I'm aware that I'm being a pain. I mean no
offense. You're contributions here, for example, are noted and definitely
*most* appreciated. But if 4D really wants bug reports, they need to:

* Make them easy to submit.
* Make them easy to monitor.
* Provide confirmation and feedback!
* Fix bugs quickly.
* Reject bugs quickly if they aren't bugs. (Quality bug reporting works
both ways, I get that.)

I may be missing something (or a lot) and I know things are different on my
side of the equator. If I'm hopelessly wrong here, I'll take my spanking
and follow the instructions I've missed to date. So, please, you're welcome
to update me and anyone following along at home, that's something you've
proven really great at in the past. If 4D doesn't have proper tools like
this now then I'd say the LAST think they should do is write some. There
are just so many great tools out there now it would be a huge waste of time
(on the calendar) and resources to write a custom one. Perhaps some of the
offerings from Australia's own Atlassian?
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[Herr Alexander Heintz via 4D_Tech]

Am 09.05.2017 um 22:58 schrieb JPR via 4D_Tech <4d_tech@lists.4d.com>:
> 
>> us has said what we think the glass is half full *of* ;-)
> For me, it will be half-full (at least) of Puligny-Montrachet 1961 from 
> Maison Faiveley ;-)

I seem to be doing something wrong, my hourly rates only allow for a 3 liter 
karton of vin de pays :-)
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[JPR via 4D_Tech]

[JPR]

Hi David,

> "Yes, 4D should trap for bad code coming into a worker and report it in a
> way that the developer can manage. 4D should not silently destroy the
> worker, lose its context, and restart it without reporting the error."
> 
> If you agree, we have no dispute.
So let's have no dispute. I agree with you. At least in a perfect world, with 
perfect products used by perfect people, I agree this is the way it should be. 
And be sure that 4D is working on it. 

Meanwhile I'll try to do defensive programming, then I'll not be trapped by 
this flaw, and I move forward... 

> Absolutely! I am also a glass half-full person. But then again, neither of
> us has said what we think the glass is half full *of* ;-)
For me, it will be half-full (at least) of Puligny-Montrachet 1961 from Maison 
Faiveley ;-)

My very best,

JPR

> Date: Tue, 9 May 2017 14:53:56 +1000
> From: David Adams <dpad...@gmail.com>
> To: 4D iNug Technical <4d_tech@lists.4d.com>
> Subject: Re: Preemptive mode flaw


**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

> I'm curious to know in which one of my postings you have found the
following sentence
> "that basic error checking is a weird thing to worry about".

No, not your words - my paraphrase. Since my entire point was about error
trapping and you objected, I didn't know how else to interpret your
opinion. It really wouldn't be hard to agree with me:

"Yes, 4D should trap for bad code coming into a worker and report it in a
way that the developer can manage. 4D should not silently destroy the
worker, lose its context, and restart it without reporting the error."

If you agree, we have no dispute. If you disagree, I guess I don't know
what your position is. If you agree, you (and others) can vote for the
feature request here:

http://forums.4d.fr/Post/EN/19391591/1/19391592

Miyako posted an excellent technical analysis (as he always does!) that may
make it possible to reduce the scope of the error request, at least
internally. I don't care about the details so long as we can catch errors
and prevent 4D from killing the existing worker context, showing error
screens, starting duplicate "unique" workers or any of the rest of it.

On the technical front, this issue is most important with CALL WORKER when
the worker is preemptive. According to the documentation on CALL WORKER:

"...they have been designed mainly for interprocess communication in the
context of preemptive processes"
http://doc.4d.com/4Dv16/4D/16.1/About-workers.300-3374909.en.html

Since this is the primary reason the feature was created, it seems entirely
reasonable that it work in a predictable, reliable, and
thoroughly documented manner. Likewise, every effort should be made to help
the developer to use these commands correctly, which includes
systematically and consistently catching and reporting on errors. This is
particularly true since both workers and preemptive mode are new features. I

> I prefer to see the glass half full than half empty,

Absolutely! I am also a glass half-full person. But then again, neither of
us has said what we think the glass is half full *of* ;-)
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[JPR via 4D_Tech]

[JPR]

Hi David,

> I find it deeply disturbing that a representative of 4D France somehow
> thinks that basic error checking is a weird thing to worry about

I just would like to correct this:

- I'm not a representative of 4D France, just a contractant.

- I do not talk in the name of 4D in general. I just express my own feeling (or 
opinion), like you do yourself. This feeling is just based on my own experience.

- I do not pretend to detain the absolute truth, and, during trainings, I 
always say that I recommend to do so and so, and never say that this is the 
only way.

- I'm curious to know in which one of my postings you have found the following 
sentence "that basic error checking is a weird thing to worry about".

I prefer to see the glass half full than half empty, this is my way of 
thinking, because I think it's a more positive way. But, here also, I just 
express my opinion, and you're perfectly right to think the contrary. This is 
the way things are going on, because thread springs light. (De la discussion 
jaillit la lumière. — Charles Monnard)

And, as my friend Thibault was used to say: "Progressons! - Spinoza"  (Let's 
get forward!)

My very best,

JPR

> On 08 May 2017, at 19:27, 4d_tech-requ...@lists.4d.com wrote:
> Date: Tue, 9 May 2017 09:04:56 +1000
> From: David Adams 
> To: 4D iNug Technical <4d_tech@lists.4d.com>
> 

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

Well, I've answered your various points but, overall, I'm disappointed that
you've again failed to offer any actual technical content. Perhaps Thomas
Maul would contribute? He often does not agree with me, but his arguments
are always well-reasoned and technically interesting. Even if Thomas scolds
someone, it's after providing good information.

I also think it may be too easy to get caught up in how cranky I sound. So
I sound cranky, so what? That's more my problem than anyone else's. Does it
make me or my arguments more convincing when I sound like that guy on the
bus that no one will sit next to? No, not really. I do myself no favor,
your best option is to ignore my frustrated sounds and focus on the
technical merits of what I'm saying. If I'm missing something or being
stupid, I invite correction. And at least I'm still here, the other 4D
people of my generation are quiet...because mostly they are gone. Hey, I
like 4D! We all do. I mean, there's no real need to convince 4D people to
use 4D. We all want it to be better.

Since this whole thing has gotten absurdly long, I'll repeat my basic point
on its own:

CALL WORKER is a remote procedure call system that injects EXECUTE(code)
into a process.
If cooperative code is injected into a preemptive process, bad things
happen.
4D should do less bad things and tell us about the problem via ON ERR call.

That's it. It strikes me as a pretty uncontroversial statement so I don't
understand what the fuss is about. Why wouldn't someone thing it makes
sense to provide an error at this point?
> Concept of Workers is very new in V16, you know that you can expect
improvements in the future.

Of course! Discussion of the subject would be pointless if there were no
chance of change. However, I cannot make any plans based on what *might*
happen in the future. So, speculation is of zero practical value for me.

I'll also note that while "workers" are new in 4D V16, the underling
concepts and tools are neither new nor unfamiliar. They're just new in 4D.
Meaning, there are many, many long-standing implementations that 4D very
likely looked at and considered while developing their design. Like any
other programming team, the crew at 4D must have considered alternative
designs when they came up with what is now CALL WORKER and CALL FORM. I can
see why they picked a remote procedure call scheme, but I can't think of a
single other messaging architecture that works that way. But either way, it
was a design choice made in France, not some sort of discovery and not some
sort of law of nature. Their is nothing sacred about the design.

Now, they had to make choices to deliver a product. Their choices have very
good results and less desirable results. I have been continuously praising
the good aspects. So much so that these features convinced me to move my
own development from V13 to V16. I was also excited enough about these
features arriving in 4D to sign up to do my first 4D presentations in a
decade. So, yeah, I'm a fan.

I'm also willing to say what is a weak spot and what I see as preferable
design choices going forward. Error handling has always been a terrible
weak spot in 4D and my latest "feature request" (it's really about a
bug/design flaw) shows me that 4D still doesn't appreciate the importance
of consistent and reliable exception handling.

> You cannot get something at this level of complexity and have all
finished at once, in a perfect way.

And you cannot expect customers not to say that are unsatisfied by an
incomplete implementation. As it stands, the current design makes a great
many normal messaging patterns difficult or impossible to implement. 4D
could have done this sort of design *without more work* and given us a very
different system:

LAUNCH WORKER("Worker_Name";"Worker_Handler_Name"")
CALL WORKER("Worker_Name";$object)
// Worker_Handler_Name (Object)
C_OBJECT($1;$message)
$message:=$1
// Do whatever you like.

That's the simplest alternate version. At that point, the problem I have
been describing *would be impossible to create.* The flaw doesn't come from
some future feature not being implemented, it's a consequence of the design
itself. 4D can improve the situation by trapping the error when running
EXECUTE inside the workerbut the underlying design is what makes it
possible. The positive side of the design is that it's very easy for people
to reuse existing code without having to mess with C_OBJECT. The downside
is that you can blow up your server.

> IMHO, it's the new way (for 4D Developers) to think their application's
programming.
> We go from a synchronous way of programming, into a totally asynchronous
way, from
> loops to messaging (CALL FORM & CALL WORKER), which is not simple for
many developers.

I can't say. Anyone that's dealt with JavaScript has dealt with a vastly
harder to deal with version of asynchronous programming. In order to
achieve many normal results in 4D you'll need to roll your own versions of
features provided by 

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

 > Thanks. Nobody ever mentions that when discussing the feature.

Yes. Has anyone heard if this is an expected feature in a future version?
That would seem like a fair thing.

I'm obviously a lot more into headless and distributed applications than
anything else, so I look at networked 4D systems architecture differently
than some. (But not all.) If I wanted to set up a tolerably stable,
 high-performance 4D Server system, it would look like this:

4D Server
4D Sand-alone merged apps
Some kind of communications layer

In that setup, all of the machines are preemptive capable.

With that said, what I would *really* like to hear are real-world stories
from people getting some advantage out of preemptive mode. And details
about their approach.

Taking a page from Miyako's book, I think that a sensible starting strategy
for a regular 4D Server would be:

* Pick a discrete, time-consuming, CPU-bound task, of course. (Nothing else
could ever be worth the bother.)
* Run the code using Execute on server.

That rules out CALL WORKER, but that's okay. I suspect we're all pretty
careful about what sort of code runs on the server machine!

To communicate with the process on the other CPU, I'd use any number of
old-fashioned tricks. Honestly, I'd be picking jobs that can be described
in either a document or a record. Then the task runner would periodically
poll for new work in either a watch folder or a job table. If we're talking
about large tasks, the polling costs should be low. In fact, if the jobs
are large and rare, just run them on demand in a new process and then let
that process die.

But it would depend on circumstances how best to proceed, of course.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[Jeffrey Kain via 4D_Tech]

Thanks. Nobody ever mentions that when discussing the feature.

> On May 8, 2017, at 8:49 AM, Epperlein, Lutz (agendo) 
>  wrote:
> 
> http://doc.4d.com/4Dv16R2/4D/16-R2.1620/Preemptive-4D-processes.300-3111846.en.html
> 
> look at "Availability of preemptive mode"
> 
> The use of preemptive mode is available in 4D 64-bit versions only. The 
> following execution contexts are currently supported:
> Preemptive execution
> 4D Server X
> 4D remote -
> 4D single-userX
> Compiled mode X
> Interpreted mode  -

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

RE: Preemptive mode flaw

[Epperlein, Lutz (agendo) via 4D_Tech]

http://doc.4d.com/4Dv16R2/4D/16-R2.1620/Preemptive-4D-processes.300-3111846.en.html

look at "Availability of preemptive mode"

The use of preemptive mode is available in 4D 64-bit versions only. The 
following execution contexts are currently supported:
Preemptive execution
4D Server   X
4D remote   -
4D single-user  X
Compiled mode   X
Interpreted mode-



> -Original Message-
> From: 4D_Tech [mailto:4d_tech-boun...@lists.4d.com] On Behalf Of Jeffrey Kain 
> via
> 4D_Tech
> Sent: Monday, May 8, 2017 2:47 PM
> To: 4D iNug Technical <4d_tech@lists.4d.com>
> Cc: Jeffrey Kain <jeffrey.k...@gmail.com>
> Subject: Re: Preemptive mode flaw
> 
> Is it still the case that client/server applications are excluded from 
> running preemptive
> 4D code?
> 
> --
> Jeffrey Kain
> jeffrey.k...@gmail.com
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[Jeffrey Kain via 4D_Tech]

Is it still the case that client/server applications are excluded from running 
preemptive 4D code?

--
Jeffrey Kain
jeffrey.k...@gmail.com

> On May 7, 2017, at 9:03 PM, Keisuke Miyako via 4D_Tech <4d_tech@lists.4d.com> 
> wrote:
> 
> then there is the second, runtime (or platform) check.
> unicode mode must be active, the app must be 64-bit, compiled, the type must 
> be either server or desktop.
> if none of these conditions are met, all processes are automatically 
> cooperative, no error.

**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

On Mon, May 8, 2017 at 11:03 AM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

Another great post, thanks. I really like the way you've broken out the
various scenarios, how they should behave and how you've observed them
behaving. Just to make it really clear, all I'm talking about is raising
errors consistently and not killing/restarting workers as a side-effect. I
in no way have _any_ problem with _any_ of 4D's restrictions on preemptive
processing. None. I looked into it a bit, and it's no simple task to make a
language run on multiple cores. So, they've got my complete respect on
that. Likewise, some of the restrictions in the language are imposed by the
underlying OS. So, it's reality imposing a constraint, not some arbitrary
choice by 4D.

Likewise, I appreciate what the compiler does for detecting errors in
advance. I'm always all in on the compiler doing more and, in this case,
it's a good effort.

But I am concerned (as you figured) about what comes up as #3 in your
taxonomy:

"CALL WORKER is used to "run cooperative code in an existing preemptive
process"

Personally, I think that 4D should *not* run the code. It should throw an
error that we can catch. Every time. It's not 4D's bug, it's my bug. But 4D
needs to tell me about it. This is very realistically the sort of flaw that
could creep into a system of almost any level of complexity from simple on
up.

Bad call: My fault
Scolding me: 4D's job
> and you can invoke that method by passing an empty string.
> http://doc.4d.com/4Dv16R2/4D/16-R2.1620/CALL-WORKER.301-3111543.en.html
> if we followed that rule and always passed an empty string, we would
never have this problem. just saying...

I guess I haven't said clearly, but I'm not primarily thinking of myself.
There is zero chance I'll start tossing unregulated code around for
execution. That's just stupid/insane/unsafe. It's a weak design, although I
can see why 4D did it. I've got a quite involved system of message passing
I'm working on that goes like this:

CallMyMagicThingy($object;{$result_callback_object})

Any method that is going to get a callback has one parameter:

C_OBJECT($1;$message)

That's it. There are formalisms and validators for the contents of the
message. (A header with routing details and action flags, a message
payload, and so on.) This, in fact, is was arguing for comprehensive JSON
scanning features a few months back. Without that, you can't write any sort
of generic validation code. Instead, you need a lot of code that embeds the
rules in the code. Ugh. I'd rather build my own meta-data/schema to
generate, validate, document, and test the formats. Can't. So, I scaled
back the system to code with only much simpler formats and had to write
individual validators. Wasteful and disappointing, but the native command
set doesn't support anything more. Anyway.

> now your feature request, if I understand correctly, talks about
detecting, on the receiving end,
> if a request to execute cooperative code has been received in a
preemptive process.
> I think it is not a problem, since there is nothing to receive since CALL
WORKER does make such requests.

I'm not parsing out what you're saying here, so I'm probably missing your
point. What I'm asking for is that the *recipient* throws an error. The
worker throws an error that I can catch. From there, I've got a lot of
options as how to proceed. There's no call-response with CALL WORKER so
there is not automatic way to report back the flaw to the code that caused
the flaw. But that's another matter:

C_BOOLEN($okay)
$okay:=Call worker("PreemptiveWorker";"BadCode")

If($okay=false)
   ALERT("Aiuto! Something went wrong!")
End if

I could go on about alternatives here all day, but that's not the point. At
the bare minimum, the preemptive worker should not terminate and should
report an error.

> * I don't get an error.
> this must be CALL WORKER asked to do something impossible. (issue #3
above)

Could be. Hard to know. Impossible to count on. The docs don't say. It has
been drilled into us for decades that when something is not in the docs *it
is not a reliable behavior.* There is absolutely no way to distinguish
between "Oh, yeah, we just didn't put that in the docs, good idea" from
"that's an undocumented behavior, don't count on it." They look the same
from the outside since there's nothing to look at!

> * The worker may be restarted in cooperative mode. Which may end up
clearing existing variables and so on (?)

> the worker must have been killed or aborted.

Yes, exactly - but not by my code - by 4D. This is a bug or design flaw.
It's not something I think is easy to defend as a good idea. "Oh, we will
keep this process running until you KILL it or quit. And that's a
guarantee! (+)"

(+) Not an actual guarantee, see the back of your ticket.

> * Other people report getting an error.
> some "eval" code must have been executed in the worker (issue #2 above)

Yeah, I'll buy that.

> * Sometimes I get an error.
> * 

Re: Preemptive mode flaw

[Keisuke Miyako via 4D_Tech]

the reason why I think

> CALL WORKER("Preemptive_Worker";"MethodWithCooperativeCode")

or more specifically

> CALL WORKER("Preemptive_Worker";"MethodWithPreemptiveCode")
> CALL WORKER("Preemptive_Worker";"MethodWithCooperativeCode")

is not a problem goes like this:

4D performs 2 checks; checking at compilation time, and checking at runtime.

the first check is well documented. every method with the "run preemptive" 
property is validated,
that no thread-unsafe commands or methods or inter-process variables or plugins 
or Begin SQL~End SQL blocks are called.
if the "use preemptive web processes" database property is activated,
web-related database methods, compiler_web, triggers and methods that are 
public on 4D Mobile are also validated.

then there is the second, runtime (or platform) check.
unicode mode must be active, the app must be 64-bit, compiled, the type must be 
either server or desktop.
if none of these conditions are met, all processes are automatically 
cooperative, no error.

and this is where we feel like we are entering uncharted territory; the list of 
"what if" s.

I try to categorise the potential issues this way:

1. references

any "references", that are "safe" in that they are not interprocess variables 
(therefore pass the first test),
only work within the same preemptive process that created it, or across 
cooperative processes.
these include DocRef, DOM ref, and yes, pointers.

http://doc.4d.com/4Dv16R2/4D/16-R2.1620/Preemptive-4D-processes.300-3111846.en.html

the code pass the first test, but breaks in that the references are invalid.
the error itself does not suggest thread-safety per-se,
they are just classic runtime errors due to invalid references.

there is a special provision for objects; an implicit OB Copy is called by CALL 
WORKER.
http://doc.4d.com/4Dv16R2/4D/16-R2.1620/CALL-WORKER.301-3111543.en.html

2. execution

using interprocess variables inside PROCESS 4D TAGS,
EXECUTE FORMULA, EXECUTE METHOD, etc.
the process has already been started as preemptive,
but an attempt to run cooperative code in the same context has been made.

the error specifically mentions that thread-safety is the problem.

3. calling

I think this is the type of issue we are discussing right now.

CALL WORKER is used to "run cooperative code in an existing preemptive process"
it is similar to issue #2 but different in the sense that it is not the current 
context that is asked to break thread safety.
when asked to "call worker", 4D already knows that the method is thread unsafe, 
and so is the target process (worker).
the instruction is simply impossible to execute.
so what happens? nothing (that has been my experience).

incidentally, what I find interesting about CALL WORKER is  that a worker 
remembers the name of the method that started it off,
and you can invoke that method by passing an empty string.

http://doc.4d.com/4Dv16R2/4D/16-R2.1620/CALL-WORKER.301-3111543.en.html

if we followed that rule and always passed an empty string, we would never have 
this problem. just saying...

now your feature request, if I understand correctly, talks about detecting, on 
the receiving end,
if a request to execute cooperative code has been received in a preemptive 
process.
I think it is not a problem, since there is nothing to receive since CALL 
WORKER does make such requests.

http://forums.4d.fr/Post//19391591/1/

> What happens when you send an illegal instruction to a worker that's running 
> in preemptive mode?
> I've done my own quick tests and asked around, and here's what I've found:
>
> * I don't get an error.
> * The worker may be restarted in cooperative mode. Which may end up clearing 
> existing variables and so on (?)
> * Other people report getting an error.
> * Sometimes I get an error.
> * Sometimes an error handler catches, sometimes it doesn't.
>
> It's wildly unpredictable.


I agree the documentation could be better, by giving more specific examples of 
each scenario.
but by and large, I think the results are pretty predictable for each case.

> * I don't get an error.

this must be CALL WORKER asked to do something impossible. (issue #3 above)

> * The worker may be restarted in cooperative mode. Which may end up clearing 
> existing variables and so on (?)

the worker must have been killed or aborted.

> * Other people report getting an error.

some "eval" code must have been executed in the worker (issue #2 above)

> * Sometimes I get an error.
> * Sometimes an error handler catches, sometimes it doesn't.


a mix of all the other problems, just presenting themselves in random order?




**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

On Mon, May 8, 2017 at 01:42 JPR via 4D_Tech <4d_tech@lists.4d.com> wrote:

> I'm not sure about this topic... About what do you try to demonstrate...

It's a pretty standard question about exception handling.

> Is it a crash test? A kind of stress test? Do you really want to find a
way to crash 4D?
I'll answer your questions in order: No. No. No. There's a likely scenario
that isn't documented, so I wanted to find out the actual behavior. That's
just a bit of routine, technical due diligence. I'd do this with any
product.

> What I said during this training is basically 

It sounds like you offered some sensible guidelines and such in your class,
but your points don't have much of anything to do with the issue I've
raised.

* I objected to EXECUTE for a very
* long time because of this very reason. Unsafe at any speed.

> That's true. But you know, Life is unsafe at any speed.
? That's just an odd thing to say.

> Why to object to a command? Just don't use it if you do not feel safe
with it,
> and let people willing to use it do it at their own risks. I feel BASE
jumping
> very unsafe, so I don't do it, and I still feel good about people who
like it...

I don't understand your response or how it applies to the issue I've
raised. You're not offering any sort of technical answer, just a confusing
metaphor. I won't follow you in this direction.

I'm talking about a pretty straightforward technical issue, not some kind
of exotic misuse of 4D's commands. For background, where did I get the
notion that CALL WORKER is meant to communicate with preemptive processes?
From watching Laurent Esnault's 2016 Summit presentation. His presentation
was clear, well reasoned, and informative. He explained how CALL WORKER
satisfies a requirement that arose out of the need to communicate with
preemptive workers. In other words, CALL WORKER was *created* to talk to
preemptive workers! So, what I'm talking about is a 100% orthodox use of
the 4D commands as explained by their creator. CALL WORKER has broader
applications, but it sounds like its genesis was to talk to preemptive
workers.

Now take another look at the title of my feature request:

   "Provide developer control when a preemptive worker receives
non-preemptive code"
   http://forums.4d.fr/Post//19391591/1/

That sounds entirely reasonable, no? If a piece of code accepts inputs,
it's basic defensive coding to have a way of dealing with exceptions. We
don't have this now in 4D for preemptive workers that receive code that
cannot be run preemptively. That's bad. 4D should fix this. This is
*entirely* within 4D's powers and absolutely *not* within our powers.

Now, it is easy enough to say (as someone already has) "don't do that!"
Well, of course! But that's as helpful as saying "Don't make mistakes!" or
"Be smarter!" It's worthless advice because there's no way to follow it. If
I could be smarter, I would be. But I can't. I would love to make fewer
mistakes but, by nature, they're something I'm unaware of. That's why
programmers learn to build tools to catch, manage, and prevent mistakes.
I'm not asking 4D to tolerate something illegal, I'm asking it to complain!
But to tell me about that complaint. Right now, there's a whole bunch of
unpredictable/inconsistent/undocumented behavior that could lead to some
seriously messed up outcomes. Since 4D is proud of this new feature (and
why not?), I'd hope that they want to make it more useful, reliable, and
easy to use successfully.

Most people aren't that far into the details on CALL WORKER, etc. yet, but
there's no need to understand those commands to appreciate the basic
quality-oriented coding issues. Back to talking about bad inputs... Take a
simple example. I've got a routine called CalculateRate

   CalculateRate (Longint) : Real

$1 must only be a whole number between 1-5, so the entire set of legal
values is:

1   2   3   4   5
That's it, all other values are illegal. Now, imagine four illegal calls
like these:

C_REAL($rate1;$rate2;$rate3;$rate4)
$rate:=CalculateRate(Pi)
$rate2:=CalculateRate("5")
$rate3:=CalculateRate(Current date)
$rate4:=CalculateRate(0)

Honestly, none of them look wrong per se. Any of those parameters could
make sense, depending on what sort of rate you're trying to calculate. But
in this case $1 has been defined as a longint, so the compiler detects that
the first three calls are illegal. However, the compiler is blind to the
out-of-range value in the $rate4 line. This is because 4D doesn't have any
syntax for declaring an input as allowing only a specific range of values,
etc. 4D's type system is non-extensible, so we're limited to 4D's built-in
types.  And even if the compiler did see the flaw in the fourth
call, there are other cases that only arise at runtime:

C_LONGINT($rate_code)
$rate_code:=Number_GetRandom(0;5)

This variable gets a longint from the series 0, 1, 2, 3, 4, 5. Ouch! It's
going to generate a bad input roughly 1/6 of the time. You'll only see this
at runtime. That's a 

Re: Preemptive mode flaw

[JPR via 4D_Tech]

[JPR]

Hi David,

I'm not sure about this topic... About what do you try to demonstrate...

> I did it on purpose to see what happens. I'm fanatical about checking
> inputs and preconditions and wanted to see what sort of error to trap for.
> In many cases, there is no error trapped - it just screws up.


Is it a crash test? A kind of stress test? Do you really want to find a way to 
crash 4D? If yes, I can tell you many ways to crash 4D, or your System, or your 
HD, or even your computer, beyond any hope of recover...

It's like if you test a car, pedal to the metal, and ask what will happen if 
you close your eyes and turn abruptly the steering wheel. Is it useful? 

What I said during this training is basically:

- Except in very exceptional cases, you do NOT need preemptive, so you can keep 
your interprocess variables and interface tricks.

- If you actually NEED preemptive tasks, then you will develop new programming 
for that, and then why not do it in a kind of thread-safe programming, just in 
case...

- Any analysis that concludes into a need to synchronize your thread with 
others is probably wrong somewhere.

- You can always try to cheat with 4D, if you like it, but in serious 
programming, don't try to mess with 4D, for I strongly doubt it will do any 
good...

> I objected to EXECUTE for a very
> long time because of this very reason. Unsafe at any speed.

That's true. But you know, Life is unsafe at any speed. 

Why to object to a command? Just don't use it if you do not feel safe with it, 
and let people willing to use it do it at their own risks. I feel BASE jumping 
very unsafe, so I don't do it, and I still feel good about people who like it...

My very best,

JPR




> On 06 May 2017, at 21:03, 4d_tech-requ...@lists.4d.com wrote:
> Date: Sun, 7 May 2017 13:38:41 +1000
> From: David Adams 



**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

On Sun, May 7, 2017 at 3:07 PM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> first up, I myself am still in the process of learning how workers work,
> and I totally agree one should probably start with small POCs or
> relatively minor tasks before going full on.
>

I think many of us here will be very interested in how you end up using
workers, so please share when you find applications that you like!


> keep in mind a CALL WORKER can create a new worker if the previous one was
> killed.
> for instance. imagine a recursive worker method, using process variables
> for context,
> aborted for whatever reason from the debugger or runtime explorer.
> the worker is killed, but a new "call" comes in from the previous context,
> starting a new worker that has no idea where the previous worker left off.
> that can be a nightmare.
>

Yes, exactly! There is no On Worker Start or On Worker Stop event - so it's
*very* important that they not be terminated accidentally.

For what it's worth, I've been working with workers, call message, and call
form intensively for a few months and am pretty deep into the details of
the mechanics. I like the commands and I think 4D did an admirable job of
providing power without complexity. I never would have come up with the
design they did as there are a lot of standard queuing features I would
never have considered dropping the way they did. I've tried to be open
minded about their design choices. But the more I get into it, the more I'm
seeing how different designs wouldn't be much more complex (if any more
complex) without taking away any of the power.

As far as preemptive mode goes, I think that CALL WORKER and preemptive
processing are wedded in people's minds because that's how 4D has been
presenting the subjects. I guess that they added workers to solve the
problem of dropping IP variables for IP communication. I didn't use IP
variables that way (too hard and expensive to use safely, too dangerous and
messy to use any other way)...I just use them for pseudo-constants/cached
items. So, CALL WORKER didn't solve a problem I had...but I love message
queues and have been trying to convert CALL WORKER into an actual message
queue system. (Callbacks and Publish/Subscribe.)

keep in mind a client process can never be preemptive.
>

Yes, I've noticed that. I don't know what the reasoning is there or if it
will change. It's just server-side processes, in compiled mode, using legal
commands, on a multi-core machine, where you've launched the process in
preemptive mode. Whew! Or stand-alone...
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

> On Sun, May 7, 2017 at 1:29 PM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

Hey, thanks for writing! I think the world of your contributions and work,
but we disagree pretty completely on this. I'll offer my thoughts and point
of view on all of this and, hopefully, you can give me a new way of looking
things. (You did manage to shift my thinking on ON ERR CALL in general
recently.) And if anyone else has real-world comments or observations, I'm
all in on hearing them, even if they make me change my opinion. My opinions
are *always* right, I just try to change them a lot ;-)

> is "code injection" even a problem?

Absolutely.

> preemptive mode is all about multi-tasking.
If by that you mean "taking advantage of idle cores", then I agree.

> I would expect minimal communication between those tasks,
> literally, they should be "preemptive".
> it's true you can start a preemptive process with CALL WORKER,
> but that is not required,
> you can also use New process or Execute on server.
> it's totally fine to have a runaway preemptive task.

Good point, thanks for pointing it out. I wasn't thinking about New
process/Execute on server, excellent point.

Just to make it clear, I have very little interest in preemptive
processing. I expect it to be a powerful and useful technique in a *very*
narrow range of problems. To justify the effort of using preemptive
threads, you need a task that is 1) taking a long time, 2) CPU-bound, and
3) doesn't require so much time to prepare and marshall that you don't get
much speed gain, and 4) has a sufficiently large non-cooperative component
to make it work the trouble. That's a lot of hurdles to clear and,
honestly, I don't have a single task in day-to-day use that makes it clear
of all of them. I've worked on systems (and am likely to work on systems
again soon) where such tasks exists, but they're not so commonplace.

> workers, on the other hand, is all about messaging.
Sadly, we *don't* have messages. We have blocks of code that are run.
That's not messaging. It only seems like messaging *if you're 4D.* As a 4D
developer, there's no queue, there's no message. If you're inside of 4D
itself, you can see a queue, etc. But as a 4D developer? You can't. All we
have is a pipe that things pop out of one-at-a-time and then execute. We
don't know if anything is in the pipe, we don't know when something
arrives, we don't have any ability to view/screen/filter/reject/trap that.
There's no event, nothing. Code is being tacked onto the end of a thread
and run blindly. It's an inherently dangerous design, but it _can_ be used
well. It just isn't built to run safely out of the box. Don't get me wrong,
I like the feature - I just don't use it without quite a bit of scaffolding
in place to minimize and/or eliminate risks.

As a practical example a very common, obvious, and good use of a preemptive
thread with CALL PROCESS, consider a log writer. It's a perfect setup.
Let's say you're logging errors/tasks/event/requests/whatever in 8
processes. Should each have its own log file? If not, it's a huge,
time-sucking waste to manage the contention between processes fighting for
a lock on the file. So, the better solution is to pipe all log entries to a
single process for writing. And since writing to disk is a relatively slow
operation, often of low value for logs, why suck up time on the main CPU?

CALL WORKER("LogWriter_Errors";"LogWriter_Start";"Errors")
LogWriter_Errors opens/appends/whatever a log file on disk *and keeps it
open*. Then any number of processes can stream log lines in for writing:

CALL WORKER("LogWriter_Errors";"LogWriter_Write";$error_details)
LogWriter_Errors catches the line, does a SEND PACKET and you're done and
dusted. You could set it up to handle different types of logs (say have
errors and requests both open), use different workers for different types
of logs, etc. None of the data has to go through 4D's data file, you've
just got a pass-through to disk. Sweet. Speed improvement over fighting
over the log file, etc.? 20x is *easy*. That's a huge optimization -
largely because of the design, the preemptive part - I don't even know how
much that helps. But the critical detail is that *the log file is opened
once at the start of the worker and closed only when the worker finishes.*
If that fails, you've got a big fat problem. (As it turns out, 4D currently
has a bug that makes race conditions on files super easy to create. It's in
the bug system somewhere but 16.2 and 16R2 make no difference.) So, yeah,
CALL WORKER is 100% appropriate, a preemptive worker is 100%
appropriate...and the ability to break the whole think with a simple
mistake makes it too dangerous to use.

In fact, that's my conclusion about preemptive mode now: Not ready for
prime time. My hat is off to 4D for trying on this as making a language
multi-threaded isn't a small thing. But I've found it just way too easy to
break so far. (I've tried a few times and ran into problems quickly 

Re: Preemptive mode flaw

[David Adams via 4D_Tech]

> Not a flaw… don’t do this… you know you are not supposed to,

I did it on purpose to see what happens. I'm fanatical about checking
inputs and preconditions and wanted to see what sort of error to trap for.
In many cases, there is no error trapped - it just screws up.

This scenario is *way* to easy to produce in the wild to simply leave it up
to developers to screen their own code. I objected to EXECUTE for a very
long time because of this very reason. Unsafe at any speed. I only started
using it under tight regulation once I could check that a method exists
before running the code.

This situation would be 100% avoidable with as lightly different design for
CALL WORKER. Instead of appending code to the workers chain of execution,
pass a message. (CALL WORKER calls are not messages in any normal use of
that word in computing.) Pass a message, let the developer collect it and
decide what to do. At that point, it's on us. As it stands, it's on them as
they haven't provided 1) proper intercept tools to screen the call before
it is executed, 2) a reliable or consistent error handling mechanism to
trap the error.

> I was told it will produce a runtime error

Sometimes it produces a runtime error, sometimes it doesn't. Sometimes it
kills the process that then restarts.

On Sun, May 7, 2017 at 12:37 PM, npdennis  wrote:

> CALL WORKER("Preeemptive_Worker";"Display_Message";"Hello world!”)
>
>
> Not a flaw… don’t do this… you know you are not supposed to, I was told it
> will produce a runtime error
>
> This is the same as
>
> C_LONGINT(myVar)
> EXECUTE FORMULA(“myVar:=\”Hello\””)
>
> It will compile just fine, but throw a runtime error.
>
>
> --
> Neil Dennis
> 4D Developer since 1990
>
> GreaText - Designing Software for the Way You Work
> 716 East 1850 N
> North Ogden, UT 84414
>
> mailto:npden...@greatext.com 
> http://www.greatext.com/
>
>
>
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**