Re: Native 4D code to convert numbers into text?

[David Adams via 4D_Tech]

Thanks!

On Wed, Jun 17, 2020 at 1:08 AM Keith Culotta 
wrote:

> This was written for v6 a while back.  The app is still being used believe
> it or not.
>
> =
>   // METHOD: DtoS - Dollar to String - kc 3/1/00
>   // INPUT: real - dollar to convert to string
>   // was -> $0:=String($1;"$###,###,###,##0.00")
>
> String(200;$0)
> _O_C_INTEGER($i;$sec)
> C_REAL($partVal)
>
> _O_ARRAY STRING(12;mag1;20)  //  this should be be declared elsewhere
> _O_ARRAY STRING(12;mag2;8)
> _O_ARRAY STRING(12;mag3;4)
> mag1{0}:=""
> mag1{1}:=" One"
> mag1{2}:=" Two"
> mag1{3}:=" Three"
> mag1{4}:=" Four"
> mag1{5}:=" Five"
> mag1{6}:=" Six"
> mag1{7}:=" Seven"
> mag1{8}:=" Eight"
> mag1{9}:=" Nine"
> mag1{10}:=" Ten"
> mag1{11}:=" Eleven"
> mag1{12}:=" Twelve"
> mag1{13}:=" Thirteen"
> mag1{14}:=" Fourteen"
> mag1{15}:=" Fifteen"
> mag1{16}:=" Sixteen"
> mag1{17}:=" Seventeen"
> mag1{18}:=" Eighteen"
> mag1{19}:=" Nineteen"
>
> mag2{1}:=" Twenty"
> mag2{2}:=" Thirty"
> mag2{3}:=" Forty"
> mag2{4}:=" Fifty"
> mag2{5}:=" Sixty"
> mag2{6}:=" Seventy"
> mag2{7}:=" Eighty"
> mag2{8}:=" Ninety"
>
> mag3{1}:=" "
> mag3{2}:=" Thousand"
> mag3{3}:=" Million"
> mag3{4}:=" Billion"
>
> $theNum:=$1  //  Num(Request("a Number"))
> $Target:=String(Trunc($theNum;0))
> $TLength:=Length($Target)
> $sec:=1
> $theOutput:=""
> While (Length($Target)>0)
> $partVal:=Num(Substring($target;Length($target)-3+1;3);$sec)
> If ($partVal=0)
> $theOutput:=" "+$theOutput
> Else
> $theOutput:=DtoS_Get3
> (Substring($target;Length($target)-3+1;3);$sec)+mag3{$sec}+$theOutput
> End if
> $target:=Substring($Target;1;Length($target)-3)
> $sec:=$sec+1
> End while
> $cents:=String(Round(Round($theNum;2)-Trunc($theNum;0);2);"#.00")
> $cents:=Substring($cents;2;2)
> $theOutput:=$theOutput+"Dollars and "+$cents+"/100"
>
>
> $theOutput:=Replace string($theOutput;"  ";" ")  // extra spaces may
> appear for "000"
> $0:=Replace string($theOutput;"  ";" ")
>
>
> =
>   // METHOD: DtoS_Get3 - return words for a 3 digit section xxx,XXX,xxx
>   // INPUT1: string - 0..999
>   // INPUT2: integer - what 3 digit section are we in ...3,2,1
>
> _O_C_STRING(3;$1)
> _O_C_STRING(200;$0)
> _O_C_INTEGER($numValueR2;$numValueR1;$numValueM2;$2)
>
> While (Length($1)<3)
> $1:="0"+$1
> End while
>
> $numValueR2:=Num(Substring($1;2;2))
> $numValueR1:=Num(Substring($1;3;1))
> $numValueM2:=Num(Substring($1;2;1))
> $numValueL1:=Num(Substring($1;1;1))
>
> If ($numValueR2<20)  //  handle the rightmost 2 digits
> $0:=mag1{$numValueR2}
> Else
> $0:=mag2{$numValueM2-1}+mag1{$numValueR1}
> End if
>
> If ($numValueL1>0)
> $0:=mag1{$numValueL1}+" Hundred"+$0
> End if
>
> Keith - CDI
>
> > On Jun 16, 2020, at 4:48 AM, David Adams via 4D_Tech <
> 4d_tech@lists.4d.com> wrote:
> >
> > I'm looking for code that converts something like this:
> >
> > 24120.10 into something like this:
> >
> > twenty-four thousand, one hundred and twenty point one
> >
> > The hope is to emulate Python's num2words.
> >
> > I figure anyone who had to write a check printing feature years ago might
> > have something along these lines, and may be willing to share. I'm not
> > trying to print checks, that's just an example.
> >
> > Definitely after native code, hoping to avoid a PHP call.
> >
> > For reference, Dani Beaubien sent me a link to an old post from Garri
> Ogata
> > with something like the code I'm asking about:
> >
> >
> >
> http://4d.1045681.n5.nabble.com/Number-To-Text-native-4D-code-td4864780.html
> >
> > I also found what looks like a version of that code here:
> >
> > https://kb.4d.com/assetid=76423
> >
> > This version has a bunch of extra ) characters, but runs once you fix
> them.
> > However, this code does *not* include decimals. I'd like to keep the
> > decimals part.
> >
> > Thanks for any help.
> > **
> > 4D Internet Users Group (4D iNUG)
> > New Forum: https://discuss.4D.com
> > Archive:  http://lists.4d.com/archives.html
> > Options: https://lists.4d.com/mailman/options/4d_tech
> > Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> > **
>
>
**
4D Internet Users Group (4D iNUG)
New Forum: https://discuss.4D.com
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Native 4D code to convert numbers into text?

[David Adams via 4D_Tech]

I'm looking for code that converts something like this:

24120.10 into something like this:

twenty-four thousand, one hundred and twenty point one

The hope is to emulate Python's num2words.

I figure anyone who had to write a check printing feature years ago might
have something along these lines, and may be willing to share. I'm not
trying to print checks, that's just an example.

Definitely after native code, hoping to avoid a PHP call.

For reference, Dani Beaubien sent me a link to an old post from Garri Ogata
with something like the code I'm asking about:


http://4d.1045681.n5.nabble.com/Number-To-Text-native-4D-code-td4864780.html

I also found what looks like a version of that code here:

https://kb.4d.com/assetid=76423

This version has a bunch of extra ) characters, but runs once you fix them.
However, this code does *not* include decimals. I'd like to keep the
decimals part.

Thanks for any help.
**
4D Internet Users Group (4D iNUG)
New Forum: https://discuss.4D.com
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Defining and documenting Objects

[David Adams via 4D_Tech]

Just went back and looked at the thread. And, again, I see that there's
huge amount of confusion and murk about what constitutes a "global"
variable. Flow of control, type, and scope are all basic language design
features. Scope in 4D ispeculiar. Scope is typically tied to a
collection of methods that share access to a variable. 4D absolutely,
positively does not have anything like that. It doesn't have any real
notion of data hiding. That's the problem with "globals".

4D has a *runtime* concept of scope where the scope boundary is a process.
That's hard to explain to most people. It's also not something that can
easily be tested in code at rest since it depends on the runtime context
and exact path of execution. It's an inherently unsafe approach compared
with well known alternatives. "Modular" programming is a concept from the
1970s, it's nothing new. And it came out of both best practice, the pain of
project failures, and some really systematic thinking and research.

The ideal goal of scope is to limit the number of methods that can directly
touch an item and the lifetime of a variable. Heck, there are languages
without even the possibility of global variables. One of the thing that
drives people crazy about JavaScript historically is the lack of block
level scope. (Not a concept we have in 4D, nor that really makes sense in
4D.) 4D's feature set is actively antagonistic to anti-global scoping:

* This is why *SET PROCESS VARIABLE* and *VARIABLE TO VARIABLE* are evil.

* With pointers to locals that can escape outside of the current method,
you've now got a hard to figure out level of scope.

* Double for 4D's reference-based dictionaries (*C_OBJECT*.)

The notion that process variables aren't globals and that inter-process
variables are globals is wrong. They're both global, but with a specific
runtime-only boundary called a process. It's a semi-porous boundary that
you can violate without any control at any time with *CALL WORKER/CALL FORM*,
*VARIABLE TO VARIABLE*, and *SET PROCESS VARIABLE. *Alternative design
include benign messaging and function calls. Also nothing new, decades old.

And, as Tim was saying, don't worry about the tiny memory and speed issues,
if there are any. I couldn't measure them in V3 or V6 and doubt that you
can now. Programmers are always guided by their own worst experiences, and
the team developing the server-side of 4D *has different experiences than
we do*. They worry about edge cases. A lot. A whole lot. I'm entirely
sympathetic to this, they really do need to worry about edge cases. Do we
need to worry about the rare cases? Only when we have them. I have them at
the expected frequency. Namely, almost never. It's just silly to design to
edge cases when you don't have to.

With that said, the new *Form* function is *excellent*. Really outstanding,
it's easily one of the greatest things I've seen in V17. I'm also happy to
see 4D finally come around on this. A day nearly 30 years ago now has
always stuck in my mind because it was so awful. Let's just say that the
Engineering team back then was *really* hostile to the idea and lost the
most knowledgeable programmer I ever worked with as a result. Glad they've
seen the light. And thanks to Cannon Smith for the feature request that
seems to have been the genesis of the feature. Oh Canada!
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Defining and documenting Objects

[David Adams via 4D_Tech]

Keisuke,

Thanks for the quick response, it's always great to hear from you.

Yes, I think you've described exactly the situation. I've got a collection
which is tied to a listbox.
I've got an automatic selected object.
I've got an object display bound to the selected object.
I change the source value in the collection and the binding doesn't fire
all the way through.

Here's where the master data lives:

collection.selected_object.element

The item I've modified via code isn't displayed in the listbox, it's
displayed in a bound variable. The bound variable isn't updating, that's my
problem. Is that the bug you're talking about being fixed? For the
listboxes themselves, I've been calling *REDRAW* and it works well. I've
got a multi-page form and if I switch to a different page and back, the
bound variable redraw is done.

I guess I could update the contents of my bound variable directly but,
honestly, that kind of subverts the whole binding system. Then it's really
unclear what the real data is...plus I don't want my code to have to know
about binding subscribers.



On Fri, Oct 12, 2018 at 3:43 PM Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Is this about a cell in a collection listbox?
>
> the reason why a redraw needed to be forced (Form.col:=Form.col)
> was related to how ref. counting works,
> in particular,
> when you have an expression defined as the "current item" property.
>
> the current item object can be referenced from another object on the same
> form,
> in addition to the listbox cell itself.
> so when you update that object it by code,
> there was an uncertainty as to whether the original object is meant to be
> updated to retained.
>
> anyway, it should be fixed in ACI0098483
>
> https://bugs.4d.fr/fixedbugslist?branch=17
>
> 2018/10/12 13:35、David Adams via 4D_Tech <4d_tech@lists.4d.com 4d_tech@lists.4d.com>>のメール:
> I was looking for the an answer to "why don't my object variables on
> fields automatically redraw when changed?"
>
>
>
> **
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: SEND PACKET in a multi-process environment

[David Adams via 4D_Tech]

I've written a lot of multi-process log writing code in my career. A lot.
Enough to run into edge cases and performance issues of various sorts that
are irrelevant to most. The standard way to implement a multi-process log
goes like this:

* Have one log writer process/thread.

* That process is the only thread to touch the file, so zero contention.

* Use an interprocess message bus (of whatever design) to pass log messages
from your various threads to the log.

So the file IO is limited to one location and you don't have to open/close
the file all the time. The performance difference is ridiculously large,
but arguably the main benefit is avoiding file contention hassles.

There are several ways to to the interprocess communications in 4D:

* NTK IPC channels. Works great, has always worked great. Not multi-core
because it's a plug-in. If anyone cares.

* IP arrays with a semaphore or mutex. Can be 100% reliable but,
potentially, kind of expensive.

* *CALL WORKER*, if you don't care about losing a few entries.

That final one seems like a great solution, and it might work perfectly in
V17 In V16, it wasn't thread-safe and you could blow up your server just by
rolling a log file. I submitted this bug with a small sample database early
in the V16 product line and it's reportedly fixed in V17.0, but I haven't
confirmed that. This might be the same bug David is hitting? Or a closely
related one? If so, V17 is worth a try, even if just as an experiment.
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Defining and documenting Objects

[David Adams via 4D_Tech]

I don't follow the list any longer, but I do check in to look for answers.
Today I was looking for the an answer to "why don't my object variables on
fields automatically redraw when changed?" Didn't find it, does anyone have
a workaround? While I was checking on my question, I ran into this massive
thread on objects. As it turns out, I've been working in this area for a
couple of weeks and have a few thoughts to share. I haven't read the whole
thread, so apologies for repetitions. I don't have time for this post...but
here it is anyway. If you care about this issue enough to work on it,
you'll find a bunch of solid ideas in the TL;DR version. I don't have the
time to edit it into something shorter and better, but here is a very short
version:

* 4D's new system leads to very easy to break, brittle, hard to correct
coding problems. By default.

* You can use tables instead of objects. But do you want to?

* 4D dropped the ball on supporting complex types, so we each have to do
all of that work on our own.

* *JSON Validate* can help. Some.

* It takes quite a bit of work to write a custom type definition and
validation system but, if you need quality code, you don't have a better
alternative.

* Use constructors!

TL;DR version:
First off, some background about types. There are only a few very core
issues in language design...you could have spare fingers on one
handtype is one of those core questions. What we've got in 4D:

* Primitive types.
* Arrays of primitives.
* Dictionaries/associative arrays.

We do *not* have compound types nor do we have objects. *C_OBJECT* is a
dictionary, not an object. We need dictionaries, so that's great, but the
name is misleading to many. Anyway, what difference does it make? Well, a
compound type has many, many advantages. It's a collection of elements into
a distinguishable format with *rules*. 4D's tables are the closest thing to
this we've got. But there are tons of times when you really don't want a
table, you want a pure, in-memory structure. Or at least I do. And every
other programmer in history does. Just saying.

What difference does it make that we don't have compound types (structs in
C, records in Pascal, whatever you prefer to call them?) It makes a huge,
huge difference. Here's some of what compound/complex types are good for:

* They're *types *(formally defined data structures) so you can pass them
around and get an interpreter, linter, or compiler to complain if you pass
the wrong type. You know how 4D uses longints for a zillion things, like
window refs, process ids numbers, etc. With a custom type, you could have
windowRef, or processId as distinguishable types. Pass in some rando number
to a method requiring a window reference and get an error from the
interpreter or, depending on how it's all coded, the compiler. I used a
simple type (longint) here as an example, but the same applies to more
complex structures.

* Type can embed rules right into the structured format. In a sense,
objects are complex types with behavior bound in. In a simpler case, you
can add ranges to elements - like "only allow numbers from 1-3" or "only
allow the strings 'Mon','Tue', or 'Wed'", force reals to round to a certain
number of decimal places, etc., etc. Again, 4D tables are the closest thing
a pure-4D programmer will be familiar with. And the field-level constraints
in 4D are pretty painful to use (required, etc.) because of how error
handling is implemented in 4D...so many people don't use field-level
constraints. I don't.

Okay, so we don't have types...how much difference does it make in V17. It
makes a huge difference, as people have clearly noticed already and are
exploring on this thread. I like the new dot notation, but there are some
massive issues with names. Such as:

* If you add a name to an object, the element is added. What if it's a
typo? You get a new element. No error. This is described as a "feature".
Well, sometimes it is a feature, sometimes it's a bug. There's no "strict
mode" choice. That's a missing feature in 4D. Or a design flaw, call it
what you will.

* Case-sensitive. I just can't say how stupid I find this. Has anyone in
history *ever* found this useful? If so, get them away from the keyboard
and put them into management or something where they can do less harm. Pet
peeve. Tabs not spaces people! Spaces, who does that? But I digress from my
digression. 4D says that this is to be "more compatible with JavaScript."
Why? I can't think of any reason. I've spent enough time with JavaScript to
really enjoy some of its features (closures!, some of the
messaging/delegation, the range of normal operators...man 4D needs ++,
etc.), but I've never heard *anyone* outside of 4D point to JavaScript as
an example of what a language should look like. Quite the opposite.

* So, an "object" can have anything in it. Or not. It's a completely
unfixed format. Sometimes that's what you want. Most of the time? I can
hardly think of *any* time I want a 

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

[David Adams via 4D_Tech]

Just for the record, my 160x speed improvement came with pretty naive code.
I loop to a record, build the text I need for that row, append the block to
the BLOB, clear the text and hit the next row. All-in-all, only a couple of
lines of extra code.

After what everyone has contributed and suggested, along with my own local
testing, my summary goes like this:

*Build a text var*
Yeah, don't do that. In my case, the final block size is ~3MB, not
huge...but the resizing is pretty aggressive.

*Build a BLOB var*
Works great. The downside is that if you ultimately need text, you (even if
briefly) need to double the amount of RAM you're consuming. I'm defaulting
to a max BLOB of 5MB and change, so this should never be a problem.

*Write to a text file*
This should never slow down, no matter how big the file gets. This has
always been true, the limiting factor being the speed of the drive. Drives
are fast now. Memory is also fine because you take no RAM to store the
document and can load it into RAM in one go very easily.

I'm going with a BLOB instead of documents. Why? Old prejudice, and I like
avoiding the file system. Permissions, file contention issues in 4D (been
there not long ago...and after *more than a year*, there's reportedly a
fix), data leakage. Plus, old prejudice. In my specific case, a BLOB should
be fine as I'm not pushing the envelope on memory usage. Under other
circumstances, I can easily see documents as being the best bet.
Particularly if you have other reasons to want the data in documents for
dispatching, archiving, etc.

Thanks for the help!
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

[David Adams via 4D_Tech]

Thanks to John for adding writing to disk to Dani's test set. You make some
good points about documents in the modern world. You can probably append to
a document all day long and never see any change in speed. Years ago, the
legend was that you didn't have to buffer before calling *SEND PACKET* because
4D was doing it already. Meaning also that you had to close/open the
document it you were writing a log to track down crashes. No idea if this
is true now.

And, yeah, then you can *DOCUMENT TO TEXT* in one go, it that's what you're
after. But, and I don't know if this is just me or not, but I hate writing
scratch files. Just...don't like it. I can't back that up with any good
reasons, I just have a bit of an allergy to the file system. Permissions
issues, etc. Maybe I should rethink that. Another nice feature of files is
that you can use them in so many ways...they can then be read by another
program for dispatch, etc. So, hmm.
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17

[David Adams via 4D_Tech]

Dani,

Thank you, for a very large value of "thank you." I woke up this morning
all excited to write some tests, and you had already done the work. It's
really great that you took the time to do the tests, to share your
conclusions and your test method. Thank you!

In my case, I'm not so kind or generous ;-) I've got a specific method that
I'm using for testing. I pass in a method and it does the work. This is a
fixed data set, so the results are always identical. The only thing I'm
changing is how I'm building up the block of text. I posted about the four
method I tired and they were all the same, about 6 minutes. I went back,
added a new test for *SET TEXT TO BLOB* in the loop and then one call to *BLOB
to text* at the end.

The results? What otherwise take about 6 minutes now takes just over 2
seconds. I had to run it a few times to convince myself I wasn't imagining
things. I wasn't. I thought that I was past the point where I'd ever see a
100x (well, x160 in this case) improvement from a small code change.

So thank you all for this information, it's a game-changer for me!

Onto the other point, memory. A Dani commented, if you need text then you
have to briefly double the RAM consumed. In my case, I need text. (The text
is passed to a plugin that doesn't accept BLOBs in this situation.) What do
people thing about RAM these days? I'm still kind of conservative. Tim
Nevels (off-line) said to me the other day, "David, why are you worrying
about memory? Everyone has tons of memory." I'm already not worrying about
performance in advance, and I *don't* feel like spending more time worrying
about security, so can't I worry about memory a bit? Where do you guys
stand on this on modern gear? When was the last time you bumped into
harmfull memory constraints?

Thanks
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

[David Adams via 4D_Tech]

Bernd,

Thanks for the report, it's interesting and believable. In this case, I
have to get the data into memory to send it... I guess I could write to a
file, load that and then send it. Interesting concept, thanks!

On Mon, Sep 17, 2018 at 4:53 PM Bernd Fröhlich via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> David Adams:
>
> > I'm checking for the simplest, least difficult default as that should
> work in the vast majority of cases.
>
>
> Here are my two Eurocents:
> it does not matter how you collect the data, but concatenating the text in
> memory vs. writing to a file makes a HUGE difference when you have lots of
> records.
>
> I have a method in my toolbox  that writes out a selection or a whole
> table into a file.
> Looping through the records and appending each record to a file.
> Works great.
>
> Then I did the same but instead of writing to a file, I appended to a
> textvariable.
> Works well for small amounts of data but is unusable for more than about
> 2 records as it slows down ever more.
> (I am lazy and did not do any pre-sizing, chunking, whatever.)
>
> So if you want to export the data anyway and look for the simplest way,
> then just loop through the data and write into a file.
>
> Greetings from Germany,
> Bernd Fröhlich
> **
> 4D Internet Users Group (4D iNUG)
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

[David Adams via 4D_Tech]

On Sun, Sep 16, 2018 at 9:09 PM Olivier Flury via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Not really an answer to your question but a more general observation:
>
> Very often, when I have to do an export function towards other systems, I
> end up in doing a bulk export.
>
> Either because I am lazy or the other side is lazy or both sides are lazy.
> It's easy, just send everything. But sometimes it is worth, doing more
> homework, on both sides, and export only what has changed since the last
> export. Sometimes this is simply not possible (too complex, the other side
> wants all data etc.), but very often it would be. And it would - most of
> the time - result in a drastic reduction of amount of data to exchange.
>

Well observed. In this case, I have the good fortune to be working with
source tables that include modification counters. (Think "sequence numbers"
on each change.) This makes it *incredibly* easy to find all records that
have been modified or added since the last push.

For deletion, records in 4D are marked with a deleted date. I've always
found this practice annoying and weird...and boy am I glad it's being used
in this. With the delete date field in place (soft deleted records), it's
simple to handle delete reconciliation. I don't push the delete date field,
just a Boolean based on it. So, you push everything up including the delete
flag, then a simple DELETE FROM table WHERE marked_for_deletion = true
takes care of it. Amazingly easy. Welsh Harris and Tim Nevels get all the
credit for setting things up this way in advance.

Even still, there are sometimes still millions of records to move,
particularly when we're first pushing a table over. In most cases, pushes
are quite manageable in size on a day-to-day basis. These tables are
getting pushed largely for aggregation and analysis, so it's not one-off
operations. Push and keep pushing as more data flows through 4D.
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

[David Adams via 4D_Tech]

Hello people, thanks for all of the suggestions and feedback. Several of
you pointed out that my thinking was in the fuzzy-to-nonsense range. I hate
you all. So much. Not really, the comments were completely on the mark. I'm
trying to find a decent default option that will work over a wide range of
conditions. Some tables are tiny, some are very large. Some pushes have a
lot of data, some have vey little. Most machines are up on AWS, a few are
not...but have GB fiber connections. So, there are variablestoo may for
a best-fit solution for every case. I'm checking for the simplest, least
difficult default as that should work in the vast majority of cases. If
something does need help, then I can get in an optimize it, as required. I
have one case like that already and it was a total bear to finish.
Sometimes you get the bear, sometimes the bear gets you...

I was doing some chores today which gave me a chance to run tests and leave
my computer alone while things ran. What I wanted to test out was nothing
but the behavior of creating my big text object. What happens with it after
that isn't relevant. The text block built is identical on each run.

So, I did milliseconds before & after and memory stats before & after. Then
looked at the results. I ran compiled, single-user, macOS 10.12.6 with 16GB
or RAM and a 3GB cache. This is a MacBook Pro from the past couple of
years...one of the ones with the shitty keyboard. Man, that thing really
makes me less productive.

Back to the tests.  I ran compiled only. I launched, ran a test, got the
results, ran the test again, got the requests and quit. Just because that's
how I normally test stuff, at a bare minimum. I didn't reboot. My test
sample pulls five smallish fields from about 30K records in a table with a
lot more fields. The methods for navigating the data are what differ. The
four versions that I checked are:

* *GOTO SELECTED RECORD*
* *SELECTION TO ARRAY* + *For* loop
* Entity selection + *For each* loop
* Entity selection + *For* loop

I'll start with the speed report as it's short: It doesn't matter. The
performance range for the four solutions on their two runs was so small
that I chalked it up to normal variability. So, *speed is not an issue*.
This leaves memory, ease of development, and stability/brittleness. For
memory, I'm not terribly confident in the numbers. The first run values are
confounded by everything else that might be happening with the first run
against that data, so I've ended up ignoring those results. The second run
results confirm what Keith said. Namely, that *SELECTION TO ARRAY* takes
more memory than *GOTO SELECTED RECORD*. Kind of has to. On the two entity
selection results, they seemed to use more memory than *GOTO SELECTED
RECORD* and less than *SELECTION TO ARRAY*. Or maybe not, I'm not testing
more. I don't want to start any superstitions here. I will say again though
that if you care and want to test it, forget first principles. Even if you
had and understood the 4D source code, you would have very poor chances of
guessing how things would behave by the time you're running your code. If
anyone does some testing, I'm sure there's an appetite for the results.

Okay, that's it. Not the kind of results that you can broadly apply. There
are just so many factors involved. For example, the memory profile on
*SELECTION
TO ARRAY* grows as you add more and more arrays. It has to, and you can
even calculate the costs pretty well. The difference for *GOTO SELECTED
RECORD* is likely very little. I only tested on a machine that runs the
engine, Client would likely show dramatically different behavior.

Regarding the entity selections, that was just out of curiosity. I won't be
using them here. They don't offer a speed or memory advantage over *GOTO
SELECTED RECORD* and they make the code vulnerable to breakage. Someone
changes the capitalization of a field, and my code breaks. My server side
code with no UI...Not worth the risk. The code clarity on *GOTO SELECTED
RECORD* and then using field references is *excellent*. It's at least as
good as the readability of the entity-dot versions. But with regular old
field references, there's no risk of weird breakage and the code is
instantly understandable to any existing 4D programmer.
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

[David Adams via 4D_Tech]

Thanks Keith, that all sounds pretty right to me too. And, yes, I'm
chunking the pushes into batches to avoid a truly massive text object, when
necessary. I don't know about with SQL Server and don't remember about
MySQL, but with Postgres it's important to avoid single row inserts where
possible. Postgres is absolutely fanatical about integrity, so each
operation is automatically wrapped in a transaction. Pushing rows
one-by-one can, therefore, be surprisingly slow. So I bundle multiple rows
into a mutli-value insert, and then several of those into a transaction. It
goes pretty quickly and for smaller pushes, it probably doesn't matter how
I build up the text, it just isn't that big. (Note for the PG people in the
crowd, I'm using an UPSERT, so COPY TO isn't an option.)

Regarding the ever-resizing text block, I don't know how that behaves
today. I used to know...and knew the sort of crazy stuff you could do to
improve performance. Like, pre-size a giant block, keep track of the last
byte written and then write into bytes by hand. So, you didn't resize the
block as you went along. You might need a final trim or an expansion...but
you pretty much kept the block as a set size and then write into it by
direct character reference, etc. It's a lot more work and I have no idea
how modern versions of 4D on modern versions of Mac/Win handle all of this.
I have a notion in the back of my mind that all of that kind of byte
tracking stuff isn't too useful any more. I notice that both 4D and
Postgres don't have VARCHAR types internally, it's all variable length
text. Haven't testing anything. Oh, we don't have any double-byte data to
worry about.

Thanks again!
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Do I want to use SELECTION TO ARRAY instead of GOTO SELECTED RECORD server-side in V17?

[David Adams via 4D_Tech]

Short version:
I need to load some fields from records into a big text thingy.

The code runs on the server-side only.

I'm keen to preserve RAM.

What are the trade-offs in V17 between *GOTO SELECTED* record and *SELECTION
TO ARRAY*? I've been using *SELECTION TO ARRAY*, but it's hard to read,
write, and maintain. And, I realized, might be de-optimized for memory
because you have to load all of the data you're processing into arrays.
(Yes, you can chunk it, but that doesn't change the fundamental point that
you pre-load a lot of data.)

Any test results or thoughts? I considered a fair range of option and did
comparison tests on none. The long version below includes more details on
the two solutions I'm down to, plus the ideas that I discarded.


TL;DR version
I'm working in V17 and I'm hoping that someone has done some real-world
tests already that could help me out with a question. Here's the setup: I
need to load up some fields from lots of records and push them into an
external system. It's going to Postgres, but that's not an important
detail, the result is a ginormous text object. The result could just as
well be a text or JSON file dump. The main constraint is available memory.
Performance matters when there are millions of records but, typically, the
only important consideration is memory. As far as the final solution goes,
it's ideally code that's easy to write, read, and maintain. As a plus, we
can position the code to run server side, so client-server optimization
isn't an issue. And, for the record, in lots of cases there isn't enough
data to make memory an issue at all, so readable reliable code is
definitely a preference.

Note: Yes, I can chunk data in ranges, etc. to keep things within my memory
footprint. I'm doing thatbut the question still remains

Here are the solutions I've come up with:

*QUERY* and a *For* loop with *GOTO SELECTED RECORD*.
Easy to read, write and maintain. But when you use *GOTO SELECTED RECORD*,
do you get the whole record in V17? Without fat fields? Since this is
server-side or stand-alone, should I care? On the upside, you're only
loading one record at a time, so only burning through memory for that
record while you use it.

*SELECTION TO ARRAY* and a *For* loop
This is what I have been doingbased on old habits as much as anything.
Yes, you only get the columns you want, but it gets _all_ of the rows at
once. So, you burn up a lot of memory with the arrays and then duplicate++
that memory when building up the output. On the code side, that kind
of *SELECTION
TO ARRAY*-loop-read by index code is ugly, tedious to write, and tedious to
maintain. It's clear(ish) and reliable, but only worth it if it pays for
itself somehow. In other words, it has to be a good deal better than *GOTO
SELECTED RECORD* to be worth it. Says the guy who has been doing all *SELECTION
TO ARRAY* forever.

Entity Selection and a *For* or *For each* loop
I have no clue why an entity selection is *C_OBJECT* instead of
*C_COLLECTION*, to give you a sense of how much I know about this stuff. I
was happy to discover that you can easily create an entity selection from a
current selection, so old style queries work fine:

*C_OBJECT*($stuff_es)
*QUERY*([Stuff];[Stuff]Counter>=1)
$stuff_es:=*Create entity selection*([Stuff])

The resulting *For*/*For each* loop code is very readable, it's == *GOTO
SELECTED RECORD*, but with a different syntax. Otherwise, same same. I
*suspect* that the memory use here is excellent. I'm guessing that as you
navigate through the entity selection, you're only really pulling the data
you use. But maybe not. If you do a For each, you get an object (entity)
with all of the fields. So, possibly this is approach is even worse than *GOTO
SELECTED RECORD* which, I'm guessing, doesn't load as many fields. I
haven't tested these points out in any way. If anyone has dug into this, it
would be great to know about the difference (if any) in what 4D loads when
you:

-- Use *GOTO SELECTED RECORD*

-- Use a *For each* loop on an entity selection, which builds an
$entity_object which you can then read/write to/from like $entity_object.ID

-- Use a *For* loop on an entity selection and then reference
$specifc_es[0].ID

It's pretty easy to imagine different ways that 4D might have implemented
things that are more or less efficient in each of these days. I have no
idea what they actually did.I'm kind of curious about this behavior in V17,
but have already talked myself out of using entity selections. Why? Because
the table and field references are brittle and *case-sensitive*. Man, I
truly hate case-sensitive names. When do I want them? Never. Not once, and
I never will. This isn't all on 4D, many languages are case-sensitive. It
makes sense if you're a computer. I'm not a computer, I'm a person...to me
its just horrible. Anyway, not exclusively a 4D problem...because in 4D you
can avoid it altogether.

For those that haven't been following along at home, here's a hello world

Re: v17 PDF-Manuals

[David Adams via 4D_Tech]

Thanks Wayne!
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: v17 PDF-Manuals

[David Adams via 4D_Tech]

I was just at the 4D site looking for PDF manualsand didn't find them.
So I searched here and found this thread.

I'm going to be away from the Internet and want to take documentation with
me. I expect that I'll have some down time, and figured it might be a nice
time to look at recoding some things as collections. (I'm coming from R3
where object access is still through the OB SET/OB GET...which make me not
use C_OBJECT all that much.)

To be clear, I won't have the Internet. At all. So, is there any way to get
an off-line version of the V17? I don't see a way to download the HTML
versions and don't see any PDF versions. If not, I'll just skip what I was
planning and do something else with the time.

Thanks!
**
4D Internet Users Group (4D iNUG)
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Who’s the Gestapo?

[David Adams via 4D_Tech]

Robert:

* This list is run by 4D.

* I've never heard that about the censoring, and it hasn't happened to me.
Which it probably would have, if it were happening at all. Are you 100%
certain that there's not technical glitch on your people' side?

* I'd suggest *not* comparing people to the Gestapo in general but, in
particular, when they're old enough that their parents can remember the
Gestapo.

On Sat, Jul 14, 2018 at 2:34 PM Robert ListMail via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> I tried to turn on a few young developers to 4D and suggested they join
> the list. They were never able to post. I take it the list is highly
> censored based on the feedback I’ve received. I didn’t know the bar was
> that high. So, who’s the gatekeeper...?
>
> Thanks,
>
> Robert
>
> Sent from my iPhone
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Chromium Embedded Framework in built applications question

[David Adams via 4D_Tech]

Thanks for the answers, I wondered if it might be to do with Blink...seems
pretty heavy if you're not using it. Actually, we might have a couple of
Web areas in there. I wonder if 4D detects that and includes/excludes the
framework accordingly.

Thanks for the R6 suggestion, we can try that out and see what happens.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Chromium Embedded Framework in built applications question

[David Adams via 4D_Tech]

A friend recently pointed out that built apps had unexpectedly increased in
sizea lot. We used to come in at something like 150MB and are now at
over 600MB.

macOS, 16.3 R3

Digging into the ressoures in the built app, this package appears:

Chromium Embedded Framework.framework

...at about 260MB. The compiled database is less than 55MB.

Does anyone know what this Chromium Embedded Framework.framework file is
and how to get rid of it? You can't just strip it out after the build as 4D
complains on launch.

Thanks!
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 7-zip from within 4D

[David Adams via 4D_Tech]

On Wed, Jan 9, 2013 at 4:30 AM, Tony Ringsmuth <ton...@bbsp.com> wrote:

> Wayne,
>
> I use 7zip extensively, driving it from 4D.
>
> What I do is this: After a 4D backup completes, from the On Backup Shutdown
> database event, I initiate an encrypted 7-zip of the backup file, and some
> info files.  After they're zipped, then 4D server ftp's them to several
> remote locations.
>
> It works rock solid.
>
> If you need, I can forward you the crucial code samples.  It's basically
> done through LEP and batch files.
>

Tony,

If your offer of code hasn't expired after more than five years, I'm
looking for exactly this functionality on Windows right now.

Hope you're well,

-- David Adams
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Bug or feature?

[David Adams via 4D_Tech]

Hey, long thread - I was even part of it at some point. I just wanted to
send out a big *thank you* to Jeff Kain for starting it. (And to Chip for
the awesome feature bug picture.)

I'm writing now because I've been banging my head against the desk for a
couple of days trying to get past a problemthe very problem Jeff
reported. I don't want to go through or rehash the whole thread, it's there
in the archives for anyone that cares. But my current issue (16R3 macOS) is
a bit the same, and a bit different. The big difference is that *there is
nothing wrong with the JSON*. It's valid JSON. But 4D blows up on some of
it, some of the time and the *ON ERR CALL* does not check. Jeff's trick
with declaring Is object has at least stopped the blocking error. Now I'll
have to see if the system is actually working.

This has all been a bit slow for me as the error only appears in compiled
mode and only in one method (more on that in a moment.) And there are
nearly 6M records in my test file and over 8M in production. When you blow
up, the process is dead, you're compiled and you can't get anything back to
identify *which* record blew up. So I grafted in some logging code to log
activity before it happens so that I could figure out what the problem is.
I didn't ;-) But at least I can look and see the record.

Tim Penner made a solid point earlier about Booleans in JSON being all
lower-case. I'm wondering if that has something to do with it? Here's a
clause from the JSON:

{
  "name": "is Name Finalized",
  "old": "False",
  "new": "True"
}

Except here's the thing, those are formatted as quoted JSON strings, not
booleans. We don't want booleans, we want strings. These are valid strings.
If we wanted bools, it would look like this:

{
  "name": "is Name Finalized",
  "old": false,
  "new": true
}

Not what we want, but Tim's point is worth keeping in mind. I'm wondering
if without the Is object hint, 4D is getting overly-aggressive about
interpreting "True" and "False" (or "true" and "false") as Booleans? No
idea.

Regarding the "but my JSON is okay!" claim, here's how I checked. The 4D
bits are all in compiled mode:

-- My main big method with an *ON ERR CALL* that doesn't have any
callbacks, current method name, etc. It's either empty or Error:=Error.
This fails, and on the same record(s).
--> This method loads the data into a text array and the value is then
parsed from there, it is not directly parsed as a field

-- A big loop that scans over the same records but only copies the data
into a variable and parse it. *I don't get any errors from the same records*
.

-- Checking an individual record in an input and clicking a button that has
a little parser script. *I don't get any errors from the same records*.

-- Copy the JSON out and putting it into a JSON format window I've got (two
text vars, one in and one out.) The JSON stringify stuff works fine to
pretty print or flatten the JSON.

-- Copy the JSON out and into any JSON tool you like. No problem.

So, it's very weird. 4D blows up on the JSON in one case and no other.

If anyone can suggest why this might be happening, I'd love to hear about
it.

More to the point, should we explicitly set Is object pretty much
everywhere in our code. We have a ton of calls to *JSON Parse* that would
need updates, if this is the case. Is there any  downside to specifying Is
Object to be aware of?

Thanks!

P.S. If possible, please CC me directly in responses on dpad...@gmail.com
as I do not monitor this list.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Warning] Settings properties values on object field by object notation

[David Adams via 4D_Tech]

Thanks to TIm and Brian for responding here, always appreciated.

There is no lack of clarity from 4D that the official channel is to post on
the forums. Before that, it was bugs.4d.fr. That's always been clear. Heck,
I can't even count the number of times I said that on this list over the
years. Of course when I said it, I didn't really think much good would come
from it. I can open up 4D today and point out tons of bugs (cosmetic,
usability, functional, and crashing) that have been known about for years
and versions. So, yeah, reporting bugs doesn't have a lot to do with bugs
getting fixed.

Pardon me for being so simple minded, but what remains completely unclear
is why customer information is ignored. See, that's the issue: You've got
customers sending you information, you know what they say, and you say
"nope, doesn't count." That's about 4D's relationship with your customers,
it's not about your customers doing things wrong. "Posting on the NUG
doesn't count" just feels like a weak cop-out. That 4D doesn't see this, I
just don't understand.

If you guys used any sort of normal, commercial bug-tracking or forum
platform, I think the position "do everything there" would be a lot more
tenable.  But a hidden bug system (which I still have no access to because
of my hemisphere) and a home-made forum system in 2017? You just have to
accept that that's not going to work for some people. For good reason.

I've tried the Forums. While 4D Engineers participate there a little, I
heard nothing form the Product Team. Ever. And from Engineering, I mostly
heard or observed that posts were being put in the "wrong" forum, and that
any sort of design critique wasn't welcome. You can say that a feature is
not working correctly, but that a feature or feature set has a design flaw?
Not a good reception. At all.

The question is, do you want to make 4D better? Do you want to make 4D more
reliable? Yes or no. Blowing off customer input = "no". Making it harder
for customers to provide input = "no". Not providing feedback on bug status
= "no." Not documenting the tools properly = "no". Calling bad design
"standard behavior" = "no." Telling people that its' down to us to track
down a bug that freezes the server and that 4D isn't actively investigating
it on their own? Yeah, that's most definitely "no." Giving people a hard
time for pointing out flaws in the *design* of features, also "no".

Your business, your choice.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Warning] Settings properties values on object field by object notation

[David Adams via 4D_Tech]

> So, if you want to officially request such a change, it needs to go on
the forums.

Does 4D realize that *officially* ignoring their customers isn't exactly an
example of great customer service.

I tried the forums for about six months, I don't think I got _any_
information from 4D Engineering, but I sure did get told I was wrong a lot.
About matters of fact and opinion.

And, to date, I have seen zero evidence that participating on the Forums
does anything at all to forward the cause of bugs or features getting fixed.

This whole business that somehow it's our fault that we're not "doing it
right" just doesn't sell. I mean, 4D can say that - but how many people
does it convince? I doubt it's more than a handful, if even that.The only
obvious interpretation is that 4D doesn't want to be bothered with feature
requests and bug reports (even on the Forums, frankly). Hey, your company,
your rules...but it doesn't make 4D come across as customer-focused.

But, again, it's entirely up to 4D how to deal with their business. But,
please, quit saying that the problems with customer service are the
customers.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Working with a 64 bit Integer in an object

[David Adams via 4D_Tech]

Yeah, if 4D isn't working, try NTK. In either case, if you can get at the
value as raw text then you'll be able to find text --> 64-bit conversion
code. The question in 4D at that point is how to store it. I know it's a
field type, but can you touch it with anything other than SQL?
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Warning] Settings properties values on object field by object notation

[David Adams via 4D_Tech]

> it is an interesting point of view,
> to frame it as an issue of the assignment operator being different,
> as opposed to how objects and their properties are different "because
they are references".

I think that the real issue for a 4D programmer is that a record is a
persistent data store and that field modifications are supposed to be
stored reliably. How 4D handles all of that internally is, literally, none
of our business. (4D refuses to discuss any of this.) The data should be
stored reliably 100% of the time. If that can be done in a more optimized
way, great!
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Warning] Settings properties values on object field by object notation

[David Adams via 4D_Tech]

On Wed, Nov 1, 2017 at 1:05 PM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

>
> I thought it was also quite clear in Tim's reply which was reposted here,
> that we failed you on this, and that we are in the process of updating the
> documentation.
>

If changing a field and then saving the record doesn't save the field
change, that's a bug.

Adding a footnote to the documentation isn't a fix, it's an admission of
failure.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Warning] Settings properties values on object field by object notation

[David Adams via 4D_Tech]

>I am going to post here a response I got from Tim Penner in the TAOW a bit
earlier today. While it does not change my mind that this issue should be
characterized as a bug, it does clarify what the issue really is.

It's a bug. If 4D wants to document it as standard behavior, that's their
business. But it's still a bug and will lead to real-world problems. This
is the kind of bug I'd rank *extremely* serious. Why? Well, imagine any
number of data change scenarios some of which change the field on one way
and some in another. Depending on the exact sequence of steps the same
modification will result in putting the data into a different state. This
is exactly the sort of situation that can run undetected for
days/weeks/months in the field. And then what do you do? There is literally
no way to unscramble the eggs. You have *no idea* what quality your data is
at that point. It has any number of undetectable, subtle flaws. The number
may be zero, the number may be .0005% of the candidate fields, it may be
7.23% of the candidate fields. Unless you've got source data to completely
rebuild everything from scratch, you'll never know and never be able to fix
it.

Object fields sound better and better to me all the time ;-) They're
non-relational, accessed through a "novel" and incomplete JSON
serialization scheme, take a lot of space (the same data in primitives
takes less room), have a bunch of adjacent/extra commands devoted to
working with themand, I'm told, are *not* designed to be used for the
majority of cases where JSON fields are commonly used in ORDBS and NoSQL
systems. I just don't get it.

Anyway, modifying and saving a field of this sort should be
*deterministic*. Making it functionally non-deterministic isn't defensible
for a RDMS in any way.

 Input --> [Function] ---> It depends

That's fine if you want something random, it's not fine in any other case.

It's a bug. But, again, 4D is free to say it isn't. Their business, their
rules.

> Am I the only one that was clueless that there is a field dirty field
flag that controls whether or not a field gets updated or not during a save
record. Guess that does make sense.

For a long time you would bump up against this on large data types and the
Old() function. That's about the only practical way I remember it being an
issue for us.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Warning] Settings properties values on object field by object notation

[David Adams via 4D_Tech]

On Wed, Nov 1, 2017 at 10:23 AM, Tim Nevels via 4D_Tech <
4d_tech@lists.4d.com> wrote:


> I can guess the answer to that. The engineer responsible looked at the
> code and said, “yeah I can fix that, but it will be a lot of work”. And
> somebody said, "OK then don’t bother. You’ve got more important things to
> do. We’ll just call it ‘Standard Behavior’”. Pitiful! :(
>

We'll never know. It could be that it is the result of a carefully thought
out design.

Then again, you could have a function like Add(2;2) that returns 5 and
claim that it's "standard behavior" because it's consistent. No matter
what, it's still a bug (except for sufficiently large values of 2.)
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Warning] Settings properties values on object field by object notation

[David Adams via 4D_Tech]

> It looks like this bug ACI0097454 was marked as Standard Behavior on
October 24th.

Smells like a bug.
Looks like a bug.
Sounds like a bug.
Walks like a bug.

I mean, if you have a light switch that gives you a shock every time you
touch it, you wouldn't say "Oh, it always does that. It's standard
behavior." I mean, that's an accurate description (ouch!), but it's not a
reasonable reaction.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Warning] Settings properties values on object field by object notation

[David Adams via 4D_Tech]

On Tue, Oct 31, 2017 at 8:21 AM, Alberto Bachler via 4D_Tech <
4d_tech@lists.4d.com> wrote:

Just a quick note of thanks for posting this behavior to the list.

On Tue, Oct 31, 2017 at 3:49 PM, John Baughman via 4D_Tech <
4d_tech@lists.4d.com> wrote:

...and to you too John for posting that you also got an official response.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Warning] Settings properties values on object field by object notation

[David Adams via 4D_Tech]

On Tue, Oct 31, 2017 at 3:49 PM, John Baughman via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> And, yes, I too was told that the engineers were leaning to declaring this
> as standard behavior. Why? I have not a clue.
>

I'm failing to come up with any good arguments in support of such a
posture.

If you make an assignment to a field and save the record, the assignment
should stick. Nothing else makes sense from a DBMS perspective. That's
exactly the sort of behavior that should be absolute and unquestionable.
(The only exception being that you have to be in a position to write any
changes to the record.)

I hope that they come in off the ledge and don't create a buggy feature
that will lead to some really, really hard to detect bugs in the field for
years to come.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Method to put a c-obj (or JSON) into a hierarchical list

[David Adams via 4D_Tech]

Jeff,

Now that's a smart idea. I'm interested.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: info.plist

[David Adams via 4D_Tech]

I've been working with BUILD APPLICATION a lot lately, so a couple of notes:

* 4D _will_ record an error in the log if the path to the license files is
wrong or, for a built single-user app, the path to the engine is wrong.

* 4D will _not_ record an error if the path to your custom icons is wrong.

* It's not that hard to clear out extra resources after a build, like .xlf
files and so on.

* 4D has a tech tip about clearing out the SVG component help file which
has a helpful description of the file paths for compiled databases and
merged apps.

* Go ahead and clear out the /Cache folders from within Components,
Plugins, and Resources after the build. They can contain 100s of MB of
stuff and you don't need it.

Check the archives for some past tips and code from Tim Nevels on tweaking
the info/about details. There are several different build keys you can use:

Tip: I also like to onboard a small build_info.json file after the build.
It's got details that I might want to read out about the build itself for
error logs, about boxes, error reports, etc. This isn't part of the
standard build project file but it's easy to implement.


I'd be very interested to hear what other people clear out post-build.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Snippet] Process_IsPreemptive

[David Adams via 4D_Tech]

Hello! The 1980s called, they want their naming conventions back! ;-)
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

PROCESS PROPERTIES origin = -29

[David Adams via 4D_Tech]

I've just discovered an unexpected process origin value of -29. It's not
any any version of the docs that I can find. Although hunting for a -term
is a challenge.

Anyway, it's coming up for the Compiler window process. Has anyone else
seen any other use of this origin code?

Thanks
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: [Snippet] Process_IsPreemptive

[David Adams via 4D_Tech]

Yeah, well, the code above doesn't work because of what the kids call a
"bug".

PROCESS PROPERTIES(Current process;$name;$state;$time;$mode)

Should (obviously) be

PROCESS PROPERTIES($process_id;$name;$state;$time;$mode)

And that's what I get for *not* writing unit tests for this "simple"
function. I tend to screw up simple routines at a way higher rate than more
complex routines. Weird, but that's how it is for me.

I also noticed that 4D late published a Tech Tip for the same functionality:

http://kb.4d.com/assetid=77860

Their code looks right. Grab theirs, if you prefer their variable naming
style.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: POP3 Email Processing (and ramblings by Tim Nevels on 4D)

[David Adams via 4D_Tech]

On Sat, Oct 28, 2017 at 3:21 PM, Tim Nevels via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> On Oct 27, 2017, at 8:11 PM, David Adams wrote:
>
>
Tim, I wasn't really making a broad comment of any kind about 4D or
criticizing - just citing a bit of the history I was there for. The
original programmer was Phil Weiss, nice guy. But I do remember how shocked
4D was when they got the code and discovered that it was not re-entrant.
Then there was ITK, which was really kind of great (again, a single
programmer  - Christian Quest?- the same person that did some of the
network layer pieces back when TCP/IP was just an option). But that
disappeared...or sort of withered on the vine. And then NTK, which I like a
lot.

All three of these tools were written by individual programmers and they
didn't spend years on them. Rob could say something about how long the
4DIC-like features of NTK took to write, not that I expect him to. (No
reason he should.). And he's got a cross-platform C++ framework as well, I
believe. Paul Carnine wrote a multi-platform framewor too - but with
support for even more platforms - for his award-winning video software.
(That's where he went, if I remember correctly, it had nothing to do with
4D.)

If 4D comes up with a native replacement 4DIC, great! For example, I'm
really happy that they've got HTTP Get and HTTP Request in the language,
very helpful.

But waiting for a future feature that hasn't been announced, shown, put
into a beta or released? Something that might or might not happen, and who
knows when? And who knows how it will work or what it will do when/if it
arrives? That's not something useful to me today or something that I can
plan around. If I need a feature, I already need it. So I have to work with
what we've got. Once there's a native and non-blocking way to do email in
4D, great! If they've got it and I need it, I'll check it out.

Until then, there are stacks of could-based mail services with nice APIs.
That's where I'd be looking, particularly since HTTP Get and HTTP Request
seem to do a decent job with no real downsides.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: POP3 Email Processing

[David Adams via 4D_Tech]

> It blocks the whole application while waiting for and downloading
messages from the server. This is because the plugin interface of 4D is
designed
> to be in cooperative mode. Even switching to v16 and using preemptive
threads will not help, the plugin interface isn't thread-safe.

Plug-ins can yield time (in two flavors) to the scheduler, so they don't
*have* to be completely blocking. The original PDM code was not re-entrant
and that's legacy seems to have been haunting us for the 25 years since 4D
took it on.

Plug-ins are not thread-safe in 4D, period. Will that change? *Can* that
even change? That's above my pay grade...not a clue. Maybe Rob, Miyako or
someone from 4D could comment. So far, I've heard nothing on this important
question.

What Kirk says. He's using a watch folder system which, now that I think
about it, is a fantastic idea in his setup.

You can use HTTP Get/Request in a non-blocking way, so that's *always* a
solid possibility for external and internal communications.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Tab bar in methods

[David Adams via 4D_Tech]

Nice!

Thanks John.
Thanks Tim.

Much appreciated.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Tab bar in methods

[David Adams via 4D_Tech]

I've been noticing a "bug" for months in the Method Editor. This might just
be an R-track feature, not sure. Anyway, if you open a method named
"My_Method", you see the name twice. It's up in the window title and then
again inside the window. It looks weird and I assumed it to be a cosmetic
bug that would get caught and go away.

Then I somehow opened up another method in the same window and realize that
it's a tab bar in the method window. That's not a feature I'm likely to use
much, but it seems like a decent feature. I use that in browsers
constantly.

Here's the question: Is there a way to hide the tab bar when there's only
one tab? Browsers tend to have this option because a one-tab tab bar kind
of makes no sense and wastes space for no purpose.

I had a look at the prefs that I could find and didn't find one that seemed
to apply.

Any suggestions?
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Server Process "Frozen"-ish

[David Adams via 4D_Tech]

On Thu, Oct 26, 2017 at 7:45 AM, Timothy Penner via 4D_Tech <
4d_tech@lists.4d.com> wrote:

Thanks for chiming in. Given the number and nature of the people reporting
this bug, it's clearly a real bug. Can you give us some insight into what
efforts 4D is making internally to track this down and fix it? The way I
interpret your remarks is "We're waiting for customers to figure out what
the problem is." Is that a mis-characterization or even a totally off-base
reading of your message?

Is there an *active* effort within to track down this particular problem?

Thanks.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Tip: Exporting to a target folder in a package

[David Adams via 4D_Tech]

Have you tried Select document? It lets you navigate inside of packages.

As far as code goes, you can do whatever you like with a package. I've just
been doing that week, doing post-build cleanup. If I remember correctly,
the path is something like:

...your long path:BuildAppName.app:Contents:Database:Resources:

On macOS, the application name is used like a folder name to get down
inside of the package. If you're doing a build, you can insert help files,
delete .xlf files, etc.

On Tue, Oct 24, 2017 at 9:29 PM, Peter Jakobsson via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Hi
>
> I have XLIFF applications that generate stuff like custom constants and
> language resources.
>
> Unfortunately, if you use 4D’s desktop navigation commands like “Create
> Document(“”)” or Select Folder and the like, to allow the user (me !) to
> locate the target folder for the export, it doesn’t let you navigate inside
> a package (which is where all my destination folders are since they are in
> the Resource folder for the structure).
>
> The only way I could do it was:
>
> 1. open the package in the MacOS desktop using the right-click contextual
> “Open Package” command
> 2. drag the resources folder to the sidebar shortcuts list
> 3. then use the sidebar shortcut as the target for the export path
>
> This seems to work. The question is, is it a feature or a bug ? Will 4D
> patch this up once they discover it so that I can no longer use this
> workaround ?
>
> Any comments, feel free !
>
> Regards
>
> Peter
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: QUERY into variable vs. Records in Table

[David Adams via 4D_Tech]

> And all it's doing is waiting for a relatively rare condition to happen
(and a record to appear in this table).  It might
>  happen at most once or twice per day, maybe only once every few days,
but when it happens it has to be dealt with
> immediately. It's such a waste to be constantly querying for a record
that 9 times out of 10 won't exist.

Can this rare condition that creates a record do anything else? Something
that would make it into a push/notification instead of record/semaphore
that you have to constantly poll for?

And, the last time I tested, Records in table takes advantage of a stored
statistic so it's speed is very fast and also very consistent. It doesn't
change based on table size, etc. As you know, InnoDB doesn't store that
stat, so SELECT COUNT(*) on a big table can kill you. Postgres, on the
other hand, stores tons of stats and has a sample-based frequency table on
board or the query analyzer to read from.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: PDF in Web Area not displaying on a Mac

[David Adams via 4D_Tech]

John,

This all sounds very familiar & is why I went with Rob's viewer plug-in.

http://www.pluggers.nl/product/pdf-viewer/

This plug-in isn't crazy expensive like some of the PDF generating tools.
(I think Rob faces high licensing costs of his own on those.)
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: PDF in Web Area not displaying on a Mac

[David Adams via 4D_Tech]

On Thu, Oct 19, 2017 at 10:51 AM, Wayne Stewart via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> John,
>
> I use Rob Laveaux's pdf viewer plugin rather than the web area.I use Rob
> Laveaux's pdf viewer plugin rather than the web area.


+1
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: v13+ unti testing UI

[David Adams via 4D_Tech]

Chip,

Your idea is vastly simpler to implement in versions of 4D with CALL FORM
and CALL WORKER. That's somewhere from the late 15 R series onwards and
V16.0+.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Server Process "Frozen"-ish

[David Adams via 4D_Tech]

Cannon,

Thanks for being so systematic about tracking down this problem and keeping
the list informed.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

> But at the risk of dating myself

John was a *very* precocious 8 year old.

On Mon, Oct 16, 2017 at 5:22 AM, John Baughman via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Hey Chuck your right of course. But at the risk of dating myself, when
> I first came to Hawaii in 1969 for my first tour in the Navy I was given a
> choice, fly or boat. A first class booking on the SS Lurline sure was
> tempting, but this sailor got sea sick far too easily so I opted to fly.
>
> John
>
>
> Sent from my iPad
> John Baughman
> Kailua,Hawaii
> john...@hawaii.rr.com
>
> > On Oct 15, 2017, at 6:32 AM, Chuck Miller via 4D_Tech <
> 4d_tech@lists.4d.com> wrote:
> >
> > I assume he flew
> >
> > Regards
> >
> > Chuck
> > 
> 
> > Chuck Miller Voice: (617) 739-0306
> > Informed Solutions, Inc. Fax: (617) 232-1064
> > mailto:cjmillerinformed-solutions.com
> > Brookline, MA 02446 USA Registered 4D Developer
> >   Providers of 4D and Sybase connectivity
> >  http://www.informed-solutions.com
> > 
> 
> > This message and any attached documents contain information which may be
> confidential, subject to privilege or exempt from disclosure under
> applicable law.  These materials are intended only for the use of the
> intended recipient. If you are not the intended recipient of this
> transmission, you are hereby notified that any distribution, disclosure,
> printing, copying, storage, modification or the taking of any action in
> reliance upon this transmission is strictly prohibited.  Delivery of this
> message to any person other than the intended recipient shall not
> compromise or waive such confidentiality, privilege or exemption from
> disclosure as to this communication.
> >
> >> On Oct 11, 2017, at 11:09 PM, Alan Tilson via 4D_Tech <
> 4d_tech@lists.4d.com> wrote:
> >>
> >> Hey John,
> >> How did you end up in Hawaii?
> >> Alan
> >>
> >> On Wed, Oct 4, 2017 at 10:02 PM, John Baughman via 4D_Tech <
> >> 4d_tech@lists.4d.com> wrote:
> >>
> >>> I am going to chime in here with a recent real world experience.
> >>>
> >>> A while back I posted to the NUG a problem that I was having with
> external
> >>> data storage using custom mode. When the data was updated and properly
> >>> saved to the external data folder, the field continued to display the
> old
> >>> version of the data until the server or standalone 4D was restarted.
> This
> >>> was a show stopper for me upgrading a client to v16.
> >>>
> >>> I called tech support and after uploading a sample db demonstrating the
> >>> problem was told that it was in fact a bug and a bug case was opened
> for
> >>> it. This bug existed in 16.1 and R4 beta.
> >>>
> >>> I then called Tracy, my sales rep, and asked her to expedite the bug.
> >>>
> >>> A couple of weeks ago I received a message from 4D TS saying the bug
> was
> >>> fixed in the latest nightly build of 16.1. I confirmed the fix, but
> noted
> >>> that it was still broken in R4 beta. TS acknowledged this and said
> that the
> >>> fix would not show up in the R releases until R5.
> >>>
> >>> 16.2 was released and as expected no bug.
> >>>
> >>> Today R4 was released and voila the bug has been fixed.
> >>>
> >>> Now in the context of this thread, this appears to be an anomaly? It
> does
> >>> say to me that 4D does heavily prioritize the bugs in their bug list.
> For
> >>> example, you might have noticed that in 16.2 the issue with the Design
> >>> Environment not remembering the open windows from the last session has
> been
> >>> fixed. Not so in R4. Interesting, yes?
> >>>
> >>> For me personally, I am not one to complain. I may be naive, but I
> trust
> >>> that the folks working on 4D are doing the best they can. Sure there
> could
> >>> be improvements, but I am one to live with what I got and be patient
> for
> >>> what is yet to come.
> >>>
> >>> I understand the frustrations, but that is my 2 cents worth.
> >>>
> >>> John
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>> On Oct 4, 2017

Re: v13 - odd compiler error

[David Adams via 4D_Tech]

Check if there is a typo and you've declared $43 instead of $3. The errors
you describe are almost exactly what I would expect in this case. Not that
I've ever done anything like that...but it happened to a friend once or
twice.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: NB: Pre-emptive mode hassle

[David Adams via 4D_Tech]

Right, about the custom code scanner/pre-processor/compiler macro idea, I
forgot to mention that you could use such a tool to remedy problems in 4D.
For background, some of this takes a syntax in your code that enables you
to identify the following:
-- Required parameters
-- Optional parameters
-- Parameters that are of magic types, such as strings that are actually
method names.

There are a ton of ways to implement that, I've settled on one I'm happy
enough with...but it involves work no matter what. (4D's compiler
declaration system isn't very expressive.)

Anyway, with that in place and a custom code scanner, here are some very
handy things that you can do:

* Count parameters and see if optional parameters are missing or if there
are too many parameters in place. (The compiler doesn't do this reliably
and has no concept of an optional parameter.)

* Interpret parameters of magical types. For example, check that a call to
ON ERR CALL is taking a proper method name, or that a call to CALL WORKER
is taking a preemptively safe method. (This second one is harder to do, but
doable.)

* Strip out calls to TRACE as they (mysteriously) are thread-unsafe. Why?
Why are they even in the compiled output? It makes no sense, they can't run
anyway.

And more...I'm sure everyone can come up with a few ideas for what you can
do by rewriting code just before compile and build.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: NB: Pre-emptive mode hassle

[David Adams via 4D_Tech]

> Also trying to justify reducing the size of the process variable table.
> Supposedly having 1 - 10 object variables was far superior to 100 process
> variables. Who the hell cares if your process variable table is 1MB in
size
> when everyone has over 4,000MB to work with? And don’t try to say “it will
> save time starting new processes”. Really? By how much real world time? By
> 4ms? Can any human notice a decrease in time by that much?

That whole business with the process variable table was and is nonsense. It
makes no measurable difference to process startup and never has.

> And if you really think about it, a C_OBJECT variable most likely takes up
> more space in memory than a C_TEXT or C_LONGINT or C_BOOLEAN. You know,
with
> all the extra overhead that a C_OBJECT has to manage to provide the fast
> lookup and access to properties.

Don't know, can't know, 4D refuses to say. Absolutely refuses. "Because it
might change." ? Such an absurd and unconvincing position to take. Here is
where I *don't* talk about the Postgres documentation other than to say
"how it's done."

> I do a lot of UI work with 4D. I love doing UI work. Windows and dialog
boxes
> that users deal with all day every day. So 4D is great for a lot of the
work
> that I do.

Right on! I wasn't saying 4D is necessarily bad for that - just that it's
not my favorite thing to do. I actually really like 4D's form editor and
think that they should have patented the field entry UI. That's genius. The
Properly List's buggy text handling has been a pain for 20 years or
sobut, overall, I like multiple pages and other stuff in there.

> Isn’t it interesting that 4D became so useful and popular because it
provided
> a great way to build very nice user interfaces to database solutions. I
never
> realized you don’t care about UI stuff and actively avoid it. It explains
a
> lot. You are a code monster and backend database monster. If 4D is not
> advancing the language and the database you think it is sitting still.
Too bad
> they can’t seem to advance both with equal speed.

Yeah, I'm happiest in a pure code situation, for sure. I very much think
about code as a series of modules with well-defined interactions. UI tends
to gum that up...plus there are those slow, erratic, inefficient
peripherals attached - people. Hey don't get me wrong! I'm a total people
personwell, apart from the people part.

> And by the way, your 4D Method presentation example database had a very
nice
> UI. Beautiful. So you do have talent or good taste there, in my opinion.

Dude, you need to raise your standards, but thanks. All I've ever really
done is steal the 4D graphic design charter from the 1990s, grab the FatCow
icon set, and make sure that things tend to be aligned. I did just pick up
"Information Dashboard Design" by Stephen Few, which I'm pretty excited
about. But that's about the visual display of statistical data and is for
my work with Domo, Loggly/SumoLogic not 4D.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Min/Max for Date Fields in 4D

[David Adams via 4D_Tech]

On Thu, Oct 12, 2017 at 6:57 AM, Arnaud de Montard via 4D_Tech <
4d_tech@lists.4d.com> wrote:

>
> It's a solution if the selection is huge (is it?). But sort selection
> (I've always found 4d sort is slow), then 2 selection to array, that's 3
> server calls, at end. I'd prefer selection to array (1 call), sort array,
> read first and last items in array.
>

Agreed. I'd want to know that there is any optimization needed past FIRST
RECORD and LAST RECORD before diving into anything else. And, past that,
you would want to balance the cost of pulling more data over the network in
once call with pulling tiny bits of data over the network with multiple
calls.

I suspect this is getting a bit silly, but another idea is to have an
method set to "Execute on server" that does the work and packs the results
in an object and returns it. Yeah, that might work.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Min/Max for Date Fields in 4D

[David Adams via 4D_Tech]

Alan,

I haven't followed this thread closely, so apologies if I say something
silly.

* Is it important to optimize this operation past the sensible solution
you've already tried?

* Since you already have a sorted selection, why no use that and SELECTION
RANGE TO ARRAY? So, use SELECTION RANGE TO ARRAY to get a single value from
the first record, that's your min. Then use SELECTION RANGE ARRAY again on
the last record to get the max.

I can't think of anything else better. You would have to test it out to see
if the double call is even worth the trouble.

On Wed, Oct 11, 2017 at 11:30 AM, Alan Tilson via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Greetings everyone,
>
> Is there a way to get the earliest and latest dates from a selection of
> records similar to using Min/Max for numerical data?
>
> I have sorted the data and captured the first and last dates but this seems
> rather a lot of work to get this info.
>
> I believe that I have also used Selection To Array to capture the dates in
> the array and then sorted the array.
>
> Surely there is something I've overlooked?
>
> Currently using v13.6.
>
> Thanks,
> Alan Tilson
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

Hey John, we ended up testing different things, I didn't ever get too far
into the memory side. (Thanks for that, good to know.) I retested again
last night and I can kill 4D from entirely legal code. I'll show you at
dinner, if you like. There is a *concurrency bug*.

I'm not convinced that everyone at 4D fully grasps the nature of a
concurrency bug and they're definitely not grasping their importance. It's
an existential bug. What's going on should be *impossible*. I don't mean
rare (it is rare, but not that rare.) There's a _reason_ I've reported this
bug and have been going on about it. When do I normally do that? I don't.
And it is extremely unlikely that I ever will again. There's a margin in
doing so, but it's pretty much all negative margin ;-)

They should email Laurent E., he understands concurrency perfectly and
should appreciate the situation. Plus, I think this feature set comes from
him. It's possible Engineering will discover that it's a highly specific
and localized bug, not a more generalized bug - let's hope so. But there is
*no* way for us to know that on the outside. I could keep probing to get a
better sense of the shape of the bug, but that's a massive misuse of my
time. I've got real work to do. So, no logging to files via a worker in 4D
for me :( I think that I can still use normal processes to do a
record-based passthrough, but that's a very different performance
proposition entirely. (There's no way that I can use that approach for the
Web logs, I've killed 4D before that way...not doing it again.)
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: NB: Pre-emptive mode hassle

[David Adams via 4D_Tech]

> This would be super useful!. Great feature that would make it much easier
to adapt methods so that they would run in
> preemptive mode. The 4D engineers probably never even thought about doing
something like this.

If you don't mind and are on the forums, would you mind putting in a
feature request? And, for what it's worth, it would be super weird if they
*hadn't* thought of this. The languages they write in have all kinds of
compiler pre-processing instrucxtions and compiler macros.

> I would suggest something like this:

> //%C-   to turn off compiling code
> SET TEXT TO PASTEBOARD($stuff_t)
> //%C+   to turn on compiling code

> That’s how you do it with range checking using //%R- and //%R+.

Nice! I was thinking of how you turn off warnings (look at
Compiler_Suppress_Warnings in our source if you don't know that one), but
your way is vastly better. It's actually a really exciting feature request
the way you have it, submit it!

> We have to remember to bring this up at 4D Summit next year in the Q
session so it is publicly directed at Laurent. You
> get him to publicly say “yeah, that is a good idea, we’ll do it”. That’s
the best way to cut through the red tape and get a
> new feature implemented.

Hmm. Yes. That sounds like an efficient way to manage software development.
Cough-cough. Hopeully, the feature request system has some value. (Time
will tell.)

> That is exactly the advice JPR gave at the 4D World Tour. If it ain’t
broke, don’t rewrite it!
> Use it for new code or if you have to redo things to get a performance
boost.

Fair point, but my point is a bit different: It is suprisingly hard to
reuse *any* code. Error handlers, utility methods, tons of stuff that we
all have around. And not for obvious reasons like showing UI. So then what?
You've got two different versions of your code to make one preemptively
safe? When the only difference is, say, the TRACE command. Duplicating
routines and making the drift apart is a bad sort of code decay. That's the
hardship, not the idea of writing fresh code for fresh problems.

> Too many developers go crazy when 4D adds some new features and get it in
their head they MUST
> rewrite everything to use the new features. That’s not perspective. I
think “nice, another tool
> in the toolbox that I can use for something new when I get the chance."

Honestly, I can't remember anyone I've met like that. Plenty of people dive
in to check out a new featureabout 6-12 months after they're
introduced. I doubt a lot of people are trying to use preemptive mode. Or,
if they are, they're not saying anything here.

> I'd love to hear from other people that are using preemptive processes
> regarding things they found that are good, bad, surprising, exciting, and
> me'h.

> CALL FORM. CALL FORM. CALL FORM. That’s the most useful new command I’ve
found.
Yes, it solves a lot of problems that I don't have ;-) I'm just not into
complicated 4D GUI work by choice. If I were doing that sort of thing, I'd
use CALL FORM for sure. (EXECUTE METHOD IN WINDOW.) The old Outside
Call/CALL PROCESS system was dangerous garbage. And don't get me started on
GET/SET PROCESS VARIABLE. Rob's IPC channels in NTK work great. The design
is better than CALL FORM and would be familiar to anyone that's used
conventially message queues. (You send messages, you don't push code into
another thread and expand it.) But plug-ins aren't thread-safe.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: NB: Pre-emptive mode hassle

[David Adams via 4D_Tech]

Thanks Tim!

More findings:

* I don't get it. I really am unclear how exactly things are supposed to be
set up. It's confusing, at least to me. I've tried diagrams and writing it
out...but I'm still not clear.

* I want to use the 'indifferent' execution mode everywhere I can, but
that's not working out.

* You cannot set execution mode in batch mode the way you can with the
invisible, etc. attributes.

* Dang, it's seriously time-consuming and hard to nail it down. Hopefully,
the tools will improve a lot in future versions. I have some optimism that
they will.

* TRACE is not thread-safe. It's also meaningless compiled. And yet, it
blocks you. Why?

* SET TEXT TO PASTEBOARD is not thread-safe. I only use it for debug stuff,
it would be nice to have an option to have a compiler declaration syntax to
say something like:

// %i C:123
... or whateverlike the range check compiler comments. The idea being
you can say "just compile this out, don't include it when you see it" and
then the command number. (Made up number above.)

* If you're building a component and you want the shared methods to be
accessible to preemptive methods in the host, it looks like you *must*
declare the component method as 'can be run in a preemptive', etc. The
'indifferent' option doesn't seem to work. That could be documented, I have
no idea.

And, as a reminder, no IP variables. For existing pseudo-constants, use
real custom constants instead:

Managing Custom Constants with Code
http://kb.4d.com/assetid=77806

All thanks and praise to Cannon!

With all of the above said, my biggest advice is *wait*. This stuff is
going to have to get better. (The compiler in particular.) If you have some
tasks where preemptive threads are going to be a big help (workers or
regular processes), fine, go for it. But it's going to be easier to write
fresh code than rewrite or reuse old code in a lot of real-world databases,
so try and start with something this is either 1) super simple or 2) mostly
based on new code.

I'd love to hear from other people that are using preemptive processes
regarding things they found that are good, bad, surprising, exciting, and
me'h.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: NB: Pre-emptive mode hassle

[David Adams via 4D_Tech]

Right, I just scanned a big code base for methods passed to CALL WORKER
(some my code, some not) and found that 5 out of 7 won't run in preemptive
mode.

Oh crap.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Post Key command not working in converted 4Dv16 database

[David Adams via 4D_Tech]

Hey John,

Yeah, I pretty well guessed something like that was going on in your
case...I mostly answered for the sake of the archives. I often ask
questions and am offered well-meaning and sensible answers...but there's
almost as often a good reason why I'm bearing down on the particular point
raised. So, understood.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: NB: Pre-emptive mode hassle

[David Adams via 4D_Tech]

Typical, my "summary" is longer than the original.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: NB: Pre-emptive mode hassle

[David Adams via 4D_Tech]

To summarize the bit above:

-- If you pass non-preemptive code to a pre-emptive worker, it won't work.

-- That's not a bug, that's a limitation.

-- 4D doesn't automatically handle this situation well, but they've said
that they'll make ON ERR CALL work, which seems like a reasonable response.

-- The Compiler should check methods passed to workers. Maybe it will,
maybe it won't. I think that it should (obviously), but I don't get to
decide.

-- For now, what you need to do is remember to *manually* check code passed
to the second parameter of CALL WORKER. It's on us.

-- The easiest way that I've found to take this step is as follows:

* Create a method marked to run in preemptive mode. You don't run it, it's
just a place to expose code to the compiler.

* Use Find in Design to locate all CALL WORKER calls.

* Copy and paste in whatever methods you pass in the second parameter.
These are what get run through EXECUTE by the worker, once received.

* Use the compiler to check syntax.

Now you should see errors on the methods that are going to fail. A bonus
from doing things this way is that you get an error the *specific* methods
that are causing trouble. For those that don't know, when you compile 4D
scans the visible call tree for each method that _might_ be called from
within a preemptive process. However, it only reports specific errors at
the top level. It's pretty frustrating trying to figure out what the
problem is (don't forget triggers). The docs say that this is the way the
Compiler works, so it's not technically a bug (unexpected behavior.) It is,
however, a giant, time-sucking hole that makes getting code into preemptive
mode harder than it needs to be. Seems like a good feature request - the
compiler gathers the necessary information during its scan and then throws
it away and gives us a report that says "Something seems not okay?"

Anyway, that's the state of the art today.

P.S. Because of the silent failures in preemptive mode, your code may not
be executing at all. Or some of it might not be. Orit might be causing
problems. There's just no way to know without looking very, very carefully.
It's early days for preemptive mode, so I'd suggest you consider any work
with it experimental.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

NB: Pre-emptive mode hassle

[David Adams via 4D_Tech]

I've finally tracked down a weird problem. I have some code that I'm
(still) trying to run in a worker. I've got the system set up so that I can
launch in cooperative or preemptive mode based on a setting. This way, if
there's a problem with preemptive mode, we can tweak a config setting and
try again in cooperative mode.

This is all working in tests.

But my overall system was failing to deliver in preemptive mode. The
preemptive worker launches just fine...by my subsequent calls to the work
fail, but only in preemptive (multi-thread/multi-core) mode.

I've got an ON ERR CALL installed that's thread-safe...but it never fires.

I think this is part of a bug I reported months ago regarding passing
non-thread safe code to a thread-safe worker. The compiler tries to stop us
from *compiling* such code, but the compiled mode runtime interpreter does
not. You can pass anything into a worker, even unsafe code. The exact
results at that point areundefined. I think that Thomas Maul said on
the forums that getting ON ERR working reliably in this situation would be
done. (I'm in R3, maybe it has been done.)

Anyway, the warning is this: You have to take extra steps to vet your code
because *the compiler does not inspect the statements that you submit
through CALL WORKER.* I think it's recently started testing that ON ERR
CALL methods exist in one situation, but that's the first and only time I'm
aware of that 4D sees arguments to EXECUTE (under whatever name) as
anything bug raw strings.

So, lesson relearned.

Anyway, what you need to do is look through your code and see any that
might be Unclean. The compiler is said to be able to dump out this
attribute for all methods. The other way is to create a method that is
marked to run in preemptive mode and then put calls to your other methods
in there. In this case, the compiler will flag any that are unsafe for you.

Kind of a hassle, but there it is.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Post Key command not working in converted 4Dv16 database

[David Adams via 4D_Tech]

Hey John, if you still need to fix what you described, CALL FORM ought to
work. Just past what method you want to run and the target window's form
will run it through EXECUTE for you.

doesn't mean that POST KEY shouldn't be fixed if it's broken, of course.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Query By Attribute vs Map NULL values to blank values

[David Adams via 4D_Tech]

> I still do not get why the R4 documentation for Query by attribute
discusses how a # query will function if the "Map null
> to blank” checkbox is or is not checked it the checkbox does not exist
for an object field.

John, random thought: Object fields aren't supported in 4D's SQL. So any
SQL-specific features and such are completely meaningless for ojject
fields. And by "not supported" I don't mean ignored. If you do

SELECT * FROM MyTableWithAnObjectField;

4D freaks out because there's an object field. At least as late as R3.

This has come up a few times here and other people have looked at this in
detail. (I think that Jeff Kain, at thel least, has the details.) In my
case, I had to rewrite my ugly little SQL browser to scan the field list
for object fields and remove them from statements. That's okay for my
needs, but I'm not using 4D's SQL for anything real any longer.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

> As a side note, I am sure you are aware of this, but by killing the
worker every 100,000 calls, out of the 2 million calls
> nearly all of the log entries are missed as they are in the queue when
the worker is killed. I am getting at most 24,000
> log entries in the text file.

I've lost track, but I think that this was the very thing I was testing -
the %/# of lost log lines I would end up getting.

The more I think about it, the more I'm convinced that what I'm trying to
do simply isn't part of the designed capacity of 4D's system. I can live
with that, I'll just move the logging out of 4D. I may lose something that
way (a bit of data that would come from a 4D looking that could inject some
extra data) and it's a pity, but 4D never promised me a log writer. It's
just not sufficient to requirement.

So, checking what NGNIX offers. On the log enrichment side, Loggly offers,
well, very little but I'll be checking out SumoLogic down the road and that
looks a whole lot better. (It's a Splunk clone, as far as I can see, but
cheaper.)
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

Add,

On the entirely likely chance that my first answer sounded insufferably
arrogant - just wanted to addWell, I'm sleep deprived like I can't
remember, and massively jet lagged. I was trying to say that this whole
area is one of my *few* areas of genuine expertise. I don't have many. Just
ask anyone ;-) It's just an accident of personal history that this is one
of my areas of interest and experience - there are *tons* of equally
important things that I'm clueless about. So, not trying to be a jerk about
it, but I figure it was worth giving you the background that I'm not
figuring any of these things out from scratch at this point...only figuring
out how to implement things within 4D's feature set.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

> I understand your thinking that 4D may not the way to go for event
logging,
> but your concern about CLOSE DOCUMENT is where we do not agree. I'am sure
you
> have seen CLOSE DOCUMENT fail, but I suspect that what you are seeing is
not
> an issue with CLOSE DOCUMENT, but something else not readily apparent. I
think
> you said at some point that 4D has recognized this as a bug, but I still
> question whether in fact it is.

Yes, you're absolutely right on this point. I don't know what the bug is.
It *manifests* as an impossible file lock error, but I think it's very,
very likely that this is a symptom of something else. Either way, it's a
show-stopper for me and I can't fix it.

> Perhaps you can show me in the sample database how to make CLOSE DOCUMENT
fail
> or have another database that demonstrates the problem?

That one already shows the problem. I've reproduced it, as has 4D and a
couple of other people. Once it happens, you get all kinds of weird stuff -
duplicate workers with "unique" names, the runtime monitor gets super buggy
and starts throwing range errors...until you're dead. If you *don't* find
this result on your gear, please post what version of 4D you're testing
with.

>> When I saw your initial post in this regard, I jumped in because I do a
lot of
> work that involves the opening and closing of documents. If a problem
exists I
> really need to know about it. Thus far my testing based on your posts has
not
> demonstrated anything that has caused me to doubt that CLOSE DOCUMENT
will in
> fact close a file.

Just try out a compiled version of the original database I sent without any
of your modifications and let it run for a couple of minutes.

Oh, I forgot to say earlier about John's finding of ~2.1M messages being a
kind of breaking point...that number may not be replicable in other tests.
You might find a different number. The payloads in my scratch database are
quite small. For all I know, if you made the payloads 10x bigger, you would
crash with ~210K messages. I won't be testing this myself.


On Mon, Oct 9, 2017 at 3:33 AM, John Baughman via 4D_Tech <
4d_tech@lists.4d.com> wrote:

>
> > On Oct 8, 2017, at 4:42 PM, David Adams via 4D_Tech <
> 4d_tech@lists.4d.com> wrote:
> >
> > What I'm concerned about is that CLOSE DOCUMENT may be
> > returning without the file *actually* being closed. That's really
> > problematic.
>
>
> I understand your thinking that 4D may not the way to go for event
> logging, but your concern about CLOSE DOCUMENT is where we do not agree.
> I'am sure you have seen CLOSE DOCUMENT fail, but I suspect that what you
> are seeing is not an issue with CLOSE DOCUMENT, but something else not
> readily apparent. I think you said at some point that 4D has recognized
> this as a bug, but I still question whether in fact it is.
>
> Perhaps you can show me in the sample database how to make CLOSE DOCUMENT
> fail or have another database that demonstrates the problem?
>
> When I saw your initial post in this regard, I jumped in because I do a
> lot of work that involves the opening and closing of documents. If a
> problem exists I really need to know about it. Thus far my testing based on
> your posts has not demonstrated anything that has caused me to doubt that
> CLOSE DOCUMENT will in fact close a file.
>
> John
>
>
> John Baughman
> Kailua, Hawaii
> (808) 262-0328
> john...@hawaii.rr.com
>
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
>
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

Hey John, thanks for the great post and all the work.

There's no reason for other people to know this, but John and I have
*totally* different coding styles. So I was really pleased he went through
my example and reworked it in his own way. That he turned up some problems
is instructive because he, for sure, approaches things form a different
angle than me.

> Open/Close Document: I think we all agree, even David, that it is now and
has always been our responsibility to close
> documents as quickly as possible after opening them.

Sorry I gave the impression at some point I ever thought otherwise. I now
and always believe that we're responsible for closing our own files. That's
not a question. What I'm concerned about is that CLOSE DOCUMENT may be
returning without the file *actually* being closed. That's really
problematic.

>  Worker processes in 4D are ready for prime time.

I suspect that this is true for many uses of workers. For UI stuff, they
seem like they'll be able to keep up with what even a complex UI in 4D can
push at it. And workers and CALL FORM are *massively* better than any
previous native solution. All of that stuff with SET PROCESS VARIABLE or
shared arrays with semaphores was just terrible - real garbage. I've said
that for years, so I was quite excited about the new features.

On the other hand, I was looking forward to taking a standard high-volume
task and leverage 4D's preemptive processing ability. John came up with a
solution, but it involves Semaphores and quite a bit of infrastructure that
either makes preemptive unavailable, or so expensive to get that it's
pointless.

For reliability, I'm going to look at sending the logs from NGNIX and cut
4D out of the process entirely.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

Add,

Thanks a lot for chiming in! It's always good to hear from you guys in the
US office.

I'm not sure what problem your solution is solving. I'm not trying to be
flip, I just want to make sure to focus on the meat of the issue. A really
common approach to log writing is as follows:

Have one process manage the log disk file. So. when that thread starts, it
opens the file, keeps it open, writes to it and can then close it neatly.
You can then have any number of other processes message the "writer" to
stream information to the disk file. Lots of OS-level files, Web servers,
etc. follow this model. Why? Because it is *substantially* faster and
easier than anything else. You completely sidestep race conditions on the
file lock - there's no chance of a race condition because only one process
*ever* opens the file. It's a nice, tidy solution.

That's what I wanted to do with 4D's worker when I first saw them - I
assumed that would be pretty much anyone's impression because it's such a
classic use for a message+worker setup. I was excited because with a
preemptive worker, you could push all of this low-value logging over onto a
secondary core. It's a satisfyingly complete, tidy, and naturally
low-conflict, high-performance solution. That doesn't work in 4D.

Unfortunately, this doesn't seem to work. I can easily generate race
conditions from opening and close the file and restarting the worker. This
should be *impossible*. Something is wrong in how 4D is managing file
locks. I don't know what. I start a worker, open a file, write to it, close
the file, kill the worker. The file should be closed. Period. Usually it
is, sometimes it isn't. I have no explanation for this, but it definitively
blows up any plan to use workers for logging via a preemptive process.

John Baughman in his tests thinks that the problem is flooding the worker
and blowing up 4D's via memory exception. Could be. I briefly did a bit of
load testing like that, but not a lot.

So, either way, it doesn't look like 4D's CALL WORKER and worker system is
designed to scale or work safely/reliably under heavy load. I've come to
suspect that it was never intended for such tasks.The lack of a pause
mechanism or a way to count 'queued' entries are telling. They're very
standard (universal) features of a queuing/messaging system, but 4D doesn't
have them or seem to appreciate why you would need them. I've got feature
requests in for this stuff from months ago. Note that if you do any kind of
heavy load, you need to be able to measure the state of a queue to see if
you need to spin up another one. That 4D lacks this feature (or seemingly
interest in such a feature) suggests that I'm off-base to even consider 4D
workers for heavy loads. Just not built for it.

4D never said that their workers were meant for heavy lifting. I think that
they were designed to handle various kinds of UI-related tasks in 4D. I
haven't got much interest in that (I did a bit of a publish-subscribe
system that I showed at 4DMETHOD, that was fun to work through), and I'll
guess that CALL WORKER and CALL FORM are just fine for that.

But for what I need? I have to conclude that 4D isn't the right tool
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

I was never focused on LOG EVENT (?) I've never used the command and would
prefer not to in this case. I only brought it up because it would
potentially well with Loggly. (I'm not committed to Loggy and may not stick
with that platform, for what that's worth. But it's worth trying for now.)

I have no access to the forums, so I don't know what your link says. But if
the title is safe to go by then, yes, that might help. I'll have to test it
out to see how well it works before I know. In theory, what I was trying to
do in the first place is a completely safe idea - even better than LEP. But
it isn't safe in practice.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

I've posted about my various worker log problems here and on some other
threads and even posted my sample database for review. Lately, John
Baughman has really been sinking his teeth into this issue and has spent a
*ton* of time working on it. Thank you John! The news is bad, but I figured
I'd post it here as part of this thread. I won't say a lot, but John might
want to explain his point of view. He's an optimistic person, with a
cheerful nature and he likes people. Despite that, I still like him ;-)

Anyway, John took a copy of the sample database (including the correction
contributed by David Porter) and tried to figure out what he could. My
summary:

* John is able to crash 4D just by flooding the queue.

* He came up with some ideas to help manage the situation, but they
involved either or semaphores or IP variables.

* John also came to see (very quickly) that you need a function to report
current queue size.

Now, it's entirely possible that 4D never intended their CALL WORKER system
to be used in a high-transaction setting. They have offered no guidance on
the systems limitations and scope. I briefly tried crashing it with a
message flood months back and thought that 4D preformed well. In fact, I
was looking to see if it created disk scratch files, but I couldn't find
any. John sounds like he's able to flood and crash the system pretty
reliably, but not every time.

One of my primary goals in using CALL WORKER was to avoid writing records
and take some work off of core 1. There appears to be no safe and reliable
way to meet those goals. This is a total bummer as even my HTTP push
solution won't be safe.

My next good idea is to see about pushing as much of the logging as I can
onto NGNIX which will absolutely, definitely have zero problems with
whatever we can throw at it.

John my have more to say, but I think I'm likely done with this subject.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

Hey John,

Thanks again for trying. I kill the worker from *within* the worker *after*
closing the file. It *usually* works. But "usually" is worthless when the
consequences of it not working are a dead server.

4D has accepted it as a bug in, I think 16.0. they just haven't fixed it.

For now, I'm not using files from workers. I'm streaming C_OBJECTS over
CALL WORKER, invoking a method that transforms the results, bundles it up
and then pushes the data in batches to Loggly. We'll see how that goes.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How do workers die?

[David Adams via 4D_Tech]

>
> Perhaps I am misunderstanding your question. Do you mean after a worker
> has been killed with KILL WORKER?
>

 Clearly, I asked my question poorly as you and Julio both found my
question unclear in the same way. (Or perhaps it's just that it's unclear
to people who can speak Portuguese?)  In any case, yes, I mean when a
worker has been killed explicitly.

I've tested out workers extensively - I've spent months on CALL WORKER,
etc. and drafted a few hundred pages about them. None of that will ever see
the light of day, but I've spent a fair amount of time studying them. I'm
now trying to put them into production, working with the limitations they
have based on the design choices that were made for this part of 4D. And
bugs.

The way I like to kill workers is to have a method that does it that is
called from *within* the worker. This allows for a more organized shutdown.
Closing files, etc. You can call the worker to execute the method in the
context of the worker very easily.

Tip: It's also easy to forbid method to run outside of the context of a
worker with a specific name.

If (Current process name # "YourWorkerNameHere")
// Deal with the illegal call.
Else
 // Do your thing
End if

That's a solution to a problem not everyone will run into, but it's a tidy
little trick when it is appropriate.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: How do workers die?

[David Adams via 4D_Tech]

Julio,

Thanks for taking a stab at it. I should have been more clear, I mean when
workers are killed. I do this from within the worker to try and have an
orderly shutdown. Your reading of the docs is correct though. Workers are a
thread of execution, like any other process, but 4D has its own magic loop
waiting for incoming 'messages'. These messages aren't really messages,
they're blocks of code that are appended to the worker's thread of
execution and then run through EXECUTE. At least, that's how they behave.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

Again:

Start worker
Open file.
Use file
Close file.
Kill worker

Start worker again
*** FIle lock conflict ***

4D's accepted it as a bug as of some months back.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

Hey Chip, thanks or chiming in.

I've always been disciplined about opening and closing files on my own, I
have no idea what 4D's defaults are on this...or if there is even a
documented (promised) behavior. The traces of old, dead workers I was
referring to *might* not actually exist and that wasn't what I was talking
about. I'm talking closing the file explicitly, killing the worker
explicitly, and then the worker is restarted. And that's when you can run
into a conflict. Which should never happen, but it does.

Regarding <>whatever, you can't use that kind of variable in a preemptive
process of any kind. That's one of the more painful costs of admission, but
it also makes sense.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

How do workers die?

[David Adams via 4D_Tech]

Does anyone know or can they find out the following?

What happens when workers die? 4D has at least two different approaches to
this:

* HTTP processes wait before dying, but then die if there are no new
requests after some time. (Or at least they used to.) Meaning, an HTTP
process may or may not have old variables and so forth in place.

* New process () / Execute on server () When they finish, they're dead.
They're supposed to release file locks, record locks, etc. If you start a
new copy of the process with the same name, it starts off in a "virginal"
state.

Which is it for workers? I thought it was like New process(), but am seeing
evidence that it's not always the case. The docs aren't explicit. I take
the implication to be that they should be fully dead, even if started
almost instantly. But this doesn't seem to be true.

The difference is pretty important.

Is there a way to find out authoritatively? I've sent it to Wayne for tech
support, so maybe that will work in time...but perhaps someone knows or has
a shortcut to finding out?
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

> Not an anomaly.

John & Neil, thanks for the reports, I find them encouraging.

We're dealing with somewhat different situations. You're going through US
Tech Support, which I have no contact with and you're using the TAOW, which
we don't currently have access to in AU/NZ. The business case for 4D
devoting resources to rolling their own bulletin boards and help desk
software instead of renting tools from Atlassian (Aussie! Aussie! Aussie!
Oi! Oi Oi) or similar escapes mebut that's a different discussion.

Anyway, our tech support is Wayne Stewart. Everyone loves Wayne. If I made
anyone think that I was criticizing Wayne, I am sorry and also sad. Wayne
is a helpful *to a fault*. Even now he's trying to come up with workaround
for my particular bug. But here's the thing, it's a *concurrency problem*
and it is *inside of 4D*. That's not a problem we can fix. The only thing
Wayne can do (and has done months ago) is kick it to France. Its their bug
and only they can fix it. Wayne told me that they accepted it as a bug
months ago, and that's the last we heard.

Why do I care about this bug? No one here will have noticed, but I don't
talk about bugs a whole lot. For the most part, I don't care. If 4D can't
do it one way and I can find another, I'll do that. If 4D can't do it, I'll
find another tool. In this case I *do* care. Here's why: It's an important
bug that blocks certain designs and leaves me mistrusting workers and
preemptive mode quite fundamentally.

My problem boils down to how workers deal with file locks. In theory, a
worker processes requests in strict order. (They're not requests, they're
EXECUTE statements run in the context of the worker, but nevermind).  So,
if you have a worker and close a file, it should be closed completely. If
you kill the worker and it restarts, the file close done explicitly in the
worker should finish before the worker dies. Managing file locks
across/amongst process is the sort of basic concurrency problem that was
worked out over 50 years ago. And when I say "worked out", I don't mean
"dude wrote some code" I mean the basic reality and mathematics of
concurrency were worked out. In that process, the semaphore was invented.
Why? Because it is necessary in many situations.

Are semaphores hard to get your head around? Yes. But what's _really_ at
issue are the issues of concurrent computing themselves, that's what's hard
to grasp. I remember how many smart people just could not accept that
semaphores were necessary to lock shared resources when V3 introduced
semaphores. Some people _still_ don't seem to grasp them. Here are the
options that you've got for a situation where a race condition might occur:

1) You develop a system that is *provably* never going to have the problem,
ever.
2) You are inevitably going to encounter the problem.

Because science. There is no third option other than "it doesn't seem
likely, so I'll risk it." See outcome 2.

In my case, I'm architecting some stuff that is going to be very high
volume, so option 2 isn't a choice. So I need a truly safe solution. I
stumbled across the file lock problem *by accident*. I don't want this bug
to exist. But here's the thing, if what we've been told about workers and
preemptive process is all correct, then this bug is *impossible*. But it's
there. So there's something most definitely wrong. Is the problem very
narrow? Is it widespread? I have no way of knowing and 4D has told me
exactly nothing. (I asked on the Forums but was a) told nothing or b)
accused of trying to "sabotage" the command, whatever that means.) So, I
have to go with what I can see:

It is not safe to rely on file open/close in workers.

I tried a delay and a bunch of other things and they don't solve the
problem. I guess never closing the file might work, but that's not an
option. (I don't want a 2GB HTTPD log file, thanks.) And here's the thing,
there is no workaround to a concurrency bug.

It's 4D's bug, only they can fix it.

So, in my case, to get my custom log data out I have to avoid files
entirely. Again, high volume - race condition possible = race condition
inevitable. I've got two different architectures:

Source process ---> CALL WORKER ---> HTTP Request --> Logging platform

Or

[Log_Record]
Read with a standard process
Write to a log file

There is so much data involved that log records aren't a sustainable
solution. (The last time we went down this road we killed 4D and then
InnoDB in MySQL. Postgres, you're next!)

As I've been unclear: The lock problem manifests differently in cooperative
and preemptive processes, but occurs in both.

There is no workaround, only alternative architectures and an actual fix to
4D.

My frustration is not with tech support - Wayne's great - it's with France
not fixing their bugs. And, as it has _always_ been for me, they're an
informational black hole. I don't get any information from them. A few
times over the years, ever. That's it. So, how am I to know if they're
working on this? 

Re: What to do about users who won't follow instructions?

[David Adams via 4D_Tech]

Steve,

That looks great! Thanks for the screen shots, they really show what you
were talking about.

Curious: Do the pictures show up on the NUG? If we can post pictures
there...I'd like to know.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: What to do about users who won't follow instructions?

[David Adams via 4D_Tech]

On Wed, Oct 4, 2017 at 5 :06 PM, Stephen J. Orth via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Pat,
>
> Completely agree on your comment about video!
>
> 
>


> Let me know if you would like to see a screen shot or anything...
>

That sounds *awesome*. Any chance that you would do a 4DMETHOD or 4D Summit
presentation on this?
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

> If you have a bug that you really want fixes:  you need to work with Tech
Support.  If you don’t: then it’s just wishful thinking.

I have done this. It's a bug that 4D should care about *way* more than me.
It crashes the server. Easily Submitted to tech support with a demo
database, etc.

It is NOT MY FAULT that 4D doesn't fix properly reported bugs. It is NOT
OUR FAULT. It is their fault.

I'm getting tired of hearing from people that someone bugs aren't being
fixed because we're somehow mysteriously doing it wrong. I also appreciate
that I'm one of the very people that spit out that line over, and over
again down the years on this list.

Here's a list of positions that I can no longer accept on face value:

"We aren't documenting that because it might change."
This is silly in the extreme. Every piece of software might change, lots of
them say "This might change in a future version." Even 4D says this clearly
in their docs. So when you hear that line, it's an excuse for something
else - who knows what.

"It isn't Tech Support's job to report bugs."
Of course it is.

"You have to <> to get you bug fixed."
1. It's not my bug, it's THEIR bug. They're not doing me a favor.
2. Jumping through the hoops makes no particular impact that I can see.

"Feature requests should be submitted through the appropriate section of
the forums where there is open voting."
True. Does this get feature requests implemented? Not as far as I can tell.
(Well, I did see one - but it was from Rob and he's got a special
relationship with 4D.) My cynical interpretation is that the feature
request forum exists to stop people complaining about missing features.

Your mileage may vary.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

> It is what it is. What’s more important, getting the bug fixed or
bitching about customer service?
>  I always vote for doing whatever it takes to get bugs fixed.

Agreed! But I've found _nothing_ that works. Ever. I've jumped through all
of the hoops down the year, it hasn't worked. My crashing bug with V16.0+
where I supplied a database to demonstrate the problem in about a minute?
Nothing. No response, no action, no fix.

I take exception to the notion that somehow our bugs aren't getting fixed
because we're "not doing them right." Sure, that explains some of it. All
of it? Not a chance. *It's not our fault.* You mentioned that it's not Tim
Penner's job to take comments here and report bugs. Why not? In fact, Tim
does just that a lot of times - which seems like good service from Tim and
the US crew. I mean, it's crazy to think that somehow it's *our* fault that
a bug isn't getting fixed even though 4D knows about it. It's their bug,
not ours. I can't wait on bug fixes anyway. With their response times and
release cycles, I need to fix a problem in an hour or a day - I can't wait
six months to forever. If I can't use 4D for a task, I just don't. When I
reported my crashing bug (with demo database), I did it for *their* sake. I
figured "Hey, they're proud of preemptive mode and workers. Guess they'd
like to know it can bring down the server." Not so much. I don't get that.
In my case, I'm using a different architecture (an inferior one) as I can't
rely on logging to disk from a worker.

I used to report bugs and make feature requests a lot. I was one of the top
submitters for some time. Then one day I realized out of hundreds (probably
thousands) of reports, I'd gotten a couple of emails. Ever. I knew how to
do a "proper" submission because I'd worked at 4D for four years in Tech
support and then running IT in the USA. I couldn't get support for crashing
servers *then*. And it was their business. But I definitely knew how to
format a bug report. Down the years I've used bugs.4d.fr and more lately
the Forums. Still, no change. No feedback, no fixes. Maybe it's just me?
I'm not kidding about that. Neil reports a very different, and
satisfactory-sounding, experience.

Part of the problem is doubtlessly that as an Aussie developer, I don't
have access to the bug system.

But here is a little thought experiment. Imagine two different company's
with different attitudes to bugs and quality. Let's call them Huey and
Dewey.

Huey is *passionate* about quality and customer service. They take is as a
core part of the business to detect bugs before they reach customers, to
improve their processes when a bug does slip through, to release fixes as
soon as possible - even before customers know that there is a bug, and to
do everything reasonable to help customers reporting a problem. They're
constantly trying to improve their already substantial documentation. (I
saw a company just like this last year, it was a thing of true beauty.)

Dewy, well Dewey just doesn't know. They don't keep customers up to date,
they add a lot of hassle between customers and problems, they don't
actively look for non-critical bugs, they have no roadmap, they don't share
their priorities or even logic, and they ignore problems that they are
fully aware of. Many key features and behaviors are undocumented or
underdocumented.

What does 4D look more like? Huey or Dewy?
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

Dennis,

That's great to hear. This is definitely not my experience...perhaps they
don't want to hear from me. I'm glad that you're getting good support,
that's encouraging.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

Tim,

I'm with John on this one. The payoff for sending in bug reports and
feature requests is incredibly low. Zero feedback and results are rare. *No
matter what hoops are jumped.*

4D the company gives the _very_ clear impression that they just don't want
to know. If they do want to know, they're doing a terrible job of making
that clear, making it possible, making it work, or making it worth out time
to even try.

IMO
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: 4D v16.2 false record locks

[David Adams via 4D_Tech]

> That’s not his job. It is the job of every 4D developer to report bugs to
4D by opening a tech support case.
> And to refuse to close that tech support case, or allow it to auto close,
until the bug is fixed.

Sigh. Brining a fresh new flavor to the concept of "world class" customer
service...
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

[Snippet] Process_IsPreemptive

[David Adams via 4D_Tech]

I realized that I wanted to know for some error logs if the current process
is preemptive (thread-safe) or not. So I checked PROCESS PROPERTIES and,
yes, it will tell you. But it uses bit math. Oh no! I always run away from
bit math...and I decided that I was being silly to do so today. The "math"
in this case is nothing but an operator, not even slightly hard. So, here's
the code which I've tested out and seems to work:

// Process_IsPreemptive
// DESCRIPTION: Tests is process is running preemptively.

C_BOOLEAN($0;$is_preemptive)
C_LONGINT($1;$process_id)

If (Count parameters>=1)
$process_id:=$1
Else
$process_id:=Current process
End if

C_TEXT($name)
C_LONGINT($state;$time;$mode)
PROCESS PROPERTIES(Current process;$name;$state;$time;$mode)

C_BOOLEAN($preemptive)
$is_preemptive:=$mode ?? 1

$0:=$is_preemptive

This is handy for error logs, as mentioned, and also just to verify what's
going on. It's surprisingly tricky to launch a process in preemptive
mode...or least I find it so. You can use New process, Execute on server or
CALL WORKER, or so the docs say. I tried this code out with a worker and a
standard process, both in single-user. (Preemptive mode isn't supported on
4D Remote becausewe don't know, 4D hasn't said. Presumably something in
the network layer?)

Tip: The order of arguments for method to run and process name are the same
for New process and Execute on server but are reversed on CALL WORKER.

Tip: The easy way to see the state of a process is from the Runtime
Explorer. There are new icons since late in the V15R series for each sort
of process. Oddly enough, a preemptive process' icon changes to a
cooperative process icon when it dies. So, don't necessarily trust the
icons for dead processes.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Love 4D's math interpretation...

[David Adams via 4D_Tech]

Crap, it's documented. And, as we all know, 4D doesn't document or discuss
things that might change in future versions for fear of frightening the
children. But this is documented, so I guess we're safe relying on it as is
;-)
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

Related wrinkleI'm now wondering about something else: Does KILL WORKER
completely kill a worker, or does it sometimes just hurt it really, really
badly? I have been working under the assumption that when KILL WORKER
completes, that the worker is *totally* gone. Meaning, no process-level
values (variables, etc.) remain anywhere.A worker can be restarted very
quickly after being killed. It gets the same name (workers have unique,
case-sensitive names, outside of buggy situations) - but the contents
should be blank. At least, that's what I think and what would be best. Web
processes, as an example, do *not* work like this - they stay alive for a
few seconds and may get reused. New process *does* work like this. Let a
process finish and restart a fresh version and you get a blank slate.

I suspect with workers that they normally die completely when killed, but
not always.

I've just reread the docs and they don't address this point explicitly.
This is the sort of question I tried using the Forums to get answered and,
frankly, I didn't get answers. So, I'm going to have to go with the
assumption that a new worker *may not be virginal.* There may be some gunk
hanging around form an earlier incarnation of the worker. This is only
relevant if you use KILL WORKER, but that's a standard command.

Does anyone know anything more about this that could help change some of my
speculations into information, one wary or another?
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

Hey, David Porter spotted a bug in my code and I wanted to push this part
of the thread up to the list for the archives, etc.:

On Tue, Sep 26, 2017 at 5:05 PM, David Porter  wrote:

> Thanks for posting the LogWorker code, it has been fun experimenting with
it.
>
> I have been testing only the preemptive test code.
> Running on a Mac Pro (Late 2013) with 6-Core Intel Xeon E5 , Mac OS X
10.12.6
> 4D 16.1 hotfix 2 , 64 bit app.  ( Not server )
>
> Yes, there is a bug in the code.
> Method: LogWorker_Kill ,need to add the line
> Log_Path:=“” // This is required to get the LogWorker_Start code to open
the log again
>
> Even so, your code would crash interpreted before the Kill was ever sent.
> Putting in delay process in LogWorker_CallABunch_Preemptive, would get it
to run interpreted.
> So, clearly there is a 4D bug here.
>
> Compiled, I am able to remove all the delay processes, and run it to
completion.
> I do get some runtime errors about broken communication.
> Again, I can make them go away with some delay processes, something I was
trying to avoid.
>
> So, I encourage you to take another look at this code.
>

My first reply, before checking the code based on what David said:

Hey! Thanks for checking the code, you're awesome!

Thanks for the bug catch on my code. I'm sure that wasn't from me. Darn
kids, getting into my code and messing things up

How long did you have to delay process compiled to get it to work? I tried
up to 3 seconds and it still didn't work. In theory (as I understand it),
DELAY PROCESS should *not* be needed. Meanin, CLOSE DOCUMENT should not
return until the document is actually closed. I think I even tried it with
SET CHANNEL and found the same problem, but don't quote me on that.

If my understanding about the synchronous nature of the file/channel
commands is correct, and a process delay is needed, it points to an
underlying problem that would leave me not trusting the solution. Even if
it only shows up after a few hours/days/weeks...that's not good enough :(


My second reply now:
Okay, I've gone back and updated the code, just as David described. That
fix makes the system *way* better. Unfortunately, even with that fix and
the 3 second delay, the problem still crops up. It takes longer, but it's
still there. I guess it's an (un)happy coincidence that my bug amplified an
underlying bug in 4D. For a logging situation, a race condition rate of
1:1,000,000 isn't even close to good enough. It needs to be zero. Always.
In this case, i was able to provoke the bug in a few minutes :( I'm told
that 4D has accepted my report but Partners in Aus don't have direct access
to the bug system. And I've never heard anything from France so I don't
have a bug report number or anything else to do on.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

Hey Tim,

> Consider the situation where you have a mission critical system that is
up 24/7 and you need logging to always work every second of the day.

> How do you handle the situation where the stand-alone log writer app
crashes, is not running due to a machine restart, or for whatever reason
>  goes deaf and stops responding to the HTTP calls?  You certainly don’t
want any possibility of losing some logging information.

Again, talking about a workaround. I want to log to disk out of 4D via SEND
PACKET.

As to your point, it's true. However, I'm seeing is a *far* more likely
that 4D Server crashes than a 4D stand-alone. But your point is good.
Perhaps it's better to find a small, stable service (Windows) that can run
in the background to handle the HTTP-push-to-disk task and leave 4D out of
it entirely.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

Keisuke,

Thanks for the background information on LOG EVENT, your details are
*always* interesting. In my case, I don't think that it has anything to do
with what I'm dealing with. I'm not talking about 4D events or 4D logs, I'm
talking about custom logs with data I'm sending out. LOG EVENT isn't my
first choice, it isn't even my secondbut my first and second choices
aren't doable because of this fatal bug in 4D.

On Tue, Sep 26, 2017 at 4:26 PM, Keisuke Miyako via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> sometimes you will see a kind of "comment" line in the request log,
> in particular "TTF" or time to forget,
> which describes a change of state in the process:
>
> a: automatic relation
> b: record stack
> c: current record
> d: current selection
> e: transaction
>
> these are not real "requests",
> in that the information is not sent by itself.
> rather, it has an impact on future requests.
>
> "INFO" is a pseudo module name
> (as opposed to srv4, cli4, dbmg, etc.)
>  for writing such entries in the request log.
>
> in v2004, 4D was cooperative,
> so you could trust that the logs are recorded sequentially.
>
> however, since v11,
> a different process could write interject between "srv4" and "INFO", for
> example,
> so you can not assume that the two subsequent lines make a pair.
>
> "process info index" and "task id" is used to link such lines.
>
> it's in the documentation.
>
> http://doc.4d.com/4Dv16/4D/16.1/Appendix-E-Description-of-
> log-files.300-3373556.en.html
>
> > 2017/09/26 23:07、David Adams via 4D_Tech <4d_tech@lists.4d.com> のメール：
> > Not sure which ID that is
>
>
>
>
> **
> 4D Internet Users Group (4D iNUG)
> FAQ:  http://lists.4d.com/faqnug.html
> Archive:  http://lists.4d.com/archives.html
> Options: http://lists.4d.com/mailman/options/4d_tech
> Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
> **
>
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

Chip's right about server-side code not needing network bandwidth. I was
jumbling a few of my stories unclearly. Yes, we're starting with
server-side only process getting logged, but that will change. Once you
want to keep error, access, and event logs and tie them together, it's nice
to be able to log from the client side as well. I'm planning ahead a bit
here.

> doesn't have to be that way; you can have a local external database on
the client side.

If I'm going to use an external database, it's going to be Postgres. 4D has
given every appearance of abandoning further work on their SQL "support",
so I'm not going to base any designs on what amounts to a dead-end feature.
Sorry :( I liked the idea of SQL in 4D and loved the idea of external
database. But, for what it's worth, they're no usable from preemptive
processes anyway, so...

> but I am slightly puzzled how LOG EVENT is in the spotlight.

I brought it up because that's one way to get events to Loggly. Loggly
describes itself as an "agentless" service. Which just means "we use system
agents and don't provide our own." So, LOG EVENT should work. File that
under "commands I've never used". So I wasn't even clear if it was a sane
idea to ask for a preemptive-safe version. Seems like it should be. If it
isn't, I'll just put that idea out of my mind forever.

> preemptively logging can make things more complicated,

My basic design with a preemptive worker collecting log events in memory
centrally and pushing them out to a disk file is *entirely* normal. This is
what high-volume Web servers do. Probably anyone that's done a lot of
working thought of the same design *instantly* when they heard about CALL
WORKER. It's just that obvious. The only reason I can't do this is because
of a *crashing server bug* in 4D. As far as I can tell. So, good design,
proven design, sensible design...doesn't work. Doesn't workwith extreme
prejudice.

> a new ID was introduced because preemptive multi-line logs can be
interwoven.
Not sure which ID that is, but I'm planning on using the unique ID from
PROCESS PROPERTIES and a unified timeline to tie together events from
different logs related to the same process.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

On Tue, Sep 26, 2017 at 10:38 AM, Jeremy Roussak via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> David,
>
> Thanks for the explanation. Your second point (no reason to have the data
> in 4D) strikes me as more theological than practical,


Depends on your situation. I nearly killed a system once from pushing the
logging data over the network. Not philosophical, existential in the
contemporary sense of "life and death threat." So, that happened. But in a
small system? Everything is easier in a small system. So, yeah, unless
you're going to have tons of data in the logs and they're very busy, I'd
say that it's a matter of taste.


> but the rest are certainly excellent reasons. I wasn’t aware that 4D
> didn’t cope well with thrashing tables, so that’s something I’ll bear in
> mind in future.
>

Well, that's been my experience, perhaps things have changed? I usually
mange to slip out of the back of the room when they're asking for
volunteers to do server admin. Many years of suffering...don't like server
admin.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

> As stupid as it sounds, another option is to push the log entries to
something else locally - even a stand-alone built 4D app - and let it write
the disk files.
>  Then you could use a service with a custom agent, like SumoLogic or
Splunk. Actually, that's not a terrible idea at all...who cares if that
data file gets bloaty?
>  Clear it out, throw it out, start a new one.

Now I'm loving this idea. Do a special build to generate a stand-alone log
writer app and message it with HTTP. This tool is functioning as a logging
"agent." There are a lot of advantages to this approach in some
environments:

* You can put it on the same machine as the server(s), or not.

* You can send in all sorts of different logging data.

* You can do something smart with the data, like aggregates, if you want
to. (I don't want to, but someone else might like to do that.)

* You completely divide up stashing something in the log "send it to the
log writer" from what happens next. Want to change formats? Services?
Transport mechanisms? It's all right there. So, you could use disk files
one day and switch to HTTP upload the next...or pushing it into Postgres,
or whatever.

* Since the target app is a stand-alone, it can run preemptive processes.
Nice when you're generating log data on 4D Remote which _cannot_ run
preemptive processes.

Yeah, I think this has gone from Plan B to Plan A.

On Tue, Sep 26, 2017 at 8:28 AM, David Adams <dpad...@gmail.com> wrote:

> > Not having my server crash sounds like a very good reason to use records.
> > At least until a better solution comes along.
>
> Plan B in this case is to push the data to Loggly via HTTP.
>
> As stupid as it sounds, another option is to push the log entries to
> something else locally - even a stand-alone built 4D app - and let it write
> the disk files. Then you could use a service with a custom agent, like
> SumoLogic or Splunk. Actually, that's not a terrible idea at all...who
> cares if that data file gets bloaty? Clear it out, throw it out, start a
> new one.
>
>
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

> At least until a better solution comes along.

I have zero reason to assume that this bug will ever be fixed.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

> Not having my server crash sounds like a very good reason to use records.
> At least until a better solution comes along.

Plan B in this case is to push the data to Loggly via HTTP.

As stupid as it sounds, another option is to push the log entries to
something else locally - even a stand-alone built 4D app - and let it write
the disk files. Then you could use a service with a custom agent, like
SumoLogic or Splunk. Actually, that's not a terrible idea at all...who
cares if that data file gets bloaty? Clear it out, throw it out, start a
new one.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

On Mon, Sep 25, 2017 at 6:06 PM, Jeremy Roussak via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Please forgive me if I’m being dim, but isn’t a solution (maybe not the
> best, but a solution) to maintain the log as records in a table, which is
> periodically emptied into a file by a process which opens, writes and
> closes that file, then deletes the records in the table?
>
> It’s not an approach which I’ve tried, and you may have good reasons for
> rejecting it. I’m just curious to know what they are.
>
>
Jeremy,

Using a 4D table as a sort of cache for log lines is a perfectly sensible
idea, but also one I'd like to avoid. I did this some years back in a very
high-volume system and ended up killing a lot of performance as a
consequence. (Just ask Justin Leavens.) Now, you could take steps to reduce
the network activity involved, to be sure, but there are still a few points
that come to mind:

* 4D doesn't do well (or at least hasn't historically done well) with
tables that get thrashed a lot. Add-delete-add-delete. Gets really slow and
horrible. So then you have to do compacts, or hijinks with reusing records,
or all kinds of special magic with never deleting records. Life is too
short for any of that, if at all avoidable.

* There's no _real_ reason to have the data in 4D at any point. 4D isn't an
analysis platform and it isn't something that I'd use for
high-volume/low-data-content material like log entries. So, there's no good
reason for it to be in 4D other than as a cache. I would consider using a
small table as a cache for storing lines that didn't upload/output to a log
file, but only if the % of failure was pretty low.

* With a post-to-table-log-and-clear system, you now need a process to poll
the table. You can do this in a way that is relatively inexpensive, but
it's not free.

* Potentially lots of network traffic for zero gain.

But there are tons of ways to think about this sort of thing, and lots of
different setups and requirements. My judgment calls and current
constraints are likely different to what other people are dealing with. So,
there are plenty of good arguments for any particular design. I'm just
trying to be clear that while I'm not keen on the approach you suggest in
this case, it's not because I think it's inherently bad - I've used it
before. It's just that in this case I don't want the log data in 4D, if at
all possible. I see that as pure cost with no gain.

Hopefully, someone will find something silly in what I'm doing as the CALL
WORKER idea is really ver nice. You have n processes doing their thing -
running code, serving Web requests, handling windows, etc. Then they can
send log data (events, behaviors, errors, etc.) over to a central log
worker. Using CALL WORKER, this is quite fast. (4D can stack up tons of
these little requests quite well.) From there, you've got one process
handling all of the log data. A single external file for a particular log
is perfect at this point. It's fast and super easy to use. The whole point
of using a single process for this is to avoid contention on the file
amongst processes. That can otherwise be a *huge* and totally unworthy
bottleneck. So, one process for logging which has the file open in
read-write all the time. So good. Such a standard design. Simple, clean,
cheap, efficient...doesn't work in 4D. Unless there's a bug in my bug
report ;-)
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

On Mon, Sep 25, 2017 at 12:04 PM, Epperlein, Lutz (agendo) via 4D_Tech <
4d_tech@lists.4d.com> wrote:
I asked because we use a little own implementation of writing log files.
This is all before V16. We use semaphores to avoid race conditions while
writing to the file.
And one of our first projects after updating to V16 should be the rewriting
of this code using workers. So I'm a bit disappointed to hear that.

Yes, centralizing a log writer seems like the most obvious use of a worker,
so I also was disappointed. But perhaps I'm wrong? It would not be the
first time that I've done something stupid. Just in case I'm missing
something obvious, I've posted a copy of the test database I've used to
demonstrate the bug here:

http://www.4dcompanion.com/LogWorker.zip

It's a crude test database. What I was doing is compiling it and running
LogWorker_CallABunch_Coop or LogWorker_Test_Preemptive.

The test isn't to see if you can use CALL WORKER to execute code that
passes a log entry to a worker.
You can.

The test isn't to see if you can then keep the log file open just in the
worker to avoid contention on the file.
You can.

The test is to see what happens when you need to "roll" the log. Namely,
close the current log, rename it and open a new log.
This is where things go sideways.

If I remember correctly, I've retested this as far as 16.2 and R3 but not
R4. Perhaps a fresh set of eyes would help here. I do see that I tried
putting in a 3 second delay when rolling the log in case that gave 4D
and/or the OS a chance to catch up. Didn't help.

So, if I'm doing something foolish and wrong *please* tell me. I hate being
wrong, but I'd also rather know when I'm wrong. So, yeah, let me know. If
there's another way to approach this that's reliable, also please say. If
that way requires semaphores then, well, nothat kind of defeats the
whole purpose. At that point, I'd use non-preemptive code with NTK IPC
channels. That's been working flawlessly for years.

One thing: Any solution has to be 100% reliable. For server-side code like
this, a success rate of 99.9% isn't even close. When an error can bring the
server down, it's a non-starter. Particularly an error that's super easy to
generate.

The truth is that I'd really like to use disk files for logs. They're
simple, lightweight, cheap and robust. You can feed them into your
aggregator of choice in a million different ways, they take up no room in
the database, you can resume posting of entries after a crash, they're just
a solid choice. Hence, their near-universal popularity.

P.S. I know that using CALL WORKER inevitably means some log entries will
be lost during restart. That's an acceptable cost in this particular case.
Crashing/freezing the server is not.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

On Mon, Sep 25, 2017 at 10:43 AM, Epperlein, Lutz (agendo) <
lutz.epperl...@agendo.de> wrote:

> A question:
> What other approaches did you try to implement a logging feature using
> preemptive threads. I'm pretty sure there is something buried in the
> threads of this mailinglist but I'm not able to find it.
>
>
I would very likely have used files and an external collector...but this
didn't work out. Within 15 minutes of trying this, I discovered that it's
*very* easy to throw 4D into a fatal race condition when rolling log files.
(It's also possible in cooperative mode, for that matter.) I created a
database that reproduces the error within about a minute of running.
Submitted and, I'm told, accepted during version 16.0. Since then, zero
news and no change. So, for now, I can't rely on files because there is
*no* way to log to files if you can't safely roll them without bringing
down the server.

On macOS, I saw the system pushing errors and launching workers over and
over until I had to force quit it. Wayne Stewart tried it out on Windows
where it crashed 4D. This is for a Windows deployment, for what it's worth.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Making LOG EVENT thread-safe

[David Adams via 4D_Tech]

I finally got around to checking, and LOG EVENT is not a thread-safe
command. Pity! It would be very handy if this command were usable from a
preemptive process as it would give us a great way to stream log data out
to the system for collection by an "agentless" (cough-cough) tool like
Loggly.

Given the open bug that prevents me from rolling log files safely, I've got
to use memory-based solutions to get logs out of 4D. Since LOG EVENT isn't
usable preemptively, I'm down to HTTP Get/Request pushing data over HTTP to
the Loggly API. Not ideal, but that's the only thing that I can figure
that's even workable from a preemptive process at the moment.

If anyone else cares and agrees, would you mind posting a feature request
on the Forums to make LOG EVENT thread-safe? Or, if not, to tell us
definitively that LOG EVENT falls into the too hard basket. If I'm never
going to get a feature, it's better to know sooner rather than later.
**
4D Internet Users Group (4D iNUG)
FAQ:  http://lists.4d.com/faqnug.html
Archive:  http://lists.4d.com/archives.html
Options: http://lists.4d.com/mailman/options/4d_tech
Unsub:  mailto:4d_tech-unsubscr...@lists.4d.com
**

Re: SVG Scale

[David Adams via 4D_Tech]

> Regarding SVG scaling, I haven’t specifically used D3, but I’m sure it
makes scaling issues easier. However, scaling in SVG is still something
> that requires you to think through the relationships of your objects. It
isn’t magic, and it can’t guess what you have in mind.

Well said. Also, I should have made the point more clearly that you can get
the scaling library from D3 _without_ the rest of D3. No need to buy the
whole pig if you just want a pork chop.



On Wed, Sep 20, 2017 at 11:55 PM, Tom DeMeo via 4D_Tech <
4d_tech@lists.4d.com> wrote:

> Hi,
>
>
> > SVG is a whole lot easier to manage in JavaScript. If for no other
> reason,
> > you've got access to the D3 libraries.
>
> I would strongly concur with David Adams point that you want to manage SVG
> with javaScript instead of the 4D language. This is an object oriented
> scheme, and its pretty difficult to work with if you don’t have an object
> oriented language to work with it. The D3 libraries or some other library
> are important too, but the language itself is truly important.
>
>
> > * He's also pretty into that freight-car-function style from jQuery
> which I
> > hated on sight. But it does make it easier to set a ton of properties
> > legibly in a compact space. Virtually everything in D3 is set to return
> in
> > a way that supports this syntax so, yeah, be aware of that.
>
>
> This style is really just accepting a single hierarchical object of
> parameters. You can write code that defines attributes in objects in a lot
> of different ways in javaScript. I’d recommend to anyone that they spend
> some time considering a coding style that appeals to them and always try to
> do it the same way. The syntax the library’s examples shows you is not
> mandatory. Its just a javascript object, and as long as you respect the
> underlying architecture of that object, you can use whatever syntax you
> prefer.
>
>
> Regarding SVG scaling, I haven’t specifically used D3, but I’m sure it
> makes scaling issues easier. However, scaling in SVG is still something
> that requires you to think through the relationships of your objects. It
> isn’t magic, and it can’t guess what you have in mind.
>
> Consider the scheme on a 4D form. In 4D, the form itself and the objects
> have sizing parameters (none/move/grow). Splitters can provide additional
> control. Its a nice, simple scheme. However, if you don’t bother to set
> these parameters, you don’t get the scaling you want.
>
> A few years ago, I worked out an HTML layout scheme using a different
> javascript library, Raphael. I wrote a code library so that I could set
> move/grow parameters just like 4D provides in its forms, which I think is a
> pretty smart scheme. I haven’t done anything with it for quite a while, but
> I’m still hosting a demo, so you can check it out:
>
> http://www.svglayout.com/svglayout/index.html <http://www.svglayout.com/
> svglayout/index.html>
>
> When you get there, click on the SL widget in the upper right corner to
> activate the editor. You will see object outlines and handles and can drag
> objects around. You can also double click on an object to see the sizing
> and scaling attribute dialog. The whole site you see is built using an SVG
> based layout scheme. Traditional HTML layout strategies are not used.
>
>
>
> Tom DeMeo
>
>
>
>
>
>
>
> > --
> >
> > Message: 5
> > Date: Wed, 20 Sep 2017 09:02:08 +1000
> > From: David Adams <dpad...@gmail.com>
> > To: 4D iNug Technical <4d_tech@lists.4d.com>
> > Subject: Re: SVG Scale
> > Message-ID:
> >