Re: Making LOG EVENT thread-safe

Hey John, we ended up testing different things, I didn't ever get too far into the memory side. (Thanks for that, good to know.) I retested again last night and I can kill 4D from entirely legal code. I'll show you at dinner, if you like. There is a *concurrency bug*. I'm not convinced that

Re: Making LOG EVENT thread-safe

One thing I forgot to mention/ask… David’s test routine runs significantly faster with 32-bit 4D than it does with 64-bit 4D. Any ideas as to why this might be the case? To refresh memories as to what the test is doing… For ($1;1;10) $prsNum:=New Process(“TestProcess”.0;”TestProcess”)

Re: Making LOG EVENT thread-safe

Got an email from Timothy Penner yesterday pointing out that in his testing of this issue he has concluded that the crash occurs because 4D is being pushed past it’s memory limits. That is if run using the 32bit version of 4D. Tim suggested that we run the test using the 64 bit version of 4D.

Re: Making LOG EVENT thread-safe

question: was your IP var a longint?, Real?, or Doublelong (whatever it is called)? This might make some difference to your testing.. BUT... based on your discovered size limit, it seems that the Worker Queue is limited by some internal reference to a long integer... MaxLong being

Re: Making LOG EVENT thread-safe

> As a side note, I am sure you are aware of this, but by killing the worker every 100,000 calls, out of the 2 million calls > nearly all of the log entries are missed as they are in the queue when the worker is killed. I am getting at most 24,000 > log entries in the text file. I've lost track,

Re: Making LOG EVENT thread-safe

> On Oct 8, 2017, at 5:53 PM, David Adams via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > Oh, I forgot to say earlier about John's finding of ~2.1M messages being a > kind of breaking point...that number may not be replicable in other tests. > You might find a different number. The payloads in

Re: Making LOG EVENT thread-safe

> Once it happens, you get all kinds of weird stuff - > duplicate workers with "unique" names, the runtime monitor gets super buggy > and starts throwing range errors...until you're dead. If you *don't* find > this result on your gear, please post what version of 4D you're testing > with. Yes I

Re: Making LOG EVENT thread-safe

Add, On the entirely likely chance that my first answer sounded insufferably arrogant - just wanted to addWell, I'm sleep deprived like I can't remember, and massively jet lagged. I was trying to say that this whole area is one of my *few* areas of genuine expertise. I don't have many. Just

Re: Making LOG EVENT thread-safe

> I understand your thinking that 4D may not the way to go for event logging, > but your concern about CLOSE DOCUMENT is where we do not agree. I'am sure you > have seen CLOSE DOCUMENT fail, but I suspect that what you are seeing is not > an issue with CLOSE DOCUMENT, but something else not

Re: Making LOG EVENT thread-safe

Hey John, thanks for the great post and all the work. There's no reason for other people to know this, but John and I have *totally* different coding styles. So I was really pleased he went through my example and reworked it in his own way. That he turned up some problems is instructive because

Re: Making LOG EVENT thread-safe

Add, Thanks a lot for chiming in! It's always good to hear from you guys in the US office. I'm not sure what problem your solution is solving. I'm not trying to be flip, I just want to make sure to focus on the meat of the issue. A really common approach to log writing is as follows: Have one

Re: Making LOG EVENT thread-safe

I was never focused on LOG EVENT (?) I've never used the command and would prefer not to in this case. I only brought it up because it would potentially well with Loggly. (I'm not committed to Loggy and may not stick with that platform, for what that's worth. But it's worth trying for now.) I

Re: Making LOG EVENT thread-safe

LAUNCH EXTERNAL PROCESS becomes thread safe in 16R5 http://forums.4d.com/Post/FR/21049933/1/21050200#21050200 which opens many possibilities, the logger command on Mac, for example. perhaps we need not focus on LOG EVENT anymore...

Re: Making LOG EVENT thread-safe

I've posted about my various worker log problems here and on some other threads and even posted my sample database for review. Lately, John Baughman has really been sinking his teeth into this issue and has spent a *ton* of time working on it. Thank you John! The news is bad, but I figured I'd

Re: Making LOG EVENT thread-safe

Related wrinkleI'm now wondering about something else: Does KILL WORKER completely kill a worker, or does it sometimes just hurt it really, really badly? I have been working under the assumption that when KILL WORKER completes, that the worker is *totally* gone. Meaning, no process-level

Re: Making LOG EVENT thread-safe

Hey, David Porter spotted a bug in my code and I wanted to push this part of the thread up to the list for the archives, etc.: On Tue, Sep 26, 2017 at 5:05 PM, David Porter wrote: > Thanks for posting the LogWorker code, it has been fun experimenting with it. > > I have

Re: Making LOG EVENT thread-safe

Hey Tim, > Consider the situation where you have a mission critical system that is up 24/7 and you need logging to always work every second of the day. > How do you handle the situation where the stand-alone log writer app crashes, is not running due to a machine restart, or for whatever reason

Re: Making LOG EVENT thread-safe

Keisuke, Thanks for the background information on LOG EVENT, your details are *always* interesting. In my case, I don't think that it has anything to do with what I'm dealing with. I'm not talking about 4D events or 4D logs, I'm talking about custom logs with data I'm sending out. LOG EVENT isn't

Re: Making LOG EVENT thread-safe

On Sep 26, 2017, at 9:08 AM, David Adams wrote: >> As stupid as it sounds, another option is to push the log entries to >> something else locally - even a stand-alone built 4D app - and let it write >> the disk files. >> Then you could use a service with a custom agent, like SumoLogic or >>

Re: Making LOG EVENT thread-safe

David, Thanks for posting the LogWorker code, it has been fun experimenting with it. I have been testing only the preemptive test code. Running on a Mac Pro (Late 2013) with 6-Core Intel Xeon E5 , Mac OS X 10.12.6 4D 16.1 hotfix 2 , 64 bit app. ( Not server ) Yes, there is a bug in the code.

Re: Making LOG EVENT thread-safe

sometimes you will see a kind of "comment" line in the request log, in particular "TTF" or time to forget, which describes a change of state in the process: a: automatic relation b: record stack c: current record d: current selection e: transaction these are not real "requests", in that the

Re: Making LOG EVENT thread-safe

Chip's right about server-side code not needing network bandwidth. I was jumbling a few of my stories unclearly. Yes, we're starting with server-side only process getting logged, but that will change. Once you want to keep error, access, and event logs and tie them together, it's nice to be able

Re: Making LOG EVENT thread-safe

well, Begin SQL/End SQL is not thread safe either so that's not going to help. but back to the original point: LOG EVENT has many modes, is there a particular type of log that you wish were thread safe? because, technically, you could just call a command with arguments and that would be

Re: Making LOG EVENT thread-safe

doesn't have to be that way; you can have a local external database on the client side. http://doc.4d.com/4Dv15/4D/15/USE-DATABASE.300-2288124.en.html but I am slightly puzzled how LOG EVENT is in the spotlight. preemptively logging can make things more complicated, look at 4DRequestsLog.txt /

Re: Making LOG EVENT thread-safe

so, if this is a logging issue, will not most/all events requiring logging pass through the server? so logging would not/should not take network bandwidth - log from trigger, log from stored methods, log from 'execute on server' >> >> >> Thanks for the explanation. Your second point (no reason

Re: Making LOG EVENT thread-safe

On Tue, Sep 26, 2017 at 10:38 AM, Jeremy Roussak via 4D_Tech < 4d_tech@lists.4d.com> wrote: > David, > > Thanks for the explanation. Your second point (no reason to have the data > in 4D) strikes me as more theological than practical, Depends on your situation. I nearly killed a system once

Re: Making LOG EVENT thread-safe

David, Thanks for the explanation. Your second point (no reason to have the data in 4D) strikes me as more theological than practical, but the rest are certainly excellent reasons. I wasn’t aware that 4D didn’t cope well with thrashing tables, so that’s something I’ll bear in mind in future.

Re: Making LOG EVENT thread-safe

> As stupid as it sounds, another option is to push the log entries to something else locally - even a stand-alone built 4D app - and let it write the disk files. > Then you could use a service with a custom agent, like SumoLogic or Splunk. Actually, that's not a terrible idea at all...who cares

Re: Making LOG EVENT thread-safe

> At least until a better solution comes along. I have zero reason to assume that this bug will ever be fixed. ** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html

Re: Making LOG EVENT thread-safe

> Not having my server crash sounds like a very good reason to use records. > At least until a better solution comes along. Plan B in this case is to push the data to Loggly via HTTP. As stupid as it sounds, another option is to push the log entries to something else locally - even a stand-alone

Re: Making LOG EVENT thread-safe

Not having my server crash sounds like a very good reason to use records. At least until a better solution comes along. My 2¢ On Mon, Sep 25, 2017 at 1:03 PM, David Adams via 4D_Tech < 4d_tech@lists.4d.com> wrote: > On Mon, Sep 25, 2017 at 6:06 PM, Jeremy Roussak via 4D_Tech < >

Re: Making LOG EVENT thread-safe

On Mon, Sep 25, 2017 at 6:06 PM, Jeremy Roussak via 4D_Tech < 4d_tech@lists.4d.com> wrote: > Please forgive me if I’m being dim, but isn’t a solution (maybe not the > best, but a solution) to maintain the log as records in a table, which is > periodically emptied into a file by a process which

Re: Making LOG EVENT thread-safe

Please forgive me if I’m being dim, but isn’t a solution (maybe not the best, but a solution) to maintain the log as records in a table, which is periodically emptied into a file by a process which opens, writes and closes that file, then deletes the records in the table? It’s not an approach

Re: Making LOG EVENT thread-safe

On Mon, Sep 25, 2017 at 12:04 PM, Epperlein, Lutz (agendo) via 4D_Tech < 4d_tech@lists.4d.com> wrote: I asked because we use a little own implementation of writing log files. This is all before V16. We use semaphores to avoid race conditions while writing to the file. And one of our first projects

RE: Making LOG EVENT thread-safe

I asked because we use a little own implementation of writing log files. This is all before V16. We use semaphores to avoid race conditions while writing to the file. And one of our first projects after updating to V16 should be the rewriting of this code using workers. So I'm a bit

Re: Making LOG EVENT thread-safe

On Mon, Sep 25, 2017 at 10:43 AM, Epperlein, Lutz (agendo) < lutz.epperl...@agendo.de> wrote: > A question: > What other approaches did you try to implement a logging feature using > preemptive threads. I'm pretty sure there is something buried in the > threads of this mailinglist but I'm not

RE: Making LOG EVENT thread-safe

A question: What other approaches did you try to implement a logging feature using preemptive threads. I'm pretty sure there is something buried in the threads of this mailinglist but I'm not able to find it. Thanks Lutz Epperlein > -Original Message- > From: 4D_Tech

Making LOG EVENT thread-safe

I finally got around to checking, and LOG EVENT is not a thread-safe command. Pity! It would be very handy if this command were usable from a preemptive process as it would give us a great way to stream log data out to the system for collection by an "agentless" (cough-cough) tool like Loggly.