On Mon, Sep 5, 2016 at 2:53 PM, Cannon Smith < [email protected]> > wrote: I also really enjoyed watching the video. It is a subject that I was > really excited about at the Summit and I appreciate the 4DMethod user group > making this possible.
> Thanks for any thoughts! I'll jump in with a few thoughts...but they don't necessarily address your problems directly. I've got a different perspective on queues because I always think of them as multi-machine systems, which totally colors how I look at queuing and message features. First off, I'd say there's merit in thinking about CALL WINDOW (let's fact it, they'll have to rename it CALL WINDOW or CALL FORM WINDOW eventually...so why don't we pretend it's happened already) and CALL WORKER as discrete features. They can interact, but they don't need to. Personally, I'm a lot more interested in CALL WORKER, but CALL WINDOW is probably vastly more important for UI work in 4D. In any case, I'm just more interested in CALL WORKER as it relates 4D to the larger world of messaging more obviously. (And because I really, really like headless tasks and they're part of my core programming style.) So, skipping CALL WINDOW for now and thinking about CALL WORKER, here are some random thoughts: * Think of tasks that can be processed without thinking how or where. * If you really want to scale, you want to bring in resources past 4D server. A generalized message bus can help with this. Instead of everything having to happen on 4D Server or it's connected clients, you can have processing happen over a range of clients. They can be written as stand-alone 4D's, PHP, Python, Ruby, etc. How nice is that? That is very nice. Very. Put another way, want a great way to get more cores/thread/optimize 4D Server? Push work out onto other machines running code outside of 4D Server. For a lot of systems, there are _heaps_ of tasks that fit well with this. If you don't need to write data back to the 4D database, you can often farm the work out. (Sending messages, composing/preparing/building Web pages, generating SMS, performing various calculations, etc.) * If you want a generalized message queue...or if you think that you might want one...pass one parameter as your message, and make that message a JSON block. This should work fine with CALL WORKER or any other message queue. The point is that the payload of the message is JSON with whatever data you need...the mechanics of message transfer can be modified, as needed. For example, you might start with CALL WORKER, then move to SQS and 4D stand-alones that read the messages. This transition could be pretty tidy. * Yeah, push rather than pull-polling. The wrong polling model for messages can really kill performance and throughput for no gain. Don't think of client-server, think of publisher & consumer. And don't think of them as bits of software think of them as roles in a message exchange. Thomas has a nice sort of example of this as an aside in his talk. When you're implementing an RPC pattern with EXECUTE ON SERVER, he says not to poll the server from the client but, instead, to use EXECUTE ON CLIENT from the server to the client to deliver finished results. Lots more efficient because there isn't any extra checking for incomplete results. In this example, the client calls the server and the server calls the client. So, "client" and "server" don't mean anything - you've got one piece of software sending a message to another. More like email than a Web server. * IP variables, semaphores, etc. Forget it. Brittle, not going to persist, 4D specific, can't scale, a real headache. I've written code to do all of that, it can work perfectly...but it's easy to break. Use records. Just to be entirely clear, I'm not saying that IP variables (etc.) can't be programmed to work perfectly, I'm saying that it's a dead-end. It hasn't made sense in server-side programing for a long time. (Read: Since day 1.) If it doesn't work in triggers, it probably doesn't make sense in other server side code. Just my opinion, but this time I'm right ;-) * Speaking of records, you can have the fanciest message bus on earth and there are _still_ times when you're going to need a database to track state/condition of a task, make message persist, log things, etc. Not strange...very normal. Given that, if you've got very complex or large data to transfer, you can stash it in a record and then your message JSON should include a reference to the record so that the worker can retrieve all of the required details. This strategy will prepare you to deal with cloud-based message queues which often have fairly restrictive message payload limits. * But what if I'm in a transaction! Not a problem for me that often, but CALL WORKER gets around that quite beautifully in most scenarios that I can think of - you just threw the problem over the wall into another process. * Regarding passing selections, etc. A traditional (non-4D) way of dealing with this is to pass enough information for the worker to select the necessary data. In the SQL world, that makes sense anyway since "selection" isn't a concept. In the 4D world, you can build an array of record IDs and put them through JSON or stash them in a record and push that into a record. Pretty 4D-specific, but there are times where that makes perfect sense. ********************************************************************** 4D Internet Users Group (4D iNUG) FAQ: http://lists.4d.com/faqnug.html Archive: http://lists.4d.com/archives.html Options: http://lists.4d.com/mailman/options/4d_tech Unsub: mailto:[email protected] **********************************************************************
