Re: New record numbering system and CALL WORKER
Hi Pat, Set the method property to "Execute on Server". This shared object/collection could also have a collection of numbers that need to be reused if you require that there are no missing numbers in your sequence. Of course, you'll have to get the database involved at some level to handle startup and shutdown and the case where the server crashed without writing the last sequence number used. John DeSoi, Ph.D. > On Aug 25, 2018, at 10:21 AM, Pat Bensky via 4D_Tech <4d_tech@lists.4d.com> > wrote: > > Thanks John. > But how would that work in a multi-user setting? Whatever method is used > for assigning the new numbers, it must run on the server, otherwise we're > back to the problem of using semaphores, checking record locking, etc. > Also I should point out that there could be a large number of tables for > which the record numbering system needs to be used. We have a RecordNumbers > table which has one record for each table that requires this function. so > we must have ONE process that accesses and updates that table. > PB ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: New record numbering system and CALL WORKER
On Aug 25, 2018, at 4:33 PM, Christian Sakowski wrote: > i mainly use Storage as „constants“. I store global attributes and user > specific options and access rights. Those are in fact mostly read only. Hi Christian, And that is a great use of “Storage”. I’ll be doing the same thing in new databases I create. But lets explore other possibilities. What could “Storage” be use for in other cases? Is it a good use of “Storage”? One of the great things about 4D is that features designed and intended to be use for one situation turn out to be very useful in other situations. And I love the “big brains” on the iNUG exposing these new uses. Tim * Tim Nevels Innovative Solutions 785-749-3444 timnev...@mac.com * ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: New record numbering system and CALL WORKER
Tim, i mainly use Storage as „constants“. I store global attributes and user specific options and access rights. Those are in fact mostly read only. -- Grüße/Regards, [heubach-media] | Christian Sakowski christian.sakow...@heubach-media.de iChat/AIM: SakowskiF Tel: +49/(0)40/52 10 59-23 > Am 25.08.2018 um 22:57 schrieb Tim Nevels : > > On Aug 25, 2018, at 2:25 PM, Christian Sakowski > wrote: > >> This is not necessary. For just reading you don’t need to use „use“. >> For generating sequence numbers, you have to use „auto increment“. Nothing >> else. >> For invoice numbers, you now can save the number to the database within >> transactions with SUSPEND/RESUME TRANSACTION (of course, locking and waiting >> if necessary…) > > Hi Christian, > > You are correct, it is not necessary to “Use” a shared object if you are only > going to read it. I’m still getting used to this new 4D feature. Thanks for > reminding me of that. > > Regarding sequence numbers and using the “auto increment” or the “Sequence > number” command, I always use that. I agree it’s the best, period. > > But this is really a discussion of how one might use shared storage. Rightly > or wrongly. Trying to understand how to use this new 4D feature and gather > some ideas on possible ways of using it. I’m still learning and trying to > understand it. > > Tim > > * > Tim Nevels > Innovative Solutions > 785-749-3444 > timnev...@mac.com > * > -- heubach media Osterfeldstr. 12-14 | Haus 1 | Eingang Nord 22529 Hamburg tel: 040 / 52 10 59 - 10 | fax: -99 mail: i...@heubach-media.de home: www.heubach-media.de Geschäftsführer|CEO: Matthias Heubach Mieten Sie Ihre Computer, iPads & Drucker für Ihre Events bei: http://www.milo-rental.com Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: New record numbering system and CALL WORKER
On Aug 25, 2018, at 2:25 PM, Christian Sakowski wrote: > This is not necessary. For just reading you don’t need to use „use“. > For generating sequence numbers, you have to use „auto increment“. Nothing > else. > For invoice numbers, you now can save the number to the database within > transactions with SUSPEND/RESUME TRANSACTION (of course, locking and waiting > if necessary…) Hi Christian, You are correct, it is not necessary to “Use” a shared object if you are only going to read it. I’m still getting used to this new 4D feature. Thanks for reminding me of that. Regarding sequence numbers and using the “auto increment” or the “Sequence number” command, I always use that. I agree it’s the best, period. But this is really a discussion of how one might use shared storage. Rightly or wrongly. Trying to understand how to use this new 4D feature and gather some ideas on possible ways of using it. I’m still learning and trying to understand it. Tim * Tim Nevels Innovative Solutions 785-749-3444 timnev...@mac.com * ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: New record numbering system and CALL WORKER
> Use (Storage) > [MyNextNumber]NextNumber:=Storage.myNextNumber > End use This is not necessary. For just reading you don’t need to use „use“. For generating sequence numbers, you have to use „auto increment“. Nothing else. For invoice numbers, you now can save the number to the database within transactions with SUSPEND/RESUME TRANSACTION (of course, locking and waiting if necessary…) -- Grüße/Regards, [heubach-media] | Christian Sakowski christian.sakow...@heubach-media.de iChat/AIM: SakowskiF Tel: +49/(0)40/52 10 59-23 > Am 25.08.2018 um 21:10 schrieb Tim Nevels via 4D_Tech <4d_tech@lists.4d.com>: > > On Aug 25, 2018, at 10:33 AM, Pat Bensky wrote: > >> Thanks John. >> But how would that work in a multi-user setting? Whatever method is used >> for assigning the new numbers, it must run on the server, otherwise we're >> back to the problem of using semaphores, checking record locking, etc. >> Also I should point out that there could be a large number of tables for >> which the record numbering system needs to be used. We have a RecordNumbers >> table which has one record for each table that requires this function. so >> we must have ONE process that accesses and updates that table. > > I like John’s idea of using the new shared object “Storage”. This has a > built-in, super efficient, locking mechanism using the “Use/End Use” system. > My first thought were that this could be the ultimate “next sequence number” > system if you have already used the “Sequence number” attribute for a table > (which is already the ultimate “next sequence number” system for a table in > terms of speed and reliability in a multi-user and transactional system. > > So here is an idea for a basic sequence number system based on shared > “Storage” object. The goals are: > > - Make it as fast as possible > - Make it work on single-user and client/server with the least amount of code > - Use built-in 4D features wherever you can > - Works anywhere and everywhere, no special cases to deal with > - any process cooperative or preemptive > - in triggers > - inside or outside transactions > - from a form or from a process with no UI > - Don’t care about “lost numbers” (holes in the sequence numbers are OK) > > Sorry Pat, you’ll have to complicate the system to make it work for your > specific needs, but that should be doable with a more complex storage object. > I just wanted to present the concept I have here in the simplest terms for > easy digestion. > > 1. Initialization - this needs to be done once for a database application. If > you are running single-user, do this in the “On Startup” method. If you are > running client/server do this in “On Server Startup” > > Use (Storage) > Storage.myNextNumber:=1 > End use > > 2. GetNextNumber - this is a method that will return the next number from the > shared storage. Set the method property for “Execute on server”. This will > make it work in single-user or client/server with no code changes. > > // Method: GetNextNumber > > C_LONGINT($0) > > Use (Storage) > $0:=Storage.myNextNumber > Storage.myNextNumber:=$0+1 // increment sequence > End use > > 3. Permanent storage of the next number - If you want this to maintain it’s > value across launches of the application, use a table in the database. So in > the “initialization” part above instead of setting "Storage.myNextNumber:=1” > you would query a table and get the next number value from a field in a > record. When the database quits you update that value so it is available for > the next “On Startup” or “On Server Startup” You would put this code in the > “On Exit” or “On Server Shutdown”. > > 4. What if my database crashes and my code does not get a chance to do a > proper save of the next number to the database? A valid consideration and one > I think should be addressed. We all know that 4D will crash sometimes, so > best to be prepared and handle it as best we can. > > How about in the “initialization” part you also start a new process whose job > is to save the next number from the storage object to a database record. Have > this process run as often as you are comfortable with. If you are journaling > your data file — and you should be — the record update will be in the journal > file so that if 4D crashes, on the next startup it will integrate the change > into the database and you’ve not lost anything. > > You could bypass the whole shared storage approach and just do database > operations on records, but I would speculate that using shared storage is > just a tad bit faster than doing a query, wait for record to be unlocked, > increment, save record, unload record operation. Might not be, I’ve not done > any testing to see which is faster. Just seems like less overhead to not call > the database engine every time you need a next number. But it does introduce > a "cache within a cache” situation. And the potential risk of losing that > last sequence
Re: New record numbering system and CALL WORKER
On Aug 25, 2018, at 10:33 AM, Pat Bensky wrote: > Thanks John. > But how would that work in a multi-user setting? Whatever method is used > for assigning the new numbers, it must run on the server, otherwise we're > back to the problem of using semaphores, checking record locking, etc. > Also I should point out that there could be a large number of tables for > which the record numbering system needs to be used. We have a RecordNumbers > table which has one record for each table that requires this function. so > we must have ONE process that accesses and updates that table. I like John’s idea of using the new shared object “Storage”. This has a built-in, super efficient, locking mechanism using the “Use/End Use” system. My first thought were that this could be the ultimate “next sequence number” system if you have already used the “Sequence number” attribute for a table (which is already the ultimate “next sequence number” system for a table in terms of speed and reliability in a multi-user and transactional system. So here is an idea for a basic sequence number system based on shared “Storage” object. The goals are: - Make it as fast as possible - Make it work on single-user and client/server with the least amount of code - Use built-in 4D features wherever you can - Works anywhere and everywhere, no special cases to deal with - any process cooperative or preemptive - in triggers - inside or outside transactions - from a form or from a process with no UI - Don’t care about “lost numbers” (holes in the sequence numbers are OK) Sorry Pat, you’ll have to complicate the system to make it work for your specific needs, but that should be doable with a more complex storage object. I just wanted to present the concept I have here in the simplest terms for easy digestion. 1. Initialization - this needs to be done once for a database application. If you are running single-user, do this in the “On Startup” method. If you are running client/server do this in “On Server Startup” Use (Storage) Storage.myNextNumber:=1 End use 2. GetNextNumber - this is a method that will return the next number from the shared storage. Set the method property for “Execute on server”. This will make it work in single-user or client/server with no code changes. // Method: GetNextNumber C_LONGINT($0) Use (Storage) $0:=Storage.myNextNumber Storage.myNextNumber:=$0+1 // increment sequence End use 3. Permanent storage of the next number - If you want this to maintain it’s value across launches of the application, use a table in the database. So in the “initialization” part above instead of setting "Storage.myNextNumber:=1” you would query a table and get the next number value from a field in a record. When the database quits you update that value so it is available for the next “On Startup” or “On Server Startup” You would put this code in the “On Exit” or “On Server Shutdown”. 4. What if my database crashes and my code does not get a chance to do a proper save of the next number to the database? A valid consideration and one I think should be addressed. We all know that 4D will crash sometimes, so best to be prepared and handle it as best we can. How about in the “initialization” part you also start a new process whose job is to save the next number from the storage object to a database record. Have this process run as often as you are comfortable with. If you are journaling your data file — and you should be — the record update will be in the journal file so that if 4D crashes, on the next startup it will integrate the change into the database and you’ve not lost anything. You could bypass the whole shared storage approach and just do database operations on records, but I would speculate that using shared storage is just a tad bit faster than doing a query, wait for record to be unlocked, increment, save record, unload record operation. Might not be, I’ve not done any testing to see which is faster. Just seems like less overhead to not call the database engine every time you need a next number. But it does introduce a "cache within a cache” situation. And the potential risk of losing that last sequence number before your process “flushes” it to the database cache and writes to the journal file. But one of my above stated goals was maximum speed, so I think this will buy you a tiny bit of extra speed. Here is some example code for saving the next number storage value to the database. Put this in a method that is called in either “On Startup” or “On Server Startup”. To make is super simple, [MyNextNumber] table only contains 1 record and 1 field. And no other process is ever allowed to do any write operations to table [MyNextNumber]. The only process to modify it is “SaveMyNextNumber” then you don’t have to worry about possible record locking issues. $processID_l:=New Process(“SaveMyNextNumber”;0) // Method: SaveMyNextNumber ALL RECORDS([MyNextNumber]) if
Re: New record numbering system and CALL WORKER
Pat, On Sat, Aug 25, 2018 at 8:25 AM Pat Bensky via 4D_Tech <4d_tech@lists.4d.com> wrote: > What you say makes sense, but the record number as key field is so integral > to the way the program works, it would be a nightmare to try and change it > Understood. But this approach still works, you just have more values in the 'returned' collection. Of course I'm assuming you're not going to have hundreds of returned counters. That could make a difference. And both methods can be run EOS to speed up the multiuser environment. Try it out. I know that years ago that much querying and checking record locking would have been slow. It's not now. -- Kirk Brooks San Francisco, CA === *We go vote - they go home* ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: New record numbering system and CALL WORKER
Hi Kirk,
What you say makes sense, but the record number as key field is so integral
to the way the program works, it would be a nightmare to try and change it
:) If I was starting a project from scratch, that would be a different
story.
Pat
On Sat, 25 Aug 2018 at 16:20, Kirk Brooks via 4D_Tech <4d_tech@lists.4d.com>
wrote:
> Pat,
> A couple of thoughts since it sounds like you're refactoring your code (all
> pseudo code, v17):
> 1) don't use this number as a key field.
>
> 2) The counter record:
>
> [COUNTER]
>
> ID
>
> name - optional but a good idea
>
> _data - object field: { nextNumber: 0, returned: []}
>
> 3) Counter_get_next(text) -> long
>
> $1 is name of counter series
>
> Put [COUNTER] in read/write
>
> Query for record
>
> Wait until record is unlocked
>
> `
>
> if(COUNTER._data.returned.length>0)` use the first one here
>
> $0:=
> COUNTER._data.returned.
> shift
> ()
>
> else
>
> $0:=
> COUNTER._data.nextNumber
>
> COUNTER._data.nextNumber:=
> COUNTER._data.nextNumber
> +1
>
> end if
>
> SAVE RECORD
>
> UNLOAD RECORD
>
>
> 4) Counter_save_unused (text; long)
>
> Same idea except you just push $2 onto 'returned'. You could add a sort
> command to make sure the smallest returned value is on top if that's an
> issue.
>
>
> Several years ago I stopped using invoice or PO numbers (or phone numbers
> or SSNs or any data value) as key fields. This greatly simplifies the issue
> you may have had about the 'unused' numbers that come up when data entry is
> canceled. If you're not using that number as a key field you don't need to
> assign it until the transaction completes. But if you've got some other
> situations that result in 'returned' numbers it's not big deal.
>
> This approach let's 4D handle the record locking and request que and as
> long as you don't call it within a transaction just blissfully works.
>
>
>
> On Sat, Aug 25, 2018 at 7:11 AM Pat Bensky via 4D_Tech <
> 4d_tech@lists.4d.com>
> wrote:
>
> > Using v17 ...
> >
> > I'm looking at ways to update our ancient record numbering system, and I
> > think I should be able to do it efficiently with CALL WORKER, but having
> a
> > bit of trouble getting my head around it.
> >
> > The record numbers are like invoice numbers: each must be unique, and we
> > don't want to have any gaps in the numbering sequence. So we need to keep
> > track of any record numbers that are created and then abandoned (eg if a
> > user creates a new record and then cancels it).
>
> --
> Kirk Brooks
> San Francisco, CA
> ===
>
> *We go vote - they go home*
> **
> 4D Internet Users Group (4D iNUG)
> Archive: http://lists.4d.com/archives.html
> Options: https://lists.4d.com/mailman/options/4d_tech
> Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
> **
--
*
CatBase - Top Dog in Data Publishing
tel: +44 (0) 207 118 7889
w: http://www.catbase.com
skype: pat.bensky
*
**
4D Internet Users Group (4D iNUG)
Archive: http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
**
Re: New record numbering system and CALL WORKER
Thanks John. But how would that work in a multi-user setting? Whatever method is used for assigning the new numbers, it must run on the server, otherwise we're back to the problem of using semaphores, checking record locking, etc. Also I should point out that there could be a large number of tables for which the record numbering system needs to be used. We have a RecordNumbers table which has one record for each table that requires this function. so we must have ONE process that accesses and updates that table. PB On Sat, 25 Aug 2018 at 15:49, John DeSoi via 4D_Tech <4d_tech@lists.4d.com> wrote: > > > On Aug 25, 2018, at 9:10 AM, Pat Bensky via 4D_Tech < > 4d_tech@lists.4d.com> wrote: > > > > As I see it, we'll need to call a worker, which calls the worker that > > assigns the record number. > > The first worker has to wait until the second one has created the new > > number. > > Then the original method, that called the first worker, must wait until > the > > first worker gets the new number. It seems that this won't be much > better > > than the old method. > > > > Am I completely barking up the wrong tree? Is this not an appropriate use > > for workers? Or have I misunderstood something? > > It would have been really nice if CALL WORKER was implemented like EXECUTE > METHOD and could return a value. You can do this on your own, but it is > quite a bit of work to do it in a general way. > > For your application, all you need is a shared object (no worker needed). > Only one process at a time will be able to modify the counter. Some startup > method could initialize the counter to the starting value. > > C_LONGINT($0) > > Use (Storage.myCounter) > Storage.myCounter.value:= Storage.myCounter.value+1 > $0:= Storage.myCounter.value > End use > > > John DeSoi, Ph.D. > > ** > 4D Internet Users Group (4D iNUG) > Archive: http://lists.4d.com/archives.html > Options: https://lists.4d.com/mailman/options/4d_tech > Unsub: mailto:4d_tech-unsubscr...@lists.4d.com > ** -- * CatBase - Top Dog in Data Publishing tel: +44 (0) 207 118 7889 w: http://www.catbase.com skype: pat.bensky * ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
Re: New record numbering system and CALL WORKER
Pat,
A couple of thoughts since it sounds like you're refactoring your code (all
pseudo code, v17):
1) don't use this number as a key field.
2) The counter record:
[COUNTER]
ID
name - optional but a good idea
_data - object field: { nextNumber: 0, returned: []}
3) Counter_get_next(text) -> long
$1 is name of counter series
Put [COUNTER] in read/write
Query for record
Wait until record is unlocked
`
if(COUNTER._data.returned.length>0)` use the first one here
$0:=
COUNTER._data.returned.
shift
()
else
$0:=
COUNTER._data.nextNumber
COUNTER._data.nextNumber:=
COUNTER._data.nextNumber
+1
end if
SAVE RECORD
UNLOAD RECORD
4) Counter_save_unused (text; long)
Same idea except you just push $2 onto 'returned'. You could add a sort
command to make sure the smallest returned value is on top if that's an
issue.
Several years ago I stopped using invoice or PO numbers (or phone numbers
or SSNs or any data value) as key fields. This greatly simplifies the issue
you may have had about the 'unused' numbers that come up when data entry is
canceled. If you're not using that number as a key field you don't need to
assign it until the transaction completes. But if you've got some other
situations that result in 'returned' numbers it's not big deal.
This approach let's 4D handle the record locking and request que and as
long as you don't call it within a transaction just blissfully works.
On Sat, Aug 25, 2018 at 7:11 AM Pat Bensky via 4D_Tech <4d_tech@lists.4d.com>
wrote:
> Using v17 ...
>
> I'm looking at ways to update our ancient record numbering system, and I
> think I should be able to do it efficiently with CALL WORKER, but having a
> bit of trouble getting my head around it.
>
> The record numbers are like invoice numbers: each must be unique, and we
> don't want to have any gaps in the numbering sequence. So we need to keep
> track of any record numbers that are created and then abandoned (eg if a
> user creates a new record and then cancels it).
--
Kirk Brooks
San Francisco, CA
===
*We go vote - they go home*
**
4D Internet Users Group (4D iNUG)
Archive: http://lists.4d.com/archives.html
Options: https://lists.4d.com/mailman/options/4d_tech
Unsub: mailto:4d_tech-unsubscr...@lists.4d.com
**
New record numbering system and CALL WORKER
Using v17 ... I'm looking at ways to update our ancient record numbering system, and I think I should be able to do it efficiently with CALL WORKER, but having a bit of trouble getting my head around it. The record numbers are like invoice numbers: each must be unique, and we don't want to have any gaps in the numbering sequence. So we need to keep track of any record numbers that are created and then abandoned (eg if a user creates a new record and then cancels it). The old system works well but it uses semaphores and SET/GET PROCESS VARIABLE and DELAY PROCESS. I isn't very efficient, especially when a large number of records are being imported and each one needs a new record number. Record numbers must be assigned synchronously - processing must not continue until the new record number has been assigned. Also, this frequently needs to be done in a context where there is no active form, so CALL FORM won't work. As I see it, we'll need to call a worker, which calls the worker that assigns the record number. The first worker has to wait until the second one has created the new number. Then the original method, that called the first worker, must wait until the first worker gets the new number. It seems that this won't be much better than the old method. Am I completely barking up the wrong tree? Is this not an appropriate use for workers? Or have I misunderstood something? Pat -- * CatBase - Top Dog in Data Publishing tel: +44 (0) 207 118 7889 w: http://www.catbase.com skype: pat.bensky * ** 4D Internet Users Group (4D iNUG) Archive: http://lists.4d.com/archives.html Options: https://lists.4d.com/mailman/options/4d_tech Unsub: mailto:4d_tech-unsubscr...@lists.4d.com **
