Re: Fast copying of records from one table to another

[Charlie Noah]

Excellent comments, Kevin, especially about needing an explanation of the system or business goals. As Johnny 5 said in Short Circuit, "Malfunction. Need input". Charlie On 05-19-2016 5:41 PM, Kevin Powick wrote: Is this some type of backup/replication solution?  If not, why are you making copies of records between the two files?  SSOT (single source of truth) should be a goal.  i.e. no duplicate records in the system. If after the copy process the two files actually do have identical records, then maybe you can just copy the item IDs to the second file and not the actual record.  One file is your SSOT and the other is simply a list of item IDs. It's a bit difficult to know the best approach on this without an explanation of the system or business goals. Other things to consider: Suppress terminal output to greatly speed-up operations. Verify that you really need a JQL SELECT statement.  Using SELECT reads every item in the file once, then add to that every READ your program does.  Imagine a files with 1000 records in it, and your SELECT results in 500 records for your program to process.  The result will be at least 1500 reads (1000 for the select and 500 for your program).  You could eliminate 500 of those reads by replacing JQL SELECT with an internal JBC SELECT, letting your program read all records in the file, and adding record eligibility logic to your processing program.  i.e. There is often no reason to use a JQL SELECT to filter data for use in a processing program unless record order is important.  (SSELECT). -- Kevin Powick On Thursday, 19 May 2016 06:00:48 UTC-4, Paweł Birgiel wrote: Hi. I have a task to make a service which will copy records from one table to another as fast as possible (using at most 5 agents). I have a problem with choosing the best way to do that. The most obvious solution seems to be using one READ and one WRITE (they appear to be a bit faster than F.READ and F.WIRTE) per each Y.ID in my main routine (after selecting list of records in SELECT routine). But it's still a bit slow way and I'm not sure I'm using service capabilities at one hundred percent. Another way is to use EXECUTE 'COPY FROM TABLE.A TO TABLE.B' for each record separately in main routine, but it doesn't seem to be faster. The most frustrating part is, when I'm using just jQL syntax in my shell and type something like: COPY FROM TABLE.A TO TABLE.B ALL, then sometimes it's even faster than my service! I was also thinking about another solution: SELECT TABLE.A WITH @ID LIKE LOC... COPY FROM TABLE.A TO TABLE.B This way we can copy multiple records at once. Maybe I should send larger chunks of id-s to the main routine of my service and then copy all of them? I made some tries and it doesn't seem to be much faster, sometimes it's even slower. Have you got any piece of advice for me? -- -- IMPORTANT: T24/Globus posts are no longer accepted on this forum.   To post, send email to jBASE@googlegroups.com To unsubscribe, send email to jbase-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/jBASE?hl=en --- You received this message because you are subscribed to the Google Groups "jBASE" group. To unsubscribe from this group and stop receiving emails from it, send an email to jbase+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- -- IMPORTANT: T24/Globus posts are no longer accepted on this forum.   To post, send email to jBASE@googlegroups.com To unsubscribe, send email to jbase-unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/jBASE?hl=en --- You received this message because you are subscribed to the Google Groups "jBASE" group. To unsubscribe from this group and stop receiving emails from it, send an email to jbase+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.