Doug, Your explanation and answer are probably correct since you are the master of AR System :-) (use the source Luke....)
But I find the figures to far off what I would expect and gonna spend some more time to pinpoint where the small amout of time goes.... Regards, Jarl 2009/8/17 Mueller, Doug <doug_muel...@bmc.com>: > Jarl, > > Well, there is something you don't know about what is really going on that > may help explain why my explaination answers your question.... > > You are making the same number of API calls (well, in the bulk case you are > making 2 more for the begin and commit). HOWEVER, what I wasn't clear about > is it is API calls that go across the network that are at issue here not > actual API calls. This is because some API calls don't go across the network. > > In your scenario, what is happening.... > > In the example for #3, you issue 10,000 CreateEntry calls. This generates > 10,000 operations across the network, one per CreateEntry call. > > In the example for #4, you issue 1 beginBulk, 10,000 CreateEntry, and 1 > commitBulk calls. This generates 1 (ONE) operation across the network. > > Notice the major difference of 10,000 operations across the network vs. the > 1 operation across the network in the two scenarios. > > The beginBulk operation just starts queueing up the operations you want to > perform until it gets the commitBulk (or endBulk or whatever the name is) and > when it gets that, it sends all the pending commands across the network to the > server. > > Now, there is 1 BIG set of data flowing vs. 10,000 little sets of data flowing > across the network for these scenarios. > > This is the major difference in operation. Within the server, the operations > are the same with each operation being processed. And, in the case of your > display only form, there is no DB operation so it is just the running through > the list of 10,000 operations, any data validation, any filter processing, and > that is it. > > So, the difference of 4 seconds vs. 20 seconds is the diffence of 10,000 > individual small interactions between your program and the server and the 1 > big interaction between your program and the server. > > With this understanding of what is really going on within your API calls (and > why you may want to do this in chunks of 100 or 1000 rather than letting the > bulk call include 100s of thousands to control memory sizes on both client and > server with the large pending list) help to now match with my explaination of > what is going on and why the perceived difference in timing is not really a > difference? > > Doug Mueller > > -----Original Message----- > From: Action Request System discussion list(ARSList) > [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng > Sent: Monday, August 17, 2009 11:38 AM > To: arslist@ARSLIST.ORG > Subject: Re: Wierd behavior with Java API -> ar-server > > Doug, > > Thanks for you long answer. (I think I have wast some of your time here...) > > I may have explain the issue a bit wrong... When I wrote bulkimport, I > ment using the beginBulkEntryTransaction() option. > > So in example 3), I create 10.000 entries in a display only form. This > should not do any commit in the database, it should do no database > transaction at all (regards the logs, it does not do anything in the > database) In example 4) I using the beginBulkEntryTransaction() when > creating entries. This also should do none database transaction (is > does one, Commit) > > However, regards the internal logic inside ar-server, example 3) and > 4) should be quite equal. Still the non bulk transaction takes 5 times > longer than the bulk transaction. > > Regards, > Jarl > > > > > 2009/8/17 Mueller, Doug <doug_muel...@bmc.com>: >> Jarl, >> >> Let's hear it for LATENCY. The network kind.... (ok, there is a bit of >> savings on basic overhead with user checking and parameter marshalling and >> fewer TCP transactions (although they are different sizes) and other things >> like this because of fewer API calls). Total volume of data across the wire >> is the same, but it is many small packets (10,000 small) or fewer larger >> packets (xxxx larger with xxxx being how many things are in the bulk >> operation >> you are doing). >> >> In your first example, the difference is about 20 seconds. >> In your second example, the difference is about 16 seconds. >> >> The difference between case 1 and case 2 is whether there is a write to the >> DB involved. Well, the DB write and commit is the majority of the time in >> these cases. In the bulk case, it is about 46 seconds of the time 4 seconds >> vs. 50 seconds. Interestingly, in the non bulk case, it is about 50 seconds >> of the time 20 seconds vs. 70 seconds. >> >> Now, with this, it looks like there is 4 seconds related to DB difference and >> 16 that is common and not related to DB difference. >> >> I would guess that the 4 second DB difference is small because of there being >> no data in the tables and probably small records and no other work so the >> extra commits you are saving (not saving anything from an insert or workflow >> perspective so it is just the multiple transaction aspect) are relatively >> small in this case. It is still 20% savings! >> >> The time savings on the network are relatively constant in this mix -- >> although >> if you have high network latency, you are saving even more. For example, >> say there were chunks of 100 operations in the bulk operations. That would >> mean 100 chunks. If you were talking about .25 second latency, you are >> talking >> about 100 * .25 or 25 seconds vs. 10000 * .25 or 2500 seconds in time for >> the savings on the network (notice how any overhead of the API calls is >> totally >> swamped by network time when there is latency). Again, fixed, but just a >> bigger number than the above. >> >> It is important to look at what the savings is in situations like this and >> where the savings is coming from. I cannot gaurantee that the numbers are >> all >> in the locations I note here, but the concept of what the bulk calls do and >> save and where the savings are (db vs. network) are correct. Also, it shows >> why the numbers that look odd at first glance are much more reasonable and >> make more sense. >> >> You are just swamped here with network and overhead issues that are taking >> the >> majority of the time while the DB side is efficient and giving you gain, but >> a lower percentage of the gain. >> >> I hope this helps explain what is likely occurring and how to interpret your >> data results. >> >> Doug Mueller >> >> -----Original Message----- >> From: Action Request System discussion list(ARSList) >> [mailto:arsl...@arslist.org] On Behalf Of Jarl Grøneng >> Sent: Monday, August 17, 2009 8:16 AM >> To: arslist@ARSLIST.ORG >> Subject: Wierd behavior with Java API -> ar-server >> >> I have a JavaAPI program that reads a csv file, and import the data into a >> form >> >> 1) Import 10.000 records into a regular form, takes appox 70 seconds >> 2) Import 10.000 records into a regular form with bulkimport enabled , >> takes appox 50 seconds: >> >> The time gap between 1 and 2 is mainly one commit to the database. >> >> >> 3) Import 10.000 records into a display only form, takes appox 20 seconds >> 4) Import 10.000 records into a display only forrm with bulkimport >> enabled, takes appox 4 seconds. >> >> >> Why should it be that large difference between 3 and 4? >> >> >> AR server 7.5 patch 2 >> Oracle 10 >> >> (all figures on arserver on my laptop) >> >> -- >> Jarl >> >> _______________________________________________________________________________ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" >> >> _______________________________________________________________________________ >> UNSUBSCRIBE or access ARSlist Archives at www.arslist.org >> Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" >> > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" > > _______________________________________________________________________________ > UNSUBSCRIBE or access ARSlist Archives at www.arslist.org > Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are" > _______________________________________________________________________________ UNSUBSCRIBE or access ARSlist Archives at www.arslist.org Platinum Sponsor:rmisoluti...@verizon.net ARSlist: "Where the Answers Are"
