Hi Matt, Let me give you the details how our application works : We have 5-7 instances of application which point to same database(i.e. Postgres) and the same S3 Bucket. We don’t have S3 Transfer Acceleration enabled on the S3 bucket. We have huge amount of data with more than 2 TB and more than 20 million of blob ids in S3.
We were not facing this issue before but in last week we started facing this issue and it has been continued ever since. Could this be the issue due to huge amount of data? We are using Oak 1.22.3 version. And also our repository takes more than 7 hours to be up and running. Could any of this be the issue for slowness? Thank you. From: Matt Ryan [mailto:[email protected]] Sent: 30 July 2020 02:32 To: Tanvi Shah Subject: Re: Slowness in Direct Binary Upload [External - Use Caution] Hi Tanvi, Your issue is a bit puzzling, something I've never seen so far in my work with direct upload. Theoretically the "completeBinaryUpload()" call should be quite fast, as it only makes three S3 API calls: - One to check to make sure the binary does not already exist - One to get the list of uncommitted blocks for the provided blob ID - One to commit the block list At this point I recommend you try writing a very simple test application that can reproduce the same behavior. In doing that I imagine either you will uncover something in your program that you need to change OR isolate a problem that we can look into in more depth. It will also provide a concrete implementation we can look at. If you end up reproducing the problem in your as-simple-as-possible test application, please file a bug and then reply on-list with the bug link so we can dig in further. -MR On Tue, Jul 28, 2020 at 12:30 AM Tanvi Shah <[email protected]<mailto:[email protected]>> wrote: Hi, So the issue we are facing is while Complete Upload when we have to save the binaries in repository. For more than 10 files it is taking 10 minutes and sometimes even more. S3 bucket is having more than 20 million of data stored in it and we haven't execute S3 garbage collection yet. Code to complete upload: final ValueFactory vf = session.getValueFactory(); if (vf instanceof JackrabbitValueFactory) { final JackrabbitValueFactory valueFactory = (JackrabbitValueFactory) vf; final Binary binary = valueFactory.completeBinaryUpload(uploadToken); final Node content = file.addNode(Property.JCR_CONTENT, NodeType.NT_RESOURCE); content.setProperty(Property.JCR_DATA, binary); } session.save(); Thank you. ________________________________ From: Tanvi Shah Sent: 27 July 2020 16:29 To: [email protected]<mailto:[email protected]> Subject: Slowness in Direct Binary Upload Hi, We are using Direct Binary S3 Upload feature to save the files in repository. We faced the issue that when we are storing more than 10 files then direct binary feature(upload token) takes around 1 hour to save the files and with the same files if we don't use the direct binary approach it takes few seconds to upload the files. Please can you let us know what could be the issue here. Also We have multiple instances of application and all use the same Postgres database and S3 bucket. Will this cause any issue. Thank you in advance. ________________________________ Disclaimer: This e-mail is confidential and should not be used by anyone who is not the original intended recipient. If you have received this e-mail in error please inform the sender and delete it from your mailbox or any other storage mechanism. Springer Nature Technology and Publishing Solutions Private Limited does not accept liability for any statements made which are clearly the sender's own and not expressly made on behalf of Springer Nature Technology and Publishing Solutions Private Limited or one of their agents. Please note that Springer Nature Technology and Publishing Solutions Private Limited and their agents and affiliates do not accept any responsibility for viruses or malware that may be contained in this e-mail or its attachments and it is your responsibility to scan the e-mail and attachments (if any). Springer Nature Technology and Publishing Solutions Private Limited. Registered office: Upper Ground Floor, Wing B, Tower 8, Magarpatta City SEZ, Hadapsar Pune MH 411013 IN Registered number: U72200PN2006FTC128967
