Hi, I couldn't find the option to set the limit for Query in S3 GC.(<https://jdbc.postgresql.org/documentation/head/connect.html#connection-parameters>).
________________________________ From: Tanvi Shah Sent: 19 October 2020 12:07 To: oak-dev@jackrabbit.apache.org Subject: Error in S3 Garbage Collection Hi , We have started Se Garbage Collection and S3 size is more than 2 TB. We are facing memory issues while executing GC even when we have given 11 GB of memory to the application. Code is: final MarkSweepGarbageCollector gc = documentNodeStore.createBlobGarbageCollector(seconds, repository.toString(), wb, new DefaultStatisticsProvider(Executors.newScheduledThreadPool(1))); gc.collectGarbage(markOnly); final OperationsStatsMBean stats = gc.getOperationStats(); log.info( "number deleted : " + stats.numDeleted() + " Size deleted : " + stats.sizeDeleted()); The exceptions are : java.lang.RuntimeException: org.postgresql.util.PSQLException: Ran out of memory retrieving query results. at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore$3.iterator(RDBDocumentStore.java:1876) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBBlobReferenceIterator.getIteratorOverDocsWithBinaries(RDBBlobReferenceIterator.java:48) at org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.loadBatch(BlobReferenceIterator.java:64) at org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:52) at org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:36) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.jackrabbit.oak.plugins.document.DocumentBlobReferenceRetriever.collectReferences(DocumentBlobReferenceRetriever.java:49) at org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.iterateNodeTree(MarkSweepGarbageCollector.java:600) at org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.mark(MarkSweepGarbageCollector.java:397) at org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.markAndSweep(MarkSweepGarbageCollector.java:347) at org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.collectGarbage(MarkSweepGarbageCollector.java:235) at com.springernature.mango.cms.common.helper.RepositoryUtil.s3GarbageCollection(RepositoryUtil.java:324) at com.springernature.mango.cms.services.rest.service.ServicesController.lambda$runS3GC$1(ServicesController.java:460) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) org.postgresql.util.PSQLException: Ran out of memory retrieving query results. at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2145) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118) at sun.reflect.GeneratedMethodAccessor49.invoke at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tomcat.jdbc.pool.interceptor.StatementDecoratorInterceptor$StatementProxy.invoke(StatementDecoratorInterceptor.java:237) at com.sun.proxy.$Proxy93.executeQuery at sun.reflect.GeneratedMethodAccessor49.invoke at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210) at com.sun.proxy.$Proxy93.executeQuery at sun.reflect.GeneratedMethodAccessor49.invoke at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114) at com.sun.proxy.$Proxy93.executeQuery at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStoreJDBC$ResultSetIterator.<init>(RDBDocumentStoreJDBC.java:591) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStoreJDBC.queryAsIterator(RDBDocumentStoreJDBC.java:558) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore$3.iterator(RDBDocumentStore.java:1865) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBBlobReferenceIterator.getIteratorOverDocsWithBinaries(RDBBlobReferenceIterator.java:48) at org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.loadBatch(BlobReferenceIterator.java:64) at org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:52) at org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:36) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.jackrabbit.oak.plugins.document.DocumentBlobReferenceRetriever.collectReferences(DocumentBlobReferenceRetriever.java:49) at org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.iterateNodeTree(MarkSweepGarbageCollector.java:600) at org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.mark(MarkSweepGarbageCollector.java:397) at org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.markAndSweep(MarkSweepGarbageCollector.java:347) at org.apache.jackrabbit.oak.plugins.blob.MarkSweepGarbageCollector.collectGarbage(MarkSweepGarbageCollector.java:235) at com.springernature.mango.cms.common.helper.RepositoryUtil.s3GarbageCollection(RepositoryUtil.java:324) at com.springernature.mango.cms.services.rest.service.ServicesController.lambda$runS3GC$1(ServicesController.java:460) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) java.lang.OutOfMemoryError: GC overhead limit exceeded at org.postgresql.core.PGStream.receiveTupleV3(PGStream.java:401) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2141) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:155) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:118) at sun.reflect.GeneratedMethodAccessor49.invoke at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tomcat.jdbc.pool.interceptor.StatementDecoratorInterceptor$StatementProxy.invoke(StatementDecoratorInterceptor.java:237) at com.sun.proxy.$Proxy93.executeQuery at sun.reflect.GeneratedMethodAccessor49.invoke at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tomcat.jdbc.pool.interceptor.AbstractQueryReport$StatementProxy.invoke(AbstractQueryReport.java:210) at com.sun.proxy.$Proxy93.executeQuery at sun.reflect.GeneratedMethodAccessor49.invoke at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114) at com.sun.proxy.$Proxy93.executeQuery at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStoreJDBC$ResultSetIterator.<init>(RDBDocumentStoreJDBC.java:591) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStoreJDBC.queryAsIterator(RDBDocumentStoreJDBC.java:558) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore$3.iterator(RDBDocumentStore.java:1865) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBBlobReferenceIterator.getIteratorOverDocsWithBinaries(RDBBlobReferenceIterator.java:48) at org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.loadBatch(BlobReferenceIterator.java:64) at org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:52) at org.apache.jackrabbit.oak.plugins.document.BlobReferenceIterator.computeNext(BlobReferenceIterator.java:36) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.jackrabbit.oak.plugins.document.DocumentBlobReferenceRetriever.collectReferences(DocumentBlobReferenceRetriever.java:49) So I wanted to ask is there any other way how can we restrict this and successfully run S3 Garbage Collection. Thanks. ********************************************************************** Disclaimer: This e-mail is confidential and should not be used by anyone who is not the original intended recipient. If you have received this e-mail in error please inform the sender and delete it from your mailbox or any other storage mechanism. Springer Nature Technology and Publishing Solutions Private Limited does not accept liability for any statements made which are clearly the sender's own and not expressly made on behalf of Springer Nature Technology and Publishing Solutions Private Limited or one of their agents. Please note that Springer Nature Technology and Publishing Solutions Private Limited and their agents and affiliates do not accept any responsibility for viruses or malware that may be contained in this e-mail or its attachments and it is your responsibility to scan the e-mail and attachments (if any). Springer Nature Technology and Publishing Solutions Private Limited. Registered office: Upper Ground Floor, Wing B, Tower 8, Magarpatta City SEZ, Hadapsar Pune MH 411013 IN Registered number: U72200PN2006FTC128967