Hi Jose, sure. If you can provide some context privately (server login, import file) I can have a look. I should definitely not take that long.
best regards, Lars On Mon, Oct 24, 2016 at 4:36 AM, Morten Olav Hansen <[email protected]> wrote: > Hi Jose > > Lars will probably remember better.. but I think in 2.22 we had preheating > where all the object were loading in before the import started, so the > lookups would be very quick (but require more memory) > > In later versions I think it now will try and "switch" over > automagically.. but I don't think it then preloades all your references > (just caching after the cache miss threshold is reached), so if you have > 42k org units... that might be very slow > > -- > Morten Olav Hansen > Senior Engineer, DHIS 2 > University of Oslo > http://www.dhis2.org > > On Mon, Oct 17, 2016 at 10:27 PM, Jose Garcia Muñoz <[email protected]> > wrote: > >> >> Well at the end got imported (sometimes throws a memory exception), but >> it took more than 1 hour to import 7973 datavalues. It was just a few >> minutes in 2.22 with same import and same server. >> >> On Mon, Oct 17, 2016 at 4:29 PM, Jose Garcia Muñoz <[email protected]> >> wrote: >> >>> >>> It seems to be memory issue happening in 2.24? If I try the same import >>> with the same org.units in 2.22, everything works well. With 2.24, I'm >>> normally getting the follow log message... >>> >>> * INFO 2016-10-17 14:03:07,531 Org unit cache heated after cache miss >>> threshold reached (DefaultDataValueSetService.java [taskScheduler-6]) >>> >>> >>> On Fri, Oct 14, 2016 at 1:51 PM, Jason Pickering < >>> [email protected]> wrote: >>> >>>> Seems like a memory issue >>>> >>>> java.lang.OutOfMemoryError: GC overhead limit exceeded >>>> >>>> On Fri, Oct 14, 2016, 13:35 Jose Garcia Muñoz <[email protected]> >>>> wrote: >>>> >>>>> >>>>> Thanks Lars, >>>>> >>>>> I have run data integrity, but the system has passed "Organisation >>>>> units with cyclic references ". I didn't get any errors importing events, >>>>> but aggregated... Will try to divide the file in small chunks to try to >>>>> figure it out... >>>>> >>>>> Thanks >>>>> Jose >>>>> >>>>> On Fri, Oct 14, 2016 at 11:02 AM, Lars Helge Øverland <[email protected]> >>>>> wrote: >>>>> >>>>> Hi Jose, >>>>> >>>>> my best guess here is a cyclic parent relationship in the org unit >>>>> hierarchy, i.e. you have one or more org units where the one of the >>>>> ancestors are also a child. >>>>> >>>>> Try running data admin > integrity checks and look for "Organisation >>>>> units with cyclic references ", or look for it through SQL. >>>>> >>>>> best regards, >>>>> >>>>> Lars >>>>> >>>>> >>>>> On Fri, Oct 14, 2016 at 1:09 AM, Jose Garcia Muñoz <[email protected] >>>>> > wrote: >>>>> >>>>> >>>>> Dear devs, >>>>> >>>>> I'm trying to import an XML file (900K) with around 8.000 datavalues >>>>> in 2.24 (build revision 1502fe2). However, I'm getting 2 different kind of >>>>> errors during the import... (attached at the end of the email) >>>>> >>>>> Any ideas? The only thing I can think of, is the huge number of org. >>>>> units that this instance has (42.000 org. units) with 8 GB. Would it be >>>>> related to a memory problem? (although the XML file is very small) >>>>> >>>>> I have also selected Preloaded Cache = No >>>>> >>>>> Many thanks! >>>>> >>>>> >>>>> *Error 1* >>>>> >>>>> * INFO 2016-10-13 21:55:55,780 [Level: INFO, category: >>>>> DATAVALUE_IMPORT, time: Thu Oct 13 21:55:55 UTC 2016, message: Importing >>>>> data values] (InMemoryNotifier.java [taskScheduler-7]) >>>>> * ERROR 2016-10-13 22:10:55,332 java.lang.RuntimeException: >>>>> org.hibernate.PropertyAccessException: Exception occurred inside >>>>> getter of org.hisp.dhis.organisationunit.OrganisationUnit.path >>>>> at org.hisp.dhis.commons.collecti >>>>> on.CachingMap.get(CachingMap.java:90) >>>>> at org.hisp.dhis.dxf2.datavaluese >>>>> t.DefaultDataValueSetService.saveDataValueSet(DefaultDataVal >>>>> ueSetService.java:954) >>>>> at org.hisp.dhis.dxf2.datavaluese >>>>> t.DefaultDataValueSetService.saveDataValueSet(DefaultDataVal >>>>> ueSetService.java:488) >>>>> at org.hisp.dhis.dxf2.datavaluese >>>>> t.tasks.ImportDataValueTask.call(ImportDataValueTask.java:102) >>>>> at org.hisp.dhis.security.Securit >>>>> yContextRunnable.run(SecurityContextRunnable.java:57) >>>>> at org.springframework.scheduling >>>>> .support.DelegatingErrorHandlingRunnable.run(DelegatingError >>>>> HandlingRunnable.java:54) >>>>> at java.util.concurrent.Executors >>>>> $RunnableAdapter.call(Executors.java:511) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>>> at java.util.concurrent.Scheduled >>>>> ThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledT >>>>> hreadPoolExecutor.java:180) >>>>> at java.util.concurrent.Scheduled >>>>> ThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPo >>>>> olExecutor.java:293) >>>>> at java.util.concurrent.ThreadPoo >>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at java.util.concurrent.ThreadPoo >>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: org.hibernate.PropertyAccessException: Exception occurred >>>>> inside getter of org.hisp.dhis.organisationunit.OrganisationUnit.path >>>>> at org.hibernate.property.access. >>>>> spi.GetterMethodImpl.get(GetterMethodImpl.java:44) >>>>> at org.hibernate.tuple.entity.Abs >>>>> tractEntityTuplizer.getPropertyValues(AbstractEntityTuplizer.java:505) >>>>> at org.hibernate.tuple.entity.Poj >>>>> oEntityTuplizer.getPropertyValues(PojoEntityTuplizer.java:217) >>>>> at org.hibernate.persister.entity >>>>> .AbstractEntityPersister.getPropertyValues(AbstractEntityPer >>>>> sister.java:4613) >>>>> at org.hibernate.event.internal.D >>>>> efaultFlushEntityEventListener.getValues(DefaultFlushEntityE >>>>> ventListener.java:178) >>>>> at org.hibernate.event.internal.D >>>>> efaultFlushEntityEventListener.onFlushEntity(DefaultFlushEnt >>>>> ityEventListener.java:135) >>>>> at org.hibernate.event.internal.A >>>>> bstractFlushingEventListener.flushEntities(AbstractFlushingE >>>>> ventListener.java:216) >>>>> at org.hibernate.event.internal.A >>>>> bstractFlushingEventListener.flushEverythingToExecutions(Abs >>>>> tractFlushingEventListener.java:85) >>>>> at org.hibernate.event.internal.D >>>>> efaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEve >>>>> ntListener.java:44) >>>>> at org.hibernate.internal.Session >>>>> Impl.autoFlushIfRequired(SessionImpl.java:1264) >>>>> at org.hibernate.internal.Session >>>>> Impl.list(SessionImpl.java:1780) >>>>> at org.hibernate.internal.Criteri >>>>> aImpl.list(CriteriaImpl.java:363) >>>>> at org.hibernate.internal.Criteri >>>>> aImpl.uniqueResult(CriteriaImpl.java:385) >>>>> at org.hisp.dhis.dataset.hibernat >>>>> e.HibernateLockExceptionStore.getCount(HibernateLockExceptio >>>>> nStore.java:167) >>>>> at org.hisp.dhis.dataset.DefaultD >>>>> ataSetService.isLockedPeriod(DefaultDataSetService.java:384) >>>>> at sun.reflect.GeneratedMethodAccessor709.invoke(Unknown >>>>> Source) >>>>> at sun.reflect.DelegatingMethodAc >>>>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> at org.springframework.aop.suppor >>>>> t.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) >>>>> at org.springframework.aop.framew >>>>> ork.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMet >>>>> hodInvocation.java:190) >>>>> at org.springframework.aop.framew >>>>> ork.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvoc >>>>> ation.java:157) >>>>> at org.springframework.transactio >>>>> n.interceptor.TransactionInterceptor$1.proceedWithInvocation >>>>> (TransactionInterceptor.java:99) >>>>> at org.springframework.transactio >>>>> n.interceptor.TransactionAspectSupport.invokeWithinTransacti >>>>> on(TransactionAspectSupport.java:281) >>>>> at org.springframework.transactio >>>>> n.interceptor.TransactionInterceptor.invoke(TransactionInter >>>>> ceptor.java:96) >>>>> at org.springframework.aop.framew >>>>> ork.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvoc >>>>> ation.java:179) >>>>> at org.springframework.aop.interc >>>>> eptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInt >>>>> erceptor.java:92) >>>>> at org.springframework.aop.framew >>>>> ork.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvoc >>>>> ation.java:179) >>>>> at org.springframework.aop.framew >>>>> ork.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) >>>>> at com.sun.proxy.$Proxy77.isLockedPeriod(Unknown Source) >>>>> at org.hisp.dhis.dxf2.datavaluese >>>>> t.DefaultDataValueSetService.lambda$saveDataValueSet$14(Defa >>>>> ultDataValueSetService.java:955) >>>>> at org.hisp.dhis.commons.collecti >>>>> on.CachingMap.get(CachingMap.java:82) >>>>> ... 12 more >>>>> Caused by: java.lang.reflect.InvocationTargetException >>>>> at sun.reflect.GeneratedMethodAccessor391.invoke(Unknown >>>>> Source) >>>>> at sun.reflect.DelegatingMethodAc >>>>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> at org.hibernate.property.access. >>>>> spi.GetterMethodImpl.get(GetterMethodImpl.java:41) >>>>> ... 42 more >>>>> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>> at java.lang.AbstractStringBuilde >>>>> r.<init>(AbstractStringBuilder.java:68) >>>>> at java.lang.StringBuilder.<init>(StringBuilder.java:101) >>>>> at org.apache.commons.lang3.Strin >>>>> gUtils.join(StringUtils.java:4121) >>>>> at org.apache.commons.lang3.Strin >>>>> gUtils.join(StringUtils.java:4177) >>>>> at org.hisp.dhis.organisationunit >>>>> .OrganisationUnit.getPath(OrganisationUnit.java:901) >>>>> at sun.reflect.GeneratedMethodAccessor391.invoke(Unknown >>>>> Source) >>>>> at sun.reflect.DelegatingMethodAc >>>>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> at org.hibernate.property.access. >>>>> spi.GetterMethodImpl.get(GetterMethodImpl.java:41) >>>>> at org.hibernate.tuple.entity.Abs >>>>> tractEntityTuplizer.getPropertyValues(AbstractEntityTuplizer.java:505) >>>>> at org.hibernate.tuple.entity.Poj >>>>> oEntityTuplizer.getPropertyValues(PojoEntityTuplizer.java:217) >>>>> at org.hibernate.persister.entity >>>>> .AbstractEntityPersister.getPropertyValues(AbstractEntityPer >>>>> sister.java:4613) >>>>> at org.hibernate.event.internal.D >>>>> efaultFlushEntityEventListener.getValues(DefaultFlushEntityE >>>>> ventListener.java:178) >>>>> at org.hibernate.event.internal.D >>>>> efaultFlushEntityEventListener.onFlushEntity(DefaultFlushEnt >>>>> ityEventListener.java:135) >>>>> at org.hibernate.event.internal.A >>>>> bstractFlushingEventListener.flushEntities(AbstractFlushingE >>>>> ventListener.java:216) >>>>> at org.hibernate.event.internal.A >>>>> bstractFlushingEventListener.flushEverythingToExecutions(Abs >>>>> tractFlushingEventListener.java:85) >>>>> at org.hibernate.event.internal.D >>>>> efaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEve >>>>> ntListener.java:44) >>>>> at org.hibernate.internal.Session >>>>> Impl.autoFlushIfRequired(SessionImpl.java:1264) >>>>> at org.hibernate.internal.Session >>>>> Impl.list(SessionImpl.java:1780) >>>>> at org.hibernate.internal.Criteri >>>>> aImpl.list(CriteriaImpl.java:363) >>>>> at org.hibernate.internal.Criteri >>>>> aImpl.uniqueResult(CriteriaImpl.java:385) >>>>> at org.hisp.dhis.dataset.hibernat >>>>> e.HibernateLockExceptionStore.getCount(HibernateLockExceptio >>>>> nStore.java:167) >>>>> at org.hisp.dhis.dataset.DefaultD >>>>> ataSetService.isLockedPeriod(DefaultDataSetService.java:384) >>>>> at sun.reflect.GeneratedMethodAccessor709.invoke(Unknown >>>>> Source) >>>>> at sun.reflect.DelegatingMethodAc >>>>> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> at org.springframework.aop.suppor >>>>> t.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) >>>>> at org.springframework.aop.framew >>>>> ork.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMet >>>>> hodInvocation.java:190) >>>>> at org.springframework.aop.framew >>>>> ork.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvoc >>>>> ation.java:157) >>>>> at org.springframework.transactio >>>>> n.interceptor.TransactionInterceptor$1.proceedWithInvocation >>>>> (TransactionInterceptor.java:99) >>>>> at org.springframework.transactio >>>>> n.interceptor.TransactionAspectSupport.invokeWithinTransacti >>>>> on(TransactionAspectSupport.java:281) >>>>> at org.springframework.transactio >>>>> n.interceptor.TransactionInterceptor.invoke(TransactionInter >>>>> ceptor.java:96) >>>>> (DefaultDataValueSetService.java [taskScheduler-3]) >>>>> >>>>> ------------------------------------------------------------ >>>>> ------------------------------------------------- >>>>> >>>>> >>>>> *Error 2:* >>>>> >>>>> * INFO 2016-10-13 22:47:36,040 [Level: INFO, category: >>>>> DATAVALUE_IMPORT, time: Thu Oct 13 22:47:36 UTC 2016, message: Importing >>>>> data values] (InMemoryNotifier.java [taskScheduler-7]) >>>>> CachingMap* INFO 2016-10-13 22:49:58,444 Org unit cache heated after >>>>> cache miss threshold reached (DefaultDataValueSetService.java >>>>> [taskScheduler-7]) >>>>> Org unit cache heated after cache miss threshold reached* ERROR >>>>> 2016-10-13 22:56:14,899 Unable to render Velocity Template, '/main.vm' >>>>> (CommonsLogger.java [http-apr-8081-exec-1]) >>>>> ClientAbortException: java.io.IOException >>>>> at org.apache.catalina.connector. >>>>> OutputBuffer.doFlush(OutputBuffer.java:371) >>>>> at org.apache.catalina.connector. >>>>> OutputBuffer.flush(OutputBuffer.java:333) >>>>> at org.apache.catalina.connector. >>>>> CoyoteOutputStream.flush(CoyoteOutputStream.java:101) >>>>> at org.springframework.security.w >>>>> eb.context.OnCommittedResponseWrapper$SaveContextServletOutp >>>>> utStream.flush(OnCommittedResponseWrapper.java:437) >>>>> at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297) >>>>> at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) >>>>> at java.io.OutputStreamWriter.flu >>>>> sh(OutputStreamWriter.java:229) >>>>> at org.apache.struts2.dispatcher. >>>>> VelocityResult.doExecute(VelocityResult.java:160) >>>>> at org.apache.struts2.dispatcher. >>>>> StrutsResultSupport.execute(StrutsResultSupport.java:191) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.executeResult(DefaultActionInvocation.java:369) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:273) >>>>> at org.hisp.dhis.webportal.interc >>>>> eptor.XWorkPortalUserInterceptor.intercept(XWorkPortalUserIn >>>>> terceptor.java:85) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.webportal.interc >>>>> eptor.XWorkPortalModuleInterceptor.intercept(XWorkPortalModu >>>>> leInterceptor.java:91) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.webportal.interc >>>>> eptor.XWorkPortalMenuInterceptor.intercept(XWorkPortalMenuIn >>>>> terceptor.java:94) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.webportal.interc >>>>> eptor.XWorkPortalParamsInterceptor.intercept(XWorkPortalPara >>>>> msInterceptor.java:150) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.interceptor.Cont >>>>> extInterceptor.intercept(ContextInterceptor.java:83) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.interceptor.User >>>>> SettingInterceptor.intercept(UserSettingInterceptor.java:81) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.security.interce >>>>> pt.LoginInterceptor.intercept(LoginInterceptor.java:85) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.security.interce >>>>> pt.XWorkSecurityInterceptor.intercept(XWorkSecurityIntercept >>>>> or.java:116) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.interceptor.Syst >>>>> emSettingInterceptor.intercept(SystemSettingInterceptor.java:115) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.interceptor.I18n >>>>> Interceptor.intercept(I18nInterceptor.java:139) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at com.opensymphony.xwork2.interc >>>>> eptor.ParametersInterceptor.doIntercept(ParametersIntercepto >>>>> r.java:229) >>>>> at com.opensymphony.xwork2.interc >>>>> eptor.MethodFilterInterceptor.intercept(MethodFilterIntercep >>>>> tor.java:98) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at com.opensymphony.xwork2.interc >>>>> eptor.StaticParametersInterceptor.intercept(StaticParameters >>>>> Interceptor.java:191) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.hisp.dhis.interceptor.Exce >>>>> ptionInterceptor.intercept(ExceptionInterceptor.java:113) >>>>> at com.opensymphony.xwork2.Defaul >>>>> tActionInvocation.invoke(DefaultActionInvocation.java:244) >>>>> at org.apache.struts2.impl.Struts >>>>> ActionProxy.execute(StrutsActionProxy.java:54) >>>>> at org.apache.struts2.dispatcher. >>>>> Dispatcher.serviceAction(Dispatcher.java:564) >>>>> at org.apache.struts2.dispatcher. >>>>> ng.ExecuteOperations.executeAction(ExecuteOperations.java:81) >>>>> at org.apache.struts2.dispatcher. >>>>> ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepa >>>>> reAndExecuteFilter.java:99) >>>>> at org.apache.catalina.core.Appli >>>>> cationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) >>>>> at org.apache.catalina.core.Appli >>>>> cationFilterChain.doFilter(ApplicationFilterChain.java:208) >>>>> at org.springframework.security.w >>>>> eb.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainP >>>>> roxy.java:330) >>>>> at org.springframework.security.w >>>>> eb.access.intercept.FilterSecurityInterceptor.invoke(FilterS >>>>> ecurityInterceptor.java:118) >>>>> at org.springframework.security.w >>>>> eb.access.intercept.FilterSecurityInterceptor.doFilter(Filte >>>>> rSecurityInterceptor.java:84) >>>>> >>>>> >>>>> _______________________________________________ >>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>> Post to : [email protected] >>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>> More help : https://help.launchpad.net/ListHelp >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Lars Helge Øverland >>>>> Lead developer, DHIS 2 >>>>> University of Oslo >>>>> Skype: larshelgeoverland >>>>> [email protected] >>>>> http://www.dhis2.org <https://www.dhis2.org/> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Mailing list: https://launchpad.net/~dhis2-devs >>>>> Post to : [email protected] >>>>> Unsubscribe : https://launchpad.net/~dhis2-devs >>>>> More help : https://help.launchpad.net/ListHelp >>>>> >>>> >>> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~dhis2-devs >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~dhis2-devs >> More help : https://help.launchpad.net/ListHelp >> >> > -- Lars Helge Øverland Lead developer, DHIS 2 University of Oslo Skype: larshelgeoverland [email protected] http://www.dhis2.org <https://www.dhis2.org/>
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : [email protected] Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
