Re: [Dhis2-devs] Errors while importing datavalues in 2.24

[Lars Helge Øverland]

Hi Jose,

sure. If you can provide some context privately (server login, import file)
I can have a look. I should definitely not take that long.

best regards,

Lars


On Mon, Oct 24, 2016 at 4:36 AM, Morten Olav Hansen 
wrote:

> Hi Jose
>
> Lars will probably remember better.. but I think in 2.22 we had preheating
> where all the object were loading in before the import started, so the
> lookups would be very quick (but require more memory)
>
> In later versions I think it now will try and "switch" over
> automagically.. but I don't think it then preloades all your references
> (just caching after the cache miss threshold is reached), so if you have
> 42k org units... that might be very slow
>
> --
> Morten Olav Hansen
> Senior Engineer, DHIS 2
> University of Oslo
> http://www.dhis2.org
>
> On Mon, Oct 17, 2016 at 10:27 PM, Jose Garcia Muñoz 
> wrote:
>
>>
>> Well at the end got imported (sometimes throws a memory exception), but
>> it took more than 1 hour to import 7973 datavalues. It was just a few
>> minutes in 2.22 with same import and same server.
>>
>> On Mon, Oct 17, 2016 at 4:29 PM, Jose Garcia Muñoz 
>> wrote:
>>
>>>
>>> It seems to be memory issue happening in 2.24? If I try the same import
>>> with the same org.units in 2.22, everything works well. With 2.24, I'm
>>> normally getting the follow log message...
>>>
>>> * INFO  2016-10-17 14:03:07,531 Org unit cache heated after cache miss
>>> threshold reached (DefaultDataValueSetService.java [taskScheduler-6])
>>>
>>>
>>> On Fri, Oct 14, 2016 at 1:51 PM, Jason Pickering <
>>> jason.p.picker...@gmail.com> wrote:
>>>
 Seems like a memory issue

  java.lang.OutOfMemoryError: GC overhead limit exceeded

 On Fri, Oct 14, 2016, 13:35 Jose Garcia Muñoz 
 wrote:

>
> Thanks Lars,
>
> I have run data integrity, but the system has passed "Organisation
> units with cyclic references ". I didn't get any errors importing events,
> but aggregated... Will try to divide the file in small chunks to try to
> figure it out...
>
> Thanks
> Jose
>
> On Fri, Oct 14, 2016 at 11:02 AM, Lars Helge Øverland 
> wrote:
>
> Hi Jose,
>
> my best guess here is a cyclic parent relationship in the org unit
> hierarchy, i.e. you have one or more org units where the one of the
> ancestors are also a child.
>
> Try running data admin > integrity checks and look for "Organisation
> units with cyclic references ", or look for it through SQL.
>
> best regards,
>
> Lars
>
>
> On Fri, Oct 14, 2016 at 1:09 AM, Jose Garcia Muñoz  > wrote:
>
>
> Dear devs,
>
> I'm trying to import an XML file (900K) with around 8.000 datavalues
> in 2.24 (build revision 1502fe2). However, I'm getting 2 different kind of
> errors during the import... (attached at the end of the email)
>
> Any ideas? The only thing I can think of, is the huge number of org.
> units that this instance has (42.000 org. units) with 8 GB. Would it be
> related to a memory problem? (although the XML file is very small)
>
> I have also selected Preloaded Cache = No
>
> Many thanks!
>
>
> *Error 1*
>
> * INFO  2016-10-13 21:55:55,780 [Level: INFO, category:
> DATAVALUE_IMPORT, time: Thu Oct 13 21:55:55 UTC 2016, message: Importing
> data values] (InMemoryNotifier.java [taskScheduler-7])
> * ERROR 2016-10-13 22:10:55,332 java.lang.RuntimeException:
> org.hibernate.PropertyAccessException: Exception occurred inside
> getter of org.hisp.dhis.organisationunit.OrganisationUnit.path
> at org.hisp.dhis.commons.collecti
> on.CachingMap.get(CachingMap.java:90)
> at org.hisp.dhis.dxf2.datavaluese
> t.DefaultDataValueSetService.saveDataValueSet(DefaultDataVal
> ueSetService.java:954)
> at org.hisp.dhis.dxf2.datavaluese
> t.DefaultDataValueSetService.saveDataValueSet(DefaultDataVal
> ueSetService.java:488)
> at org.hisp.dhis.dxf2.datavaluese
> t.tasks.ImportDataValueTask.call(ImportDataValueTask.java:102)
> at org.hisp.dhis.security.Securit
> yContextRunnable.run(SecurityContextRunnable.java:57)
> at org.springframework.scheduling
> .support.DelegatingErrorHandlingRunnable.run(DelegatingError
> HandlingRunnable.java:54)
> at java.util.concurrent.Executors
> $RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Scheduled
> ThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledT
> hreadPoolExecutor.java:180)
> at java.util.concurrent.Scheduled
> ThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPo
> olExecutor.java:293)

Re: [Dhis2-devs] Errors while importing datavalues in 2.24

[Morten Olav Hansen]

Hi Jose

Lars will probably remember better.. but I think in 2.22 we had preheating
where all the object were loading in before the import started, so the
lookups would be very quick (but require more memory)

In later versions I think it now will try and "switch" over automagically..
but I don't think it then preloades all your references (just caching after
the cache miss threshold is reached), so if you have 42k org units... that
might be very slow

-- 
Morten Olav Hansen
Senior Engineer, DHIS 2
University of Oslo
http://www.dhis2.org

On Mon, Oct 17, 2016 at 10:27 PM, Jose Garcia Muñoz 
wrote:

>
> Well at the end got imported (sometimes throws a memory exception), but it
> took more than 1 hour to import 7973 datavalues. It was just a few minutes
> in 2.22 with same import and same server.
>
> On Mon, Oct 17, 2016 at 4:29 PM, Jose Garcia Muñoz 
> wrote:
>
>>
>> It seems to be memory issue happening in 2.24? If I try the same import
>> with the same org.units in 2.22, everything works well. With 2.24, I'm
>> normally getting the follow log message...
>>
>> * INFO  2016-10-17 14:03:07,531 Org unit cache heated after cache miss
>> threshold reached (DefaultDataValueSetService.java [taskScheduler-6])
>>
>>
>> On Fri, Oct 14, 2016 at 1:51 PM, Jason Pickering <
>> jason.p.picker...@gmail.com> wrote:
>>
>>> Seems like a memory issue
>>>
>>>  java.lang.OutOfMemoryError: GC overhead limit exceeded
>>>
>>> On Fri, Oct 14, 2016, 13:35 Jose Garcia Muñoz 
>>> wrote:
>>>

 Thanks Lars,

 I have run data integrity, but the system has passed "Organisation
 units with cyclic references ". I didn't get any errors importing events,
 but aggregated... Will try to divide the file in small chunks to try to
 figure it out...

 Thanks
 Jose

 On Fri, Oct 14, 2016 at 11:02 AM, Lars Helge Øverland 
 wrote:

 Hi Jose,

 my best guess here is a cyclic parent relationship in the org unit
 hierarchy, i.e. you have one or more org units where the one of the
 ancestors are also a child.

 Try running data admin > integrity checks and look for "Organisation
 units with cyclic references ", or look for it through SQL.

 best regards,

 Lars


 On Fri, Oct 14, 2016 at 1:09 AM, Jose Garcia Muñoz 
 wrote:


 Dear devs,

 I'm trying to import an XML file (900K) with around 8.000 datavalues in
 2.24 (build revision 1502fe2). However, I'm getting 2 different kind of
 errors during the import... (attached at the end of the email)

 Any ideas? The only thing I can think of, is the huge number of org.
 units that this instance has (42.000 org. units) with 8 GB. Would it be
 related to a memory problem? (although the XML file is very small)

 I have also selected Preloaded Cache = No

 Many thanks!


 *Error 1*

 * INFO  2016-10-13 21:55:55,780 [Level: INFO, category:
 DATAVALUE_IMPORT, time: Thu Oct 13 21:55:55 UTC 2016, message: Importing
 data values] (InMemoryNotifier.java [taskScheduler-7])
 * ERROR 2016-10-13 22:10:55,332 java.lang.RuntimeException:
 org.hibernate.PropertyAccessException: Exception occurred inside
 getter of org.hisp.dhis.organisationunit.OrganisationUnit.path
 at org.hisp.dhis.commons.collection.CachingMap.get(CachingMap.j
 ava:90)
 at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.s
 aveDataValueSet(DefaultDataValueSetService.java:954)
 at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.s
 aveDataValueSet(DefaultDataValueSetService.java:488)
 at org.hisp.dhis.dxf2.datavalueset.tasks.ImportDataValueTask.ca
 ll(ImportDataValueTask.java:102)
 at org.hisp.dhis.security.SecurityContextRunnable.run(SecurityC
 ontextRunnable.java:57)
 at org.springframework.scheduling.support.DelegatingErrorHandli
 ngRunnable.run(DelegatingErrorHandlingRunnable.java:54)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executor
 s.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
 tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
 tureTask.run(ScheduledThreadPoolExecutor.java:293)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
 Executor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
 lExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: org.hibernate.PropertyAccessException: Exception occurred
 inside getter of org.hisp.dhis.organisationunit.OrganisationUnit.path
 at 

Re: [Dhis2-devs] Errors while importing datavalues in 2.24

[Jose Garcia Muñoz]

Well at the end got imported (sometimes throws a memory exception), but it
took more than 1 hour to import 7973 datavalues. It was just a few minutes
in 2.22 with same import and same server.

On Mon, Oct 17, 2016 at 4:29 PM, Jose Garcia Muñoz 
wrote:

>
> It seems to be memory issue happening in 2.24? If I try the same import
> with the same org.units in 2.22, everything works well. With 2.24, I'm
> normally getting the follow log message...
>
> * INFO  2016-10-17 14:03:07,531 Org unit cache heated after cache miss
> threshold reached (DefaultDataValueSetService.java [taskScheduler-6])
>
>
> On Fri, Oct 14, 2016 at 1:51 PM, Jason Pickering <
> jason.p.picker...@gmail.com> wrote:
>
>> Seems like a memory issue
>>
>>  java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>> On Fri, Oct 14, 2016, 13:35 Jose Garcia Muñoz  wrote:
>>
>>>
>>> Thanks Lars,
>>>
>>> I have run data integrity, but the system has passed "Organisation
>>> units with cyclic references ". I didn't get any errors importing events,
>>> but aggregated... Will try to divide the file in small chunks to try to
>>> figure it out...
>>>
>>> Thanks
>>> Jose
>>>
>>> On Fri, Oct 14, 2016 at 11:02 AM, Lars Helge Øverland 
>>> wrote:
>>>
>>> Hi Jose,
>>>
>>> my best guess here is a cyclic parent relationship in the org unit
>>> hierarchy, i.e. you have one or more org units where the one of the
>>> ancestors are also a child.
>>>
>>> Try running data admin > integrity checks and look for "Organisation
>>> units with cyclic references ", or look for it through SQL.
>>>
>>> best regards,
>>>
>>> Lars
>>>
>>>
>>> On Fri, Oct 14, 2016 at 1:09 AM, Jose Garcia Muñoz 
>>> wrote:
>>>
>>>
>>> Dear devs,
>>>
>>> I'm trying to import an XML file (900K) with around 8.000 datavalues in
>>> 2.24 (build revision 1502fe2). However, I'm getting 2 different kind of
>>> errors during the import... (attached at the end of the email)
>>>
>>> Any ideas? The only thing I can think of, is the huge number of org.
>>> units that this instance has (42.000 org. units) with 8 GB. Would it be
>>> related to a memory problem? (although the XML file is very small)
>>>
>>> I have also selected Preloaded Cache = No
>>>
>>> Many thanks!
>>>
>>>
>>> *Error 1*
>>>
>>> * INFO  2016-10-13 21:55:55,780 [Level: INFO, category:
>>> DATAVALUE_IMPORT, time: Thu Oct 13 21:55:55 UTC 2016, message: Importing
>>> data values] (InMemoryNotifier.java [taskScheduler-7])
>>> * ERROR 2016-10-13 22:10:55,332 java.lang.RuntimeException:
>>> org.hibernate.PropertyAccessException: Exception occurred inside getter
>>> of org.hisp.dhis.organisationunit.OrganisationUnit.path
>>> at org.hisp.dhis.commons.collection.CachingMap.get(CachingMap.
>>> java:90)
>>> at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.s
>>> aveDataValueSet(DefaultDataValueSetService.java:954)
>>> at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.s
>>> aveDataValueSet(DefaultDataValueSetService.java:488)
>>> at org.hisp.dhis.dxf2.datavalueset.tasks.ImportDataValueTask.
>>> call(ImportDataValueTask.java:102)
>>> at org.hisp.dhis.security.SecurityContextRunnable.run(SecurityC
>>> ontextRunnable.java:57)
>>> at org.springframework.scheduling.support.DelegatingErrorHandli
>>> ngRunnable.run(DelegatingErrorHandlingRunnable.java:54)
>>> at java.util.concurrent.Executors$RunnableAdapter.call(
>>> Executors.java:511)
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1142)
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.hibernate.PropertyAccessException: Exception occurred
>>> inside getter of org.hisp.dhis.organisationunit.OrganisationUnit.path
>>> at org.hibernate.property.access.spi.GetterMethodImpl.get(Gette
>>> rMethodImpl.java:44)
>>> at org.hibernate.tuple.entity.AbstractEntityTuplizer.getPropert
>>> yValues(AbstractEntityTuplizer.java:505)
>>> at org.hibernate.tuple.entity.PojoEntityTuplizer.getPropertyVal
>>> ues(PojoEntityTuplizer.java:217)
>>> at org.hibernate.persister.entity.AbstractEntityPersister.getPr
>>> opertyValues(AbstractEntityPersister.java:4613)
>>> at org.hibernate.event.internal.DefaultFlushEntityEventListener
>>> .getValues(DefaultFlushEntityEventListener.java:178)
>>> at org.hibernate.event.internal.DefaultFlushEntityEventListener
>>> .onFlushEntity(DefaultFlushEntityEventListener.java:135)
>>> at 

Re: [Dhis2-devs] Errors while importing datavalues in 2.24

[Jose Garcia Muñoz]

It seems to be memory issue happening in 2.24? If I try the same import
with the same org.units in 2.22, everything works well. With 2.24, I'm
normally getting the follow log message...

* INFO  2016-10-17 14:03:07,531 Org unit cache heated after cache miss
threshold reached (DefaultDataValueSetService.java [taskScheduler-6])


On Fri, Oct 14, 2016 at 1:51 PM, Jason Pickering <
jason.p.picker...@gmail.com> wrote:

> Seems like a memory issue
>
>  java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> On Fri, Oct 14, 2016, 13:35 Jose Garcia Muñoz  wrote:
>
>>
>> Thanks Lars,
>>
>> I have run data integrity, but the system has passed "Organisation units
>> with cyclic references ". I didn't get any errors importing events, but
>> aggregated... Will try to divide the file in small chunks to try to figure
>> it out...
>>
>> Thanks
>> Jose
>>
>> On Fri, Oct 14, 2016 at 11:02 AM, Lars Helge Øverland 
>> wrote:
>>
>> Hi Jose,
>>
>> my best guess here is a cyclic parent relationship in the org unit
>> hierarchy, i.e. you have one or more org units where the one of the
>> ancestors are also a child.
>>
>> Try running data admin > integrity checks and look for "Organisation
>> units with cyclic references ", or look for it through SQL.
>>
>> best regards,
>>
>> Lars
>>
>>
>> On Fri, Oct 14, 2016 at 1:09 AM, Jose Garcia Muñoz 
>> wrote:
>>
>>
>> Dear devs,
>>
>> I'm trying to import an XML file (900K) with around 8.000 datavalues in
>> 2.24 (build revision 1502fe2). However, I'm getting 2 different kind of
>> errors during the import... (attached at the end of the email)
>>
>> Any ideas? The only thing I can think of, is the huge number of org.
>> units that this instance has (42.000 org. units) with 8 GB. Would it be
>> related to a memory problem? (although the XML file is very small)
>>
>> I have also selected Preloaded Cache = No
>>
>> Many thanks!
>>
>>
>> *Error 1*
>>
>> * INFO  2016-10-13 21:55:55,780 [Level: INFO, category: DATAVALUE_IMPORT,
>> time: Thu Oct 13 21:55:55 UTC 2016, message: Importing data values]
>> (InMemoryNotifier.java [taskScheduler-7])
>> * ERROR 2016-10-13 22:10:55,332 java.lang.RuntimeException: 
>> org.hibernate.PropertyAccessException:
>> Exception occurred inside getter of org.hisp.dhis.organisationunit.
>> OrganisationUnit.path
>> at org.hisp.dhis.commons.collection.CachingMap.get(
>> CachingMap.java:90)
>> at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.
>> saveDataValueSet(DefaultDataValueSetService.java:954)
>> at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.
>> saveDataValueSet(DefaultDataValueSetService.java:488)
>> at org.hisp.dhis.dxf2.datavalueset.tasks.
>> ImportDataValueTask.call(ImportDataValueTask.java:102)
>> at org.hisp.dhis.security.SecurityContextRunnable.run(
>> SecurityContextRunnable.java:57)
>> at org.springframework.scheduling.support.
>> DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnabl
>> e.java:54)
>> at java.util.concurrent.Executors$RunnableAdapter.
>> call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$
>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.hibernate.PropertyAccessException: Exception occurred
>> inside getter of org.hisp.dhis.organisationunit.OrganisationUnit.path
>> at org.hibernate.property.access.spi.GetterMethodImpl.get(
>> GetterMethodImpl.java:44)
>> at org.hibernate.tuple.entity.AbstractEntityTuplizer.
>> getPropertyValues(AbstractEntityTuplizer.java:505)
>> at org.hibernate.tuple.entity.PojoEntityTuplizer.
>> getPropertyValues(PojoEntityTuplizer.java:217)
>> at org.hibernate.persister.entity.AbstractEntityPersister.
>> getPropertyValues(AbstractEntityPersister.java:4613)
>> at org.hibernate.event.internal.DefaultFlushEntityEventListene
>> r.getValues(DefaultFlushEntityEventListener.java:178)
>> at org.hibernate.event.internal.DefaultFlushEntityEventListene
>> r.onFlushEntity(DefaultFlushEntityEventListener.java:135)
>> at org.hibernate.event.internal.AbstractFlushingEventListener.
>> flushEntities(AbstractFlushingEventListener.java:216)
>> at org.hibernate.event.internal.AbstractFlushingEventListener.
>> flushEverythingToExecutions(AbstractFlushingEventListener.java:85)
>> at org.hibernate.event.internal.DefaultAutoFlushEventListener.
>> onAutoFlush(DefaultAutoFlushEventListener.java:44)
>>

Re: [Dhis2-devs] Errors while importing datavalues in 2.24

[Jason Pickering]

Seems like a memory issue

 java.lang.OutOfMemoryError: GC overhead limit exceeded

On Fri, Oct 14, 2016, 13:35 Jose Garcia Muñoz  wrote:

>
> Thanks Lars,
>
> I have run data integrity, but the system has passed "Organisation units
> with cyclic references ". I didn't get any errors importing events, but
> aggregated... Will try to divide the file in small chunks to try to figure
> it out...
>
> Thanks
> Jose
>
> On Fri, Oct 14, 2016 at 11:02 AM, Lars Helge Øverland 
> wrote:
>
> Hi Jose,
>
> my best guess here is a cyclic parent relationship in the org unit
> hierarchy, i.e. you have one or more org units where the one of the
> ancestors are also a child.
>
> Try running data admin > integrity checks and look for "Organisation units
> with cyclic references ", or look for it through SQL.
>
> best regards,
>
> Lars
>
>
> On Fri, Oct 14, 2016 at 1:09 AM, Jose Garcia Muñoz 
> wrote:
>
>
> Dear devs,
>
> I'm trying to import an XML file (900K) with around 8.000 datavalues in
> 2.24 (build revision 1502fe2). However, I'm getting 2 different kind of
> errors during the import... (attached at the end of the email)
>
> Any ideas? The only thing I can think of, is the huge number of org. units
> that this instance has (42.000 org. units) with 8 GB. Would it be related
> to a memory problem? (although the XML file is very small)
>
> I have also selected Preloaded Cache = No
>
> Many thanks!
>
>
> *Error 1*
>
> * INFO  2016-10-13 21:55:55,780 [Level: INFO, category: DATAVALUE_IMPORT,
> time: Thu Oct 13 21:55:55 UTC 2016, message: Importing data values]
> (InMemoryNotifier.java [taskScheduler-7])
> * ERROR 2016-10-13 22:10:55,332 java.lang.RuntimeException:
> org.hibernate.PropertyAccessException: Exception occurred inside getter of
> org.hisp.dhis.organisationunit.OrganisationUnit.path
> at
> org.hisp.dhis.commons.collection.CachingMap.get(CachingMap.java:90)
> at
> org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSet(DefaultDataValueSetService.java:954)
> at
> org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSet(DefaultDataValueSetService.java:488)
> at
> org.hisp.dhis.dxf2.datavalueset.tasks.ImportDataValueTask.call(ImportDataValueTask.java:102)
> at
> org.hisp.dhis.security.SecurityContextRunnable.run(SecurityContextRunnable.java:57)
> at
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.hibernate.PropertyAccessException: Exception occurred
> inside getter of org.hisp.dhis.organisationunit.OrganisationUnit.path
> at
> org.hibernate.property.access.spi.GetterMethodImpl.get(GetterMethodImpl.java:44)
> at
> org.hibernate.tuple.entity.AbstractEntityTuplizer.getPropertyValues(AbstractEntityTuplizer.java:505)
> at
> org.hibernate.tuple.entity.PojoEntityTuplizer.getPropertyValues(PojoEntityTuplizer.java:217)
> at
> org.hibernate.persister.entity.AbstractEntityPersister.getPropertyValues(AbstractEntityPersister.java:4613)
> at
> org.hibernate.event.internal.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:178)
> at
> org.hibernate.event.internal.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:135)
> at
> org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:216)
> at
> org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:85)
> at
> org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:44)
> at
> org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1264)
> at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1780)
> at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:363)
> at
> org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:385)
> at
> org.hisp.dhis.dataset.hibernate.HibernateLockExceptionStore.getCount(HibernateLockExceptionStore.java:167)
> at
> org.hisp.dhis.dataset.DefaultDataSetService.isLockedPeriod(DefaultDataSetService.java:384)

Re: [Dhis2-devs] Errors while importing datavalues in 2.24

[Jose Garcia Muñoz]

Thanks Lars,

I have run data integrity, but the system has passed "Organisation units
with cyclic references ". I didn't get any errors importing events, but
aggregated... Will try to divide the file in small chunks to try to figure
it out...

Thanks
Jose

On Fri, Oct 14, 2016 at 11:02 AM, Lars Helge Øverland 
wrote:

> Hi Jose,
>
> my best guess here is a cyclic parent relationship in the org unit
> hierarchy, i.e. you have one or more org units where the one of the
> ancestors are also a child.
>
> Try running data admin > integrity checks and look for "Organisation units
> with cyclic references ", or look for it through SQL.
>
> best regards,
>
> Lars
>
>
> On Fri, Oct 14, 2016 at 1:09 AM, Jose Garcia Muñoz 
> wrote:
>
>>
>> Dear devs,
>>
>> I'm trying to import an XML file (900K) with around 8.000 datavalues in
>> 2.24 (build revision 1502fe2). However, I'm getting 2 different kind of
>> errors during the import... (attached at the end of the email)
>>
>> Any ideas? The only thing I can think of, is the huge number of org.
>> units that this instance has (42.000 org. units) with 8 GB. Would it be
>> related to a memory problem? (although the XML file is very small)
>>
>> I have also selected Preloaded Cache = No
>>
>> Many thanks!
>>
>>
>> *Error 1*
>>
>> * INFO  2016-10-13 21:55:55,780 [Level: INFO, category: DATAVALUE_IMPORT,
>> time: Thu Oct 13 21:55:55 UTC 2016, message: Importing data values]
>> (InMemoryNotifier.java [taskScheduler-7])
>> * ERROR 2016-10-13 22:10:55,332 java.lang.RuntimeException:
>> org.hibernate.PropertyAccessException: Exception occurred inside getter
>> of org.hisp.dhis.organisationunit.OrganisationUnit.path
>> at org.hisp.dhis.commons.collection.CachingMap.get(CachingMap.
>> java:90)
>> at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.s
>> aveDataValueSet(DefaultDataValueSetService.java:954)
>> at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.s
>> aveDataValueSet(DefaultDataValueSetService.java:488)
>> at org.hisp.dhis.dxf2.datavalueset.tasks.ImportDataValueTask.
>> call(ImportDataValueTask.java:102)
>> at org.hisp.dhis.security.SecurityContextRunnable.run(SecurityC
>> ontextRunnable.java:57)
>> at org.springframework.scheduling.support.DelegatingErrorHandli
>> ngRunnable.run(DelegatingErrorHandlingRunnable.java:54)
>> at java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:293)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.hibernate.PropertyAccessException: Exception occurred
>> inside getter of org.hisp.dhis.organisationunit.OrganisationUnit.path
>> at org.hibernate.property.access.spi.GetterMethodImpl.get(Gette
>> rMethodImpl.java:44)
>> at org.hibernate.tuple.entity.AbstractEntityTuplizer.getPropert
>> yValues(AbstractEntityTuplizer.java:505)
>> at org.hibernate.tuple.entity.PojoEntityTuplizer.getPropertyVal
>> ues(PojoEntityTuplizer.java:217)
>> at org.hibernate.persister.entity.AbstractEntityPersister.getPr
>> opertyValues(AbstractEntityPersister.java:4613)
>> at org.hibernate.event.internal.DefaultFlushEntityEventListener
>> .getValues(DefaultFlushEntityEventListener.java:178)
>> at org.hibernate.event.internal.DefaultFlushEntityEventListener
>> .onFlushEntity(DefaultFlushEntityEventListener.java:135)
>> at org.hibernate.event.internal.AbstractFlushingEventListener.f
>> lushEntities(AbstractFlushingEventListener.java:216)
>> at org.hibernate.event.internal.AbstractFlushingEventListener.f
>> lushEverythingToExecutions(AbstractFlushingEventListener.java:85)
>> at org.hibernate.event.internal.DefaultAutoFlushEventListener.o
>> nAutoFlush(DefaultAutoFlushEventListener.java:44)
>> at org.hibernate.internal.SessionImpl.autoFlushIfRequired(Sessi
>> onImpl.java:1264)
>> at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1780)
>> at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:
>> 363)
>> at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImp
>> l.java:385)
>> at org.hisp.dhis.dataset.hibernate.HibernateLockExceptionStore.
>> getCount(HibernateLockExceptionStore.java:167)
>> at org.hisp.dhis.dataset.DefaultDataSetService.isLockedPeriod(D
>> efaultDataSetService.java:384)
>> at sun.reflect.GeneratedMethodAccessor709.invoke(Unknown Source)

Re: [Dhis2-devs] Errors while importing datavalues in 2.24

[Lars Helge Øverland]

Hi Jose,

my best guess here is a cyclic parent relationship in the org unit
hierarchy, i.e. you have one or more org units where the one of the
ancestors are also a child.

Try running data admin > integrity checks and look for "Organisation units
with cyclic references ", or look for it through SQL.

best regards,

Lars


On Fri, Oct 14, 2016 at 1:09 AM, Jose Garcia Muñoz 
wrote:

>
> Dear devs,
>
> I'm trying to import an XML file (900K) with around 8.000 datavalues in
> 2.24 (build revision 1502fe2). However, I'm getting 2 different kind of
> errors during the import... (attached at the end of the email)
>
> Any ideas? The only thing I can think of, is the huge number of org. units
> that this instance has (42.000 org. units) with 8 GB. Would it be related
> to a memory problem? (although the XML file is very small)
>
> I have also selected Preloaded Cache = No
>
> Many thanks!
>
>
> *Error 1*
>
> * INFO  2016-10-13 21:55:55,780 [Level: INFO, category: DATAVALUE_IMPORT,
> time: Thu Oct 13 21:55:55 UTC 2016, message: Importing data values]
> (InMemoryNotifier.java [taskScheduler-7])
> * ERROR 2016-10-13 22:10:55,332 java.lang.RuntimeException: 
> org.hibernate.PropertyAccessException:
> Exception occurred inside getter of org.hisp.dhis.organisationunit.
> OrganisationUnit.path
> at org.hisp.dhis.commons.collection.CachingMap.get(
> CachingMap.java:90)
> at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.
> saveDataValueSet(DefaultDataValueSetService.java:954)
> at org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.
> saveDataValueSet(DefaultDataValueSetService.java:488)
> at org.hisp.dhis.dxf2.datavalueset.tasks.ImportDataValueTask.call(
> ImportDataValueTask.java:102)
> at org.hisp.dhis.security.SecurityContextRunnable.run(
> SecurityContextRunnable.java:57)
> at org.springframework.scheduling.support.
> DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnabl
> e.java:54)
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.hibernate.PropertyAccessException: Exception occurred
> inside getter of org.hisp.dhis.organisationunit.OrganisationUnit.path
> at org.hibernate.property.access.spi.GetterMethodImpl.get(
> GetterMethodImpl.java:44)
> at org.hibernate.tuple.entity.AbstractEntityTuplizer.
> getPropertyValues(AbstractEntityTuplizer.java:505)
> at org.hibernate.tuple.entity.PojoEntityTuplizer.
> getPropertyValues(PojoEntityTuplizer.java:217)
> at org.hibernate.persister.entity.AbstractEntityPersister.
> getPropertyValues(AbstractEntityPersister.java:4613)
> at org.hibernate.event.internal.DefaultFlushEntityEventListene
> r.getValues(DefaultFlushEntityEventListener.java:178)
> at org.hibernate.event.internal.DefaultFlushEntityEventListene
> r.onFlushEntity(DefaultFlushEntityEventListener.java:135)
> at org.hibernate.event.internal.AbstractFlushingEventListener.
> flushEntities(AbstractFlushingEventListener.java:216)
> at org.hibernate.event.internal.AbstractFlushingEventListener.
> flushEverythingToExecutions(AbstractFlushingEventListener.java:85)
> at org.hibernate.event.internal.DefaultAutoFlushEventListener.
> onAutoFlush(DefaultAutoFlushEventListener.java:44)
> at org.hibernate.internal.SessionImpl.autoFlushIfRequired(
> SessionImpl.java:1264)
> at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1780)
> at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:363)
> at org.hibernate.internal.CriteriaImpl.uniqueResult(
> CriteriaImpl.java:385)
> at org.hisp.dhis.dataset.hibernate.HibernateLockExceptionStore.
> getCount(HibernateLockExceptionStore.java:167)
> at org.hisp.dhis.dataset.DefaultDataSetService.isLockedPeriod(
> DefaultDataSetService.java:384)
> at sun.reflect.GeneratedMethodAccessor709.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.springframework.aop.support.AopUtils.
> invokeJoinpointUsingReflection(AopUtils.java:302)
> at org.springframework.aop.framework.ReflectiveMethodInvocation.
> invokeJoinpoint(ReflectiveMethodInvocation.java:190)
> at 

[Dhis2-devs] Errors while importing datavalues in 2.24

[Jose Garcia Muñoz]

Dear devs,

I'm trying to import an XML file (900K) with around 8.000 datavalues in
2.24 (build revision 1502fe2). However, I'm getting 2 different kind of
errors during the import... (attached at the end of the email)

Any ideas? The only thing I can think of, is the huge number of org. units
that this instance has (42.000 org. units) with 8 GB. Would it be related
to a memory problem? (although the XML file is very small)

I have also selected Preloaded Cache = No

Many thanks!


*Error 1*

* INFO  2016-10-13 21:55:55,780 [Level: INFO, category: DATAVALUE_IMPORT,
time: Thu Oct 13 21:55:55 UTC 2016, message: Importing data values]
(InMemoryNotifier.java [taskScheduler-7])
* ERROR 2016-10-13 22:10:55,332 java.lang.RuntimeException:
org.hibernate.PropertyAccessException: Exception occurred inside getter of
org.hisp.dhis.organisationunit.OrganisationUnit.path
at
org.hisp.dhis.commons.collection.CachingMap.get(CachingMap.java:90)
at
org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSet(DefaultDataValueSetService.java:954)
at
org.hisp.dhis.dxf2.datavalueset.DefaultDataValueSetService.saveDataValueSet(DefaultDataValueSetService.java:488)
at
org.hisp.dhis.dxf2.datavalueset.tasks.ImportDataValueTask.call(ImportDataValueTask.java:102)
at
org.hisp.dhis.security.SecurityContextRunnable.run(SecurityContextRunnable.java:57)
at
org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.hibernate.PropertyAccessException: Exception occurred inside
getter of org.hisp.dhis.organisationunit.OrganisationUnit.path
at
org.hibernate.property.access.spi.GetterMethodImpl.get(GetterMethodImpl.java:44)
at
org.hibernate.tuple.entity.AbstractEntityTuplizer.getPropertyValues(AbstractEntityTuplizer.java:505)
at
org.hibernate.tuple.entity.PojoEntityTuplizer.getPropertyValues(PojoEntityTuplizer.java:217)
at
org.hibernate.persister.entity.AbstractEntityPersister.getPropertyValues(AbstractEntityPersister.java:4613)
at
org.hibernate.event.internal.DefaultFlushEntityEventListener.getValues(DefaultFlushEntityEventListener.java:178)
at
org.hibernate.event.internal.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:135)
at
org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:216)
at
org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:85)
at
org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:44)
at
org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1264)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1780)
at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:363)
at
org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:385)
at
org.hisp.dhis.dataset.hibernate.HibernateLockExceptionStore.getCount(HibernateLockExceptionStore.java:167)
at
org.hisp.dhis.dataset.DefaultDataSetService.isLockedPeriod(DefaultDataSetService.java:384)
at sun.reflect.GeneratedMethodAccessor709.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at