Hi Mohiuddin, This bug has already existed for a year or so ( https://issues.apache.org/jira/browse/ASTERIXDB-2125?filter=-2). It seems during the RTree bulkloading process some calculation of MBRs are wrong (but it's hard to be reproduced). If one merge operation fails, then in the future all retries would fail as well since the same bug would always be triggered...
Best regards, Chen Luo On Sun, Jul 15, 2018 at 2:22 AM Mohiuddin Qader <[email protected]> wrote: > After I debugged myself and looked up in logs, I tracked down the problem. > It looks like during merge compaction it fails due to data type issue for > points during MBR calculation at the end of bulk loading (don't know why) > and throws an exception. But the main bug here was that there is no follow > up to clean up the failed merged file and thus no subsequent compaction can > occur. I attach the screenshot. > > > org.apache.hyracks.algebricks.common.exceptions.NotImplementedException: > Value provider for type bigint is not implemented. > > > > > On Sat, Jul 14, 2018 at 9:53 PM, Mohiuddin Qader <[email protected]> wrote: > >> Hello everyone, >> >> I have been running some experiments on current master AsterixDB with >> default merge policy. I am using simple OpenStreetMap and Twitter dataset >> with id used as primary key and a Rtree index on location attribute. >> >> Then I used a feed to ingest the data to the database. After running >> some time (about 2 million insertion), the Rtree index >> (LSMRtreeWithAntiMatterTuples) are throwing a strange exception like file >> already exists: After this exception, cluster infinitely giving this >> exception repetitively and become unusable. >> >> 20:36:55.980 [Executor-24:asterix_nc1] ERROR >> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness - Failed merge >> operation on {"class" : "LSMRTreeWithAntiMatterTuples", "dir" : >> "/home/mohiuddin/asterix-hyracks/asterixdb/target/io/dir/asterix_nc1/target/tmp/asterix_nc1/iodevice1/storage/partition_0/experiments/OpenStreetMap/0/OSMlocation", >> "memory" : 2, "disk" : 5} >> org.apache.hyracks.api.exceptions.HyracksDataException: HYR0082: Failed >> to create the file >> /home/mohiuddin/asterix-hyracks/asterixdb/target/io/dir/asterix_nc1/target/tmp/asterix_nc1/iodevice1/storage/partition_0/experiments/OpenStreetMap/0/OSMlocation/2018-07-14-20-36-09-733_2018-07-14-20-34-55-555 >> because it already exists >> at >> org.apache.hyracks.api.exceptions.HyracksDataException.create(HyracksDataException.java:55) >> ~[classes/:?] >> at org.apache.hyracks.api.util.IoUtil.create(IoUtil.java:87) >> ~[classes/:?] >> at >> org.apache.hyracks.storage.common.buffercache.BufferCache.createFile(BufferCache.java:809) >> ~[classes/:?] >> at >> org.apache.hyracks.storage.am.common.impls.AbstractTreeIndex.create(AbstractTreeIndex.java:83) >> ~[classes/:?] >> at >> org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMDiskComponent.activate(AbstractLSMDiskComponent.java:158) >> ~[classes/:?] >> at >> org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.createDiskComponent(AbstractLSMIndex.java:427) >> ~[classes/:?] >> at >> org.apache.hyracks.storage.am.lsm.rtree.impls.LSMRTreeWithAntiMatterTuples.doMerge(LSMRTreeWithAntiMatterTuples.java:237) >> ~[classes/:?] >> at >> org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.merge(AbstractLSMIndex.java:728) >> ~[classes/:?] >> at >> org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:645) >> [classes/:?] >> at >> org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.merge(LSMTreeIndexAccessor.java:128) >> [classes/:?] >> at >> org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:45) >> [classes/:?] >> at >> org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:30) >> [classes/:?] >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> [?:1.8.0_45-internal] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [?:1.8.0_45-internal] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [?:1.8.0_45-internal] >> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45-internal] >> >> >> I have tried to reinstall everything multiple times, removed all old >> storage files, giving pause during and before ingestion through feeds. >> Nothing seems to be working, every time this exception is occurring after >> some ingestion. Can any of you guys have idea on what is happening? I have >> attached the DDL I was using. >> >> >> >> -- >> Regards, >> Mohiuddin Abdul Qader >> Dept of Computer Science >> University of California Riverside >> > > > > -- > Regards, > Mohiuddin Abdul Qader > Dept of Computer Science > University of California Riverside >
