I think we might have "file GC issues" - I vaguely remember that we
don't (or at least didn't once upon a time) proactively remove
unnecessary run files - removing all of them at end-of-job instead of at
the end of the execution phase that uses their contents. We may also
have an "Amdahl problem" right now with our sort since we serialize
phase two of parallel sorts - though this is not a query, it's index
build, so that shouldn't be it. It would be interesting to put a
df/sleep script on each of the nodes when this is happening - actually a
script that monitors the temp file directory - and watch the lifecycle
happen and the sizes change....
On 8/23/16 2:06 AM, Chris Hillery wrote:
When you get the "disk full" warning, do a quick "df -i" on the device -
possibly you've run out of inodes even if the space isn't all used up. It's
unlikely because I don't think AsterixDB creates a bunch of small files,
but worth checking.
If that's not it, then can you share the full exception and stack trace?
Ceej
aka Chris Hillery
On Tue, Aug 23, 2016 at 1:59 AM, Wail Alkowaileet <[email protected]>
wrote:
I just cleared the hard drives to get 80% free space. I still get the same
issue.
The data contains:
1- 2887453794 records.
2- Schema:
create type CDRType as {
id:uuid,
'date':string,
'time':string,
'duration':int64,
'caller':int64,
'callee':int64,
location:point?
}
On Tue, Aug 23, 2016 at 9:06 AM, Wail Alkowaileet <[email protected]>
wrote:
Dears,
I have a dataset of size 290GB loaded in a 3 NCs each of which has
2x500GB
SSD.
Each of NC has two IODevices (partitions) in each hard drive (i.e the
total is 4 iodevices per NC). After loading the data, each Asterix
partition occupied 31GB.
The cluster has about 50% free space in each hard drive (approximately
about 250GB free space in each hard drive). However, when I tried to
create
an index of type RTree, I got an exception that no space left in the hard
drive during the External Sort phase.
Is that normal ?
--
*Regards,*
Wail Alkowaileet
--
*Regards,*
Wail Alkowaileet
