welcome to 824pm on sunday evening
it's been hard to move near the code i started. tomorrow morning i have
appointment again.
maybe i can paste in a snippet and comment on a concern,
this is part of where i left off. last time i elided the in-progress
commenting, here it's included:
idx = 0
while idx < len(offset_lengths):
#for idx in pbar:#range(len(offset_lengths)):
#offset, length = offset_lengths[idx]
#tail = min(offset + length, len(self.mmap))
#aligned_offset = (offset // self.blksize) * self.blksize
#aligned_tail = min(self.size(), (((tail - 1) // self.blksize)
+ 1) * self.blksize)
aligned_offset = aligned_offsets[idx].item()
if aligned_offset > min_hole:
next_hole = self._next_sparse(aligned_offset, os.SEEK_HOLE)
else:
next_hole = min_hole
missing_idcs = (next_hole < tails[idx:]).nonzero()[:,0]
# here we are handling all undownloaded indices before the next
cached ones
# there could be many pages between them that don't need to be
fetched
num_missing_idcs = missing_idcs.shape[0]
if num_missing_idcs > 0:
i expect the logic in the above is not quite correct. i've been trying to
re-understand `missing_idcs = (next_hole < tails[idx:]).nonzero()[:,0]`. i
think this line was ported from the non-batched loop in a naive manner.
roughly, i need to figure out how to correct process chunks of idcs that are
cached and are not cached. when they are cached, then they are represented by a
data region in the sparse mmap file. when they are not cached, then they are
represented by a hole region.
`self. _next_sparse(self, off, region)` is a wrapper around `os.lseek` -- given
an offset, and a region type, it returns the offset of the next byte with the
given region type, such that it returns the passed offset if it is of the given
region type. this is presently the only interface into determining what is
cached and what is not, and it works fine for procedural scans and should work
fine here for now. (the caller of read_many needs to provide the offsets and
lengths of the requested data already). [it's also reasonable to add an index
to the cache, although it would add a concern of syncing sparsity to manage
diskspace. a sparse file is a special kind of file where regions of the file do
not occupy disk space and cannot contain nonzero data, these regions are called
holes, and they can be written to which uses more disk space when done.]
here's the current head of read_many:
def read_many(self, offset_lengths, progress, validate_sorted=True):
if validate_sorted:
sorted_offset_lengths = list(offset_lengths)
sorted_offset_lengths.sort()
assert sorted_offset_lengths == offset_lengths
OP_FETCH = 1
OP_PLACE = 2
OP_OUTPUT = 4
offset_length_tail_idx_ops =
torch.zeros([offset_lengths.shape[0]*2, 5])
OFFSET, LENGTH, TAIL, IDX, OP =
range(offset_length_tail_ops.shape[-1])
op_ct = 0
my idea to prototype things was to have the structure
offset_length_tail_idx_ops contain all the data in one parallel block that is
needed to operate. so first this structure is filled with data in batches using
vector operations, and then as much of it as possible is operated on at once.
the abstraction bounds can change. it's just what i have atm, and obviously
everything's confusing and slow for me so i'm working off it part by part.
the name `offset_length_tail_idx_ops` is named to directly represent the
content of the rows in order. this is a convention i use to code using less of
my working memory. i'm happy to add or remove data to this structure, and i do
a find/replace to update the name when i do so.
so the next step i was talking about, i was planning to implement involved
preparing for filling this structure with `OP_OUTPUT` entries for each of the
`offset_lengths` specifying what values to output back to the user. this is
maybe not necessary, but it focuses me around the next logical concern -- which
is ensuring uncached data is properly fetched, whereas cached data is not. so
maybe not quite needed yet.
been confused all day around ` missing_idcs = (next_hole <
tails[idx:]).nonzero()[:,0]` despite writing this line earlier myself. [after
success on finding region bounds]. given the loop is at `idx`, i gotta figure
out which `idcs` to consider cached, and which to fetch, and how to organize
that into a larger loop that repeats. i'm just getting really confused around
that simple concern
