Hi Kenton,
Thanks for the reply. I didn't want to overwhelm you guys, its been hard to
decide what to share.
Perhaps this will be a good peak into the main function giving me problems:
class DagDataset(Dataset):
def __init__(self, path2dataprep, path2hash2idx, split):
self.split = split
self.path2dataprep = path2dataprep
db = torch.load(self.path2dataprep)
self.data_prep = db['data_prep']
self.list_files_current_split =
self.data_prep.flatten_lst_files_split[self.split]
self.list_counts_current_split = self.data_prep.counts[self.split]
self.list_cummulative_sum_current_split =
self.data_prep.cummulative_sum[self.split]
self.list_cummulative_end_index_current_split =
self.data_prep.cummulative_end_index[self.split]
self.length = sum(self.list_counts_current_split)
#
self.path2hash2idx = path2hash2idx
db = torch.load(self.path2hash2idx)
self.hash2idx = db['hash2idx']
def __len__(self):
return self.length
def __getitem__(self, idx: int) -> DagNode:
# gets the file idx for the value we want
file_idx =
bisect.bisect_left(self.list_cummulative_end_index_current_split, idx)
# now get the actual file name
file_name = self.list_files_current_split[file_idx]
# get the file with proof steps
file_name = self.convert_to_local_home_path(file_name)
f = open(file_name)
current_dag_file = dag_api_capnp.Dag.read_packed(f,
traversal_limit_in_words=2 ** 64 - 1)
# current_dag_file = dag_api_capnp.Dag.read_packed(f)
# - global idx 2 idx relative to this file
prev_cummulative_sum = self.get_previous_cummulative_sum(file_idx)
idx_rel_this_file = idx - prev_cummulative_sum
# - data point
node_idx = current_dag_file.proofSteps[idx_rel_this_file].node
tactic_hash = current_dag_file.proofSteps[idx_rel_this_file].tactic
tactic_label = self.hash2idx[tactic_hash]
# - get Node obj
node_ref = NodeRef(node_idx, 0) # indicates it's in the
current file this cased named current_dag_file
node = DagNode(current_dag_file, node_ref)
# node = current_dag_file
f.close()
return node, tactic_label
my suspicion is that its the this line of code that is giving me issues.:
current_dag_file = dag_api_capnp.Dag.read_packed(f,
traversal_limit_in_words=2 ** 64 - 1)
I think if I return the current_dag_file reference directly and do a del
and gc.collect() instead of the wrapper around it that I defined called
DagNode, it seems to solve the memory problem (perhaps though I am not 100%
but fairly sure).
I will confirm if this is true in a sec.
On Wed, Mar 31, 2021 at 10:56 AM Kenton Varda <[email protected]> wrote:
> Hi Brando,
>
> It's hard for us to guess what might be the problem without seeing more
> code.
>
> -Kenton
>
> On Tue, Mar 30, 2021 at 12:56 PM Brando Miranda <[email protected]>
> wrote:
>
>> Hi,
>>
>> I am doing machine learning with captain proto (because captain proto is
>> good at communicating between python and different languages).
>>
>> The challenge I have is that my data is represented in captain proto
>> structs. I load a batch of these proto structs every so often to process
>> them with a Neural Network. However, eventually after a certain number of
>> iterations it seems I allocated all the system's memory and I get a SIGKILL
>> from OOM.
>>
>> I am unsure why this would happen or where (I've been memory profiling my
>> code all day but it is difficult to figure out what part is breaking). I am
>> fairly sure it has to do with captain proto because I used to have a
>> version of the data set with json files and I didn't have this error but
>> now I do. I could directly use the captain proto dataset to create a new
>> json file data set to really figure out if that is the case but it seems
>> redundant.
>>
>> I thought it could be that I open a captain proto struct and then I close
>> it:
>>
>>
>> file_name = self.convert_to_local_home_path(file_name)
>> f = open(file_name)
>> ...
>> bunch of processing to make it into my python class
>> ....
>> f.close()
>> return x, y
>>
>> I am explicitly closing the file from captain proto so I'd assume that
>> isn't the case. But anyway, is there a way to really check if the memory
>> errors I am getting are due to captain proto or not?
>>
>> Thanks, Brando
>>
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Cap'n Proto" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/capnproto/ac8e4eba-11e9-44cc-9095-4313e4b7e544n%40googlegroups.com
>> <https://groups.google.com/d/msgid/capnproto/ac8e4eba-11e9-44cc-9095-4313e4b7e544n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/CAO-aKn7c7B_5r9CJcTM5eCpooZ7EOy-Ow%3Ddn%3DiFf0ot8qGp6rA%40mail.gmail.com.
