[
https://issues.apache.org/jira/browse/MADLIB-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Frank McQuillan reassigned MADLIB-1406:
---------------------------------------
Assignee: Nikhil Kak
> DL: fit multiple takes up unnecessary disk space
> ------------------------------------------------
>
> Key: MADLIB-1406
> URL: https://issues.apache.org/jira/browse/MADLIB-1406
> Project: Apache MADlib
> Issue Type: Bug
> Components: Deep Learning
> Reporter: Nikhil Kak
> Assignee: Nikhil Kak
> Priority: Major
> Fix For: v1.17
>
>
> While testing places10 with fit multiple (gpdb5, 10 iterations and 20 msts),
> we ran out of disk space although we had at least 1.5T left at the beginning
> of the query. There is no reason for us to use this much space and this
> probably means that there is a bug in the code
> Here is the query and the failure
> {code:java}
> DROP TABLE IF EXISTS mst_table, mst_table_summary;
> SELECT load_model_selection_table(
> 'model_arch_places10',
> 'mst_table',
> ARRAY[1],
> ARRAY[
> $$loss='categorical_crossentropy', optimizer='SGD(lr=0.1, decay=1e-6,
> nesterov=True)', metrics=['accuracy']$$,
> $$loss='categorical_crossentropy', optimizer='SGD(lr=0.01,
> decay=1e-6, nesterov=True)', metrics=['accuracy']$$,
> $$loss='categorical_crossentropy', optimizer='SGD(lr=0.001,
> decay=1e-6, nesterov=True)', metrics=['accuracy']$$,
> $$loss='categorical_crossentropy', optimizer='SGD(lr=0.0001,
> decay=1e-6, nesterov=True)', metrics=['accuracy']$$,
> $$loss='categorical_crossentropy', optimizer='SGD(lr=0.001,
> decay=1e-6, nesterov=False)', metrics=['accuracy']$$
> ],
> ARRAY[
> $$batch_size=16, epochs=1, verbose=0$$,
> $$batch_size=20, epochs=1, verbose=0$$,
> $$batch_size=32, epochs=1, verbose=0$$,
> $$batch_size=40, epochs=1, verbose=0$$
> ]
> );
> DROP TABLE if exists places10_train_mult_model,
> places10_train_mult_model_summary, places10_train_mult_model_info;
> SELECT madlib_keras_fit_multiple_model(
> 'places10_train_bytea_batched',
> 'places10_train_mult_model',
> 'mst_table',
> 10,
> TRUE
> );
> -- failed in the 7th iteration
> ....
> Time for training in iteration 6: 6403.70687222 sec
> ERROR: plpy.SPIError: could not extend relation 1663/3721274/1121877: No
> space left on device (seg1){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
