[
https://issues.apache.org/jira/browse/MADLIB-1345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16936205#comment-16936205
]
Ekta Khanna commented on MADLIB-1345:
-------------------------------------
w/ [~nikhilkak]
After rethinking the implementation of {{madlib_keras_predict()}} using GD, the
GD seems a little brittle mainly due to the concern around the process
availability between the queries where GD is set and used while calling keras
predict.
Instead, we came up with re-writing the predict query calling
{{internal_keras_predict()}}, where we pass in the model weights only for the
first row of each segment(by querying minimum ctid on each segment) and NULL
for the remaining rows.
With this approach below are the performance runs for
{{madlib_keras_predict()}} on places10 dataset (20 segment GPDB cluster) :
*On master*
{{SELECT madlib.madlib_keras_predict(
'out_places10_bytea_1iter','places10_train', 'id','x',
'places10_predict_out_master', NULL,4);}}
{{Time: 1285697.150 ms}}
*using GD*
{{SELECT madlib.madlib_keras_predict(
'out_places10_bytea_1iter','places10_train', 'id','x',
'places10_predict_out_GD', NULL,4);}}
{{Time: 345466.193 ms}}
*Query re-write*
{{SELECT madlib.madlib_keras_predict(
'out_places10_bytea_1iter','places10_train', 'id','x',
'places10_predict_out_CASE', NULL,4);}}
{{Time: 354660.193 ms}}
Looking at the above runtimes, it is comparable to the GD approach {{~4x}}
faster than the current master code and more reliable than it.
> DL: Performance improvement in DL functions
> -------------------------------------------
>
> Key: MADLIB-1345
> URL: https://issues.apache.org/jira/browse/MADLIB-1345
> Project: Apache MADlib
> Issue Type: Improvement
> Components: Deep Learning
> Reporter: Ekta Khanna
> Priority: Major
> Fix For: v2.0
>
>
> Currently, we pass around model_data, model_arch, etc.. for each buffer/image
> for fit(), predict() and evaluate(). This causes a lot of overhead and slows
> down the query considerable.
> We tried to set model_data and model_arch using GD for predict. Following
> were the runtimes:
> with GD
> ~707 sec(with CPU) - 50K places10_20seg
> without GD
> ~1650 sec(with CPU) - 50K places10_20seg
> Below is the patch for GD changes:
> {code}
> def set_predict_GD(model_architecture, model_data,
> is_response, normalizing_const, seg_ids,
> images_per_seg, gpus_per_host, segments_per_host,
> **kwargs):
> GD = kwargs['GD']
> GD['model_architecture'] = model_architecture
> GD['model_data'] = model_data
> GD['is_response'] = is_response
> GD['normalizing_const'] = normalizing_const
> #GD['current_seg_id'] = current_seg_id
> GD['seg_ids'] = seg_ids
> GD['images_per_seg'] = images_per_seg
> GD['gpus_per_host'] = gpus_per_host
> GD['segments_per_host'] = segments_per_host
> def predict()
> ....
> set_gd_query=plpy.prepare("""
> SELECT set_predict_GD
> ($MAD${model_arch}$MAD$,
> $1,
> {is_response},
> {normalizing_const},
> -- gp_segment_id,
> ARRAY{seg_ids_test},
> ARRAY{images_per_seg_test},
> {gpus_per_host},
> {segments_per_host}
> ) from gp_dist_random('gp_id')
> """.format(**locals()), ["bytea"]) #Using gp_dist_random('gp_id')
> in the query makes the UDF run on each segment
> plpy.execute(set_gd_query, [model_data])
> predict_query = plpy.execute("""
> CREATE TABLE {output_table} AS
> SELECT {id_col}, {prediction_select_clause}
> FROM (
> SELECT {test_table}.{id_col},
> ({schema_madlib}.internal_keras_predict
> ({independent_varname}, {gp_segment_id_col})
> ) AS {intermediate_col}
> FROM {test_table}
> ) q distributed by ({id_col})
> """.format(**locals()))
> def internal_keras_predict(independent_var, current_seg_id, **kwargs):
> start = time.time()
> SD = kwargs['SD']
> GD = kwargs['GD']
> is_response = GD['is_response']
> normalizing_const = GD['normalizing_const']
> #current_seg_id = GD['current_seg_id']
> seg_ids = GD['seg_ids']
> images_per_seg = GD['images_per_seg']
> gpus_per_host = GD['gpus_per_host']
> segments_per_host = GD['segments_per_host']
> device_name = get_device_name_and_set_cuda_env(gpus_per_host,
> current_seg_id)
> ...
> {code}
> With the above changes , we found out that GD is not reliable for GPDB
> because of the following reasons:
> Consider a single node gpdb cluster with 3 segments
> Calling set_gd using gp_dist_random(), creates 1 process per seg and sets GD
> on each of these processes.
> seg1 - pid 100 - gd is set here for seg1
> seg2 - pid 200 - gd is set here for seg2
> seg3 - pid300- gd is set here for seg3
> Now, CREATE TABLE.. in predict(), spins up 2 processes per seg, (the old
> processes where GD was set) + 1 new process per seg.
> seg1 - pid 100 - gd is set here for seg1 (reused from before)
> seg1 - pid 101 - gd is read here for seg1
> seg2 - pid 200 - gd is set here for seg2 (reused from before)
> seg1 - pid 201 - gd is read here for seg2
> seg3 - pid300 - gd is set here for seg3 (reused from before)
> seg1 - pid 301- gd is read here for seg3
> This causes problems because , the processes where GD is read from is not
> same as the process where it was set.
> Couple of ways to avoid this problem
> # Change predict code to run two plpy execute queries, the first one being
> the internal predict query and the second one being the create table query.
> # Distribute the source table by the id column and while creating the predict
> output table use that id column as the distribution key.
> We are not sure if this is good enough for all use cases like what if the
> source table has an index which might do the same thing as the create table
> command. Our goal is to avoid the query from creating multiple processes.
> # Explore the GD option
> # Explore alternatives so that we don't have to pass the model data for every
> row/buffer/image in the transition function/udf
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
