[
https://issues.apache.org/jira/browse/MADLIB-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Frank McQuillan reassigned MADLIB-1259:
---------------------------------------
Assignee: Orhan Kislal
> PostgreSQL out of memory issue with Neural networks training
> ------------------------------------------------------------
>
> Key: MADLIB-1259
> URL: https://issues.apache.org/jira/browse/MADLIB-1259
> Project: Apache MADlib
> Issue Type: Bug
> Components: Module: Neural Networks
> Reporter: Nandish Jayaram
> Assignee: Orhan Kislal
> Priority: Major
> Fix For: v2.0
>
>
> Neural network training results in an out of memory exception in the
> following scenario:
> * 16 GB RAM
> * Dataset: Same as the one used in
> https://issues.apache.org/jira/browse/MADLIB-1257.
> 23K instances / 300 features in 263 groups
> * PostgreSQL memory setup.
> checkpoint_completion_target = '0.9';
> default_statistics_target = '500';
> effective_cache_size = '12GB';
> effective_io_concurrency = '200';
> maintenance_work_mem = '2GB';
> max_connections = '20';
> max_parallel_workers = '4';
> max_parallel_workers_per_gather = '2';
> max_wal_size = '8GB';
> max_worker_processes = '4';
> min_wal_size = '4GB';
> random_page_cost = '1.1';
> shared_buffers = '4GB';
> wal_buffers = '16MB';
> work_mem = '52428kB';
> sysctl -w vm.overcommit_memory=2 to avoid the crash of postmaster
> With the above database settings and dataset size, the following query
> resulted in an error:
> {code:java}
> SELECT madlib.mlp_classification(
> 'train_data_sub', -- Source table
> 'mlp_model', -- Destination table
> 'features', -- Input features
> 'positive', -- Label
> ARRAY[5], -- Number of units per layer
> 'learning_rate_init=0.003,
> n_iterations=500,
> tolerance=0', -- Optimizer params
> 'tanh', -- Activation function
> NULL, -- Default weight (1)
> FALSE, -- No warm start
> true, -- verbose
> 'case_icd' -- Grouping
> );
> ERROR: spiexceptions.OutOfMemory: out of memory
> DETAIL: Failed on request of size 32800.
> CONTEXT: Traceback (most recent call last):
> PL/Python function "mlp_classification", line 36, in <module>
> grouping_col
> PL/Python function "mlp_classification", line 45, in wrapper
> PL/Python function "mlp_classification", line 325, in mlp
> PL/Python function "mlp_classification", line 580, in update
> PL/Python function "mlp_classification"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
