[
https://issues.apache.org/jira/browse/MADLIB-960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123588#comment-15123588
]
Frank McQuillan edited comment on MADLIB-960 at 1/29/16 3:24 PM:
-----------------------------------------------------------------
Recent update from Rahul including questions sent to HAWQ team:
Current state of affairs. We have some failures and in some cases it's just too
slow. The kmeans failure is MADlib's fault - I'm in the process of fixing that.
For others, it could be combination of issues with HAWQ, HDFS and MADlib not
doing the right thing for HAWQ.
All modules with + have passed, with - have either failed or are too slow. The
X implies those modules have been removed since they're deprecated. I can keep
them but it would require some work to make them pass.
{code}
Modules:
+ array_ops
- bayes (view leads to crash in hdfs)
- crf (pg_temp in case statement #4)
+ elastic_net
- linalg (too slow)
- pmml (too slow)
+ prob
+ sketch
X svd_mf (deleted)
- svm (too slow)
- tsa (too slow)
- conjugate_gradient
- lda ()
- stats (gp_segment_configuration #1)
+ svec_util
+ utilities
- assoc_rules (too slow)
X cart (deleted)
+ convex
- glm (node type issue #3)
+ linear_systems ()
- recursive_partitioning ()
- regress ()
+ sample
+ summary
- kmeans (closest_column still has exception for HAWQ)
<--------------------
- pca ()
- validation (pg_temp in aggregate function #5)
{code}
Questions:
1. gp_segment_configuration has changed. What's the best way to compute the
number
of segments?
2. hdfs crashes on many of the queries. A particular case is the view creation
in bayes module.
3. psql:/tmp/madlib.eVW4q7/glm/test/gaussian.sql_in.tmp:92: ERROR:
plpy.SPIError: could not serialize unrecognized node type: 44867360
(plpython.c:4651)
CONTEXT: Traceback (most recent call last):
PL/Python function "glm", line 23, in <module>
return glm.glm(**globals())
PL/Python function "glm", line 90, in glm
PL/Python function "glm", line 223, in __glm_compute
PL/Python function "glm", line 52, in __compute_glm
PL/Python function "glm", line 164, in final
PL/Python function "glm"
4. plpy.SPIError: invalid reference to FROM-clause entry for table
"_madlib_tmp1_feature" (plpython.c:4651)
LINE 4: ... CASE WHEN pg_temp._m...
^
HINT: There is an entry for table "_madlib_tmp1_feature", but it cannot be
referenced from this part of the query.
QUERY: INSERT INTO pg_temp._madlib_tmp_rtbl(start_pos,doc_id,feature)
SELECT start_pos, doc_id, array_cat(fset.feature,
ARRAY[f_index,start_pos,
CASE WHEN
pg_temp._madlib_tmp1_feature.feature = fset.feature THEN 1
ELSE 0
END] )
FROM pg_temp._madlib_tmp1_feature, train_featureset fset
WHERE pg_temp._madlib_tmp1_feature.f_name = fset.f_name
AND fset.f_name <> 'E.';
5. psql:/tmp/madlib.kHS01Q/validation/test/cross_validation.sql_in.tmp:1425:
NOTICE: table "cv_linregr_result" does not exist, skipping
CONTEXT: SQL statement "drop table if exists cv_linregr_result"
PL/pgSQL function "check_cv0" line 5 at execute statement
psql:/tmp/madlib.kHS01Q/validation/test/cross_validation.sql_in.tmp:1425:
ERROR: plpy.SPIError: invalid reference to FROM-clause entry for table
"__madlib_temp_15692241_1454031660_13288785__" (plpython.c:4651)
LINE 4: avg((pg_temp.__madlib_temp_15692241_1454031660_1...
^
HINT: There is an entry for table
"__madlib_temp_15692241_1454031660_13288785__", but it cannot be referenced
from this part of the query.
QUERY:
CREATE TABLE pg_temp.__madlib_temp_68223508_1454031660_3362198__ AS
SELECT
avg((pg_temp.__madlib_temp_15692241_1454031660_13288785__.prediction -
("Y"))^2) as mean_squared_error
FROM
pg_temp.__madlib_temp_15692241_1454031660_13288785__,
__madlib_temp_77009712_1454031660_23171700__
WHERE
pg_temp.__madlib_temp_15692241_1454031660_13288785__.__madlib_temp_84152653_1454031660_15203995__
=
__madlib_temp_77009712_1454031660_23171700__.__madlib_temp_84152653_1454031660_15203995__
CONTEXT: Traceback (most recent call last):
PL/Python function "cross_validation_general", line 23, in <module>
return cross_validation.cross_validation_general(**globals())
PL/Python function "cross_validation_general", line 360, in
cross_validation_general
PL/Python function "cross_validation_general", line 265, in _one_step_cv
PL/Python function "cross_validation_general", line 121, in
__cv_funcall_general
PL/Python function "cross_validation_general"
SQL statement "SELECT cross_validation_general( 'cv_linregr_train',
ARRAY['%data%', '"X"', '"Y"', '%model%']::varchar[], '{varchar, varchar,
varchar, varchar}'::varchar[], NULL::varchar, NULL, 'cv_linregr_predict',
ARRAY['%model%', '%data%', '"X"', '%id%', '%prediction%']::varchar[],
'{varchar, varchar, varchar, varchar, varchar}'::varchar[], 'mse_error',
ARRAY['%prediction%', '%data%', '%id%', '"Y"', '%error%']::varchar[],
'{varchar, varchar, varchar, varchar, varchar}'::varchar[], '"Lin_housing_wi"',
NULL, False, 'cv_linregr_result', ARRAY['"X"', '"Y"']::varchar[], 10 )"
PL/pgSQL function "check_cv0" line 6 at perform
was (Author: fmcquillan):
Recent update from Rahul including questions sent to HAWQ team:
Current state of affairs. We have some failures and in some cases it's just too
slow. The kmeans failure is MADlib's fault - I'm in the process of fixing that.
For others, it could be combination of issues with HAWQ, HDFS and MADlib not
doing the right thing for HAWQ.
All modules with + have passed, with - have either failed or are too slow. The
X implies those modules have been removed since they're deprecated. I can keep
them but it would require some work to make them pass.
{panel}
Modules:
+ array_ops
- bayes (view leads to crash in hdfs)
- crf (pg_temp in case statement #4)
+ elastic_net
- linalg (too slow)
- pmml (too slow)
+ prob
+ sketch
X svd_mf (deleted)
- svm (too slow)
- tsa (too slow)
- conjugate_gradient
- lda ()
- stats (gp_segment_configuration #1)
+ svec_util
+ utilities
- assoc_rules (too slow)
X cart (deleted)
+ convex
- glm (node type issue #3)
+ linear_systems ()
- recursive_partitioning ()
- regress ()
+ sample
+ summary
- kmeans (closest_column still has exception for HAWQ)
<--------------------
- pca ()
- validation (pg_temp in aggregate function #5)
{panel}
Questions:
1. gp_segment_configuration has changed. What's the best way to compute the
number
of segments?
2. hdfs crashes on many of the queries. A particular case is the view creation
in bayes module.
3. psql:/tmp/madlib.eVW4q7/glm/test/gaussian.sql_in.tmp:92: ERROR:
plpy.SPIError: could not serialize unrecognized node type: 44867360
(plpython.c:4651)
CONTEXT: Traceback (most recent call last):
PL/Python function "glm", line 23, in <module>
return glm.glm(**globals())
PL/Python function "glm", line 90, in glm
PL/Python function "glm", line 223, in __glm_compute
PL/Python function "glm", line 52, in __compute_glm
PL/Python function "glm", line 164, in final
PL/Python function "glm"
4. plpy.SPIError: invalid reference to FROM-clause entry for table
"_madlib_tmp1_feature" (plpython.c:4651)
LINE 4: ... CASE WHEN pg_temp._m...
^
HINT: There is an entry for table "_madlib_tmp1_feature", but it cannot be
referenced from this part of the query.
QUERY: INSERT INTO pg_temp._madlib_tmp_rtbl(start_pos,doc_id,feature)
SELECT start_pos, doc_id, array_cat(fset.feature,
ARRAY[f_index,start_pos,
CASE WHEN
pg_temp._madlib_tmp1_feature.feature = fset.feature THEN 1
ELSE 0
END] )
FROM pg_temp._madlib_tmp1_feature, train_featureset fset
WHERE pg_temp._madlib_tmp1_feature.f_name = fset.f_name
AND fset.f_name <> 'E.';
5. psql:/tmp/madlib.kHS01Q/validation/test/cross_validation.sql_in.tmp:1425:
NOTICE: table "cv_linregr_result" does not exist, skipping
CONTEXT: SQL statement "drop table if exists cv_linregr_result"
PL/pgSQL function "check_cv0" line 5 at execute statement
psql:/tmp/madlib.kHS01Q/validation/test/cross_validation.sql_in.tmp:1425:
ERROR: plpy.SPIError: invalid reference to FROM-clause entry for table
"__madlib_temp_15692241_1454031660_13288785__" (plpython.c:4651)
LINE 4: avg((pg_temp.__madlib_temp_15692241_1454031660_1...
^
HINT: There is an entry for table
"__madlib_temp_15692241_1454031660_13288785__", but it cannot be referenced
from this part of the query.
QUERY:
CREATE TABLE pg_temp.__madlib_temp_68223508_1454031660_3362198__ AS
SELECT
avg((pg_temp.__madlib_temp_15692241_1454031660_13288785__.prediction -
("Y"))^2) as mean_squared_error
FROM
pg_temp.__madlib_temp_15692241_1454031660_13288785__,
__madlib_temp_77009712_1454031660_23171700__
WHERE
pg_temp.__madlib_temp_15692241_1454031660_13288785__.__madlib_temp_84152653_1454031660_15203995__
=
__madlib_temp_77009712_1454031660_23171700__.__madlib_temp_84152653_1454031660_15203995__
CONTEXT: Traceback (most recent call last):
PL/Python function "cross_validation_general", line 23, in <module>
return cross_validation.cross_validation_general(**globals())
PL/Python function "cross_validation_general", line 360, in
cross_validation_general
PL/Python function "cross_validation_general", line 265, in _one_step_cv
PL/Python function "cross_validation_general", line 121, in
__cv_funcall_general
PL/Python function "cross_validation_general"
SQL statement "SELECT cross_validation_general( 'cv_linregr_train',
ARRAY['%data%', '"X"', '"Y"', '%model%']::varchar[], '{varchar, varchar,
varchar, varchar}'::varchar[], NULL::varchar, NULL, 'cv_linregr_predict',
ARRAY['%model%', '%data%', '"X"', '%id%', '%prediction%']::varchar[],
'{varchar, varchar, varchar, varchar, varchar}'::varchar[], 'mse_error',
ARRAY['%prediction%', '%data%', '%id%', '"Y"', '%error%']::varchar[],
'{varchar, varchar, varchar, varchar, varchar}'::varchar[], '"Lin_housing_wi"',
NULL, False, 'cv_linregr_result', ARRAY['"X"', '"Y"']::varchar[], 10 )"
PL/pgSQL function "check_cv0" line 6 at perform
> Test MADlib on HAWQ 2.0 beta build
> ----------------------------------
>
> Key: MADLIB-960
> URL: https://issues.apache.org/jira/browse/MADLIB-960
> Project: Apache MADlib
> Issue Type: Task
> Components: All Modules
> Reporter: Frank McQuillan
> Assignee: Rahul Iyer
>
> Story
> As a MADlib developer, I want to ensure that MADlib 1.8 runs on HAWQ 2.0
> beta. This includes svec. Any fixes needed in MADlib 1.8 will be ported
> forward to the 1.9 build; TBD if there will be any.
> Acceptance
> 1) MADlib installs correctly in usual way for HAWQ 2.0 beta (install check
> passes)
> 2) Toy svec example works OK.
> 3) Regular functional tests pass.
> 4) Feed back any findings to Apache HAWQ team.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
