[GitHub] madlib pull request #286: Build: Remove symlinks during rpm uninstall
Github user orhankislal commented on a diff in the pull request: https://github.com/apache/madlib/pull/286#discussion_r200793992 --- Diff: pom.xml --- @@ -66,6 +66,7 @@ deploy/preflight.sh deploy/RPM/CMakeLists.txt deploy/rpm_post.sh + deploy/rpm_post_uninstall.sh --- End diff -- Since this is a new file, we should add the apache license instead of adding it to the exclude list. ---
[GitHub] madlib pull request #288: Jira:1239: Converts features from multiple columns...
Github user iyerr3 commented on a diff in the pull request: https://github.com/apache/madlib/pull/288#discussion_r200790286 --- Diff: src/ports/postgres/modules/cols_vec/cols2vec.py_in --- @@ -0,0 +1,104 @@ +""" +@file cols2vec.py_in + +@brief Utility to convert Columns to array + +""" + +import plpy +from utilities.control import MinWarning +from utilities.utilities import split_quoted_delimited_str +from utilities.utilities import _string_to_array +from utilities.utilities import _assert +from utilities.validate_args import columns_exist_in_table +from utilities.validate_args import is_var_valid +from utilities.validate_args import get_cols +from utilities.validate_args import quote_ident +from utilities.utilities import py_list_to_sql_string + + +m4_changequote(`') + + +def validate_cols2vec_args(source_table, output_table, + list_of_features, list_of_features_to_exclude, cols_to_output, **kwargs): +""" +Function to validate input parameters +""" +if list_of_features.strip() != '*': +if not (list_of_features and list_of_features.strip()): +plpy.error("Features to include is empty") +_assert( +columns_exist_in_table( +source_table, split_quoted_delimited_str(list_of_features)), +"Invalid columns to list of features {0}".format(list_of_features)) + +if cols_to_output and cols_to_output.strip() != '*': +_assert( +columns_exist_in_table( +source_table, _string_to_array(cols_to_output)), +"Invalid columns to output list {0}".format(cols_to_output)) + + +def cols2vec(schema_madlib, source_table, output_table, list_of_features, + list_of_features_to_exclude=None, cols_to_output=None, **kwargs): +""" +Args: +@param schema_madlib: Name of MADlib schema +@param model: Name of table containing the tree model +@param source_table:Name of table containing prediction data +@param output_table:Name of table to output the results +@param list_of_features:Comma-separated string of column names or +expressions to put into feature array. +Can also be a '*' implying all columns +are to be put into feature array. +@param list_of_features_to_exclude: Comma-separated string of column names +to exclude from the feature array +@param cols_to_output: Comma-separated string of column names +from the source table to keep in the output table, +in addition to the feature array. + +Returns: +None + +""" + +with MinWarning('warning'): +validate_cols2vec_args(source_table, output_table, list_of_features, + list_of_features_to_exclude, cols_to_output, **kwargs) + +all_cols = '' +feature_cols = '' +feature_list = '' +if list_of_features.strip() == '*': +all_cols = get_cols(source_table, schema_madlib) +all_col_set = set(list(all_cols)) +exclude_set = set(split_quoted_delimited_str( +list_of_features_to_exclude)) +feature_list = list(all_col_set - exclude_set) +else: +feature_list = split_quoted_delimited_str(list_of_features) + +feature_cols = py_list_to_sql_string( +list(feature_list), "text", False) +filtered_list_of_features = ",".join( --- End diff -- `filtered_list_of_features = ",".join(feature_list)` ---
[GitHub] madlib pull request #288: Jira:1239: Converts features from multiple columns...
Github user iyerr3 commented on a diff in the pull request: https://github.com/apache/madlib/pull/288#discussion_r200790240 --- Diff: src/ports/postgres/modules/cols_vec/cols2vec.py_in --- @@ -0,0 +1,104 @@ +""" +@file cols2vec.py_in + +@brief Utility to convert Columns to array + +""" + +import plpy +from utilities.control import MinWarning +from utilities.utilities import split_quoted_delimited_str +from utilities.utilities import _string_to_array +from utilities.utilities import _assert +from utilities.validate_args import columns_exist_in_table +from utilities.validate_args import is_var_valid +from utilities.validate_args import get_cols +from utilities.validate_args import quote_ident +from utilities.utilities import py_list_to_sql_string + + +m4_changequote(`') + + +def validate_cols2vec_args(source_table, output_table, + list_of_features, list_of_features_to_exclude, cols_to_output, **kwargs): +""" +Function to validate input parameters +""" +if list_of_features.strip() != '*': +if not (list_of_features and list_of_features.strip()): +plpy.error("Features to include is empty") +_assert( +columns_exist_in_table( +source_table, split_quoted_delimited_str(list_of_features)), +"Invalid columns to list of features {0}".format(list_of_features)) + +if cols_to_output and cols_to_output.strip() != '*': +_assert( +columns_exist_in_table( +source_table, _string_to_array(cols_to_output)), +"Invalid columns to output list {0}".format(cols_to_output)) + + +def cols2vec(schema_madlib, source_table, output_table, list_of_features, + list_of_features_to_exclude=None, cols_to_output=None, **kwargs): +""" +Args: +@param schema_madlib: Name of MADlib schema +@param model: Name of table containing the tree model +@param source_table:Name of table containing prediction data +@param output_table:Name of table to output the results +@param list_of_features:Comma-separated string of column names or +expressions to put into feature array. +Can also be a '*' implying all columns +are to be put into feature array. +@param list_of_features_to_exclude: Comma-separated string of column names +to exclude from the feature array +@param cols_to_output: Comma-separated string of column names +from the source table to keep in the output table, +in addition to the feature array. + +Returns: +None + +""" + +with MinWarning('warning'): +validate_cols2vec_args(source_table, output_table, list_of_features, + list_of_features_to_exclude, cols_to_output, **kwargs) + +all_cols = '' --- End diff -- We can delete lines 70, 71, 72 since we don't need those anymore. ---
[GitHub] madlib issue #288: Jira:1239: Converts features from multiple columns into a...
Github user asfgit commented on the issue: https://github.com/apache/madlib/pull/288 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/madlib-pr-build/536/ ---
[GitHub] madlib pull request #288: Jira:1239: Converts features from multiple columns...
Github user hpandeycodeit commented on a diff in the pull request: https://github.com/apache/madlib/pull/288#discussion_r200787816 --- Diff: src/ports/postgres/modules/cols_vec/cols2vec.py_in --- @@ -0,0 +1,110 @@ +""" +@file cols2vec.py_in + +@brief Utility to convert Columns to array + +""" + +import plpy +from utilities.control import MinWarning +from utilities.utilities import split_quoted_delimited_str +from utilities.utilities import _string_to_array +from utilities.validate_args import columns_exist_in_table +from utilities.validate_args import is_var_valid +from utilities.validate_args import get_cols +from utilities.validate_args import quote_ident +from utilities.utilities import py_list_to_sql_string + + +m4_changequote(`') + + +def validate_cols2vec_args(source_table, output_table, + list_of_features, list_of_features_to_exclude, cols_to_output, **kwargs): +""" +Function to validate input parameters +""" +if list_of_features.strip() != '*': --- End diff -- This is done. ---
[GitHub] madlib pull request #288: Jira:1239: Converts features from multiple columns...
Github user hpandeycodeit commented on a diff in the pull request: https://github.com/apache/madlib/pull/288#discussion_r200787876 --- Diff: src/config/Modules.yml --- @@ -50,3 +50,4 @@ modules: - name: validation depends: ['array_ops', 'regress'] - name: stemmer +- name: cols_vec --- End diff -- Discussed it with @fmcquillan99 and I will move it under Utilities module in next commit. ---
[GitHub] madlib issue #288: Jira:1239: Converts features from multiple columns into a...
Github user fmcquillan99 commented on the issue: https://github.com/apache/madlib/pull/288 update my comment above to remove the rows processed and skipped. ---
[GitHub] madlib issue #288: Jira:1239: Converts features from multiple columns into a...
Github user hpandeycodeit commented on the issue: https://github.com/apache/madlib/pull/288 @fmcquillan99 , What is total_rows_processed and total_rows_skipped ? Can you provide more details on these? ---
[GitHub] madlib issue #288: Jira:1239: Converts features from multiple columns into a...
Github user fmcquillan99 commented on the issue: https://github.com/apache/madlib/pull/288 Since we are writing out a summary table, may as well add more info in it. {code} A summary table named _summary is also created at the same time, which has the following columns: source_table TEXT. Source table name. feature_names TEXT[]. Array of names of features. total_rows_processedINTEGER. Total numbers of rows processed. total_rows_skippedINTEGER. Total numbers of rows skipped due to failures. {code} Is this do-able @hpandeycodeit ? ---