[GitHub] madlib issue #318: Madpack: Add a script for automating changelist creation
Github user kaknikhil commented on the issue: https://github.com/apache/madlib/pull/318 @orhankislal The new changes look good. Apart from the 2 minor comments LGTM +1 ---
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user kaknikhil commented on a diff in the pull request:
https://github.com/apache/madlib/pull/318#discussion_r217842739
--- Diff: src/madpack/create_changelist.py ---
@@ -0,0 +1,239 @@
+#!/usr/bin/python
+#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+
+# Create changelist for any two branches/tags
+
+# Prequisites:
+# The old version has to be installed in the "madlib_old_vers" schema
+# The new version has to be installed in the "madlib" (default) schema
+# Two branches/tags must exist locally (run 'git fetch' to ensure you have
the latest version)
+# The current branch does not matter
+
+# Usage (must be executed in the src/madpack directory):
+# python create_changelist.py
+# If you are using the master branch, plase make sure to edit the
branch/tag in the output file
+
+# Example (should be equivalent to changelist_1.13_1.14.yaml):
+# python create_changelist.py madlib rel/v1.13 rel/v1.14 chtest1.yaml
+
+import sys
+import os
+
+database = sys.argv[1]
+old_vers = sys.argv[2]
+new_vers = sys.argv[3]
+ch_filename = sys.argv[4]
+
+if os.path.exists(ch_filename):
+print "{0} already exists".format(ch_filename)
+raise SystemExit
+
+err1 = os.system("""psql {0} -l > /dev/null""".format(database))
+if err1 != 0:
+print "Database {0} does not exist".format(database)
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib_old_vers.version()" >
/dev/null
+ """.format(database))
+if err1 != 0:
+print "MADlib is not installed in the madlib_old_vers schema. Please
refer to the Prequisites."
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib.version()" > /dev/null
+ """.format(database))
+if err1 != 0:
+print "MADlib is not installed in the madlib schema. Please refer to
the Prequisites."
+raise SystemExit
+
+print "Creating changelist {0}".format(ch_filename)
+os.system("rm -f /tmp/madlib_tmp_nm.txt /tmp/madlib_tmp_udf.txt
/tmp/madlib_tmp_udt.txt")
+try:
+# Find the new modules using the git diff
+err1 = os.system("git diff {old_vers} {new_vers} --name-only
--diff-filter=A > /tmp/madlib_tmp_nm.txt".format(**locals()))
+if err1 != 0:
+print "Git diff failed. Please ensure that branches/tags are
fetched."
+raise SystemExit
+
+f = open("/tmp/madlib_tmp_cl.yaml", "w")
+f.write(
+"""#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+""")
+
+f.write(
+"""
+# Changelist for MADlib version {old_vers} to {new_vers}
+
+# This file contains all changes that were introduced in a new version of
+# MADlib. This changelist is used by the upgrade script to detect what
objects
+# should be upgraded (while retaining all other objects from the previous
version)
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user kaknikhil commented on a diff in the pull request:
https://github.com/apache/madlib/pull/318#discussion_r217842978
--- Diff: src/madpack/create_changelist.py ---
@@ -0,0 +1,239 @@
+#!/usr/bin/python
+#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+
+# Create changelist for any two branches/tags
+
+# Prequisites:
+# The old version has to be installed in the "madlib_old_vers" schema
+# The new version has to be installed in the "madlib" (default) schema
+# Two branches/tags must exist locally (run 'git fetch' to ensure you have
the latest version)
+# The current branch does not matter
+
+# Usage (must be executed in the src/madpack directory):
+# python create_changelist.py
+# If you are using the master branch, plase make sure to edit the
branch/tag in the output file
+
+# Example (should be equivalent to changelist_1.13_1.14.yaml):
+# python create_changelist.py madlib rel/v1.13 rel/v1.14 chtest1.yaml
+
+import sys
+import os
+
+database = sys.argv[1]
+old_vers = sys.argv[2]
+new_vers = sys.argv[3]
+ch_filename = sys.argv[4]
+
+if os.path.exists(ch_filename):
+print "{0} already exists".format(ch_filename)
+raise SystemExit
+
+err1 = os.system("""psql {0} -l > /dev/null""".format(database))
+if err1 != 0:
+print "Database {0} does not exist".format(database)
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib_old_vers.version()" >
/dev/null
+ """.format(database))
+if err1 != 0:
+print "MADlib is not installed in the madlib_old_vers schema. Please
refer to the Prequisites."
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib.version()" > /dev/null
+ """.format(database))
+if err1 != 0:
+print "MADlib is not installed in the madlib schema. Please refer to
the Prequisites."
+raise SystemExit
+
+print "Creating changelist {0}".format(ch_filename)
+os.system("rm -f /tmp/madlib_tmp_nm.txt /tmp/madlib_tmp_udf.txt
/tmp/madlib_tmp_udt.txt")
+try:
+# Find the new modules using the git diff
+err1 = os.system("git diff {old_vers} {new_vers} --name-only
--diff-filter=A > /tmp/madlib_tmp_nm.txt".format(**locals()))
+if err1 != 0:
+print "Git diff failed. Please ensure that branches/tags are
fetched."
+raise SystemExit
+
+f = open("/tmp/madlib_tmp_cl.yaml", "w")
+f.write(
+"""#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+""")
+
+f.write(
+"""
+# Changelist for MADlib version {old_vers} to {new_vers}
+
+# This file contains all changes that were introduced in a new version of
+# MADlib. This changelist is used by the upgrade script to detect what
objects
+# should be upgraded (while retaining all other objects from the previous
version)
[GitHub] madlib issue #318: Madpack: Add a script for automating changelist creation
Github user asfgit commented on the issue: https://github.com/apache/madlib/pull/318 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/madlib-pr-build/681/ ---
[GitHub] madlib issue #318: Madpack: Add a script for automating changelist creation
Github user orhankislal commented on the issue: https://github.com/apache/madlib/pull/318 @kaknikhil I checked the 1.11 -> 1.12 scenario. The tool is missing the`tree_train` and `forest_train` entries. The change seems to be the removal of `surrogate_params` and the addition of `null_handling_params`. Since both of them are `text` type, it does not get picked up by the `diff_udf.sql` script. ---
