[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user asfgit closed the pull request at: https://github.com/apache/madlib/pull/318 ---
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user orhankislal commented on a diff in the pull request:
https://github.com/apache/madlib/pull/318#discussion_r217953051
--- Diff: src/madpack/create_changelist.py ---
@@ -0,0 +1,239 @@
+#!/usr/bin/python
+#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+
+# Create changelist for any two branches/tags
+
+# Prequisites:
+# The old version has to be installed in the "madlib_old_vers" schema
+# The new version has to be installed in the "madlib" (default) schema
+# Two branches/tags must exist locally (run 'git fetch' to ensure you have
the latest version)
+# The current branch does not matter
+
+# Usage (must be executed in the src/madpack directory):
+# python create_changelist.py
+# If you are using the master branch, plase make sure to edit the
branch/tag in the output file
+
+# Example (should be equivalent to changelist_1.13_1.14.yaml):
+# python create_changelist.py madlib rel/v1.13 rel/v1.14 chtest1.yaml
+
+import sys
+import os
+
+database = sys.argv[1]
+old_vers = sys.argv[2]
+new_vers = sys.argv[3]
+ch_filename = sys.argv[4]
+
+if os.path.exists(ch_filename):
+print "{0} already exists".format(ch_filename)
+raise SystemExit
+
+err1 = os.system("""psql {0} -l > /dev/null""".format(database))
+if err1 != 0:
+print "Database {0} does not exist".format(database)
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib_old_vers.version()" >
/dev/null
+ """.format(database))
+if err1 != 0:
+print "MADlib is not installed in the madlib_old_vers schema. Please
refer to the Prequisites."
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib.version()" > /dev/null
+ """.format(database))
+if err1 != 0:
+print "MADlib is not installed in the madlib schema. Please refer to
the Prequisites."
+raise SystemExit
+
+print "Creating changelist {0}".format(ch_filename)
+os.system("rm -f /tmp/madlib_tmp_nm.txt /tmp/madlib_tmp_udf.txt
/tmp/madlib_tmp_udt.txt")
+try:
+# Find the new modules using the git diff
+err1 = os.system("git diff {old_vers} {new_vers} --name-only
--diff-filter=A > /tmp/madlib_tmp_nm.txt".format(**locals()))
+if err1 != 0:
+print "Git diff failed. Please ensure that branches/tags are
fetched."
+raise SystemExit
+
+f = open("/tmp/madlib_tmp_cl.yaml", "w")
+f.write(
+"""#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+""")
+
+f.write(
+"""
+# Changelist for MADlib version {old_vers} to {new_vers}
+
+# This file contains all changes that were introduced in a new version of
+# MADlib. This changelist is used by the upgrade script to detect what
objects
+# should be upgraded (while retaining all other objects from the previous
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user kaknikhil commented on a diff in the pull request:
https://github.com/apache/madlib/pull/318#discussion_r217842978
--- Diff: src/madpack/create_changelist.py ---
@@ -0,0 +1,239 @@
+#!/usr/bin/python
+#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+
+# Create changelist for any two branches/tags
+
+# Prequisites:
+# The old version has to be installed in the "madlib_old_vers" schema
+# The new version has to be installed in the "madlib" (default) schema
+# Two branches/tags must exist locally (run 'git fetch' to ensure you have
the latest version)
+# The current branch does not matter
+
+# Usage (must be executed in the src/madpack directory):
+# python create_changelist.py
+# If you are using the master branch, plase make sure to edit the
branch/tag in the output file
+
+# Example (should be equivalent to changelist_1.13_1.14.yaml):
+# python create_changelist.py madlib rel/v1.13 rel/v1.14 chtest1.yaml
+
+import sys
+import os
+
+database = sys.argv[1]
+old_vers = sys.argv[2]
+new_vers = sys.argv[3]
+ch_filename = sys.argv[4]
+
+if os.path.exists(ch_filename):
+print "{0} already exists".format(ch_filename)
+raise SystemExit
+
+err1 = os.system("""psql {0} -l > /dev/null""".format(database))
+if err1 != 0:
+print "Database {0} does not exist".format(database)
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib_old_vers.version()" >
/dev/null
+ """.format(database))
+if err1 != 0:
+print "MADlib is not installed in the madlib_old_vers schema. Please
refer to the Prequisites."
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib.version()" > /dev/null
+ """.format(database))
+if err1 != 0:
+print "MADlib is not installed in the madlib schema. Please refer to
the Prequisites."
+raise SystemExit
+
+print "Creating changelist {0}".format(ch_filename)
+os.system("rm -f /tmp/madlib_tmp_nm.txt /tmp/madlib_tmp_udf.txt
/tmp/madlib_tmp_udt.txt")
+try:
+# Find the new modules using the git diff
+err1 = os.system("git diff {old_vers} {new_vers} --name-only
--diff-filter=A > /tmp/madlib_tmp_nm.txt".format(**locals()))
+if err1 != 0:
+print "Git diff failed. Please ensure that branches/tags are
fetched."
+raise SystemExit
+
+f = open("/tmp/madlib_tmp_cl.yaml", "w")
+f.write(
+"""#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+""")
+
+f.write(
+"""
+# Changelist for MADlib version {old_vers} to {new_vers}
+
+# This file contains all changes that were introduced in a new version of
+# MADlib. This changelist is used by the upgrade script to detect what
objects
+# should be upgraded (while retaining all other objects from the previous
version)
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user orhankislal commented on a diff in the pull request:
https://github.com/apache/madlib/pull/318#discussion_r217361584
--- Diff: src/madpack/create_changelist.py ---
@@ -0,0 +1,229 @@
+#!/usr/bin/python
+#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+
+# Create changelist for any two branches/tags
+
+# Prequisites:
+# The old version has to be installed in the "madlib_old_vers" schema
+# The new version has to be installed in the "madlib" (default) schema
+# Two branches/tags must exist locally (run 'git fetch' to ensure you have
the latest version)
+# The current branch does not matter
+
+# Usage (must be executed in the src/madpack directory):
+# python create_changelist.py
+
+# Example (should be equivalent to changelist_1.13_1.14.yaml):
+# python create_changelist.py madlib rel/v1.13 rel/v1.14 chtest1.yaml
+
+import sys
+import os
+
+database = sys.argv[1]
+old_vers = sys.argv[2]
+new_vers = sys.argv[3]
+ch_filename = sys.argv[4]
+
+if os.path.exists(ch_filename):
+print "{0} already exists".format(ch_filename)
+raise SystemExit
+
+err1 = os.system("""psql {0} -l > /dev/null""".format(database))
+if err1 != 0:
+print "Database {0} does not exist".format(old_vers)
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib_old_vers.version()" >
/dev/null
+ """.format(database))
+if err1 != 0:
+print "Schema {0} does not exist".format(old_vers)
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib.version()" > /dev/null
+ """.format(database))
+if err1 != 0:
+print "Schema {0} does not exist".format(new_vers)
+raise SystemExit
+
--- End diff --
That would be tricky for branches. `madlib.version()` gives a on a branch (and just a tag on tags). We can use `git
rev-parse --short HEAD` to get the commit tag but it seems complicating the
code for handholding the developer that uses it.
---
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user kaknikhil commented on a diff in the pull request:
https://github.com/apache/madlib/pull/318#discussion_r217152555
--- Diff: src/madpack/create_changelist.py ---
@@ -0,0 +1,229 @@
+#!/usr/bin/python
+#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+
+# Create changelist for any two branches/tags
+
+# Prequisites:
+# The old version has to be installed in the "madlib_old_vers" schema
+# The new version has to be installed in the "madlib" (default) schema
+# Two branches/tags must exist locally (run 'git fetch' to ensure you have
the latest version)
+# The current branch does not matter
+
+# Usage (must be executed in the src/madpack directory):
+# python create_changelist.py
+
+# Example (should be equivalent to changelist_1.13_1.14.yaml):
+# python create_changelist.py madlib rel/v1.13 rel/v1.14 chtest1.yaml
+
+import sys
+import os
+
+database = sys.argv[1]
+old_vers = sys.argv[2]
+new_vers = sys.argv[3]
+ch_filename = sys.argv[4]
+
+if os.path.exists(ch_filename):
+print "{0} already exists".format(ch_filename)
+raise SystemExit
+
+err1 = os.system("""psql {0} -l > /dev/null""".format(database))
+if err1 != 0:
+print "Database {0} does not exist".format(old_vers)
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib_old_vers.version()" >
/dev/null
+ """.format(database))
+if err1 != 0:
+print "Schema {0} does not exist".format(old_vers)
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib.version()" > /dev/null
+ """.format(database))
+if err1 != 0:
+print "Schema {0} does not exist".format(new_vers)
+raise SystemExit
+
--- End diff --
Can we add another error check to ensure that the
`madlib_old_vers.schema()` is the same as `old_vers branch/tag`? This does
assume that the branch/tag will always be a release tag.
---
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user kaknikhil commented on a diff in the pull request:
https://github.com/apache/madlib/pull/318#discussion_r217143101
--- Diff: src/madpack/create_changelist.py ---
@@ -0,0 +1,229 @@
+#!/usr/bin/python
+#
--
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
--
+
+# Create changelist for any two branches/tags
+
+# Prequisites:
+# The old version has to be installed in the "madlib_old_vers" schema
+# The new version has to be installed in the "madlib" (default) schema
+# Two branches/tags must exist locally (run 'git fetch' to ensure you have
the latest version)
+# The current branch does not matter
+
+# Usage (must be executed in the src/madpack directory):
+# python create_changelist.py
+
+# Example (should be equivalent to changelist_1.13_1.14.yaml):
+# python create_changelist.py madlib rel/v1.13 rel/v1.14 chtest1.yaml
+
+import sys
+import os
+
+database = sys.argv[1]
+old_vers = sys.argv[2]
+new_vers = sys.argv[3]
+ch_filename = sys.argv[4]
+
+if os.path.exists(ch_filename):
+print "{0} already exists".format(ch_filename)
+raise SystemExit
+
+err1 = os.system("""psql {0} -l > /dev/null""".format(database))
+if err1 != 0:
+print "Database {0} does not exist".format(old_vers)
+raise SystemExit
+
+err1 = os.system("""psql {0} -c "select madlib_old_vers.version()" >
/dev/null
+ """.format(database))
+if err1 != 0:
+print "Schema {0} does not exist".format(old_vers)
--- End diff --
This error message prints the branch name instead of the schema name. Same
with the previous and next error checks. Maybe we should change the var name to
be more descriptive
---
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user kaknikhil commented on a diff in the pull request: https://github.com/apache/madlib/pull/318#discussion_r216443402 --- Diff: src/madpack/create_changelist.py --- @@ -0,0 +1,132 @@ +#!/usr/bin/python +# -- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# -- + +# Create changelist for any two branches + +# Prequisites: +# The old version has to be installed in the "madlib_old_vers" schema +# The new version has to be installed in the "madlib" (default) schema +# Two branches must exist locally (run 'git fetch' to ensure you have the latest version) --- End diff -- Can we expand on this comment ? Also mention that the old/new version branches should be the release tags and not branches. ---
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
Github user kaknikhil commented on a diff in the pull request:
https://github.com/apache/madlib/pull/318#discussion_r216438294
--- Diff: src/madpack/diff_udf.sql ---
@@ -142,9 +142,12 @@ DROP TABLE IF EXISTS functions_madlib_new_version;
SELECT get_functions('madlib_old_vers');
SELECT
+type,
--'\t-' || name || ':' || '\n\t\t-rettype: ' || retype ||
'\n\t\t-argument: ' || argtypes
-'- ' || name || ':' || '\nrettype: ' || retype || '\n
argument: ' || argtypes AS "Dropped UDFs"
-, type
+'- ' || name || ':' AS "Dropped UDF part1",
--- End diff --
I think it would be helpful to add comments in the commit description
explaining the reasoning behind modifying the `diff_udf.sql` and
`diff_udt.sql`. Just looking at these changes, it's hard to know why for ex
`rettype` was removed and why we need `part1` at the end
---
[GitHub] madlib pull request #318: Madpack: Add a script for automating changelist cr...
GitHub user orhankislal opened a pull request: https://github.com/apache/madlib/pull/318 Madpack: Add a script for automating changelist creation You can merge this pull request into a Git repository by running: $ git pull https://github.com/madlib/madlib madpack/auto-changelist Alternatively you can review and apply these changes as the patch at: https://github.com/apache/madlib/pull/318.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #318 commit 28528b21ecb9b53d03de7683ff7e8db2bb409675 Author: Orhan Kislal Date: 2018-09-04T12:50:50Z Madpack: Add a script for automating changelist creation ---
