#14480: switch sage to the new directory layout
-----------------------------------------+-------------------------------
Reporter: ohanar | Owner: tbd
Type: task | Status: needs_info
Priority: major | Milestone: sage-6.0
Component: distribution | Resolution:
Keywords: | Merged in:
Authors: R. Andrew Ohana | Reviewers:
Report Upstream: N/A | Work issues:
Branch: u/ohanar/build_system | Dependencies: #13015, #14781
Stopgaps: |
-----------------------------------------+-------------------------------
Comment (by ohanar):
Replying to [comment:12 tkluck]:
> Maybe it makes sense for at least one more person to go through this
diff.
Definitely. And hopefully someone who is already familiar with Sage's
build system.
> why do we clean more directories than before?
In order to make sure distclean still works as expected. Previously
cleaning up everything wasn't important for this since distclean would
just delete the whole devel directory.
> What happened to this parallel_make.cfg file?
There is no other reference to it in the source code.
> Do I understand correctly that, and is it okay that the micro_release
code now
> only strips the binaries and does not remove build directories?
It seems like I didn't revert some cleanup I made to the Makefile in
response to #14721.
> {{{
> diff --git a/build/gen_html b/build/gen_html
> deleted file mode 100755
> index 6dbad06..0000000
> --- a/build/gen_html
> +++ /dev/null
> }}}
> What did this script do? Why don't we need it anymore?
it is used to generate the webpages
http://sagemath.org/packages/{standard,optional,experimental}, however the
actual version of the script that is used in generating these patches
disagrees from the version that is included in Sage. Since I was
overhauling the build system (and this was in the directory related to the
build system), I thought it wise to remove it to avoid any confusion.
> Should we make sure that this does not break any optional/experimental
packages?
> Personally, I'm fine with any package that installs anything into this
data directory,
> which I'm guessing has been deprecated for some time now. (how long?)
>
> This reminds me: will we, at some point, test the installation of the
> optional packages into the git framework? And, since we're already
distributing
> them officially, couldn't we just as well put their metadata in the
repository, too?
> (We don't need to do that right away).
We should at least make sure that all optional packages work. Basically
all changes that might be needed should be fixable within the mercurial
distribution. Since we will have to through the packages anyway, it made
perfect sense to me to cleanup the whole SAGE_DATA thing. I've opened
#14962 for this task.
> This change has been made in quite some places. Why wasn't GREP_OPTIONS
being set
> a problem before?
It was, however I didn't bother fixing it on the mercurial side because
either you have to unset it in sage-env or you have to do it in a bunch of
packages (although it just came to me that you could unset it in sage-
spkg). Unsetting in sage-env is less than ideal because it means that
commands like sage -grep don't respect GREP_OPTIONS, and changing a simple
thing like this across a bunch of packages is incredibly painful in the
current workflow.
> It feels wrong to me to prefer the src/bin version of the sage script to
> the local/bin version. Why is this?
I've gone back and forth on this a few times. The reason why I ended up
with this was because the latest version of the sage script will always be
at src/bin, where as the version at local/bin will only be updated if you
have rerun make.
> {{{
> diff --git a/src/bin/sage b/src/bin/sage
> index 5c886e3..3762d5d 100755
> --- a/src/bin/sage
> +++ b/src/bin/sage
> <SNIP>
> @@ -864,12 +825,8 @@ if [ "$1" = '-pkg_nc' -o "$1" = "--pkg_nc" ]; then
> fi
>
> if [ "$1" = '-sdist' -o "$1" = "--sdist" ]; then
> - if [ $# -ne 2 ]; then
> - echo >&2 "** MISSING VERSION NUMBER! **"
> - exit 2
> - fi
> - maybe_sage_location
> - exec sage-sdist $2 "$SAGE_ROOT"
> + shift
> + exec sage-sdist "$@"
> }}}
> Why doesn't this need sage location anymore?
This was a merging error, sage-location should still be here.
> Why don't we call sage-location anymore? and sage-starts?
I moved the sage-location call to sage-real-upgrade, and I think that
sage-starts should probably be moved to sage-real-upgrade as well (which I
think I meant to do, but it seems like I forgot).
> We need to set this to the trac server. Also, what branch does this
clone? It
> should probably be the current one, because those are the binaries it
copies.
> Is it?
Without an argument git clone will clone HEAD.
> {{{
> diff --git a/src/bin/sage-download-file b/src/bin/sage-download-file
> index cf7dce0..a3428df 100755
> --- a/src/bin/sage-download-file
> +++ b/src/bin/sage-download-file
> @@ -33,7 +33,7 @@ if [ -z "$URL_GRABBER" ]; then
> if [ -n "$SAGE_LOCAL" ] && [ -x "$SAGE_LOCAL/bin/python" ]; then
> URL_GRABBER="$SAGE_LOCAL/bin/python"
> elif command -v curl &>/dev/null; then
> - URL_GRABBER="curl"
> + URL_GRABBER="curl --fail"
> elif command -v wget &>/dev/null; then
> URL_GRABBER="wget -nv -O-"
> # Pick Python last because we don't know which version it is,
> }}}
> This looks fine, but why wasn't it an issue before?
It was, but it is more of an issue now because sage-spkg now tries a few
urls for each package, which requires sage-download-file to bail if the
server returns an error.
> {{{
> diff --git a/src/bin/sage-sync-build.py b/src/bin/sage-sync-build.py
> index 8e45386..b5d476a 100755
> --- a/src/bin/sage-sync-build.py
> +++ b/src/bin/sage-sync-build.py
> <SNIP>
> }}}
> Why does sage-sync-build need so many changes? Naively, it would need
only a few
> changes to some directory names.
Currently we are symlinking stuff into SAGE_LOCAL from the devel directory
(this is to make cloning fast), however this is no longer happening in the
git repository. This caused some silly breaking with the sage-sync-build,
which was in need of some heavy revision anyway. There is now another
ticket out there (don't remember which one) dedicated to fixing + updating
this script.
> {{{
> diff --git a/src/bin/sage-upgrade b/src/bin/sage-upgrade
> index 46c1935..55432d1 100755
> --- a/src/bin/sage-upgrade
> +++ b/src/bin/sage-upgrade
> -./install
> +if [ "$#" -gt 0 ]; then
> + BRANCH="$1"
> + shift
> +else
> + BRANCH=$(git symbolic-ref HEAD) ||
> + BRANCH=$(git describe --exact-match --tags HEAD | \
> + sed -e 's;^;refs/tags/;')
> + if [ -n "$BRANCH" ]; then
> + case $BRANCH in
> + refs/heads/master) BRANCH=master;;
> + refs/heads/release) BRANCH=release;;
> + refs/heads/beta) BRANCH=beta;;
> + refs/tags/*) BRANCH=release;;
> + esac
> + fi
> +fi
> }}}
> Maybe do a `git fetch` first?
Good idea.
> Also, should we add pushing to these branches
> in the repo to a script somewhere? Maybe to sage-sdist?
Pushing should only be done by the release manager. I think the best place
for such stuff would be as part of the development scripts, although I
think adding the release manager portion to these scripts should be lower
priority than the general developer portion of these scripts.
> {{{
> diff --git a/src/c_lib/SConstruct b/src/c_lib/SConstruct
> index fd4719a..1813de8 100644
> --- a/src/c_lib/SConstruct
> +++ b/src/c_lib/SConstruct
> -#Here we only copy the files over if we are on Cygwin. Otherwise, the
> -#library will be handled by the symlinks created in
> -#$SAGE_ROOT/devel/sage/spkg-install
> -if os.environ['UNAME'] == 'CYGWIN':
> - env.Alias("install", "$SAGE_LOCAL/lib")
> -else:
> - env.Alias("install", [lib])
> -
> +env.Alias("install", "$SAGE_LOCAL")
> }}}
> Will this still work on Cygwin?
I think so, although we really should have someone on Cygwin test. As I
mentioned, the symlinks that the comment refers to are no more.
> {{{
> diff --git a/src/sage/tests/cmdline.py b/src/sage/tests/cmdline.py
> index 342c3ff..7e45f0f 100644
> --- a/src/sage/tests/cmdline.py
> +++ b/src/sage/tests/cmdline.py
> @@ -208,15 +198,9 @@ def test_executable(args, input="", timeout=100.0,
**kwds):
> Test ``sage --info [packages]``, unless this is a binary (bdist)
> distribution which doesn't ship spkgs::
>
> - sage: if os.path.isfile(os.path.join(SAGE_ROOT, 'spkg',
'standard', '.from_bdist')):
> - ... out = "Found package sqlalchemy in
spkg/standard/sqlalchemy-...spkg\n= SQLAlchemy =\n...\nSQLAlchemy is the
Python SQL toolkit..."
> - ... err = ''
> - ... ret = 0
> - ... else:
> - ... (out, err, ret) = test_executable(["sage", "--info",
"sqlalchemy"])
> - ...
> + sage: out, err, ret = test_executable(["sage", "--info",
"sqlalchemy"])
> sage: print out
> - Found package sqlalchemy in spkg/standard/sqlalchemy-...spkg
> + Using local scripts to install sqlalchemy-...
> = SQLAlchemy =
> ...
> SQLAlchemy is the Python SQL toolkit...
> }}}
> Do I understand correctly that we install sqlalchemy when running
doctests? What is
> happening here?
This can be changed in sage-spkg. That message just prints when it
realizes it has the package metadata locally.
> {{{
> diff --git a/src/spkg-delauto b/src/spkg-delauto
> deleted file mode 100755
> index 61d071d..0000000
> --- a/src/spkg-delauto
> +++ /dev/null
> }}}
> Why don't we need this anymore?
no other reference in the source code (except for MANIFEST.in, which
should be updated)
--
Ticket URL: <http://trac.sagemath.org/ticket/14480#comment:13>
Sage <http://www.sagemath.org>
Sage: Creating a Viable Open Source Alternative to Magma, Maple, Mathematica,
and MATLAB
--
You received this message because you are subscribed to the Google Groups
"sage-trac" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sage-trac.
For more options, visit https://groups.google.com/groups/opt_out.
