This is an automated notification sent by LCG Savannah.
It relates to:
task #13958, project CDS Invenio
==============================================================================
LATEST MODIFICATIONS of task #13958:
==============================================================================
Follow-up Comment #10, task #13958 (project cdsware):
Again, the point is not the run time. It is the _fact_ that every developer
has to do this every single time they want to test a change. That's probably
a few hundred to a few thousand times per day. And if anyone forgets, they
are guaranteed to waste time because they have to at least understand why the
results are not as expected and run the installation + whatever they wanted to
do again.
I'm sure this is streamlined in your Emacs, but C-c e is undefined in my
version, I'll bet your last run command isn't always this build command, and
not everyone uses Emacs.
Also, please don't put words in my mouth. I never said this was urgent.
==============================================================================
OVERVIEW of task #13958:
==============================================================================
URL:
<http://savannah.cern.ch/task/?13958>
Summary: Imports should look in current tree before
site-packages / dist-packages
Project: CDS Invenio
Submitted by: vengmark
Submitted on: 2010-02-16 15:00
Should Start On: 2010-02-16 00:00
Should be Finished on: 2010-02-16 00:00
Category: None
Priority: 5 - Normal
Status: None
Privacy: Public
Percent Complete: 0%
Assigned to: None
Open/Closed: Open
Discussion Lock: Any
Effort: 0.00
_______________________________________________________
Symptom: Whenever you run code in the cds-invenio repo, it will import from
the Python site-packages / dist-packages.
Problem: This makes the development cycle longer since any code changed by
git pull / merge would have to be copied to the Invenio source directory
before running.
Workaround: using "sudo -u <www-data/apache> make install" on the right
level.
Fix 1: Use relative imports across the board.
Fix 2: Use the same directory structure in site-packages / dist-packages as
in cds-invenio/modules, so that absolute imports look in the current
directory. Someone more skilled with imports would have to check whether this
even makes sense.
_______________________________________________________
Follow-up Comments:
-------------------------------------------------------
Date: 2010-02-18 13:08 By: Victor Engmark <vengmark>
Again, the point is not the run time. It is the _fact_ that every developer
has to do this every single time they want to test a change. That's probably
a few hundred to a few thousand times per day. And if anyone forgets, they
are guaranteed to waste time because they have to at least understand why the
results are not as expected and run the installation + whatever they wanted to
do again.
I'm sure this is streamlined in your Emacs, but C-c e is undefined in my
version, I'll bet your last run command isn't always this build command, and
not everyone uses Emacs.
Also, please don't put words in my mouth. I never said this was urgent.
-------------------------------------------------------
Date: 2010-02-18 12:07 By: Tibor Simko <simko>
In reply to comment #8:
Easier and faster? It takes 1 keypress and 0.6 sec for me to install
the complete WebSearch module. (Or 0.3 sec to install websearch/lib
stuff only.) Do we need to economize on these timings?
Not difficult to implement? Please post a proposal, but as I said I
think we have more urgent tasks on the agenda than this.
Besides, in your comment #5, you mention a use case of unit tests.
For this, you don't need to reload your WSGI application, you just
install your files and run your test file. Here, `C-c e' would run an
equivalent of:
cd modules/websearch/lib && \
make -s && \
sudo -u www-data make -s install && \
python ./search_engine_tests.py --yes-i-know
which takes 1.38 sec for me, including the time to run 31 test cases.
If we strip the installation bit away, and only run the tests, then
the timing drops to 1.18 sec. So the installation part costs 0.2 sec
only. Hardly noticeable.
Let me mention one efficiency tip. When I'm working on something that
has a large test suite, say there are 20 time-consuming tests like:
TEST_SUITE = make_test_suite(TestFoo,
TestBar,
[...],
TestBaz)
and I know that I'm only working on the fuuux stuff, then I either add
temporarily the line
TEST_SUITE = make_test_suite(TestFuux,)
in order to run only the tests I want, or I simply run the fuux tests
inside REPL via an equivalent of:
from invenio.testutils import make_test_suite, run_test_suite
from invenio.webfoo_engine_tests import TestFuux
run_test_suite(make_test_suite(TestFuux,))
Moreover, as I said in my comment #3, for local Python testing you can
ofter stay inside Emacs ipython REPL, so you re-edit your function,
you re-evaluate it via `C-M-x', and you re-run your temporary test
line via `C-p RET', without having to install anything to /opt.
-------------------------------------------------------
Date: 2010-02-18 10:45 By: Victor Engmark <vengmark>
In reply to comment #4 and comment #6:
The point is not that it is easy and fast, once you've got Invenio installed
and you know the shortcuts. My points are simply these:
* It would be _easier_ and _faster_ to be able to work without having to
install and update the installation for each code change.
* The change necessary to be able to do this would not be difficult (a change
in the import syntax and/or moving files).
It's not a big deal, and it could make development just a little bit smoother
for everybody.
-------------------------------------------------------
Date: 2010-02-18 10:26 By: Victor Engmark <vengmark>
In reply to comment #3: This should be easy with the VM that we're working on
here. The only remaining issue is with the fact that more tests fail than on
the production machine, but I think Tibor is onto that.
-------------------------------------------------------
Date: 2010-02-18 10:25 By: Tibor Simko <simko>
In reply to comment #5: on my modest laptop, for the search_engine.py
use case I mentioned, installing takes 0.3 second exactly. If I have
to install from Invenio root, then it takes 15 seconds.
Note that you rarely need to reinstall every module from root: you
usually work on WebFoo, and so you have to install stuff from WebFoo
only. Should take well under a second.
If it takes a minute for you, then maybe you are installing all
modules unnecessarily (try cd webfoo first), or maybe you have a slow
scrolling terminal (try `make -s' or redirect output), or maybe the
issue is connected to your virtual machine parameters (why not to use
native OS anyway).
-------------------------------------------------------
Date: 2010-02-18 10:09 By: Victor Engmark <vengmark>
In reply to comment #2: It takes about 1 minute to run "sudo -u apache make
install && sudo -u apache /opt/cds-invenio/bin/inveniocfg --update-all &&
sudo /sbin/service httpd restart" on inspirenightmaster. That's not a lot,
but it adds up when you have to run that after each change to the code to be
able to run e.g. unit tests. I'm not saying that's unacceptable, but it would
be nice to be able to run tests immediately after changes.
-------------------------------------------------------
Date: 2010-02-17 21:56 By: Tibor Simko <simko>
I don't think that installing files onto /opt/cds-invenio would make
the development cycle that much longer. For example, when I edit
search_engine.py, I just press `C-c e' to install my changes to be
able to see them in the browser. Simple and quick. (Alternatively,
if I test Pythonic bits and pieces outside of the web context, I just
press `C-M-x' on the function in question and perform tests in an
ipython buffer.)
Behind the scenes, `C-c e' would exec my last entered shell command,
that would most of the time perform the following operations in the
given source directory (i.e. websearch/lib in my example):
make && sudo -u www-data make install && \
sudo -u www-data /opt/cds-invenio/bin/inveniocfg --update-all && \
sudo -u www-data touch /opt/cds-invenio/var/www-wsgi/invenio.wsgi
This is all that one needs to do to see the consequence of edits in
the browser. I have sent this tip to you and/or to the list in the
past.
So, when I want to see the consequence of my edits, I press literally
`C-c e' to install them, `Alt-TAB' to switch to the browser window,
and `r' to reload the page. Three hotkey presses. Now lather, rinse,
and repeat, until satisfaction.
Would such a model work for you?
I guess everyone has created some kind of shortcuts of this kind to
perform often-used operations from their editors of choice. We can
start a wiki page listing various tips for inspiration or something.
I have posted quite a few Emacs tips on the EmacsTips wiki page
already, but I have not posted some concrete Invenio tips anywhere. I
can do that if it helps.
That said, we can think of altering import policies and stuff, but
dunno how much priority I would allocate to that, considering the
benefit/cost relation and the importance of other `regular' tasks in
moving forward with both Invenio and INSPIRE releases.
P.S. Concerning the sub-topic of easy installing of Invenio sites,
I'll add some comments about that onto #13992, where it seems more
appropriate.
-------------------------------------------------------
Date: 2010-02-17 20:05 By: Joe Blaylock <jblayloc>
Of course I use an alias like Travis's too. Lately I've been getting
frustrated working across both the Invenio and Inspire trees, because Inspire
stomps on parts of Invenio, and the test installations have different document
sets. So testing something for both means having to do a lot of work to get
your installation into the required state.
I'd love to just have two installations, and be able to pivot between them
conveniently. This is no trouble on disk, but I haven't the faintest notion
how to make MySQL behave properly in this regime.
-------------------------------------------------------
Date: 2010-02-17 19:42 By: Travis Brooks <tbrooks>
For me, I have an alias like:
reinstall='sudo make install;sudo inveniocfg --update-all; sudo /sbin/service
httpd restart'
and this is part of my development process (i.e. edit, reinstall, test,
debug, etc)
If you are using ipython, of course, you can import directly from the current
directory, but this only gets one file. better to just use something like
reinstall and just test/debug modules in their eventual installed location to
avoid a nightmare of "oh, I've been testing against the wrong file."
YMMV of course...
-------------------------------------------------------
Date: 2010-02-17 13:42 By: Victor Engmark <vengmark>
Checking Stack Overflow
<http://stackoverflow.com/questions/2280761/python-imports-with-different-directory-structures>
for opinions on the subject.
_______________________________________________________
Carbon-Copy List:
CC Address | Comment
------------------------------------+-----------------------------
1576 | -COM-
5878 | -COM-
3364 | -COM-
3964 | -SUB-
==============================================================================
This item URL is:
<http://savannah.cern.ch/task/?13958>
_______________________________________________
Message sent via/by LCG Savannah
http://savannah.cern.ch/
