Re: [Python-Dev] setUpClass and setUpModule in unittest

[Robert Collins]

On Sat, 2010-02-13 at 01:04 +, Michael Foord wrote:


> > However from this example I *cannot* guess whether those resources are
> > set up and torn down per test or per test class.
> This particular example is the equivalent of setUpClass - so by 
> declaring the resource as a class attribute it will created before the 
> first test for the class is run and disposed of after the last test for 
> the class.
> 
> You could *also* create a single resource and share it between several 
> test classes, or even across classes in several modules, and have it 
> created and disposed of at the right point. I've copied Rob Collins in 
> on this email in case I've misunderstood.

Yes, precisely.

> > Also the notation
> >
> >resources = [('workdir', MyTempDir())]
> >
> > looks pretty ugly -- if 'workdir' ends up being an instance attribute,
> > why not make it a dict instead of a list of tuples? Or even better, a
> > could each resource become a class variable?

I have two key 'todos' planned in the notation for testresources:
 - I want to make a decorator to let individual tests (rather
than /just/ class scope as currently happens) get resources easily.

Something like

@resource(workdir=MyTempDir())
def test_foo(self):
pass


Secondly, I want to make the class level 'resources' list into a dict.
It was a list initially, for misguided reasons that no longer apply, and
it would be much nicer and clearer as a dict.


> >
> I guess we could introspect the class for every attribute that is a 
> resource, but I prefer some way of explicitly declaring which resources 
> a TestCase is using.

I could see doing the below as an alternative:
@resource(workdir=MyTempDir())
class TestFoo(TestCase):
...

I'm not personally very keen on inspecting everything in self.__dict__,
I suspect it would tickle bugs in other unittest extensions. However I'm
not really /against/ it - I don't think it will result in bad test
behaviour or isolation issues. So if users would like it, lets do it.

-Rob


signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Robert Collins]

On Fri, 2010-02-12 at 12:27 -0800, Guido van Rossum wrote:
> On Fri, Feb 12, 2010 at 12:20 PM,   wrote:
> > The idea is that you're declaring what the tests need in order to work.
> > You're not explicitly defining the order in which things are set up and torn
> > down.  That is left up to another part of the library to determine.
> >
> > One such other part, OptimisingTestSuite, will look at *all* of your tests
> > and find an order which involves the least redundant effort.
> 
> So is there a way to associate a "cost" with a resource? I may have
> one resource which is simply a /tmp subdirectory (very cheap) and
> another that requires starting a database service (very expensive).

From the pydoc:
  :ivar setUpCost: The relative cost to construct a resource of this type.
   One good approach is to set this to the number of seconds it normally
   takes to set up the resource.
  :ivar tearDownCost: The relative cost to tear down a resource of this
   type. One good approach is to set this to the number of seconds it
   normally takes to tear down the resource.
  
> > You might have something else that breaks up the test run across multiple
> > processes and uses the resource declarations to run all tests requiring one
> > thing in one process and all tests requiring another thing somewhere else.
> 
> I admire the approach, though I am skeptical. We have a thing to split
> up tests at Google which looks at past running times for tests to make
> an informed opinion. Have you thought of that?

I think thats a great way to do it; in fact doing the same thing to
assign setup and teardown costs would be lovely; I should write some
glue to do that automatically for people.

-Rob


signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Chris Withers]

R. David Murray wrote:

On Thu, 11 Feb 2010 12:41:37 +, Michael Foord  
wrote:

On 11/02/2010 12:30, Nick Coghlan wrote:

The test framework might promise to do the following for each test:

   with get_module_cm(test_instance): # However identified
 with get_class_cm(test_instance): # However identified
   with test_instance: # **
 test_instance.test_method()

Well that is *effectively* how they would work (the semantics) but I
don't see how that would fit with the design of unittest to make them
work *specifically* like that - especially not if we are to remain
compatible with existing unittest extensions.

If you can come up with a concrete proposal of how to do this then I'm
happy to listen. I'm not saying it is impossible, but it isn't
immediately obvious. I don't see any advantage of just using context
managers for the sake of it and definitely not at the cost of backwards
incompatibility.


I suspect that Nick is saying that it is worth doing for the sake of it,
as being more "Pythonic" in some sense.

That is, it seems to me that in a modern Python writing something like:


@contextlib.contextmanager
def foo_cm(testcase):
testcase.bar = some_costly_setup_function()
yield
testcase.bar.close()

@contextlib.contextmanager
def foo_test_cm(testcase):
testcase.baz = Mock(testcase.bar)
yield


@unittest.case_context(foo_cm)
@unittest.test_context(foo_test_cm)
class TestFoo(unittest.TestCase):

def test_bar:
foo = Foo(self.baz, testing=True)
self.assertTrue("Context managers are cool")


This reminds me of the decorators I have available in testfixtures:

http://packages.python.org/testfixtures/mocking.html
http://packages.python.org/testfixtures/logging.html
http://packages.python.org/testfixtures/files.html

(the last of which is a lot prettier in svn, not had a chance to release 
:-S)


Anyway, these I've ended up making available as context managers as well 
as decorators...


But yes, something similar for sharing state between tests and/or doing 
setup for each test would be nice :-)


cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Antoine Pitrou]

Robert Collins  robertcollins.net> writes:
> 
> I'm not personally very keen on inspecting everything in self.__dict__,
> I suspect it would tickle bugs in other unittest extensions. However I'm
> not really /against/ it - I don't think it will result in bad test
> behaviour or isolation issues. So if users would like it, lets do it.

Why not take all resource_XXX attributes?

By the way, how does a given test access the allocated resource? Say, the DB
connection. Does it become an attribute of the test case instance?

Thank you

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Robert Collins]

On Sat, 2010-02-13 at 10:42 +, Antoine Pitrou wrote:
> Robert Collins  robertcollins.net> writes:
> > 
> > I'm not personally very keen on inspecting everything in self.__dict__,
> > I suspect it would tickle bugs in other unittest extensions. However I'm
> > not really /against/ it - I don't think it will result in bad test
> > behaviour or isolation issues. So if users would like it, lets do it.
> 
> Why not take all resource_XXX attributes?

Sure, though if we're just introspecting I don't see much risk
difference between resource_XXX and all XXX. As I say above I'm happy to
do it if folk think it will be nice.

> By the way, how does a given test access the allocated resource? Say, the DB
> connection. Does it become an attribute of the test case instance?

yes. Given 

class Foo(TestCase):
resources = {'thing', MyResourceManager()}
def test_foo(self):
self.thing

self.thing will access the resource returned by MyResourceManager.make()

-Rob


signature.asc
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 385: Auditing

[Martin v. Löwis]

I recently set up a Mercurial hosting solution myself, and noticed that
there is no audit trail of who had been writing to the "master" clone.
There are commit messages, but they could be fake (even misleading to a
different committer).

The threat I'm concerned about is that of a stolen SSH key. If that is
abused to push suspicious changes into the repository, it is really
difficult to find out whose key had been used.

The solution I came up with is to define an "incoming" hook on the
repository which will log the SSH user along with the pack ID of the
pack being pushed.

I'd like to propose that a similar hook is installed on repositories
hosted at hg.python.org (unless Mercurial offers something better
already). Whether or not this log should be publicly visible can be
debated; IMO it would be sufficient if only sysadmins can inspect it in
case of doubt.

Alterntively, the email notification sent to python-checkins could could
report who the pusher was.

Dirkjan: if you agree to such a strategy, please mention that in the PEP.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385: Auditing

[Antoine Pitrou]

Martin v. Löwis  v.loewis.de> writes:
> 
> Alterntively, the email notification sent to python-checkins could could
> report who the pusher was.

This sounds reasonable, assuming it doesn't disclose any private information.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385: Auditing

[Rafael Villar Burke]

Antoine Pitrou  pitrou.net> writes:

> 
> Martin v. Löwis  v.loewis.de> writes:
> > 
> > Alterntively, the email notification sent to python-checkins could could
> > report who the pusher was.
> 
> This sounds reasonable, assuming it doesn't disclose any private information.

There are already made solutions for that, as the pushlog hooks used by Mozilla,
OpenJDK and others.

Mozilla's pushlog can be seen here:

http://hg.mozilla.org/mozilla-central/pushloghtml

And its code is avaliable here:
http://hg.mozilla.org/users/bsmedberg_mozilla.com/hgpoller/file/tip/pushlog-feed.py

Dirkjan is its author, so I suppose he was already thinking about having a
similar hook for Python repos.

Regards,

Rafael

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385: Auditing

[Martin v. Löwis]

> Mozilla's pushlog can be seen here:
> 
> http://hg.mozilla.org/mozilla-central/pushloghtml
> 
> And its code is avaliable here:
> http://hg.mozilla.org/users/bsmedberg_mozilla.com/hgpoller/file/tip/pushlog-feed.py
> 
> Dirkjan is its author, so I suppose he was already thinking about having a
> similar hook for Python repos.

This seems to just be the code that generates the feed, out of a
database pushlog2.db that somehow must be created. So where is the code
to actually fill that database?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385: Auditing

[Rafael Villar Burke (Pachi)]

On 13/02/2010 15:25, "Martin v. Löwis" wrote:

Mozilla's pushlog can be seen here:

http://hg.mozilla.org/mozilla-central/pushloghtml

And its code is avaliable here:
http://hg.mozilla.org/users/bsmedberg_mozilla.com/hgpoller/file/tip/pushlog-feed.py

Dirkjan is its author, so I suppose he was already thinking about having a
similar hook for Python repos.
 

This seems to just be the code that generates the feed, out of a
database pushlog2.db that somehow must be created. So where is the code
to actually fill that database?
   
There's some more content here: 
http://hg.mozilla.org/users/bsmedberg_mozilla.com/hgpoller/file/tip
But I don't use it myself, just knew about its existance. Surely Dirkjan 
can make all the pieces fit nicely :).


Rafael
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385: Auditing

[Rafael Villar Burke (Pachi)]

On 13/02/2010 16:03, Rafael Villar Burke (Pachi) wrote:
There's some more content here: 
http://hg.mozilla.org/users/bsmedberg_mozilla.com/hgpoller/file/tip
But I don't use it myself, just knew about its existance. Surely 
Dirkjan can make all the pieces fit nicely :).
The hook code looks like it's here: 
http://hg.mozilla.org/users/bsmedberg_mozilla.com/hghooks/file/tip

The previous repository link is the hgwebdir integration code.

Regards,

Rafael
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385 progress report

[Barry Warsaw]

On Feb 13, 2010, at 1:31 AM, Martin v. Löwis wrote:

>>> On Fri, Feb 12, 2010 at 11:17, "Martin v. Löwis"  wrote:
 IMO, it is realistic to predict that this will not actually happen. If
 we can agree to give up the 2to3 sandbox, we should incorporate
 find_pattern into the tree, and perhaps test.py as well.
>>> I vote on giving up the 2to3 sandbox.
>> 
>> Besides, if we're using hg, it should make it much easier for someone
>> else to branch that part of the stdlib
> 
> Actually - no: hg doesn't support branching of parts of a repository.
> You would need to branch all of Python. Then, there wouldn't be a
> straight-forward place to setup.py and any other top-level files
> (although you could hack them into Lib, and work with a distutils manifest).

Does hg support an equivalent of 'bzr split'?

% bzr split --help
Purpose: Split a subdirectory of a tree into a separate tree.
Usage:   bzr split TREE

Options:
  --usageShow usage message and options.
  -v, --verbose  Display more information.
  -q, --quietOnly display errors and warnings.
  -h, --help Show help message.

Description:
  This command will produce a target tree in a format that supports
  rich roots, like 'rich-root' or 'rich-root-pack'.  These formats cannot be
  converted into earlier formats like 'dirstate-tags'.
  
  The TREE argument should be a subdirectory of a working tree.  That
  subdirectory will be converted into an independent tree, with its own
  branch.  Commits in the top-level tree will not apply to the new subtree.

See also: join

-Barry

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385 progress report

[Benjamin Peterson]

2010/2/13 "Martin v. Löwis" :
>> I personally like 2to3 in a separate repo because it fits well with my
>> view that 2to3 is an extra application that happens to also be
>> distributed with python.
>
> But isn't that just a theoretical property? I know that's how 2to3
> started, but who, other than the committers, actually accesses the 2to3
> repo?

It's used in 3to2 for example.

>
> I would be much more supportive of that view if there had been a single
> release of 2to3 at any point in time (e.g. to PyPI). Alas, partially due
> to me creating lib2to3, you actually couldn't release it as an extra
> application and run it on 2.6 or 2.7, as the builtin lib2to3 would take
> precedence over the lib2to3 bundled with the application.

It could be distributed under another name or provide a way to
override the stdlib version.


-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385 progress report

[Martin v. Löwis]

>> But isn't that just a theoretical property? I know that's how 2to3
>> started, but who, other than the committers, actually accesses the 2to3
>> repo?
> 
> It's used in 3to2 for example.

That doesn't really seem to be the case. AFAICT, 3to2 is a hg
repository, with no inherent connection to the 2to3 svn sandbox. It does
use lib2to3, but that could come either from an installed Python, from a
trunk/3k checkout, or from the sandbox. Correct?

So if the 2.x trunk became the official master for (lib)2to3, nothing
would really change for 3to3, right? (except for the comment in the
readme that you should get 2to3 from the sandbox if the trunk copy
doesn't work; this comment would become obsolete as changes *would*
propagate immediately into the Python trunk).

>> I would be much more supportive of that view if there had been a single
>> release of 2to3 at any point in time (e.g. to PyPI). Alas, partially due
>> to me creating lib2to3, you actually couldn't release it as an extra
>> application and run it on 2.6 or 2.7, as the builtin lib2to3 would take
>> precedence over the lib2to3 bundled with the application.
> 
> It could be distributed under another name or provide a way to
> override the stdlib version.

Sure. However, I'm still claiming that this is theoretical. The only
person who has shown a slight interest in having this as a separate
project (since Collin Winter left) is you, and so far, you haven't made
any efforts to produce a stand-alone release. I don't blame you at all
for that, in fact, I think Python is better off with the status quo
(i.e. changes to 2to3 get liberally released even with bug fix releases,
basically in an exemption from the "no new features" policy - similar to
-3 warnings).

I still think that the best approach for projects to use 2to3 is to run
2to3 at install time from a single-source release. For that, projects
will have to adjust to whatever bugs certain 2to3 releases have, rather
than requiring users to download a newer version of 2to3 that fixes
them. For this use case, a tightly-integrated lib2to3 (with that name
and sole purpose) is the best thing.

Regards,
Martin



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] setUpClass and setUpModule in unittest

[Guido van Rossum]

On Fri, Feb 12, 2010 at 8:01 PM, Glyph Lefkowitz
 wrote:
> On Feb 11, 2010, at 1:11 PM, Guido van Rossum wrote:
>
>> I have skimmed this thread (hence this reply to the first rather than
>> the last message), but in general I am baffled by the hostility of
>> testing framework developers towards their users. The arguments
>> against class- and module-level seUp/tearDown functions seems to be
>> inspired by religion or ideology more than by the zen of Python. What
>> happened to Practicality Beats Purity?
>
> My sentiments tend to echo Jean-Paul Calderone's in this regard, but I think 
> what he's saying bears a lot of repeating.  We really screwed up this feature 
> in Twisted and I'd like to make sure that the stdlib doesn't repeat the 
> mistake.  (Granted, we screwed it up extra bad 
> , but I do think many of the 
> problems we encountered are inherent.)

Especially since you screwed up extra bad, the danger exists that
you're overreacting.

> The issue is not that we test-framework developers don't like our users, or 
> want to protect them from themselves.  It is that our users - ourselves chief 
> among them - desire features like "I want my tests to be transparently 
> optimized across N cores and N disks".

Yeah, users ask for impossible features all the time. ;-)

Seriously, we do this at Google on a massive scale, for many languages
including Python. It's works well but takes getting used to: while
time is saved waiting for tests, some time is wasted debugging tests
that run fine on the developer's workstation but not in the test
cluster. We've developed quite a few practices around this, which
include ways to override and control the test distribution, as well as
reports showing the historical "flakiness" for each tests.

> I can understand how resistance to setUp/tearDown*Class/Module comes across 
> as user-hostility, but I can assure you this is not the case.  It's subtle 
> and difficult to explain how incompatible with these advanced features the 
> *apparently* straightforward semantics of setting up and tearing down classes 
> and modules.  Most questions of semantics can be resolved with a simple 
> decision, and it's not clear how that would interfere with other features.
>
> In Twisted's implementation of setUpClass and tearDownClass, everything 
> seemed like it worked right up until the point where it didn't.  The test 
> writer thinks that they're writing "simple" setUpClass and tearDownClass 
> methods to optimize things, except almost by definition a setUpClass method 
> needs to manipulate global state, shared across tests.  Which means that said 
> state starts getting confused when it is set up and torn down concurrently 
> across multiple processes.  These methods seem simple, but do they touch the 
> filesystem?  Do they touch a shared database, even a little?  How do they 
> determine a unique location to do that?  Without generally available tools to 
> allow test writers to mess with the order and execution environment of their 
> tests, one tends to write tests that rely on these implementation and 
> ordering accidents, which means that when such a tool does arrive, things 
> start breaking in unpredictable ways.

Been there, done that. The guideline should be that setUpClass and
friends save time but should still isolate themselves from other
copies that might run concurrently. E.g. if you have to copy a ton of
stuff into the filesystem, you should still put it in a temp dir with
a randomized name, and store that name as a class variable.

When there's a global resource (such as a database) that really can't
be shared, well, you have to come up with a way to lock it -- that's
probably necessary even if your tests ran completely serialized,
unless there's only one developer and she never multitasks. :-)

>> The argument that a unittest framework shouldn't be "abused" for
>> regression tests (or integration tests, or whatever) is also bizarre
>> to my mind. Surely if a testing framework applies to multiple kinds of
>> testing that's a good thing, not something to be frowned upon?
>
> For what it's worth, I am a big fan of abusing test frameworks in generally, 
> and pyunit specifically, to perform every possible kind of testing.  In fact, 
> I find setUpClass more hostile to *other* kinds of testing, because this 
> convenience for simple integration tests makes more involved, 
> performance-intensive integration tests harder to write and manage.

That sounds odd, as if the presence of this convenience would prohibit
you from also implement other features.

>> On the other hand, I think we should be careful to extend unittest in
>> a consistent way. I shuddered at earlier proposals (on python-ideas)
>> to name the new functions (variations of) set_up and tear_down "to
>> conform with PEP 8" (this would actually have violated that PEP, which
>> explicitly prefers local consistency over global consistency).
>
> This is a very impo

Re: [Python-Dev] PEP 385 progress report

[Dirkjan Ochtman]

On Sat, Feb 13, 2010 at 17:14, Barry Warsaw  wrote:
> Does hg support an equivalent of 'bzr split'?
>
> % bzr split --help
> Purpose: Split a subdirectory of a tree into a separate tree.
> Usage:   bzr split TREE
>
> Options:
>  --usage        Show usage message and options.
>  -v, --verbose  Display more information.
>  -q, --quiet    Only display errors and warnings.
>  -h, --help     Show help message.
>
> Description:
>  This command will produce a target tree in a format that supports
>  rich roots, like 'rich-root' or 'rich-root-pack'.  These formats cannot be
>  converted into earlier formats like 'dirstate-tags'.
>
>  The TREE argument should be a subdirectory of a working tree.  That
>  subdirectory will be converted into an independent tree, with its own
>  branch.  Commits in the top-level tree will not apply to the new subtree.

Is that like a clone/branch of a subdir of the original repository? We
don't have what we usually call "narrow clones" yet (nor "shallow
clones", the other potentially useful form of partial clones). We do
have the convert extension, which allows you to create a new
repository from a subtree of an old repository, but it changes all the
hashes (and I don't know if we have a way to splice them back
together).

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385: Auditing

[Dirkjan Ochtman]

On Sat, Feb 13, 2010 at 12:53, "Martin v. Löwis"  wrote:
> Dirkjan: if you agree to such a strategy, please mention that in the PEP.

Having a pushlog and/or including the pusher in the email sounds like
a good idea, I'll add something to that effect to the PEP. I slightly
prefer adding it to the commit email because it would seem to require
less infrastructure, and it can be handy at times to know who pushed
something right off the bat.

Cheers,

Dirkjan
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385 progress report

[Benjamin Peterson]

2010/2/13 Dirkjan Ochtman :
> On Sat, Feb 13, 2010 at 17:14, Barry Warsaw  wrote:
>> Does hg support an equivalent of 'bzr split'?
>>
>> % bzr split --help
>> Purpose: Split a subdirectory of a tree into a separate tree.
>> Usage:   bzr split TREE
>>
>> Options:
>>  --usage        Show usage message and options.
>>  -v, --verbose  Display more information.
>>  -q, --quiet    Only display errors and warnings.
>>  -h, --help     Show help message.
>>
>> Description:
>>  This command will produce a target tree in a format that supports
>>  rich roots, like 'rich-root' or 'rich-root-pack'.  These formats cannot be
>>  converted into earlier formats like 'dirstate-tags'.
>>
>>  The TREE argument should be a subdirectory of a working tree.  That
>>  subdirectory will be converted into an independent tree, with its own
>>  branch.  Commits in the top-level tree will not apply to the new subtree.
>
> Is that like a clone/branch of a subdir of the original repository? We
> don't have what we usually call "narrow clones" yet (nor "shallow
> clones", the other potentially useful form of partial clones). We do
> have the convert extension, which allows you to create a new
> repository from a subtree of an old repository, but it changes all the
> hashes (and I don't know if we have a way to splice them back
> together).

It is not a partial clone, but rather similar to what you are
referring to with the convert extension.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385 progress report

[Benjamin Peterson]

2010/2/13 "Martin v. Löwis" :
>>> But isn't that just a theoretical property? I know that's how 2to3
>>> started, but who, other than the committers, actually accesses the 2to3
>>> repo?
>>
>> It's used in 3to2 for example.
>
> That doesn't really seem to be the case. AFAICT, 3to2 is a hg
> repository, with no inherent connection to the 2to3 svn sandbox. It does
> use lib2to3, but that could come either from an installed Python, from a
> trunk/3k checkout, or from the sandbox. Correct?

It has to be from the sandbox (or trunk I suppose) because it requires
changes that haven't been released.

>
> So if the 2.x trunk became the official master for (lib)2to3, nothing
> would really change for 3to3, right? (except for the comment in the
> readme that you should get 2to3 from the sandbox if the trunk copy
> doesn't work; this comment would become obsolete as changes *would*
> propagate immediately into the Python trunk).

Right, except you would have to clone the entire history of Python in
order to get at the trunk version.
>
>>> I would be much more supportive of that view if there had been a single
>>> release of 2to3 at any point in time (e.g. to PyPI). Alas, partially due
>>> to me creating lib2to3, you actually couldn't release it as an extra
>>> application and run it on 2.6 or 2.7, as the builtin lib2to3 would take
>>> precedence over the lib2to3 bundled with the application.
>>
>> It could be distributed under another name or provide a way to
>> override the stdlib version.
>
> Sure. However, I'm still claiming that this is theoretical. The only
> person who has shown a slight interest in having this as a separate
> project (since Collin Winter left) is you, and so far, you haven't made
> any efforts to produce a stand-alone release. I don't blame you at all
> for that, in fact, I think Python is better off with the status quo
> (i.e. changes to 2to3 get liberally released even with bug fix releases,
> basically in an exemption from the "no new features" policy - similar to
> -3 warnings).
>
> I still think that the best approach for projects to use 2to3 is to run
> 2to3 at install time from a single-source release. For that, projects
> will have to adjust to whatever bugs certain 2to3 releases have, rather
> than requiring users to download a newer version of 2to3 that fixes
> them. For this use case, a tightly-integrated lib2to3 (with that name
> and sole purpose) is the best thing.

Alright. That is reasonable.

The other thing is that we will loose some vcs history and some
history granularity by switching development to the trunk version,
since just the svnmerged revisions will be converted.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385 progress report

[Martin v. Löwis]

> The other thing is that we will loose some vcs history and some
> history granularity by switching development to the trunk version,
> since just the svnmerged revisions will be converted.

I suppose it might be possible to fake the history of Lib/lib2to3 with
commits that didn't actually happen, although this is probably a cure
worse than the disease.

We are not going to throw away the subversion repository, so it would
always be possible to go back and look at the actual history.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385 progress report

[Barry Warsaw]

On Feb 13, 2010, at 1:43 PM, Benjamin Peterson wrote:

> 2010/2/13 Dirkjan Ochtman :
>> On Sat, Feb 13, 2010 at 17:14, Barry Warsaw  wrote:
>>> Does hg support an equivalent of 'bzr split'?
>>> 
>>> % bzr split --help
>>> Purpose: Split a subdirectory of a tree into a separate tree.
>>> Usage:   bzr split TREE
>>> 
>>> Options:
>>>  --usageShow usage message and options.
>>>  -v, --verbose  Display more information.
>>>  -q, --quietOnly display errors and warnings.
>>>  -h, --help Show help message.
>>> 
>>> Description:
>>>  This command will produce a target tree in a format that supports
>>>  rich roots, like 'rich-root' or 'rich-root-pack'.  These formats cannot be
>>>  converted into earlier formats like 'dirstate-tags'.
>>> 
>>>  The TREE argument should be a subdirectory of a working tree.  That
>>>  subdirectory will be converted into an independent tree, with its own
>>>  branch.  Commits in the top-level tree will not apply to the new subtree.
>> 
>> Is that like a clone/branch of a subdir of the original repository? We
>> don't have what we usually call "narrow clones" yet (nor "shallow
>> clones", the other potentially useful form of partial clones). We do
>> have the convert extension, which allows you to create a new
>> repository from a subtree of an old repository, but it changes all the
>> hashes (and I don't know if we have a way to splice them back
>> together).
> 
> It is not a partial clone, but rather similar to what you are
> referring to with the convert extension.

Right, that's what it sounds like.  I've used it on a bzr-svn converted 
repository to split a monolithic tree after Subversion->Bazaar conversion into 
separately managed subtrees.  Note that 'bzr join' is the inverse.  The 
interesting thing (both good and bad) is that after the split both subtrees 
have the full history of the original.

-Barry

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 385 progress report

[Joe Amenta]

On Sat, Feb 13, 2010 at 12:14 PM, "Martin v. Löwis" wrote:

> >> But isn't that just a theoretical property? I know that's how 2to3
> >> started, but who, other than the committers, actually accesses the 2to3
> >> repo?
> >
> > It's used in 3to2 for example.
>
> That doesn't really seem to be the case. AFAICT, 3to2 is a hg
> repository, with no inherent connection to the 2to3 svn sandbox. It does
> use lib2to3, but that could come either from an installed Python, from a
> trunk/3k checkout, or from the sandbox. Correct?
>
> So if the 2.x trunk became the official master for (lib)2to3, nothing
> would really change for 3to3, right? (except for the comment in the
> readme that you should get 2to3 from the sandbox if the trunk copy
> doesn't work; this comment would become obsolete as changes *would*
> propagate immediately into the Python trunk).
>
> >> I would be much more supportive of that view if there had been a single
> >> release of 2to3 at any point in time (e.g. to PyPI). Alas, partially due
> >> to me creating lib2to3, you actually couldn't release it as an extra
> >> application and run it on 2.6 or 2.7, as the builtin lib2to3 would take
> >> precedence over the lib2to3 bundled with the application.
> >
> > It could be distributed under another name or provide a way to
> > override the stdlib version.
>
> Sure. However, I'm still claiming that this is theoretical. The only
> person who has shown a slight interest in having this as a separate
> project (since Collin Winter left) is you, and so far, you haven't made
> any efforts to produce a stand-alone release. I don't blame you at all
> for that, in fact, I think Python is better off with the status quo
> (i.e. changes to 2to3 get liberally released even with bug fix releases,
> basically in an exemption from the "no new features" policy - similar to
> -3 warnings).
>
> I still think that the best approach for projects to use 2to3 is to run
> 2to3 at install time from a single-source release. For that, projects
> will have to adjust to whatever bugs certain 2to3 releases have, rather
> than requiring users to download a newer version of 2to3 that fixes
> them. For this use case, a tightly-integrated lib2to3 (with that name
> and sole purpose) is the best thing.
>
> Regards,
> Martin
>
>
>
Yes, if the trunk were the official master for lib2to3, then 3to2 would not
change at all.  If fixes to lib2to3 were immediately propagated to the
trunk, 3to2 would benefit from that.
I support lib2to3's integration with the trunk... it's too confusing
otherwise and kind of defeats the idea of "trunk": if lib2to3 is provided
with Python, then shouldn't its latest version be in Python's trunk?
--Joe Amenta
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] What to intern (e.g. func_code.co_filename)?

[Jake McGuire]

Has anyone come up with rules of thumb for what to intern and what the
performance implications of interning are?

I'm working on profiling App Engine again, and since they don't allow
marshall I have to modify pstats to save the profile via pickle.
While trying to get profiles under 1MB, I noticed that each function
has its own copy of the filename in which it is defined, and sometimes
these strings can be rather long.

Creating a code object already interns a bunch of stuff; argument
names, variable names, etc.  Interning the filename will add some CPU
overhead during function creation, should save a decent amount of
memory, and ought to have minimal overall performance impact.

I have a local patch, but wanted to see if anyone had ideas or
experience weighing these tradeoffs.

-jake
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What to intern (e.g. func_code.co_filename)?

[Benjamin Peterson]

2010/2/13 Jake McGuire :
> I have a local patch, but wanted to see if anyone had ideas or
> experience weighing these tradeoffs.

Interning is really only useful because it speeds up dictionary
lookups for identifiers. A better idea would be to just attach the
same filename object in compiling and unmarshaling.



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What to intern (e.g. func_code.co_filename)?

[Martin v. Löwis]

Benjamin Peterson wrote:
> 2010/2/13 Jake McGuire :
>> I have a local patch, but wanted to see if anyone had ideas or
>> experience weighing these tradeoffs.
> 
> Interning is really only useful because it speeds up dictionary
> lookups for identifiers. A better idea would be to just attach the
> same filename object in compiling and unmarshaling.

I would try to do the sharing during marshaling already. I agree that
the file names shouldn't be interned, though, so I propose to create a
new code TYPE_SHAREDSTRING, similar to TYPE_INTERNED. It would use the
same numbering as TYPE_INTERNED, so backreferences could continue to use
TYPE_STRINGREF.

Alternatively, a general sharing feature could be added to marshal,
sharing all hashable objects. However, before that gets added, I'd like
to see statistics how many objects get considered for sharing, and how
many back-references then get actually generated.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com