[pytest-dev] devpi-server/web-3.0: generalized mirroring, speed, new backends

[holger krekel]

We just released devpi-{server,web}-3.0 and devpi-client-2.5.0 with some
major new features, see the announcement below.  didn't want to cram
too many mailing lists into it in case someone hits "reply all" :)

have fun,
holger

- Forwarded message from holger krekel  -

Date: Fri, 12 Feb 2016 11:13:36 +
From: holger krekel 
To: python announce ,
Testing in Python ,
disutils-sig 
Subject: devpi-server/web-3.0: generalized mirroring, speed, new backends


The 3.0 releases of devpi-server and devpi-web, the python packaging and 
work flow system for handling release files, documentation, testing and staging,
bring several major improvements:

- Due to popular demand we now support generalized mirroring, i.e. you can 
  create mirror indexes which proxy and cache release files from other pypi 
  servers.  Even if the mirror goes down, pip-installing will continue to work
  with your devpi-server instance.  Previously we only supported mirroring 
  of pypi.python.org.  Using it is simple:
  http://doc.devpi.net/3.0/userman/devpi_indices.html#mirror-index

- For our enterprise clients we majorly worked on improving the speed
  of serving simple pages which is now several times faster with
  private indexes.  We now also support multiple worker processes
  both on master and replica sites.
  http://doc.devpi.net/3.0/adminman/server.html#multiple-server-instances

- For our enterprise clients we also introduced a new backend
  architecture which allows to store server state in sqlite or
  postgres (which is supported through a separately released plugin).
  The default remains to use the "sqlite" backend and store files
  on the filesystem. See
  http://doc.devpi.net/3.0/adminman/server.html#storage-backend-selection

- we started a new "admin" manual for devpi-server which describes
  features relating to server configuration, replication and security
  aspects.  It's a bit work-in-progress but should already be helpful.
  http://doc.devpi.net/3.0/adminman/

- A few option names changed and we also released devpi-client-2.5 
  where we took great care to keep it forward and backward compatible
  so it should run against devpi-server-2.1 and upwards all the way
  to 3.0.

- The "3.0" major release number increase means that you will need to run 
  through an export/import cycle to upgrade your devpi-2.X installation.

For more details, see the changelog and the referenced documentation
with the main entry point here:

http://doc.devpi.net

Many thanks to my partner Florian Schulze and to the several companies
who funded parts of the work on 3.0.  We are especially grateful for
their support to not only cover their own direct issues but also support
community driven demands.  I'd also like to express my gratitude to
Rackspace and Jesse Noller who provide VMs for our open source work and
which help a lot with the testing of our releases.

We are open towards entering more support contracts to make sure you get
what you need out of devpi, tox and pytest which together provide a
mature tool chain for professional python development.  And speaking of
showing support, if you or your company is interested to donate to or
attend the largest python testing sprint in history with a particular
focus to pytest or tox, please see

https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/

have fun,

holger krekel, http://merlinux.eu



server-3.0.0 (2016-02-12)
-

- dropped support for python2.6

- block most ascii symbols for user and index names except ``-.@_``.
  unicode characters are fine.

- add ``--no-root-pypi`` option which prevents the creation of the
  ``root/pypi`` mirror instance on first startup.

- added optional ``title`` and ``description`` options to users and indexes.

- new indexes have no bases by default anymore. If you want to be able to
  install pypi packages, then you have to explicitly add ``root/pypi`` to
  the ``bases`` option of your index.

- added optional ``custom_data`` option to users.

- generalized mirroring to allow adding mirror indexes other than only PyPI

- renamed ``pypi_whitelist`` to ``mirror_whitelist``

- speed up simple-page serving for private indexes. A private index
  with 200 release files should now be some 5 times faster.

- internally use normalized project names everywhere, simplifying
  code and slightly speeding up some operations.

- change {name} in route_urls to {project} to disambiguate.
  This is potentially incompatible for plugins which have registered
  on existing route_urls.

- use "project" variable naming consistently in APIs

- drop calling of devpi_pypi_initial hook in favor of
  the new "devpi_mirror_initialnames(stage, projectnames)" hook
  which is called when a mirror is initialized.

- introduce new "devpiserver_stage_created(stage)" hook which is
  

Re: [pytest-dev] How to sandbox test execution with pytest

[Alessandro Amici]

Hi,

while researching the use of pytest-xdist for my CI-like use case and I
realised that having a "disposable" environment is as important as having a
isolated one, so that every test or every session runs from the same
initial state and it is not influenced by what older sessions might have
left on folders writable by the *testuser* (/home/testuser, /tmp, /var/tmp,
etc).

This made me go back searching the web again for a way to use pytest with
docker and I'm currently evaluating the following projects:

https://github.com/search?q=pytest+docker

Does anybody have experience with pytest/docker integration or with any of
these projects?

I'll report back what I find.

Thanks,
Alessandro

On Thu, 11 Feb 2016 at 11:07 Alessandro Amici  wrote:

> Ionel and Holger,
>
> I had considered the --box option, but it only implements process-level
> isolation which is not enough for my use case as I want to protect also
> from unrestricted filesystem read/write access. Sorry, I should have
> mentioned it in the question.
>
> Holger's suggestion instead looks almost good enough as user-level
> isolation is strong and highly customizable. The only problem is that you
> need administration access to the machine to create and configure a new
> user. I'll look into it as I've never used the --tx option.
>
> Thanks,
> Alessandro
>
>
> On Thu, 11 Feb 2016 at 10:44 holger krekel  wrote:
>
>> On Thu, Feb 11, 2016 at 11:39 +0200, Ionel Cristian Mărieș wrote:
>> > There's the `--boxed` option (from
>> https://pypi.python.org/pypi/pytest-xdist)
>> > but you can easily do your own "insulator" by using fork syscall.
>>
>> Also one can use plain "xdist" which provides process-level isolation.
>> It shouldn't be too hard to also use another unix user via e.g.:
>>
>> py.test --tx ssh=OTHERUSER@localhost --dist=each
>>
>> but you need to have setup rsyncing, see
>> https://pypi.python.org/pypi/pytest-xdist
>> for some more info.  The current "remote" test running is a bit limited
>> and we plan to improve xdist distribution during the upcoming pytest
>> sprint,
>> see
>> https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/x/4034848#/
>> creating an issue for "user-level separation of running tests" or
>> something
>> similar and adding a little donation is one way to help it :)
>>
>> best,
>> holger
>>
>>
>> >
>> > Thanks,
>> > -- Ionel Cristian Mărieș, http://blog.ionelmc.ro
>> >
>> > On Thu, Feb 11, 2016 at 10:33 AM, Alessandro Amici > >
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > I asked the following question on StackOverflow first
>> > > http://stackoverflow.com/questions/35322452, but someone quickly (and
>> > > wrongly in my opinion) marked that as duplicate of the general
>> "sandboxing
>> > > in python", so I can't get answers there anymore :(
>> > >
>> > > Here is the question:
>> > >
>> > > I'm interested in executing potentially untrusted tests with pytest in
>> > > some kind of sandbox, similarly to what continuous integration
>> services do.
>> > >
>> > > I understand that to properly sandbox a python process you need
>> OS-level
>> > > isolation, like running the tests in a disposable chroot/container,
>> but in
>> > > my use case I don't need to protect against intentionally malicious
>> code,
>> > > only from dangerous behaviour of pairing "randomly" functions with
>> > > arguments. So lesser strict sandboxing may still be acceptable. But I
>> > > didn't find any plugin that enables any form of sandboxing.
>> > >
>> > > What is the best way to sandbox tests execution in pytest?
>> > >
>> > > **Update**: This question is not about [python sandboxing in general](
>> > >
>> http://stackoverflow.com/questions/3068139/how-can-i-sandbox-python-in-pure-python
>> )
>> > > as the tests' code is run by pytest and I can't change the way it is
>> > > executed to use `exec` or `ast` or whatever. Also using pypy-sandbox
>> is not
>> > > an option unfortunately as it is "a prototype only" as per the [PyPy
>> > > feature page](http://pypy.org/features.html).
>> > >
>> > > End of question.
>> > >
>> > > For context I'm writing a pytest plugin that lets you search installed
>> > > modules for functions that pass the given feature-specification tests
>> > > (still alpha):
>> > >
>> > > https://github.com/alexamici/pytest-wish
>> > >
>> > > If you find the question useful and not answered already on
>> stackoverflow,
>> > > the best course of action would be to reopen it and answer there, but
>> I'll
>> > > take also an answer here :)
>> > >
>> > > Alessandro
>> > >
>> > >
>> > > ___
>> > > pytest-dev mailing list
>> > > pytest-dev@python.org
>> > > https://mail.python.org/mailman/listinfo/pytest-dev
>> > >
>> > >
>>
>> > ___
>> > pytest-dev mailing list
>> > pytest-dev@python.org
>> > https://mail.python.org/mailman/listinfo/pytest-dev
>>
>>
>> --
>> about me:

Re: [pytest-dev] Sprint participants writing a blog post?

[Bruno Oliveira]

On Sat, Feb 13, 2016 at 2:06 AM Brianna Laugher 
wrote:

> Your writing is great Bruno! If you wanted I would be happy to edit a
> draft though.
>

> On 12 February 2016 at 09:02, Holger Krekel  wrote:
>
>> Your written English is pretty good so you feel youbare rather missing
>> practice in pronunciation? But when we did the hangout I wasn't aware of it
>> :)
>>
>
Thanks for the compliments! :)

I will setup a blog post about the sprint tomorrow, just have to remember
how to use Pelican. :)

Cheers,
Bruno.

>
___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev

[pytest-dev] Move pytest-runner to github

[Jason R. Coombs]

I’ve been moving most of my projects to Github. I’d like to do the same with 
pytest-runner. Would someone create the repo and/or grant me the necessary 
access, including admin access to the created repo? Probably 
/pytest-dev/pytest-runner.

Thanks,
Jason (jaraco)
___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev

[pytest-dev] Visibility of project members

[Bruno Oliveira]

Hi all,

I was browsing the peoples page at
https://github.com/orgs/pytest-dev/people and
noticed that most members have their membership set as "private". I believe
most people left that setting as private because it is the default. You can
easily see your setting to members outside the organization by browsing
that page in incognito mode.

Just a heads up in case anyone wants their membership as public. :)

Thanks,
Bruno.
___
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev

Re: [pytest-dev] Sprint participants writing a blog post?

[Brianna Laugher]

Your writing is great Bruno! If you wanted I would be happy to edit a draft
though.

Andreas, how about you? You could speak about your company's support too :)

Brianna

On 12 February 2016 at 09:02, Holger Krekel  wrote:

> Your written English is pretty good so you feel youbare rather missing
> practice in pronunciation? But when we did the hangout I wasn't aware of it
> :)
>
>
>
> On February 11, 2016 10:01:52 PM GMT+01:00, Bruno Oliveira <
> nicodde...@gmail.com> wrote:
>>
>> Hi Holger,
>>
>> TBH I'm a little shy of my english speaking skills, so I rather don't do
>> the interview. :S
>>
>> Someone else might be more suited!
>>
>>
>> On Thu, Feb 11, 2016 at 7:10 AM holger krekel  wrote:
>>
>>>
>>> Hey Bruno,
>>>
>>> what about you?  You have been incredibly active and it'd be cool to see
>>> you interviewed :)  I think apart from the campaign it'd be interesting
>>> to talk about pytest-3.0 and xdist improvements that we are looking into
>>> ...
>>>
>>> best,
>>> holger
>>>
>>>
>>> On Wed, Feb 10, 2016 at 20:14 +, Dave Hunt wrote:
>>> > We have an invitation to be discuss/promote the sprint on the Python
>>> Test Podcast if anyone wants to volunteer!
>>> > https://twitter.com/brianokken/status/697419561797160960 <
>>> https://twitter.com/brianokken/status/697419561797160960>
>>> >
>>> > Cheers,
>>> > Dave
>>> >
>>> > > On 9 Feb 2016, at 17:16, Dave Hunt  wrote:
>>> > >
>>> > > Published:
>>> > > http://blargon7.com/2016/02/python-testing-sprint-2016/ <
>>> http://blargon7.com/2016/02/python-testing-sprint-2016/>
>>> > >
>>> > > Dave
>>> > >
>>> > >> On 9 Feb 2016, at 14:09, holger krekel > hol...@merlinux.eu>> wrote:
>>> > >>
>>> > >> On Tue, Feb 09, 2016 at 14:07 +, Dave Hunt wrote:
>>> > >>> I have a public preview of my blog post. Feedback welcome:
>>> > >>> http://blargon7.com/?p=204=1&_ppp=65d7c71c07 <
>>> http://blargon7.com/?p=204=1&_ppp=65d7c71c07> <
>>> http://blargon7.com/?p=204=1&_ppp=65d7c71c07 <
>>> http://blargon7.com/?p=204=1&_ppp=65d7c71c07>>
>>> > >>
>>> > >> reads good!
>>> > >>
>>> > >> holger
>>> > >>
>>> > >>> Dave
>>> > >>>
>>> >  On 9 Feb 2016, at 10:40, Dave Hunt > dh...@mozilla.com>> wrote:
>>> > 
>>> >  I think this is a great idea! I have had approval to join you all
>>> during the sprint, so I’d be happy to write a blog post about how I came to
>>> pytest and what I’d like to work on during the week.
>>> > 
>>> >  Cheers,
>>> >  Dave
>>> > 
>>> > > On 8 Feb 2016, at 20:05, Brianna Laugher <
>>> brianna.laug...@gmail.com  >> brianna.laug...@gmail.com >> wrote:
>>> > >
>>> > > Heh, I had the exact same idea last night :)
>>> > >
>>> > > I was thinking
>>> > > 1) how did you get involved with pytest?
>>> > > 2) What is your favourite feature/plugin of pytest that is
>>> relatively unknown?
>>> > > 3) What are you planning to work on during the sprint?
>>> > >
>>> > > Cheers
>>> > > Brianna
>>> > >
>>> > > On 09/02/2016 6:04 AM, "Holger Krekel" >>  > hol...@merlinux.eu>>> wrote:
>>> > > What about if each of us who wants to participate writes a small
>>> blog (lacking a blog a pytest-dev) post of where he/she comes from, past
>>> core or plugin work for example, and main areas of interest for the sprint?
>>> > >
>>> > > We could publish them via updates every few days. Any takers?
>>> > > --
>>> > > Sent using mobile touch keys, increased chances for errors and
>>> misunderstandings. Enjoy!
>>> > > ___
>>> > > pytest-dev mailing list
>>> > > pytest-dev@python.org  >> pytest-dev@python.org >
>>> > > https://mail.python.org/mailman/listinfo/pytest-dev <
>>> https://mail.python.org/mailman/listinfo/pytest-dev> <
>>> https://mail.python.org/mailman/listinfo/pytest-dev <
>>> https://mail.python.org/mailman/listinfo/pytest-dev>>
>>> > >
>>> > > ___
>>> > > pytest-dev mailing list
>>> > > pytest-dev@python.org  >> pytest-dev@python.org >
>>> > > https://mail.python.org/mailman/listinfo/pytest-dev <
>>> https://mail.python.org/mailman/listinfo/pytest-dev>
>>> > 
>>> > >>>
>>> > >>
>>> > >> --
>>> > >> about me:http://holgerkrekel.net/about-me/ <
>>> http://holgerkrekel.net/about-me/>
>>> > >> contracting: http://merlinux.eu 
>>> > >
>>> >
>>>
>>> --
>>> about me:http://holgerkrekel.net/about-me/
>>> contracting: http://merlinux.eu
>>> ___
>>> pytest-dev mailing list
>>> pytest-dev@python.org
>>>