Re: [matplotlib-devel] Strange bug when saving animations (with streamplot)

[Tony Yu]

On Mon, Jul 2, 2012 at 11:42 PM, Tony Yu  wrote:

>
>
> On Sun, Jun 24, 2012 at 10:02 AM, Tony Yu  wrote:
>
>> I'm having a tough time figuring this out: Saving an animation seems to
>> hang when using `streamplot`. The exact same animation runs without issue
>> when calling show.
>>
>> On my system, the example below hangs consistently at frame 173. However,
>> the number of saved frames (before hanging) varies with density of stream
>> lines (i.e. number of line segments in streamplot): More frames saved for
>> lower density.
>>
>> Seems to be a memory or file-size issue, but
>>
>>* Memory use doesn't seem to grow. (this suggests that ffmpeg adds
>> frames to disk as opposed to memory.)
>>* Saving the animation up to the frame just before it hangs gives
>> modest (~1MB) videos.
>> Maybe this is a limitation of `ffmpeg`?
>>
>> Can someone confirm that the example hangs on their system? (Somehow, I
>> often run into bugs that are not reproducible.)
>>
>> In case this is system dependent:
>> * OSX 10.6
>> * Python 2.6.1 (system install)
>> * matplotlib master
>> * ffmpeg 0.10
>>
>> Simple plots seem to work fine, but I should probably try to reproduce
>> using something other than streamplot (maybe create the equivalent number
>> of line and arrow collections).
>>
>> -Tony
>>
>> #~~~ Example
>> import numpy as np
>> import matplotlib.pyplot as plt
>> import matplotlib.animation as animation
>>
>> fig, ax = plt.subplots()
>>
>> # do nothing function that prints frame
>> def update(frame_num):
>> print frame_num
>> return []
>>
>> Y, X = np.mgrid[:1:100j, :1:100j]
>> U = np.ones_like(X)
>> V = np.ones_like(X)
>> ax.streamplot(X, Y, U, V, density=1)
>>
>> ani = animation.FuncAnimation(fig, update, save_count=500)
>>
>> # save hangs at frame 173 (this probably varies by system, if it's
>> reproducible).
>> ani.save('animation.avi')
>> # show animates all frames w/o issue.
>> #plt.show()
>>
>>
>>
> I finally had some time to revisit this issue. It seems that subprocess
> PIPEs have fairly limited buffers [1] (also see docs for
> `subprocess.Popen.communicate` [2]) and ffmpeg tends to generate a decent
> amount of output to stderr. A simple solution is to redirect stderr to a
> temporary file. This change fixes the issue on my system.
>
> If this bug is reproducible, I can submit a PR. The temporary file here
> could be created using tempfile so it gets cleaned up automatically, or
> maybe a more permanent log file for debugging. Thoughts?
>
> -Tony
>
> [1]
> http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/
>
> [2]
> http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
>
>
And just to clarify: My original email mentioned that saving would hang
only when `streamplot` was used to create the figure. It turns out that
ffmpeg prints more output (keyframes?) for more complex images, so when I
ran simpler plots, they didn't produce enough output to fill the subprocess
`PIPE` (although they would eventually).

Also, theres's a simpler fix than piping ffmpeg output to a file: Just
set `-loglevel quiet` in the ffmpeg command.

-Tony
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Strange bug when saving animations (with streamplot)

[Ryan May]

On Tue, Jul 3, 2012 at 9:14 AM, Tony Yu  wrote:
> On Mon, Jul 2, 2012 at 11:42 PM, Tony Yu  wrote:
>> On Sun, Jun 24, 2012 at 10:02 AM, Tony Yu  wrote:
>>>
>>> I'm having a tough time figuring this out: Saving an animation seems to
>>> hang when using `streamplot`. The exact same animation runs without issue
>>> when calling show.
>>>
>>> On my system, the example below hangs consistently at frame 173. However,
>>> the number of saved frames (before hanging) varies with density of stream
>>> lines (i.e. number of line segments in streamplot): More frames saved for
>>> lower density.
>>>
>>> Seems to be a memory or file-size issue, but
>>>
>>>* Memory use doesn't seem to grow. (this suggests that ffmpeg adds
>>> frames to disk as opposed to memory.)
>>>* Saving the animation up to the frame just before it hangs gives
>>> modest (~1MB) videos.
>>> Maybe this is a limitation of `ffmpeg`?
>>>
>>> Can someone confirm that the example hangs on their system? (Somehow, I
>>> often run into bugs that are not reproducible.)
>>>
>>> In case this is system dependent:
>>> * OSX 10.6
>>> * Python 2.6.1 (system install)
>>> * matplotlib master
>>> * ffmpeg 0.10
>>>
>>> Simple plots seem to work fine, but I should probably try to reproduce
>>> using something other than streamplot (maybe create the equivalent number of
>>> line and arrow collections).
>>>
>>> -Tony
>>>
>>> #~~~ Example
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import matplotlib.animation as animation
>>>
>>> fig, ax = plt.subplots()
>>>
>>> # do nothing function that prints frame
>>> def update(frame_num):
>>> print frame_num
>>> return []
>>>
>>> Y, X = np.mgrid[:1:100j, :1:100j]
>>> U = np.ones_like(X)
>>> V = np.ones_like(X)
>>> ax.streamplot(X, Y, U, V, density=1)
>>>
>>> ani = animation.FuncAnimation(fig, update, save_count=500)
>>>
>>> # save hangs at frame 173 (this probably varies by system, if it's
>>> reproducible).
>>> ani.save('animation.avi')
>>> # show animates all frames w/o issue.
>>> #plt.show()
>>>
>>>
>>
>> I finally had some time to revisit this issue. It seems that subprocess
>> PIPEs have fairly limited buffers [1] (also see docs for
>> `subprocess.Popen.communicate` [2]) and ffmpeg tends to generate a decent
>> amount of output to stderr. A simple solution is to redirect stderr to a
>> temporary file. This change fixes the issue on my system.
>>
>> If this bug is reproducible, I can submit a PR. The temporary file here
>> could be created using tempfile so it gets cleaned up automatically, or
>> maybe a more permanent log file for debugging. Thoughts?
>>
>> -Tony
>>
>> [1]
>> http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/
>>
>> [2]
>> http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
>>
>
> And just to clarify: My original email mentioned that saving would hang only
> when `streamplot` was used to create the figure. It turns out that ffmpeg
> prints more output (keyframes?) for more complex images, so when I ran
> simpler plots, they didn't produce enough output to fill the subprocess
> `PIPE` (although they would eventually).
>
> Also, theres's a simpler fix than piping ffmpeg output to a file: Just set
> `-loglevel quiet` in the ffmpeg command.
>
> -Tony

It's not a bad idea to have it logged to a file in a temp directory.
We could potentially just do this if debug is set to verbose, however,
and just use -loglevel quiet otherwise. Can you manage PR for this or
do I need to?

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Strange bug when saving animations (with streamplot)

[Tony Yu]

On Tue, Jul 3, 2012 at 10:22 AM, Ryan May  wrote:

> On Tue, Jul 3, 2012 at 9:14 AM, Tony Yu  wrote:
> > On Mon, Jul 2, 2012 at 11:42 PM, Tony Yu  wrote:
> >> On Sun, Jun 24, 2012 at 10:02 AM, Tony Yu  wrote:
> >>>
> >>> I'm having a tough time figuring this out: Saving an animation seems to
> >>> hang when using `streamplot`. The exact same animation runs without
> issue
> >>> when calling show.
> >>>
> >>> On my system, the example below hangs consistently at frame 173.
> However,
> >>> the number of saved frames (before hanging) varies with density of
> stream
> >>> lines (i.e. number of line segments in streamplot): More frames saved
> for
> >>> lower density.
> >>>
> >>> Seems to be a memory or file-size issue, but
> >>>
> >>>* Memory use doesn't seem to grow. (this suggests that ffmpeg adds
> >>> frames to disk as opposed to memory.)
> >>>* Saving the animation up to the frame just before it hangs gives
> >>> modest (~1MB) videos.
> >>> Maybe this is a limitation of `ffmpeg`?
> >>>
> >>> Can someone confirm that the example hangs on their system? (Somehow, I
> >>> often run into bugs that are not reproducible.)
> >>>
> >>> In case this is system dependent:
> >>> * OSX 10.6
> >>> * Python 2.6.1 (system install)
> >>> * matplotlib master
> >>> * ffmpeg 0.10
> >>>
> >>> Simple plots seem to work fine, but I should probably try to reproduce
> >>> using something other than streamplot (maybe create the equivalent
> number of
> >>> line and arrow collections).
> >>>
> >>> -Tony
> >>>
> >>> #~~~ Example
> >>> import numpy as np
> >>> import matplotlib.pyplot as plt
> >>> import matplotlib.animation as animation
> >>>
> >>> fig, ax = plt.subplots()
> >>>
> >>> # do nothing function that prints frame
> >>> def update(frame_num):
> >>> print frame_num
> >>> return []
> >>>
> >>> Y, X = np.mgrid[:1:100j, :1:100j]
> >>> U = np.ones_like(X)
> >>> V = np.ones_like(X)
> >>> ax.streamplot(X, Y, U, V, density=1)
> >>>
> >>> ani = animation.FuncAnimation(fig, update, save_count=500)
> >>>
> >>> # save hangs at frame 173 (this probably varies by system, if it's
> >>> reproducible).
> >>> ani.save('animation.avi')
> >>> # show animates all frames w/o issue.
> >>> #plt.show()
> >>>
> >>>
> >>
> >> I finally had some time to revisit this issue. It seems that subprocess
> >> PIPEs have fairly limited buffers [1] (also see docs for
> >> `subprocess.Popen.communicate` [2]) and ffmpeg tends to generate a
> decent
> >> amount of output to stderr. A simple solution is to redirect stderr to a
> >> temporary file. This change fixes the issue on my system.
> >>
> >> If this bug is reproducible, I can submit a PR. The temporary file here
> >> could be created using tempfile so it gets cleaned up automatically, or
> >> maybe a more permanent log file for debugging. Thoughts?
> >>
> >> -Tony
> >>
> >> [1]
> >>
> http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/
> >>
> >> [2]
> >>
> http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
> >>
> >
> > And just to clarify: My original email mentioned that saving would hang
> only
> > when `streamplot` was used to create the figure. It turns out that ffmpeg
> > prints more output (keyframes?) for more complex images, so when I ran
> > simpler plots, they didn't produce enough output to fill the subprocess
> > `PIPE` (although they would eventually).
> >
> > Also, theres's a simpler fix than piping ffmpeg output to a file: Just
> set
> > `-loglevel quiet` in the ffmpeg command.
> >
> > -Tony
>
> It's not a bad idea to have it logged to a file in a temp directory.
> We could potentially just do this if debug is set to verbose, however,
> and just use -loglevel quiet otherwise. Can you manage PR for this or
> do I need to?
>
> Ryan
>
>
Hey Ryan,

If you have time, that'd be great. Otherwise, I should have some time at
the end of the week to submit a PR.

-Tony
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] Strange bug when saving animations (with streamplot)

[Ryan May]

On Tue, Jul 3, 2012 at 10:03 AM, Tony Yu  wrote:
>
>
> On Tue, Jul 3, 2012 at 10:22 AM, Ryan May  wrote:
>>
>> On Tue, Jul 3, 2012 at 9:14 AM, Tony Yu  wrote:
>> > On Mon, Jul 2, 2012 at 11:42 PM, Tony Yu  wrote:
>> >> On Sun, Jun 24, 2012 at 10:02 AM, Tony Yu  wrote:
>> >>>
>> >>> I'm having a tough time figuring this out: Saving an animation seems
>> >>> to
>> >>> hang when using `streamplot`. The exact same animation runs without
>> >>> issue
>> >>> when calling show.
>> >>>
>> >>> On my system, the example below hangs consistently at frame 173.
>> >>> However,
>> >>> the number of saved frames (before hanging) varies with density of
>> >>> stream
>> >>> lines (i.e. number of line segments in streamplot): More frames saved
>> >>> for
>> >>> lower density.
>> >>>
>> >>> Seems to be a memory or file-size issue, but
>> >>>
>> >>>* Memory use doesn't seem to grow. (this suggests that ffmpeg adds
>> >>> frames to disk as opposed to memory.)
>> >>>* Saving the animation up to the frame just before it hangs gives
>> >>> modest (~1MB) videos.
>> >>> Maybe this is a limitation of `ffmpeg`?
>> >>>
>> >>> Can someone confirm that the example hangs on their system? (Somehow,
>> >>> I
>> >>> often run into bugs that are not reproducible.)
>> >>>
>> >>> In case this is system dependent:
>> >>> * OSX 10.6
>> >>> * Python 2.6.1 (system install)
>> >>> * matplotlib master
>> >>> * ffmpeg 0.10
>> >>>
>> >>> Simple plots seem to work fine, but I should probably try to reproduce
>> >>> using something other than streamplot (maybe create the equivalent
>> >>> number of
>> >>> line and arrow collections).
>> >>>
>> >>> -Tony
>> >>>
>> >>> #~~~ Example
>> >>> import numpy as np
>> >>> import matplotlib.pyplot as plt
>> >>> import matplotlib.animation as animation
>> >>>
>> >>> fig, ax = plt.subplots()
>> >>>
>> >>> # do nothing function that prints frame
>> >>> def update(frame_num):
>> >>> print frame_num
>> >>> return []
>> >>>
>> >>> Y, X = np.mgrid[:1:100j, :1:100j]
>> >>> U = np.ones_like(X)
>> >>> V = np.ones_like(X)
>> >>> ax.streamplot(X, Y, U, V, density=1)
>> >>>
>> >>> ani = animation.FuncAnimation(fig, update, save_count=500)
>> >>>
>> >>> # save hangs at frame 173 (this probably varies by system, if it's
>> >>> reproducible).
>> >>> ani.save('animation.avi')
>> >>> # show animates all frames w/o issue.
>> >>> #plt.show()
>> >>>
>> >>>
>> >>
>> >> I finally had some time to revisit this issue. It seems that subprocess
>> >> PIPEs have fairly limited buffers [1] (also see docs for
>> >> `subprocess.Popen.communicate` [2]) and ffmpeg tends to generate a
>> >> decent
>> >> amount of output to stderr. A simple solution is to redirect stderr to
>> >> a
>> >> temporary file. This change fixes the issue on my system.
>> >>
>> >> If this bug is reproducible, I can submit a PR. The temporary file here
>> >> could be created using tempfile so it gets cleaned up automatically, or
>> >> maybe a more permanent log file for debugging. Thoughts?
>> >>
>> >> -Tony
>> >>
>> >> [1]
>> >>
>> >> http://thraxil.org/users/anders/posts/2008/03/13/Subprocess-Hanging-PIPE-is-your-enemy/
>> >>
>> >> [2]
>> >>
>> >> http://docs.python.org/library/subprocess.html#subprocess.Popen.communicate
>> >>
>> >
>> > And just to clarify: My original email mentioned that saving would hang
>> > only
>> > when `streamplot` was used to create the figure. It turns out that
>> > ffmpeg
>> > prints more output (keyframes?) for more complex images, so when I ran
>> > simpler plots, they didn't produce enough output to fill the subprocess
>> > `PIPE` (although they would eventually).
>> >
>> > Also, theres's a simpler fix than piping ffmpeg output to a file: Just
>> > set
>> > `-loglevel quiet` in the ffmpeg command.
>> >
>> > -Tony
>>
>> It's not a bad idea to have it logged to a file in a temp directory.
>> We could potentially just do this if debug is set to verbose, however,
>> and just use -loglevel quiet otherwise. Can you manage PR for this or
>> do I need to?
>>
>> Ryan
>>
>
> Hey Ryan,
>
> If you have time, that'd be great. Otherwise, I should have some time at the
> end of the week to submit a PR.
>
> -Tony
>

I might be able to squeeze it in. If you don't see anything by the end
of the week, hit me up.

Ryan

-- 
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Re: [matplotlib-devel] v1.1.1rc2 tarballs are up

[Christoph Gohlke]

On 6/30/2012 1:24 PM, Christoph Gohlke wrote:
> On 6/30/2012 12:55 PM, John Hunter wrote:
>>
>>
>> On Sat, Jun 30, 2012 at 2:40 PM, Fernando Perez > > wrote:
>>
>>  On Sat, Jun 30, 2012 at 11:46 AM, John Hunter >  > wrote:
>>  > Well, looks like we better get moving then ;-)
>>
>>  Go MPL!  It would be great to have matching releases of IPython and
>>  MPL, just in time for the Debian freeze and SciPy 2012 :)
>>
>>
>> OK, the v1.1.1 tarball is up at
>> https://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-1.1.1
>> and this is now the download folder the main site points to.  I'm
>> leaving up the rc2 binaries til Russell and Christoph can build v1.1.1
>> binaries and we get them uploaded.  Sandro, if you're around, you are
>> good to go for including this in debian, hopefully squeaking in under
>> the freeze (sorry for the last minute push).
>>
>> I will hold off on the users and announce list emails til the updated
>> binaries are up.
>>
>> Tagged: git tag v1.1.1 7e47149a7b05f8e5cf1cc899a7e4e7c90dd4244f
>>
>> Thanks to all!
>> JDH
>
> Here are the Windows installers, built against numpy 1.6.2. Sorry, I can
> not upload them to SF. There seems to be some permission problems that
> the SF admins would need to fix manually.
>
> 
>
> Christoph
>

Sourceforge has fixed the permissions. I uploaded the Windows installers.

Christoph

--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel