On Fri, Oct 17, 2014 at 11:23 AM, Weeble <clockworksa...@gmail.com> wrote: > I'm having some trouble with flake8 when it's invoked from another > process, such as in vim. Sometimes its output goes missing - I just > get back an empty string. The problem vanishes if I > PYTHONUNBUFFERED=1, so I suspect it is related to how flake8 creates > multiple processes. I looked through the code and could see that it > creates multiprocessing sub-processes with daemon=True. Since these > sub-processes perform output on stdout, I believe they need to flush > it before telling the main thread that they are done, otherwise there > is a risk their output will be lost entirely if they are terminated > when the main process exits. > > I find my problem goes away if I add "sys.stdout.flush()" to the > finally block in reporter.py's process_main(). I think this is a good > thing to do for the reason I described above. > > However, I'm not *entirely* sure I understand what's going on. Every > single time I run flake8 from the command-line, all its output appears > as normal. So while there's a theoretical possibility that output > buffers might not be flushed, that doesn't actually seem to be what's > happening here. My suspicion was that the parent process was exiting > first, then the calling process (vim using the system() function) saw > it had exited and stopped reading its output, then the child process > flushed its output, but it was too late. That would explain why output > appears on the command-line, but is sometimes missed by vim. But I > can't actually figure out a way for this to happen. The parent process > should always terminate its daemon children as part of its exit > handler. So as far as I can tell they should all be dead by the time > it exits, and either they have flushed or their output is lost. > > I posted about my problems on the Syntastic google group. You can find > a bit more detail about my original problem and what I tried there: > > https://groups.google.com/forum/#!topic/vim-syntastic/Pdnp43Ok-3Y > > Does anyone better understand what might be going on here? > > Regards, > > Weeble.
Hey Weeble, I'm rather certain you've hit the nail directly on the head. The problem here is the default use of multiprocessing. If you're open to sending a PR to fix this (since you seem to have found a solution, if I understand you correctly), I'll happily and gratefully merge it and cut a new minor release for this. I'm sincerely sorry you've run into this problem. In case you missed the announcement earlier this month, the repository is https://gitlab.com/pycqa/flake8 now. Cheers, Ian _______________________________________________ code-quality mailing list code-quality@python.org https://mail.python.org/mailman/listinfo/code-quality
