Re: [Python-Dev] Reading Python source file

On 18 November 2015 at 15:57, Hrvoje Niksic wrote: > If this never really worked in Python, feel free to drop the issue. I may be > misremembering the language in which scripts I saw using this techniques > years ago were written - most likely sh or Perl. It was Perl. In

Re: [Python-Dev] Reading Python source file

On 17.11.2015 16:22, Guido van Rossum wrote: > On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg wrote: >>> [moving from read source line by line to reading all in one go] >> We use the same simplification in eGenix PyRun's emulation of >> the Python command line interface and it

Re: [Python-Dev] Reading Python source file

Yeah, let's kill this undefined behavior. On Thu, Nov 19, 2015 at 4:10 AM, Chris Angelico wrote: > On Thu, Nov 19, 2015 at 10:51 PM, Serhiy Storchaka > wrote: >> http://bugs.python.org/issue20115 > > Interestingly, the file linked in the last comment on

Re: [Python-Dev] Reading Python source file

On Thu, Nov 19, 2015 at 3:51 AM, Serhiy Storchaka wrote: > On 17.11.15 18:50, Guido van Rossum wrote: >> >> On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka >> wrote: >>> >>> Current implementation of import system went the same way. As a result >>>

Re: [Python-Dev] Reading Python source file

On Thu, Nov 19, 2015 at 10:51 PM, Serhiy Storchaka wrote: > http://bugs.python.org/issue20115 Interestingly, the file linked in the last comment on that issue [1] ties in with another part of this thread, regarding binary blobs in Python scripts. It uses

Re: [Python-Dev] Reading Python source file

On 17.11.15 18:50, Guido van Rossum wrote: On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka wrote: Current implementation of import system went the same way. As a result importing the script as a module and running it with command line can have different behaviours in

Re: [Python-Dev] Reading Python source file

On Thu, Nov 19, 2015 at 1:47 AM, M.-A. Lemburg wrote: > On 17.11.2015 16:22, Guido van Rossum wrote: >> On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg wrote: [moving from read source line by line to reading all in one go] >>> We use the same simplification

Re: [Python-Dev] Reading Python source file

Well, not quite the same thing, but https://github.com/kirbyfan64/pfbuild/blob/master/pfbuild embeds the compressed version of 16k LOC. Would it be affected negatively in any way be this? Since all the data is on one line, I'd think the old (current) parser would end up reading in the whole

Re: [Python-Dev] Reading Python source file

On Wed, Nov 18, 2015 at 4:15 AM, Hrvoje Niksic wrote: > On 11/18/2015 03:31 AM, Nick Coghlan wrote: >> >> That behaviour is then inherited at the command line by both the -m >> switch and the support for executing directories and zip archives. >> When we consider that the

Re: [Python-Dev] Reading Python source file

On 11/18/2015 04:48 PM, Guido van Rossum wrote: That trick doesn't work unless the data looks like Python comments or data (e.g. a docstring). Python has always insisted on being able to parse until EOF. The only extreme case would be a small script followed by e.g. 4 GB of comments (where the

Re: [Python-Dev] Reading Python source file

On 11/18/2015 03:31 AM, Nick Coghlan wrote: That behaviour is then inherited at the command line by both the -m switch and the support for executing directories and zip archives. When we consider that the "-c" switch also executes an in-memory string, direct script execution is currently the odd

Re: [Python-Dev] Reading Python source file

On 19 November 2015 at 02:50, Ryan Gonzalez wrote: > Well, not quite the same thing, but > https://github.com/kirbyfan64/pfbuild/blob/master/pfbuild embeds the > compressed version of 16k LOC. Would it be affected negatively in any way be > this? > > Since all the data is on one

Re: [Python-Dev] Reading Python source file

On 17.11.15 17:22, Guido van Rossum wrote: But more important is the interactive REPL, which parses your input fully each time you hit ENTER. Interactive REPL runs different code. It is simpler that the code for reading from file, because it have no care about BOM or coding cookie.

Re: [Python-Dev] Reading Python source file

On 17.11.15 18:06, Guido van Rossum wrote: OK, but what are you going to do about the interactive REPL? Nothing (except some simplification). This is a separate branch of the code. ___ Python-Dev mailing list Python-Dev@python.org

Re: [Python-Dev] Reading Python source file

OK, but what are you going to do about the interactive REPL? On Tue, Nov 17, 2015 at 7:40 AM, Serhiy Storchaka wrote: > On 17.11.15 05:00, Guido van Rossum wrote: >> >> If you free the memory used for the source buffer before starting code >> generation you should be good. >

Re: [Python-Dev] Reading Python source file

On 17.11.15 05:05, MRAB wrote: As I understand it, *nix expects the shebang to be b'#!', which means that the first line should be ASCII-compatible (it's possible that the UTF-8 BOM might be present). This kind of suggests that encodings like UTF-16 would cause a problem on such systems. The

Re: [Python-Dev] Reading Python source file

On 17.11.15 11:59, M.-A. Lemburg wrote: I don't think these situations are all that common, though, so reading in the full source code before compiling it sounds like a reasonable approach. We use the same simplification in eGenix PyRun's emulation of the Python command line interface and it

Re: [Python-Dev] Reading Python source file

On Tue, Nov 17, 2015 at 1:59 AM, M.-A. Lemburg wrote: > On 17.11.2015 02:53, Serhiy Storchaka wrote: >> I'm working on rewriting Python tokenizer (in particular the part that reads >> and decodes Python >> source file). The code is complicated. For now there are such cases: >>

Re: [Python-Dev] Reading Python source file

On 17.11.15 05:00, Guido van Rossum wrote: If you free the memory used for the source buffer before starting code generation you should be good. Thank you. The buffer is freed just after the end of generating AST. On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka wrote:

Re: [Python-Dev] Reading Python source file

On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka wrote: > On 17.11.15 11:59, M.-A. Lemburg wrote: >> >> I don't think these situations are all that common, though, >> so reading in the full source code before compiling it >> sounds like a reasonable approach. >> >> We use

Re: [Python-Dev] Reading Python source file

Oh, cool! Sorry for the disturbance. On Tue, Nov 17, 2015 at 8:27 AM, Serhiy Storchaka wrote: > On 17.11.15 18:06, Guido van Rossum wrote: >> >> OK, but what are you going to do about the interactive REPL? > > > Nothing (except some simplification). This is a separate branch

Re: [Python-Dev] Reading Python source file

Aha, so the only code path that's being replaced is the code that reads the script file when invoking "python FILE" or "python

Re: [Python-Dev] Reading Python source file

On 18 November 2015 at 02:50, Guido van Rossum wrote: > On Tue, Nov 17, 2015 at 8:20 AM, Serhiy Storchaka wrote: >> On 17.11.15 11:59, M.-A. Lemburg wrote: >>> >>> I don't think these situations are all that common, though, >>> so reading in the full source

Re: [Python-Dev] Reading Python source file

On 17.11.2015 02:53, Serhiy Storchaka wrote: > I'm working on rewriting Python tokenizer (in particular the part that reads > and decodes Python > source file). The code is complicated. For now there are such cases: > > * Reading from the string in memory. > * Interactive reading from the file.

Re: [Python-Dev] Reading Python source file

MRAB writes: > As I understand it, *nix expects the shebang to be b'#!', > which means that the first line should be ASCII-compatible > (it's possible that the UTF-8 BOM might be present). The UTF-8 BOM interferes with it on Mac OSX and Linux, at least.

Re: [Python-Dev] Reading Python source file

If you free the memory used for the source buffer before starting code generation you should be good. On Mon, Nov 16, 2015 at 5:53 PM, Serhiy Storchaka wrote: > I'm working on rewriting Python tokenizer (in particular the part that reads > and decodes Python source file).

Re: [Python-Dev] Reading Python source file

On 2015-11-17 01:53, Serhiy Storchaka wrote: I'm working on rewriting Python tokenizer (in particular the part that reads and decodes Python source file). The code is complicated. For now there are such cases: * Reading from the string in memory. * Interactive reading from the file. * Reading