Re: Write this accumuator in a functional style

[Gregory Ewing]

Steve D'Aprano wrote:
- Greg's dict comprehension version requires N+1 passes through the data, 
  one to convert to a list, and 1 per each possible key.


Just to be clear, my solution was a response to the requirement
that it be written in a purely functional style. It's not now I
would actually recommend doing it!

While a purely functional single-pass solution is possible,
in Python it would probably be just as inefficient, maybe
even worse.

--
Greg
--
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Steve D'Aprano]

On Tue, 11 Jul 2017 04:58 pm, Chris Angelico wrote:

> On Tue, Jul 11, 2017 at 4:11 PM, Steven D'Aprano  wrote:
[...]
>> accumulator = {'blue': [], 'green': [], 'red': []}
>> for parrot in parrots:
>> accumulator[parrot.colour].append(parrot)
[...]

> It's a partitioning filter. (Three way, not the usual two, but same
> same.) I've actually often wanted a quick way to write that - where
> you divide a list into two according to "passes predicate" vs "fails
> predicate". So if you find a really nice solution, I'm interested.

That is one of my colleague's complaints too: he says this is a common task and
there ought to be a built-in or at least std lib functional solution for it,
akin to map/filter/itertools.

Although I often say "not every two or three line function needs to be a
built-in", in this case I'm inclined to agree with him. Now that I have a name
for it, I am even more inclined to agree.

It's an N-way partitioning filter. My example shows only three, but the real
code we're using has more than that.

def partition(iterable, keyfunc=bool):
accumulator = {}
for item in iterable:
accumulator.setdefault(keyfunc(item), []).append(item)
return accumulator



Alternatives:

- itertools.groupby requires you to sort the entire input stream 
  before starting; that's expensive, O(N log N) rather than just O(N).

- Greg's dict comprehension version requires N+1 passes through the data, 
  one to convert to a list, and 1 per each possible key.

- Terry's solution scares me :-)

- Alain's solution appears to require list concatenation, which implies 
  that in the worst case this will be O(N**2).

Any other thoughts?



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Compiling Python 3.6.1 on macOS 10.12.5

[Pavol Lisy]

On 7/10/17, Nigel Palmer  wrote:
> Hi
>
> I am trying to compile Python 3.6.1  on macOS 10.12.5 with xcode 8.8.3 using
> the instructions at
> https://docs.python.org/devguide/setup.html#build-dependencies but I am
> getting the error
>
> ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
> if test $? -ne 0 ; then \
> echo "generate-posix-vars failed" ; \
> rm -f ./pybuilddir.txt ; \
> exit 1 ; \
> fi
> /bin/sh: line 1: 96973 Killed: 9   ./python.exe -E -S -m
> sysconfig --generate-posix-vars
> generate-posix-vars failed
> make: *** [pybuilddir.txt] Error 1
>
> When I manually run that command using dbll I get:
>
> lldb ./python.exe -- -E -S -m sysconfig --generate-posix-vars
> (lldb) target create "./python.exe"
> Current executable set to './python.exe' (x86_64).
> (lldb) settings set -- target.run-args  "-E" "-S" "-m" "sysconfig"
> "--generate-posix-vars"
> (lldb) r
> Process 96978 launched: './python.exe' (x86_64)
> Could not find platform dependent libraries 
> Consider setting $PYTHONHOME to [:]
> Process 96978 exited with status = 0 (0x)
> (lldb)
>
>
> The commands I ran to configure and build python are:
> brew install openssl xz
> CPPFLAGS="-I$(brew --prefix openssl)/include" LDFLAGS="-L$(brew --prefix
> openssl)/lib" ./configure --prefix=`pwd`/../build
> make
>
> Any ideas on what I am doing wrong?
>
> Many Thanks
> Nigel
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Something like this is working?

python  -c "from distutils.sysconfig  import get_config_var as gv; px
= 'prefix' ;epx = 'exec_prefix' ;LIBDEST = 'LIBDEST' ;LD = 'LIBDIR'
;print(f'LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:{gv(LD)}
PYTHONPATH=\$PYTHONPATH:{gv(LIBDEST)}
PYTHONHOME=\$PYTHONHOME:{gv(px)}:{gv(epx)} ')"

or maybe python.exe in your case ->

python.exe  -c "from distutils.sysconfig  import get_config_var as gv;
px = 'prefix' ;epx = 'exec_prefix' ;LIBDEST = 'LIBDEST' ;LD = 'LIBDIR'
;print(f'LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:{gv(LD)}
PYTHONPATH=\$PYTHONPATH:{gv(LIBDEST)}
PYTHONHOME=\$PYTHONHOME:{gv(px)}:{gv(epx)} ')"
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Pavol Lisy]

On 7/11/17, Steven D'Aprano  wrote:
> I have a colleague who is allergic to mutating data structures. Yeah, I
> know, he needs to just HTFU but I thought I'd humour him.
>
> Suppose I have an iterator that yields named tuples:
>
> Parrot(colour='blue', species='Norwegian', status='tired and shagged out')
>
> and I want to collect them by colour:
>
> accumulator = {'blue': [], 'green': [], 'red': []}
> for parrot in parrots:
> accumulator[parrot.colour].append(parrot)
>
>
> That's pretty compact and understandable, but it require mutating a bunch
> of pre-allocated lists inside an accumulator. Can we re-write this in a
> functional style?
>
> The obvious answer is "put it inside a function, then pretend it works by
> magic" but my colleague's reply to that is "Yes, but I'll know that its
> actually doing mutation inside the function".
>
>
> Help me humour my colleague.

This seems to be philosophical question to me:

How to (or "what colleague could accept" or "what do you mean by")
collect elements if not adding them to lists?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Test 0 and false since false is 0

[Steve D'Aprano]

On Tue, 11 Jul 2017 11:16 pm, Albert-Jan Roskam wrote:

> >>> False == 0
> True
> >>> False is 0
> False
> 
> 
> => Just wondering: Is this 'is' test depending on an implementation detail
> of cPython (small ints, I forgot how small 0-255 maybe, are  singletons)?

No. But the test 0 is 0 will be.

True and False are guaranteed singletons: there will only ever be a single
builtin False.

The small ints including 0 *may* be cached, so that they will be singletons. The
definition of "small" will vary from version to version.



-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Better Regex and exception handling for this small code

[Cameron Simpson]

On 11Jul2017 22:01, Ganesh Pal  wrote:

I am trying to open a file and check if there is a pattern  has  changed
after the task got completed?

file data:


#tail -f /file.txt
..
Note: CRC:algo = 2, split_crc = 1, unused = 0, initiator_crc = b6b20a65,
journal_crc = d2097b00
Note: Task completed successfully.
Note: CRC:algo = 2, split_crc = 1, unused = 0, initiator_crc = d976d35e,
journal_crc = a176af10


I  have the below piece of code  but would like to make this better more
pythonic , I found regex pattern and exception handling poor here , any
quick suggestion in your spare time is welcome.


#open the existing file if the flag is set and check if there is a match

log_file='/file.txt'
flag_is_on=1


Use "True" instead of "1". A flag is a Boolean thing, and should use a Boolean 
value. This lets you literally speak "true" and 'false" rather than imoplicitly 
saying that "0 means false and nonzero means true".



data = None


There is no need to initialise data here because you immediately overwrite it 
below.



with open(log_file, 'r') as f:
data = f.readlines()

if flag_is_on:


Oh yes. Just name this variable "flag". "_is_on" is kind of implicit.


   logdata = '\n'.join(data)


Do other parts of your programme deal with the file data as lines? If not, 
there is little point to reading the file and breaking it up into lines above, 
then joining them together against here. Just go:


 with open(log_file) as f:
 log_data = f.read()


   reg = "initiator_crc =(?P[\s\S]*?), journal_crc"


Normally we write regular expressions as "raw" python strings, thus:

   reg = r'initiator_crc =(?P[\s\S]*?), journal_crc'

because backslashes etc are punctuation inside normal strings. Within a "raw" 
string started with r' nothing is special until the closing ' character. This 
makes writing regular expressions more reliable.


Also, why the character range "[\s\S]"? That says whitespace or nonwhitespace 
i.e. any character. If you want any character, just say ".".



   crc = re.findall(re.compile(reg), logdata)


It is better to compile a regexp just the once, getting a Regexp object, and 
then you just use the compiled object.



   if not crc:
   raise Exception("Pattern not found in  logfile")


ValueError would be a more appropriate exception here; plain old "Exception" is 
pretty vague.



   checksumbefore = crc[0].strip()
   checksumafter = crc[1].strip()


Your regexp cannot start or end with whitespace. Those .strip calls are not 
doing anything for you.


This reads like you expect there to be exactly 2 matches in the file. What if 
there are more or fewer?



   logging.info("checksumbefore :%s and  checksumafter:%s"
 % (checksumbefore, checksumafter))

   if checksumbefore == checksumafter:
  raise Exception("checksum not macthing")


Don't you mean != here?

I wouldn't be raising exceptions in this code. Personally I would make this a 
function that returns True or False. Exceptions are a poor way of returning 
"status" or other values. They're really for "things that should not have 
happened", hence their name.


It looks like you're scanning a log file for multiple lines and wanting to know 
if successive ones change. Why not write a function like this (untested):


 RE_CRC_LINE = re.compile(r'initiator_crc =(?P[\s\S]*?), journal_crc')

 def check_for_crc_changes(logfile):
 old_crc_text = ''
 with open(logfile) as f:
 for line in f:
 m = RE_CRC_LINE.match(line)
 if not m:
 # uninteresting line
 continue
 crc_text = m.group(0)
 if crc_text != old_crc_text:
 # found a change
 return True
 if old_crc_text == '':
 # if this is really an error, you might raise this exception
 # but maybe no such lines is just normal but boring
 raise ValueError("no CRC lines seen in logfile %r" % (logfile,))
 # found no changes
 return False

See that there is very little sanity checking. In an exception supporting 
language like Python you can often write code as if it will always succeed by 
using things which will raise exceptions if things go wrong. Then _outside_ the 
function you can catch any exceptions that occur (such as being unable to open 
the log file).


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: ezdxf type of spline

[Pavol Lisy]

It seems to be:

http://pythonhosted.org/ezdxf/entities.html?highlight=spline#Spline

->

https://knowledge.autodesk.com/support/autocad/learn-explore/caas/CloudHelp/cloudhelp/2015/ENU/AutoCAD-Core/files/GUID-58316136-30EB-499C-ACAD-31D0C653B2B2-htm.html

->

https://en.wikipedia.org/wiki/Non-uniform_rational_B-spline

On 7/10/17, amka1...@gmail.com  wrote:
> Hi,
>
> Can someone says please to me which kind are the splines of the ezdxf python
> module ? Is it bezier curves ?
>
> Thanks,
>
> dylan
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

RE: Compiling Python 3.6.1 on macOS 10.12.5

[Nigel Palmer]

Hi

The python process only goes to around 4.8 MB before it dies and the machine 
has 4GB of RAM so I do not think it’s a memory issue.

Thanks,
Nigel

-Original Message-
From: INADA Naoki [mailto:songofaca...@gmail.com] 
Sent: 11 July 2017 02:57
To: Nigel Palmer 
Cc: python-list@python.org
Subject: Re: Compiling Python 3.6.1 on macOS 10.12.5

>  Killed: 9

It looks like not segmentation fault.
Maybe, RAM shortage?
INADA Naoki  


On Mon, Jul 10, 2017 at 10:24 PM, Nigel Palmer  wrote:
> Hi
>
> I am trying to compile Python 3.6.1  on macOS 10.12.5 with xcode 8.8.3 
> using the instructions at 
> https://docs.python.org/devguide/setup.html#build-dependencies but I 
> am getting the error
>
> ./python.exe -E -S -m sysconfig --generate-posix-vars ;\
> if test $? -ne 0 ; then \
> echo "generate-posix-vars failed" ; \
> rm -f ./pybuilddir.txt ; \
> exit 1 ; \
> fi
> /bin/sh: line 1: 96973 Killed: 9   ./python.exe -E -S -m 
> sysconfig --generate-posix-vars
> generate-posix-vars failed
> make: *** [pybuilddir.txt] Error 1
>
> When I manually run that command using dbll I get:
>
> lldb ./python.exe -- -E -S -m sysconfig --generate-posix-vars
> (lldb) target create "./python.exe"
> Current executable set to './python.exe' (x86_64).
> (lldb) settings set -- target.run-args  "-E" "-S" "-m" "sysconfig" 
> "--generate-posix-vars"
> (lldb) r
> Process 96978 launched: './python.exe' (x86_64) Could not find 
> platform dependent libraries  Consider setting 
> $PYTHONHOME to [:] Process 96978 exited with 
> status = 0 (0x)
> (lldb)
>
>
> The commands I ran to configure and build python are:
> brew install openssl xz
> CPPFLAGS="-I$(brew --prefix openssl)/include" LDFLAGS="-L$(brew 
> --prefix openssl)/lib" ./configure --prefix=`pwd`/../build make
>
> Any ideas on what I am doing wrong?
>
> Many Thanks
> Nigel
> --
> https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Ian Kelly]

On Tue, Jul 11, 2017 at 12:47 AM, Wolfgang Maier
 wrote:
> On 07/11/2017 08:11 AM, Steven D'Aprano wrote:
>>
>> I have a colleague who is allergic to mutating data structures. Yeah, I
>> know, he needs to just HTFU but I thought I'd humour him.
>>
>> Suppose I have an iterator that yields named tuples:
>>
>> Parrot(colour='blue', species='Norwegian', status='tired and shagged out')
>>
>> and I want to collect them by colour:
>>
>> accumulator = {'blue': [], 'green': [], 'red': []}
>> for parrot in parrots:
>>  accumulator[parrot.colour].append(parrot)
>>
>>
>> That's pretty compact and understandable, but it require mutating a bunch
>> of pre-allocated lists inside an accumulator. Can we re-write this in a
>> functional style?
>>
>> The obvious answer is "put it inside a function, then pretend it works by
>> magic" but my colleague's reply to that is "Yes, but I'll know that its
>> actually doing mutation inside the function".
>>
>>
>> Help me humour my colleague.
>>
>>
>>
>
> Hmm, isn't this just asking for itertools.groupby on the parrots sorted by
> colour?

That's one solution, but the sorting makes it O(n log n) for a task
that should really just be O(n).
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Better Regex and exception handling for this small code

[Ganesh Pal]

I am trying to open a file and check if the pattern  i.e  initiator_crc has
 changed after the task got completed? *

On Tue, Jul 11, 2017 at 10:01 PM, Ganesh Pal  wrote:

> Dear Python friends
>
> I am trying to open a file and check if there is a pattern  has  changed
> after the task got completed?
>
> file data:
> 
>
> #tail -f /file.txt
> ..
> Note: CRC:algo = 2, split_crc = 1, unused = 0, initiator_crc = b6b20a65,
> journal_crc = d2097b00
> Note: Task completed successfully.
> Note: CRC:algo = 2, split_crc = 1, unused = 0, initiator_crc = d976d35e,
> journal_crc = a176af10
>
>
>  I  have the below piece of code  but would like to make this better more
> pythonic , I found regex pattern and exception handling poor here , any
> quick suggestion in your spare time is welcome.
>
>
> #open the existing file if the flag is set and check if there is a match
>
> log_file='/file.txt'
> flag_is_on=1
>
> data = None
> with open(log_file, 'r') as f:
>  data = f.readlines()
>
>
> if flag_is_on:
> logdata = '\n'.join(data)
> reg = "initiator_crc =(?P[\s\S]*?), journal_crc"
> crc = re.findall(re.compile(reg), logdata)
> if not crc:
> raise Exception("Pattern not found in  logfile")
>
> checksumbefore = crc[0].strip()
> checksumafter = crc[1].strip()
> logging.info("checksumbefore :%s and  checksumafter:%s"
>   % (checksumbefore, checksumafter))
>
> if checksumbefore == checksumafter:
>raise Exception("checksum not macthing")
>
> I am  on Linux and Python 2.7
>
> Regards,
> Ganesh
>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Better Regex and exception handling for this small code

[Ganesh Pal]

Dear Python friends

I am trying to open a file and check if there is a pattern  has  changed
after the task got completed?

file data:


#tail -f /file.txt
..
Note: CRC:algo = 2, split_crc = 1, unused = 0, initiator_crc = b6b20a65,
journal_crc = d2097b00
Note: Task completed successfully.
Note: CRC:algo = 2, split_crc = 1, unused = 0, initiator_crc = d976d35e,
journal_crc = a176af10


 I  have the below piece of code  but would like to make this better more
pythonic , I found regex pattern and exception handling poor here , any
quick suggestion in your spare time is welcome.


#open the existing file if the flag is set and check if there is a match

log_file='/file.txt'
flag_is_on=1

data = None
with open(log_file, 'r') as f:
 data = f.readlines()


if flag_is_on:
logdata = '\n'.join(data)
reg = "initiator_crc =(?P[\s\S]*?), journal_crc"
crc = re.findall(re.compile(reg), logdata)
if not crc:
raise Exception("Pattern not found in  logfile")

checksumbefore = crc[0].strip()
checksumafter = crc[1].strip()
logging.info("checksumbefore :%s and  checksumafter:%s"
  % (checksumbefore, checksumafter))

if checksumbefore == checksumafter:
   raise Exception("checksum not macthing")

I am  on Linux and Python 2.7

Regards,
Ganesh
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Compiling Python 3.6.1 on macOS 10.12.5

[Viktor Hagström]

>Why are you trying to compile Python manually? You should use Homebrew to
>install Python in 99% of cases. (The package is python3)

I'm not the person you answered, but I can explain why I do things that are not 
"optimal" or "easy" or "best".

I am interested, I want to learn something, I think it's fun to do things the 
hard way. I want to learn how it *really* works. I want to modify it, break it, 
and fix it again. And I think it would be good for this community to encourage 
that (as long as it doesn't hinder their progress).

2017-07-11 8:48 GMT+02:00 Chris Warrick 
mailto:kwpol...@gmail.com>>:
Why are you trying to compile Python manually? You should use Homebrew to
install Python in 99% of cases. (The package is python3)

--
Chris Warrick 
--
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Test 0 and false since false is 0

[Grant Edwards]

On 2017-07-11, Albert-Jan Roskam  wrote:
> From: Python-list  on 
> behalf of Dan Sommers 
> Sent: Friday, July 7, 2017 2:46 AM
> To: python-list@python.org
> Subject: Re: Test 0 and false since false is 0
>     
> On Thu, 06 Jul 2017 19:29:00 -0700, Sayth Renshaw wrote:
>
>> I have tried or conditions of v == False etc but then the 0's being
>> false also aren't moved. How can you check this at once?
>
> Maybe this will help:
>
>     Python 3.5.3+ (default, Jun  7 2017, 23:23:48) 
>     [GCC 6.3.0 20170516] on linux
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>> False == 0
>     True
>     >>> False is 0
>     False
>
>
> Just wondering: Is this 'is' test depending on an implementation
> detail of cPython (small ints, I forgot how small 0-255 maybe, are
> singletons)?

No.

False is required to be a singleton.

Therefore, if you want to know if  is the boolean object
False, you can use ' is False' with predictable results..

Integer values are not required to be singletons, so you cannot depend
on the value of  is 0, or  is 12345678.  As you
mention, in the current version(s) of CPython, small integer values
are cached, but larger ones are not:

$ python
Python 2.7.12 (default, Jan  3 2017, 10:08:10) 
[GCC 4.9.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> x = 0
>>> x is 0
True
>>> x = 12345678
>>> x is 12345678
False
>>> 

That could change tomorrow.

-- 
Grant Edwards   grant.b.edwardsYow! PEGGY FLEMMING is
  at   stealing BASKET BALLS to
  gmail.comfeed the babies in VERMONT.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Test 0 and false since false is 0

[Rhodri James]

On 11/07/17 14:16, Albert-Jan Roskam wrote:

From: Python-list  on behalf 
of Dan Sommers 
Sent: Friday, July 7, 2017 2:46 AM
To: python-list@python.org
Subject: Re: Test 0 and false since false is 0
 
On Thu, 06 Jul 2017 19:29:00 -0700, Sayth Renshaw wrote:



I have tried or conditions of v == False etc but then the 0's being
false also aren't moved. How can you check this at once?


Maybe this will help:

 Python 3.5.3+ (default, Jun  7 2017, 23:23:48)
 [GCC 6.3.0 20170516] on linux
 Type "help", "copyright", "credits" or "license" for more information.
 >>> False == 0
 True
 >>> False is 0
 False


=> Just wondering: Is this 'is' test depending on an implementation detail 
of cPython (small ints, I forgot how small 0-255 maybe, are  singletons)?


No, it's not an implementation detail.  True and False are singletons, 
and are different objects from 1 and 0 (they behave differently when 
converted to strings, for example).  Now, relying on "0 is 0" being true 
would be relying on an implementation detail, but you weren't going to 
do that, were you?


--
Rhodri James *-* Kynesim Ltd
--
https://mail.python.org/mailman/listinfo/python-list

RE: Compiling Python 3.6.1 on macOS 10.12.5

[Nigel Palmer]

Hi Chris

I am planning on embedding Python into a C++ application and I wanted to have 
my own build of Python to do that. I know that eventually I will need to use 
--enable-shared or --enable-framework but for now I am trying to get a the 
simpler static build to compile first.

Thanks,
Nigel

From: Chris Warrick [mailto:kwpol...@gmail.com]
Sent: 11 July 2017 07:48
To: Nigel Palmer 
Cc: python-list@python.org
Subject: Re: Compiling Python 3.6.1 on macOS 10.12.5

Why are you trying to compile Python manually? You should use Homebrew to 
install Python in 99% of cases. (The package is python3)
--
Chris Warrick 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Test 0 and false since false is 0

[Albert-Jan Roskam]

From: Python-list  on 
behalf of Dan Sommers 
Sent: Friday, July 7, 2017 2:46 AM
To: python-list@python.org
Subject: Re: Test 0 and false since false is 0
    
On Thu, 06 Jul 2017 19:29:00 -0700, Sayth Renshaw wrote:

> I have tried or conditions of v == False etc but then the 0's being
> false also aren't moved. How can you check this at once?

Maybe this will help:

    Python 3.5.3+ (default, Jun  7 2017, 23:23:48) 
    [GCC 6.3.0 20170516] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> False == 0
    True
    >>> False is 0
    False


=> Just wondering: Is this 'is' test depending on an implementation detail 
of cPython (small ints, I forgot how small 0-255 maybe, are  singletons)?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Compiling Python 3.6.1 on macOS 10.12.5

[Viktor Hagström]

>Why are you trying to compile Python manually? You should use Homebrew to
>install Python in 99% of cases. (The package is python3)

I'm not the person you answered, but I can explain why I do things that are not 
"optimal" or "easy" or "best".

I am interested, I want to learn something, I think it's fun to do things the 
hard way. I want to learn how it *really* works. I want to modify it, break it, 
and fix it again. And I think it would be good for this community to encourage 
that (as long as it doesn't hinder their progress).

2017-07-11 8:48 GMT+02:00 Chris Warrick 
mailto:kwpol...@gmail.com>>:
Why are you trying to compile Python manually? You should use Homebrew to
install Python in 99% of cases. (The package is python3)

--
Chris Warrick 
--
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Can anybody help me retrieve how to retrieve output from this Python code below!

[Peter Otten]

ksatish@gmail.com wrote:

[snip code]

Wasn't there any documentation to go with that script? That's the preferable 
method to use software written by someone else ;)


Anyway -- First you have to undo what was probably changed by yourself:

$ diff -u json2csv_orig.py json2csv.py
--- json2csv_orig.py2017-07-11 15:15:06.527571509 +0200
+++ json2csv.py 2017-07-11 15:14:17.878514787 +0200
@@ -132,14 +132,6 @@
 
 return parser
 
-json_file = input("Type Json input file name: ")
-
-key_map = input("Type Key value : ")
-
-MultiLineJson2Csv(Json2Csv).init_parser()
-
-Json2Csv.load(json_file)
-
 
 if __name__ == '__main__':
 parser = init_parser()
@@ -159,4 +151,4 @@
 fileName, fileExtension = os.path.splitext(args.json_file.name)
 outfile = fileName + '.csv'
 
-loader.write_csv(filename=outfile, make_strings=args.strings)
+loader.write_csv(filename=outfile, make_strings=args.strings)

Then you have to create a file containing the data in json format, e. g.

$ cat data.json
[
["alpha", "beta", {"one": {"two": "gamma"}}],
["zeta", "eta", {"one": {"two": "theta"}}]
]

...and a file describing the conversion, also in json, like

$ cat key_map.json
{
   "map": [
   ["foo", "0"],
   ["bar", "1"],
   ["baz", "2.one.two"]
   ]
}

Now you can run the script, first to look at the command line help

$ python json2csv.py -h
usage: json2csv.py [-h] [-e] [-o OUTPUT_CSV] [--strings] json_file key_map

Converts JSON to CSV

positional arguments:
  json_file Path to JSON data file to load
  key_map   File containing JSON key-mapping file to load

optional arguments:
  -h, --helpshow this help message and exit
  -e, --each-line   Process each line of JSON file separately
  -o OUTPUT_CSV, --output-csv OUTPUT_CSV
Path to csv file to output
  --strings Convert lists, sets, and dictionaries fully to 
comma-separated strings.

and then to process your data:

$ python json2csv.py data.json key_map.json
INFO:root:[u'alpha', u'beta', {u'one': {u'two': u'gamma'}}]
INFO:root:[u'zeta', u'eta', {u'one': {u'two': u'theta'}}]

Finally, let's have a look at the resulting csv file:

$ cat data.csv
foo,bar,baz
alpha,beta,gamma
zeta,eta,theta


-- 
https://mail.python.org/mailman/listinfo/python-list

Can anybody help me retrieve how to retrieve output from this Python code below!

[ksatish . dtc]

try:
import unicodecsv as csv
except ImportError:
import csv

import json
import operator
import os
from collections import OrderedDict
import logging

logging.basicConfig(level=logging.DEBUG)

class Json2Csv(object):
"""Process a JSON object to a CSV file"""
collection = None

# Better for single-nested dictionaries
SEP_CHAR = ', '
KEY_VAL_CHAR = ': '
DICT_SEP_CHAR = '\r'
DICT_OPEN = ''
DICT_CLOSE = ''

# Better for deep-nested dictionaries
# SEP_CHAR = ', '
# KEY_VAL_CHAR = ': '
# DICT_SEP_CHAR = '; '
# DICT_OPEN = '{ '
# DICT_CLOSE = '} '

def __init__(self, outline):
self.rows = []

if not isinstance(outline, dict):
raise ValueError('You must pass in an outline for JSON2CSV to 
follow')
elif 'map' not in outline or len(outline['map']) < 1:
raise ValueError('You must specify at least one value for "map"')

key_map = OrderedDict()
for header, key in outline['map']:
splits = key.split('.')
splits = [int(s) if s.isdigit() else s for s in splits]
key_map[header] = splits

self.key_map = key_map
if 'collection' in outline:
self.collection = outline['collection']

def load(self, json_file):
self.process_each(json.load(json_file))

def process_each(self, data):
"""Process each item of a json-loaded dict
"""
if self.collection and self.collection in data:
data = data[self.collection]

for d in data:
logging.info(d)
self.rows.append(self.process_row(d))

def process_row(self, item):
"""Process a row of json data against the key map
"""
row = {}

for header, keys in self.key_map.items():
try:
row[header] = reduce(operator.getitem, keys, item)
except (KeyError, IndexError, TypeError):
row[header] = None

return row

def make_strings(self):
str_rows = []
for row in self.rows:
str_rows.append({k: self.make_string(val)
 for k, val in row.items()})
return str_rows

def make_string(self, item):
if isinstance(item, list) or isinstance(item, set) or isinstance(item, 
tuple):
return self.SEP_CHAR.join([self.make_string(subitem) for subitem in 
item])
elif isinstance(item, dict):
return self.DICT_OPEN + 
self.DICT_SEP_CHAR.join([self.KEY_VAL_CHAR.join([k, self.make_string(val)]) for 
k, val in item.items()]) + self.DICT_CLOSE
else:
return unicode(item)

def write_csv(self, filename='output.csv', make_strings=False):
"""Write the processed rows to the given filename
"""
if (len(self.rows) <= 0):
raise AttributeError('No rows were loaded')
if make_strings:
out = self.make_strings()
else:
out = self.rows
with open(filename, 'wb+') as f:
writer = csv.DictWriter(f, self.key_map.keys())
writer.writeheader()
writer.writerows(out)


class MultiLineJson2Csv(Json2Csv):
def load(self, json_file):
self.process_each(json_file)

def process_each(self, data, collection=None):
"""Load each line of an iterable collection (ie. file)"""
for line in data:
d = json.loads(line)
if self.collection in d:
d = d[self.collection]
self.rows.append(self.process_row(d))


def init_parser():
import argparse
parser = argparse.ArgumentParser(description="Converts JSON to CSV")
parser.add_argument('json_file', type=argparse.FileType('r'),
help="Path to JSON data file to load")
parser.add_argument('key_map', type=argparse.FileType('r'),
help="File containing JSON key-mapping file to load")
parser.add_argument('-e', '--each-line', action="store_true", default=False,
help="Process each line of JSON file separately")
parser.add_argument('-o', '--output-csv', type=str, default=None,
help="Path to csv file to output")
parser.add_argument(
'--strings', help="Convert lists, sets, and dictionaries fully to 
comma-separated strings.", action="store_true", default=True)

return parser

json_file = input("Type Json input file name: ")

key_map = input("Type Key value : ")

MultiLineJson2Csv(Json2Csv).init_parser()

Json2Csv.load(json_file)


if __name__ == '__main__':
parser = init_parser()
args = parser.parse_args()

key_map = json.load(args.key_map)
loader = None
if args.each_line:
loader = MultiLineJson2Csv(key_map)
else:
loader = Json2Csv(key_map)

loader.load(args.json_file)

outfile = args.output_csv
if outfile is None:
fileName, fileExtension = os.path.splitext(a

Re: Compiling Python 3.6.1 on macOS 10.12.5

[Viktor Hagström]

>Why are you trying to compile Python manually? You should use Homebrew to
>install Python in 99% of cases. (The package is python3)

I'm not the person you answered, but I can explain why I do things that are not 
"optimal" or "easy" or "best".

I am interested, I want to learn something, I think it's fun to do things the 
hard way. I want to learn how it *really* works. I want to modify it, break it, 
and fix it again. And I think it would be good for this community to encourage 
that (as long as it doesn't hinder their progress).

2017-07-11 8:48 GMT+02:00 Chris Warrick 
mailto:kwpol...@gmail.com>>:
Why are you trying to compile Python manually? You should use Homebrew to
install Python in 99% of cases. (The package is python3)

--
Chris Warrick 
--
https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Alain Ketterlin]

Steven D'Aprano  writes:

> I have a colleague who is allergic to mutating data structures. Yeah, I 
> know, he needs to just HTFU but I thought I'd humour him.
>
> Suppose I have an iterator that yields named tuples:
>
> Parrot(colour='blue', species='Norwegian', status='tired and shagged out')
>
> and I want to collect them by colour:
>
> accumulator = {'blue': [], 'green': [], 'red': []}
> for parrot in parrots:
> accumulator[parrot.colour].append(parrot)
>
>
> That's pretty compact and understandable, but it require mutating a bunch 
> of pre-allocated lists inside an accumulator. Can we re-write this in a 
> functional style?

Here is a sketch in OCaml-style (incomplete of course):

type color = Blue | Green | Red;;
type parrot = { c: color; ... };;

let rec collect list_of_parrots =
match list_of_parrots with
| nil -> (nil,nil,nil)
| h :: q ->
let b,g,r = collect q in
match h with
| {c=Blue}  -> (h::b,g,r)
| {c=Green} -> (b,h::g,r)
| {c=Red}   -> (b,g,h::r)
;;

The function returns a triple of lists in this case. The first match
drives list-traversal, the second selects between colors. Both can be
(sub-optimally) turned into cascades of "if", i.e., in Python:

def collect(list_of_parrots):
if not list_of_parrots:
return [],[],[]
else
b,g,r = collect(list_of_parrots[1:])
h = list_of_parrots[0]
if h.color == 'blue':
return ([h]+b,g,r)
elif h.color == 'green':
... and so on

-- Alain.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Terry Reedy]

On 7/11/2017 2:11 AM, Steven D'Aprano wrote:

I have a colleague who is allergic to mutating data structures. Yeah, I
know, he needs to just HTFU but I thought I'd humour him.

Suppose I have an iterator that yields named tuples:

Parrot(colour='blue', species='Norwegian', status='tired and shagged out')

and I want to collect them by colour:

accumulator = {'blue': [], 'green': [], 'red': []}
for parrot in parrots:
 accumulator[parrot.colour].append(parrot)


That's pretty compact and understandable, but it require mutating a bunch
of pre-allocated lists inside an accumulator. Can we re-write this in a
functional style?

The obvious answer is "put it inside a function, then pretend it works by
magic" but my colleague's reply to that is "Yes, but I'll know that its
actually doing mutation inside the function".


Help me humour my colleague.


To truly not mutate anything, not even hidden (as in a python list comp, 
which buries .append), replace list with linked-list and .append with 
head-linkage (Lisp's cons).  Something like


blue, green, red = None, None, None
for parrot in parrots:
color = parrot.color
if color == 'blue':
blue = (parrot, blue)
elif color = 'red':
red = (parrot, red)
elif color = 'green':
green = (parrot, green)
else:
raise ValueError(f'parrot {parrot} has unknown color {color}')

At this point, blue, green, and red are linked lists of parrots of the 
corresponding color.


Of course, for loops mutate the iterator.  Replace that with a tail 
recursive function.  To make this easy, the input 'parrots' should be a 
linked list, which is a recursive data structure.  Assignment is also 
form of mutation (of a namespace dict), so to be really strict, all the 
assignments should be replaced by function calls and returns.  That 
really requires the use of recursion.


Since this is a thought experiment we can make this easier:

Make accumulator a linked list, perhaps
 (('blue':None),(('green',None),(( 'red', None), None)))
Specify that parrots is also a linked list.
Now stipulate the we are using func_py, which compiles Python syntax 
such as

for parrot in parrots:
  accumulator[parrot.colour].append(parrot)
into a set of recursive functional functions, including tail recursive 
'for' such that for(parrots, accumulator)  returns a new linked-list.


Note that real functional language compilers do the opposite of this, 
compiling tail-recursive syntax into while loops.


--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Peter Otten]

Steven D'Aprano wrote:

> I have a colleague who is allergic to mutating data structures. Yeah, I
> know, he needs to just HTFU but I thought I'd humour him.
> 
> Suppose I have an iterator that yields named tuples:
> 
> Parrot(colour='blue', species='Norwegian', status='tired and shagged out')
> 
> and I want to collect them by colour:
> 
> accumulator = {'blue': [], 'green': [], 'red': []}
> for parrot in parrots:
> accumulator[parrot.colour].append(parrot)
> 
> 
> That's pretty compact and understandable, but it require mutating a bunch
> of pre-allocated lists inside an accumulator. Can we re-write this in a
> functional style?
> 
> The obvious answer is "put it inside a function, then pretend it works by
> magic" but my colleague's reply to that is "Yes, but I'll know that its
> actually doing mutation inside the function".
> 
> 
> Help me humour my colleague.

Wouldn't it be on your colleague to provide a competetive "functional" 
version?

However, reusing Gregory's sample data:

>>> def color(p): return p.color
... 
>>> {c: list(ps) for c, ps in groupby(
...  sorted(parrot_generator(), key=color), key=color)}
{'red': [Hawaiian/red/laid back], 'blue': [Norwegian/blue/tired and shagged 
out, Norwegian/blue/dead, French/blue/tres bon], 'green': 
[Portugese/green/perky, Italian/green/excited]}


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Write this accumuator in a functional style

[Chris Angelico]

On Tue, Jul 11, 2017 at 4:11 PM, Steven D'Aprano  wrote:
> I have a colleague who is allergic to mutating data structures. Yeah, I
> know, he needs to just HTFU but I thought I'd humour him.
>
> Suppose I have an iterator that yields named tuples:
>
> Parrot(colour='blue', species='Norwegian', status='tired and shagged out')
>
> and I want to collect them by colour:
>
> accumulator = {'blue': [], 'green': [], 'red': []}
> for parrot in parrots:
> accumulator[parrot.colour].append(parrot)
>
>
> That's pretty compact and understandable, but it require mutating a bunch
> of pre-allocated lists inside an accumulator. Can we re-write this in a
> functional style?
>
> The obvious answer is "put it inside a function, then pretend it works by
> magic" but my colleague's reply to that is "Yes, but I'll know that its
> actually doing mutation inside the function".

It's a partitioning filter. (Three way, not the usual two, but same
same.) I've actually often wanted a quick way to write that - where
you divide a list into two according to "passes predicate" vs "fails
predicate". So if you find a really nice solution, I'm interested.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list