Re: txtorcon 23.5.0

[Benjamin Schollnick]

> *Top-posting? Really?

In that case, yes, because the conversion did not depend on what was originally 
said.  Unlike this conversation.

> Actually, a very cursory look through the intarweb would have yielded this -

And that’s the point, if you want to me to consider your package, I should have 
some idea that it’s worth me investing more time into seeing if it can help me. 
 As I mentioned simply stating that it was a “Tor Control management API” would 
have been enough for me to say “That’s not helpful to me”.  On the other hand 
if it said “Django management API/Tool”, I would have gone to the git 
repository and investigated.

Effectively, posting a release notification is an advertisement for the 
package, as well as a notice saying “Here’s some updates”.

When done correctly, it can bring more people to use your package / software…

> While there are one or two graphical package managers (yumex and yumex-dnf 
> that I know of), have a go at the cli manager(s) that may live on your distro 
> and see if you can get pkg info about it. If you're not sure which one(s) you 
> have, try these - Yum, Dnf, Rpm,Apt, Apt-Get, Deb, pacman, dpkg, and zypper 
> for starters. *dpkg is my go-to pkgmgr of choice...
> 
> Also, the names of many linux-centric utility plugins, extensions, and other 
> goodies, while seemingly cryptic, are usually a prominent clue as to what the 
> package is used for. The world of *NIX has never been for the faint of heart, 
> but there are lots of very helping hands in its communities... :)

This has nothing to do with package managers, this has to do with “Why should I 
use this package” or “Should I use this package / software”.  If the release 
notes are being send to people that don’t already have the package then it’s 
worth the couple of sentences or a short paragraph to allow someone that is 
unfamiliar with the package to be able to see if they should investigate the 
package.

Cryptic names maybe cute, but if they are not descriptive, then they are not 
really that helpful other than being unique.

- Benjamin

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: txtorcon 23.5.0

[Benjamin Schollnick]

Meejah,

Sorry, but, would you please consider adding a short description to this of 
what txtorcon is.
It’s great that your announcing the update, and highlighting the changes…if I 
used it, I’d be upgrading it.

But I have absolutely no idea why I would want to investigate using it, because 
there is no description of the package.

Well, that’s not 100% true.  From my knowledge of twisted, I know it’s network 
based, and that’s it.  I can’t tell if it’s a server, or client application.

Even just simply adding your 10,000 feet summary:

txtorcon is an implementation of the control-spec 
 for Tor 
 using the Twisted 
 networking library for Python 
.

would have been extremely helpful.

I hope you will take this as the creative criticism that I am offering it as.  

- Benjamin


> On May 18, 2023, at 9:37 PM, mee...@meejah.ca wrote:
> 
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
> 
> I'm happy to announce txtorcon 23.5.0 with the following changes:
> 
>  * twisted.web.client.Agent instances now use the same HTTPS policy
>by default as twisted.web.client.Agent.  It is possible to
>override this policy with the tls_context_factory= argument, the
>equivalent to Agent's contextFactory=
>(Thanks to Itamar Turner-Trauring)
>  * Added support + testing for Python 3.11.
>  * No more ipaddress dependency
> 
> You can download the release from PyPI or GitHub (or of
> course "pip install txtorcon"):
> 
>  https://pypi.python.org/pypi/txtorcon/23.5.0
>  https://github.com/meejah/txtorcon/releases/tag/v23.5.0
> 
> Releases are also available from the hidden service:
> 
>  
> http://fjblvrw2jrxnhtg67qpbzi45r7ofojaoo3orzykesly2j3c2m3htapid.onion/txtorcon-23.5.0.tar.gz
>  
> http://fjblvrw2jrxnhtg67qpbzi45r7ofojaoo3orzykesly2j3c2m3htapid.onion/txtorcon-23.5.0.tar.gz.asc
> 
> You can verify the sha256sum of both by running the following 4 lines
> in a shell wherever you have the files downloaded:
> 
> cat < 93fd80a9dd505f698d0864fe93db8b6a9c1144b5feb91530820b70ed8982651c  
> dist/txtorcon-23.5.0.tar.gz
> 987f0a91184f98cc3f0a7eccaa42f5054063744d6ac15e325cfa666403214208  
> dist/txtorcon-23.5.0-py3-none-any.whl
> EOF
> 
> thanks,
> meejah
> -BEGIN PGP SIGNATURE-
> 
> iQFFBAEBCgAvFiEEnVor1WiOy4id680/wmAoAxKAaacFAmRm0DoRHG1lZWphaEBt
> ZWVqYWguY2EACgkQwmAoAxKAaae/+wgAw3gAm65npc7+yMdGFixNmCd+RUXorJq9
> Hy76hK3BWdtNIA6TZF20QFYs3CX5Vepa0vCJOK1N40fYgxoZTb1/828Zp6Zq2+Gn
> piJGvQ0Z1S95ww7lwSV77o67Xf7PozhLR+k7DaOdY8ugvLb/0Rdp15BykF5DWIo8
> PRgqB8uZ418ebmDLLrYtqYTdlcUMxFTji4CHXc4N55/2hVHiFiuFt59os6kJ3iG1
> u90lmQH8vbDyVF7N6tpgEAdWeb7OdgDbtzhVBdBWHrPg+vDO+UL7WZU8ZjDAcdEr
> YzzmK3fIiCH7ngG2E/VIebiJfrjAA9G4eZXltIm7VcWh5css9MXY1Q==
> =TeQp
> -END PGP SIGNATURE-
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: HTTP server benchmarking/load testing in Python

[Benjamin Schollnick]

> On Jan 26, 2023, at 11:02 AM, Grant Edwards  wrote:
> 
> On 2023-01-26, Thomas Passin  wrote:
>> On 1/25/2023 7:38 PM, Peter J. Holzer wrote:
>>> On 2023-01-25 16:30:56 -0500, Thomas Passin wrote:
 Great!  Don't forget what I said about potential overheating if you
 hit the server with as many requests as it can handle.
>>> 
>>> Frankly, if you can overheat a server by hitting it with HTTP requests,
>>> get better hardware and/or put it into a place with better airflow.
>>> 
>> 
>> Frankly, if you have a server-grade machine then well and good but if 
>> you are running a nice quiet consumer grade laptop - my development 
>> machine - you need to be careful.
> 
> A properly designed laptop with a non-broken OS will not overheat
> regardless of the computing load you throw at it. The fan might get
> annoying loud, but if it overheats either your hardware or OS needs
> to be fixed.

Exactly.  

But what he might be thinking about is Thermal Throttling, which I keep seeing 
people attribute
to overheating….  

Overheating is not thermal throttling, it’s the OS and CPU protecting 
themselves from overheating.
Usually because the manufacturer didn’t add enough cooling to keep the system 
cool enough with a continuous load.  (Which to be honest, almost no laptop 
designers do, because they assuming you are going to be having a spiky load 
instead…  

- Benjamin

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Automatic Gain Control in Python?

[Benjamin Schollnick]

> I doubt you even need to write any code to do that. Sox can normalize
> audio levels in files with a single command.

Correct.  

My phrasing was slightly misleading.

There are plenty of applications that should do that.  I was thinking of one 
that Marco Arment makes, but I can’t locate it for reference.

- Benjamin



> On May 29, 2022, at 3:18 PM, Grant Edwards  wrote:
> 
> On 2022-05-29, Benjamin Schollnick  wrote:
> 
>> Why not just right a 3rd party package to normalize the audio levels
>> in the digital file?  It’ll be faster, and probably easier than
>> trying to do it in real time…
> 
> I doubt you even need to write any code to do that. Sox can normalize
> audio levels in files with a single command.
> 
> http://sox.sourceforge.net/
> 
> --
> Grant
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Automatic Gain Control in Python?

[Benjamin Schollnick]

Well, maybe a different approach might be better?

https://www.amazon.com/LEMONKTV-Control-Device-Karaoke-Machine/dp/B07H49JB8S 
<https://www.amazon.com/LEMONKTV-Control-Device-Karaoke-Machine/dp/B07H49JB8S>

I’m unclear on how bringing the audio track into a computer to examine detect 
the audio level, and then adjusting the volume out of the Smart Speaker is 
really doing more than adding complexity.

An all analog solution might be the better route, although finding something 
that is inexpensive might be an issue as well.

- Benjamin



> On May 29, 2022, at 11:32 AM, Steve GS  wrote:
> 
> No, not a digital file. I am playing the podcast/broadcast live to the 
> community using a separate network of smart speakers (Amazon Echo). Commands 
> are being sent hourly through a speaker to the SS from an excel program. I 
> want to monitor the audio level between the line-out of the SS and the input 
> to another computer which then records the audio using Audacity for a single 
> replay during the week.
> 
> I think my first post should have started “Fasten your seat belts, it is 
> going to be a bumpy night…)
>  
>  
> Genie: You have three wishes.
> Me: I wish I had more wishes.
> Genie: You cannot wish for more wishes.
> Me: I wish I could.
>  
> From: Benjamin Schollnick  <mailto:bscholln...@schollnick.net>> 
> Sent: Sunday, May 29, 2022 11:18 AM
> To: Steve GS mailto:Gronicus@SGA.Ninja>>
> Cc: Richard Damon  <mailto:rich...@damon-family.org>>; Python  <mailto:python-list@python.org>>
> Subject: Re: Automatic Gain Control in Python?
>  
> Okay, you are capturing the audio stream as a digital file somewhere, correct?
>  
> Why not just right a 3rd party package to normalize the audio levels in the 
> digital file?  It’ll be faster, and probably easier than trying to do it in 
> real time…
>  
> eg. 
> https://campus.datacamp.com/courses/spoken-language-processing-in-python/manipulating-audio-files-with-pydub?ex=8
>  
> <https://campus.datacamp.com/courses/spoken-language-processing-in-python/manipulating-audio-files-with-pydub?ex=8>
>  
>> Normalizing an audio file with PyDub
>> 
>> Sometimes you'll have audio files where the speech is loud in some portions 
>> and quiet in others. Having this variance in volume can hinder transcription.
>> Luckily, PyDub's effects module has a function called normalize() which 
>> finds the maximum volume of an AudioSegment, then adjusts the rest of the 
>> AudioSegment to be in proportion. This means the quiet parts will get a 
>> volume boost.
>> You can listen to an example of an audio file which starts as loud then goes 
>> quiet, loud_then_quiet.wav, here 
>> <https://assets.datacamp.com/production/repositories/4637/datasets/9251c751d3efccf781f3e189d68b37c8d22be9ca/ex3_datacamp_loud_then_quiet.wav>.
>> In this exercise, you'll use normalize() to normalize the volume of our 
>> file, making it sound more like this 
>> <https://assets.datacamp.com/production/repositories/4637/datasets/f0c1ba35ff99f07df8cfeee810c7b12118d9cd0f/ex3_datamcamp_normalized_loud_quiet.wav>.
> or
>  
> https://stackoverflow.com/questions/57925304/how-to-normalize-a-raw-audio-file-with-python
>  
> <https://stackoverflow.com/questions/57925304/how-to-normalize-a-raw-audio-file-with-python>
> 
> 
>  
> - Benjamin
> 
> 
>> On May 29, 2022, at 11:04 AM, Steve GS > <mailto:Gronicus@SGA.Ninja>> wrote:
>>  
>>>> From your description, your fundamental problem is you are trying to 
>>>> automatically "control" things that weren't designed to be automatically 
>>>> controlled in the way you are attempting.
>> 
>> How so? I am sending commands to a smart speaker and it plays podcasts and 
>> broadcasts.
>> How is this a violation of SS design?
>> 
>> ===
>> 
>>>> The smart speaker assumes the "user" will adjust the volume either with 
>>>> the controls or with verbal commands, So things will be a bit "clunky" in 
>>>> your results as you command the smart speaker volume level.
>> 
>> So, you want me to sit here for every hour of the weekend and monitor the 
>> audio levels for a result that will get, at best, one replay when I believe 
>> it can be automated.
>> 
>> ===
>> 
>>>> Yes, you have an automated system that does most of what you want, but it 
>>>> is based on pieces not designed to be automated in this way, and you are 
>>>> running into the limitations caused by that.
>> 
>> Again, what limitations 

Re: Automatic Gain Control in Python?

[Benjamin Schollnick]

Okay, you are capturing the audio stream as a digital file somewhere, correct?

Why not just right a 3rd party package to normalize the audio levels in the 
digital file?  It’ll be faster, and probably easier than trying to do it in 
real time…

eg. 
https://campus.datacamp.com/courses/spoken-language-processing-in-python/manipulating-audio-files-with-pydub?ex=8
 


Normalizing an audio file with PyDub

Sometimes you'll have audio files where the speech is loud in some portions and 
quiet in others. Having this variance in volume can hinder transcription.

Luckily, PyDub's effects module has a function called normalize() which finds 
the maximum volume of an AudioSegment, then adjusts the rest of the 
AudioSegment to be in proportion. This means the quiet parts will get a volume 
boost.

You can listen to an example of an audio file which starts as loud then goes 
quiet, loud_then_quiet.wav, here 
.

In this exercise, you'll use normalize() to normalize the volume of our file, 
making it sound more like this 
.

or

https://stackoverflow.com/questions/57925304/how-to-normalize-a-raw-audio-file-with-python
 



- Benjamin

> On May 29, 2022, at 11:04 AM, Steve GS  wrote:
> 
>>> From your description, your fundamental problem is you are trying to 
>>> automatically "control" things that weren't designed to be automatically 
>>> controlled in the way you are attempting.
> 
> How so? I am sending commands to a smart speaker and it plays podcasts and 
> broadcasts.
> How is this a violation of SS design?
> 
> ===
>>> The smart speaker assumes the "user" will adjust the volume either with the 
>>> controls or with verbal commands, So things will be a bit "clunky" in your 
>>> results as you command the smart speaker volume level.
> 
> So, you want me to sit here for every hour of the weekend and monitor the 
> audio levels for a result that will get, at best, one replay when I believe 
> it can be automated.
> 
> ===
>>> Yes, you have an automated system that does most of what you want, but it 
>>> is based on pieces not designed to be automated in this way, and you are 
>>> running into the limitations caused by that.
> 
> Again, what limitations of the SS am I violating? It is designed to receive 
> commands and play the audio.
> Also, what makes you think that you know how my program is based?
> 
> ===
>>> Yes, you could split the aux-out to bring it into another computer to 
>>> listen to the sound level, and then using a sound input package get samples 
>>> of what is playing, and analyze that data to get an average volume, and 
>>> then issues the command to adjust the volume level.
> 
> Hmmm, is that not my original question? Are you suggesting to monitor the 
> audio, sense it for volume changes and apply those changes to the original 
> audio? One thing that may have to happen is a timed-delay to all for the AGC 
> to work.   This may require a correlation circuit.
> 
> ==
>>> What you seem to be missing is that you could get the podcasts from a 
>>> browser, and all a browser is is a program. It isn't that much work to 
>>> write a rudimentary browser in python, especially if you don't actually 
>>> need to display the results to a user, but are only trying to automate a 
>>> particular task.
> 
> Writing my own browser in Python might work. Do you have a sample one that I 
> could twerk to fit my needs?
> I would have to be able to invoke it and an hour later devoke it least I end 
> up with multiple audio channels playing.
> 
> Either way, I would still need an AGC program which was my original question. 
>  
> 
> ===
>>> You seem to feel strongly invested in your current code base, which is 
>>> understandable, but it seems you have reached a point where you don't want 
>>> to live with the limitations CAUSED by that system. 
> 
> The changes in volume are not CAUSED by my program. The want to fix them is a 
> new development to improve the product. The volume fluctuations are causes, 
> or neglections, by the engineers at the sources of podcasts and broadcasts. 
> 
>>> Yes, there is likely a way to tack on another layer of "stuff" to adjust 
>>> for this issue, but it likely is going to require some real programming.
> 
>>> It may well be the design I am suggesting, of writing a program to fetch 
>>> the podcast and save it requires a bit more work to get to the level you 
>>> currently are at, but the results are a system 

Re: Best way to check if there is internet?

[Benjamin Schollnick]

>>> Thanks for the in-between. I really like the Python comunity as,
>>> even though it's a 'scripting' language,
> 
> And we like you, even though you're only a ...
> 
> In English, a statement like that is considered rather insulting.
> 
>> To me, it's a programming language. In my usage, a "script" is a
>> bunch of OS commands.
> 
> Exactly. A script is a set of commands that you would have normally
> entered by hand at a command prompt. But, after having gone through
> the process a few times, you decided to shove them into a file to save
> on typing.
> 
> Python is a programming language. Period.

Exactly.  In my opinion “It’s a scripting language”, is a devaluation of the 
language.
It’s an attempt to throw python in to a “trivial” classification.  

Heck, the same argument can be made of Bash, PHP, Perl, and a few other 
languages as well.

How many “scripts” have been throw quickly together in Perl, or PHP?  

Quite a damn few, yet, would anyone call Wordpress a “script”?

It’s effectively damning with faint praise.

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pyinstaller wrong classified as Windows virus

[Benjamin Schollnick]

>> Windows Defender has a setting to also use “Reputation Scoring”.
>> What that simply means is that WDef will report back a hash to microsoft 
>> which is then checked to see if it is known.  If it is known, then it has a 
>> reputation and based off that reputation Defender will either allow it to 
>> run or not.
>> But if there is no reputation (eg no one has ever run it), that’s 
>> suspicious.  And that’s what you are running into.
>> You can submit the EXE to the defender team, which should allow it to 
>> operate properly without any issue.
>>  - Benjamin
> 
> sure... "that's suspicious". Unless you're a developer compiling your own 
> code. In which case every fresh build will be something "unknown". You have 
> to set every setting you can find to "developer mode" to help with this kind 
> of thing, and sometimes it's still not enough.

Understandable, and Steve Gibson (GRC.com , creator of 
Spinrite) has mentioned that he has had this problem with every Beta of his 
applications.

I agree completely with you, but that’s how Microsoft has set things up.

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pyinstaller wrong classified as Windows virus

[Benjamin Schollnick]

Windows Defender has a setting to also use “Reputation Scoring”.

What that simply means is that WDef will report back a hash to microsoft which 
is then checked to see if it is known.  If it is known, then it has a 
reputation and based off that reputation Defender will either allow it to run 
or not.

But if there is no reputation (eg no one has ever run it), that’s suspicious.  
And that’s what you are running into.  

You can submit the EXE to the defender team, which should allow it to operate 
properly without any issue.

- Benjamin



> On Nov 29, 2021, at 1:57 PM, Barry  wrote:
> 
> 
> 
>> On 29 Nov 2021, at 00:03, anthony.flury via Python-list 
>>  wrote:
>> 
>> 
>> On 26/11/2021 07:13, Ulli Horlacher wrote
 But consider another possibility that your compiler software is compromised
>>> Then https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe
>>> is infected. I doubt this.
>> 
>> But you aren't using python3.10 to 'compile' the code to the executable that 
>> windows complains about: you are using pyinstaller, which if memory serves 
>> is a 3rd party application.
>> 
>> I assume that you have no problem running the script without pyinstaller ?
>> 
>> so Might pyinstaller be compromised in some way ?
> 
> Not likely.
> 
> On windows pyinstall, and other tools like it, create .exe files on windows.
> I would guess it’s that .exe that is triggering the malware detector false 
> positive.
> 
> Barry
>> 
>> 
>>> 
 Is this happening to only one set of code?
>>> This is happening SOMETIMES, not always. With the SAME source code. When I
>>> call pyinstaller often enough, then the virus scanner is quiet. In about 1
>>> of 20 compile runs.
>>> 
>>> 
>>> 
>> -- 
>> Anthony Flury
>> *Moble*: +44 07743 282707
>> *Home*: +44 (0)1206 391294
>> *email*: anthony.fl...@btinternet.com 
>> -- 
>> https://mail.python.org/mailman/listinfo/python-list
>> 
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Why so fast a web framework?

[Benjamin Schollnick]

>> Sometimes this group reminds me of a certain large company I worked for.
>> If they didn't have a solution addressing a problem, they'd pretend it
>> didn't matter and belittle anyone who questioned that version of reality.
>> 
> 
> That's not strictly true; what's happening here is that someone's
> published a cool-looking bar graph but nobody knows what it really
> means. I don't feel the need to delve into everyone's benchmarks to
> explain why Python is purportedly worse. If someone uses those kinds
> of numbers to decide which programming language to use, they have
> bigger problems.

If you dig a bit, the benchmark is scary…
As in stupid-scary.

It consists of, 7 tests, and then a composite score is generated:

JSON Serialization - In this test, each response is a JSON serialization of a 
freshly-instantiated object that maps the key message to the value Hello, World!
Single Query - In this test, each request is processed by fetching a single row 
from a simple database table. That row is then serialized as a JSON response.
Multiple Queries - In this test, each request is processed by fetching multiple 
rows from a simple database table and serializing these rows as a JSON 
response. The test is run multiple times: testing 1, 5, 10, 15, and 20 queries 
per request. All tests are run at 512 concurrency.
Cached Queries - In this test, each request is processed by fetching multiple 
cached objects from an in-memory database (the cache having been populated from 
a database table either as needed or prior to testing) and serializing these 
objects as a JSON response. The test is run multiple times: testing 1, 5, 10, 
15, and 20 cached object fetches per request. All tests are run at 512 
concurrency. Conceptually, this is similar to the multiple-queries test except 
that it uses a caching layer.
Fortunes - In this test, the framework's ORM is used to fetch all rows from a 
database table containing an unknown number of Unix fortune cookie messages 
(the table has 12 rows, but the code cannot have foreknowledge of the table's 
size). An additional fortune cookie message is inserted into the list at 
runtime and then the list is sorted by the message text. Finally, the list is 
delivered to the client using a server-side HTML template. The message text 
must be considered untrusted and properly escaped and the UTF-8 fortune 
messages must be rendered properly.  Whitespace is optional and may comply with 
the framework's best practices.
data Updates - This test exercises database writes. Each request is processed 
by fetching multiple rows from a simple database table, converting the rows to 
in-memory objects, modifying one attribute of each object in memory, updating 
each associated row in the database individually, and then serializing the list 
of objects as a JSON response. The test is run multiple times: testing 1, 5, 
10, 15, and 20 updates per request. Note that the number of statements per 
request is twice the number of updates since each update is paired with one 
query to fetch the object. All tests are run at 512 concurrency.  The response 
is analogous to the multiple-query test. 
plain text - In this test, the framework responds with the simplest of 
responses: a "Hello, World" message rendered as plain text. The size of the 
response is kept small so that gigabit Ethernet is not the limiting factor for 
all implementations. HTTP pipelining is enabled and higher client-side 
concurrency levels are used for this test (see the "Data table" view).

Here, I instead benchmark my django gallery app, using Apache Bench, and so 
forth.  I guess I’ve been over-achieving…

I have to admit, though, that these benchmarks certainly allow everyone to 
play.  

431 cherrypy587 0.0%(0.0%)

Even cherrypy with it’s 587 per second replies with plain-text.  

The tasks seem deceptively (?) simple?  

But looking closer, the data table for each task, gives more details.  For 
example the plain text is run 4 different times, at 4 different client-side 
concurrency levels are used…  But the levels are: 256, 1024, 4096, and 16384.   
That can’t be the concurrency/thread count??!?!?!??  I can believe 1,000 - 
3,000, outrageously high, but believable.  But 16K worth of 
concurrency/threads?  I doubt that Wikipedia even has to dial it that high?

I have to give them points for providing API latency, and framework overhead….  

- Benjamin



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Does loading PDB slow down execution?

[Benjamin Schollnick]

> On Oct 27, 2021, at 1:01 PM, Unixnut  wrote:
> 
> On 06/10/2021 18:30, Dieter Maurer wrote:
>> Unixnut wrote at 2021-10-3 22:03 +0100:
>>> If I run a python3 program with "import pdb" in the code, would it
>>> execute slower than without loading the debugger?
>> Importing `pdb` does not slow down the application significantly
>> (it just adds the import time but does not otherwise affect the
>> application).
>> Only when you execute code under debugger control, the
>> execution is significantly slowed down (because the debugger
>> gets informed (via callbacks) about important "event"s (entering
>> a line, function call, function return, exception) during
>> the execution).
> Excellent, many thanks for confirming. I can leave the execution running then.

But it’s pointless to import pdb, if you aren’t going to use it.

I would suggest that a best practice would be to only import pdb, if and when 
you were going to be performing the debugging.
(So maybe set a boolean constant, that would cause the import, and debugging 
code to be executed.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Optimizing Small Python Code

[Benjamin Schollnick]

How would you measure the steps that it takes?

- Benjamin



> On Jun 22, 2021, at 7:04 PM, Greg Ewing  wrote:
> 
> On 23/06/21 3:03 am, Kais Ayadi wrote:
>> for n in range(1, 7):
>> print (n)
>> for x in range(0, n):
>> print(" ", x)
>> can this code be more optimised?
> 
> Optimised for what? Readability? Execution speed? Code size?
> Memory usage?
> 
> -- 
> Greg
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: neoPython : Fastest Python Implementation: Coming Soon

[Benjamin Schollnick]

> Why? The currently extant Python implementations contribute to climate change 
> as they are so inefficient; 

That same argument can be made for every triple-AAA video game, game console, 
etc.

Python is more efficient for some problem sets, and Python is less efficient 
for other problem sets.

Please feel free to come out with NeoPython.  When you are done, and it is 
backward compatible with existing Python code, I’ll be happy to benchmark it 
against Native python.  But don’t blame Python for global climate change.  
There are plenty of bigger “causations” to Global climate change, then a 
freakin’ computer programming language.

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: not able to download PyAudio

[Benjamin Schollnick]

> On Apr 2, 2021, at 4:40 AM, ᗷᑌᑎᑎY  wrote:
> 
>   Hello  everyone
>   I am not able to download PyAudio. I tried by typing pip install in
>   PyAudio in cmd but it show's no compatible version availbale. What should
>   I do? .

There seems to be some confusion, which I can understand…  

“No compatible version”, simply means that Pip was unable to find a version of 
 for your version of python.

Please note, that does not mean that a version *EXISTS*, just that it could not 
find a version for your platform.

So go to https://pypi.org  and do a search.  

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Benjamin Schollnick]

>> That's not true for digraphs where there is a third, distinct and
>> different "title" case. I think the doc should state that the initial
>> character is converted to titlecase. A parenthentical statement that
>> titlecase is usually but not always equal to uppercase would be nice,
>> but the current statement is obsolete and not correct in all, um...
>> cases.
> 
> Fair enough, but the trouble is that getting too pedantic in a
> docstring just makes it read like IBM documentation. :)

And actually conversely makes it harder to keep updating, because you’ll need 
to document every and all edge-cases, and then need to know when one of those 
edge cases breaks, etc.  

The core concept is documented, and it’s pretty straight-forward.  

I’m sorry, but it’s as if he’s arguing for the sake of arguing.  It’s starting 
to feel very unproductive, and unnecessary.

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Benjamin Schollnick]

>> I guess it depends on what you mean by "character". In my mind, the
>> first character of string s is s[1], and I would then expect that
>> 
>> s.title()[1] == s[1].upper()
>> 
> 
> I presume you mean [0], but no, that's not the case. A single
> character can titlecase to two characters, or to a single character
> that isn't the same as if you uppercase or lowercase it. See examples
> in previous post.

Or Kanji, etc.  Where a single character can represent more than one in a 
different unicode standard, as I understand. 

- Benjamin

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Benjamin Schollnick]

Robert,

I certainly see your point.  

> My only issue is that I completely fail to see how this function would be
> useful enough to warrant the inclusion into the *core* of a general-purpose
> language, including its misleading name.

But that’s where we have to disagree.

Sure, the name cold be a bit less “misleading”, I guess we could petition to 
have it renamed to:

def nonlanguage_aware_title or  def 
title_but_not_language_aware

But I’m sorry for the majority of us, we haven’t even realized that title 
wasn’t language aware.   That has to tell you something (mainly that we’ve 
never hit these problems).  

Now that doesn’t diminish your issue here.  

But you’re a programmer, people have already suggested someway to work around 
this with RE.

The other python specific might be to actually SPLIT on the ` symbol, title 
each segment returning from split, and the JOIN them together again.

Regarding the issue of THE in an actual title, I’d suggest that RE would be the 
best solution for that, it should be easy enough to come up with RE that’ll 
work for word “xxx The xxx” and convert it to “xxx the xxx”.  

> The fact that the function's behavior
> is correctly documented doesn't make its very existence less bewildering to 
> me.

Okay, I don’t recall you actually accepting at how large of a problem you are 
asking title to address.

The standard library is not going to fix and solve everything, like Apple I 
would expect them to target ~90% of the functionality, and then let people grow 
and expand on it’s functionality.  (Sort of like my CSV wrapper)

title is roughly comparable to PHP’s ucwords.

Wow, perl takes the prize though, it’s called ucwords.

$foo = 'hello world!';
$foo = ucwords($foo); // Hello World!

$bar = 'HELLO WORLD!';
$bar = ucwords($bar); // HELLO WORLD!
$bar = ucwords(strtolower($bar)); // Hello World!

perl appears to only manipulate the characters that it’s changing, so if the 
string is already uppercase, then it’ll just look like the entire string is 
uppercase when it’s done.  

So honestly that’s a mostly useless function if you need to pass a lowercase 
string into it.

What’s in common here?  both perl and php also seem to take the ` as a word 
separator.  

Neither of them appear to be language aware.

> Consider this function:
> 
> def add_seventeen(n): '''Return n with 16.8 added''' return n + 16.8
> 
> It's like .title(): It does almost the thing its name suggests, it is 
> correctly
> documented, it is useful to anybody who happens to want 16.8 added to numbers,
> and it might erroneously be used by someone who wants exactly 17 added and
> didn't bother to read the docs.

And would be laughed at significantly.  

I agree with your example, but I don’t consider it to be equivalent.  

title does at first, and second glance work as expected.  You have 
significantly higher expectations than we do, evidently.

At minimum, what I would suggest is come up with some test case examples, and 
toss that over to the python dev team.  Show them that this isn’t working as 
you expect, and see what they say.  

While there might be a few of the python developers on this list, I doubt it.  
Instead bring this up in a productive manner.  Show the harm, and then show a 
solution.  A Proposed change, an alternative framework.

We can’t solve the problem, but if you present it properly, then maybe you can 
be part of the solution.

> 
>> And as I mentioned the sheer amount of work that would be needed would
>> probably cover a phd dissertation or more…  It’s a huge problem set to
>> respect one language, let alone all languages.  
> 
> And that's why I believe that such a function should be delegated to a natural
> language-processing library and not the core of Python.

I have to disagree.  While I might expect a better version in a natural 
language processing library, every programming language that I’m aware of 
except for BASIC, offers an equivalent to title, as long as they offer an 
equivalent to upper and lower.

Why should python not offer title in light of this?

> said, I doubt that .title() would make it into Python today if it weren't 
> there
> already. I'm having fun with this.

Ah, so while being a bit serious, I’m reading a bit too much into this.  

At this point, it’s become an interesting thought experiment for you.  

Good luck,

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Benjamin Schollnick]

>> Heck, how do we prevent it from titlecasing abbreviations?  (This is plain 
>> text not XML….  If it was XML it would be easy!)
> 
> We haven't managed to teach humans how to do that, so I doubt we'll
> ever teach a simple standard library function to do it.
> 
> *cough*XMLHttpRequest*cough*

True, but I was thinking that it would be easy in XML because the XML/HTML 
would be able to have metadata helping to define the components in the text 
stream.  

And this is probably more of a pseduo-html, but that’s because I do more work 
in HTML then I do in pure XML.  XML I usually just tell the system to output, 
instead of craft by hand…  


blah blah this is a sentence that has the abbreviation 
NASA in it.


But that doesn’t help title, because it only handles pure plaintext.  

As many people have pointed out, or I think they meant to point out, there is 
no place in the text string to put metadata that would help assist parsing the 
string.  By definition the text can’t have metadata, since it’s plaintext.  

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Benjamin Schollnick]

> I agree with everything you say. Especially the open source part. But wouldn't
> you agree that .title() with all its arbitrary specificity to appear in the
> very core of a general purpose language is quite an oddity?

No, because it book ends the issue.

Upper - Converts everything to uppercase
Lower - Converts everything to lowercase

Title - Covers the cases in-between upper/lower.  

I’ll agree that if title was to respect language definitions, that there would 
be a problem here…  But it specifically states otherwise.

And as I mentioned the sheer amount of work that would be needed would probably 
cover a phd dissertation or more…  It’s a huge problem set to respect one 
language, let alone all languages.  

So the only answer is to not respect the languages, and leave that up to a 
later implementation or for someone else to assist in adding in support.

Heck, how do we prevent it from titlecasing abbreviations?  (This is plain text 
not XML….  If it was XML it would be easy!)

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: .title() - annoying mistake

[Benjamin Schollnick]

>> The problem is that you haven't read the documentation :) It very carefully
>> does NOT define itself by language, and its behaviour is identical regardless
>> of the language used.
> 
> The documentation says: "The algorithm uses a simple language-independent
> definition of a word as groups of consecutive letters..."
> 
> Yes, I get that. But the purpose it (improperly) serves only makes sense in 
> the
> English language. Which is also the reason they called it title() and not
> capitalize_words(). Frankly, I can't think of any situation where this method
> would have any use -- in any language, including English. It is just a
> completely arbitrary feature, as would be a function that capitalizes only the
> last letter of each word.

I’m sorry Robert, but just because it doesn’t meet your requirements, doesn’t 
mean it’s useless.

I use .title to normalize strings for data comparison, all the time.  It’s a 
perfect alternative to
using .UPPER or .lower.  

Right in the documentation, it specifically acknowledges .title working in 
foreign character sets. 

So while I feel for the fact that it doesn’t met your requirements, please keep 
in mind, it does meet other peoples requirements.  

As with everything here, it’s open source.  If you feel that there should be a 
revised version that does met your requirements create it, or gather a bunch of 
people and go the route of SCANDIR and open-source it, and petition that it be 
moved into the standard library.

Since this seems to be bugging you this much, come up with a solution.  

I suspect the problem you are going to have is that in effect you’ll be 
creating a multi-language parser, even worse, you may have to add nameparsing 
into this.

Good luck.

- Benjamin



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How do I read .csv files

[Benjamin Schollnick]

The best answer is to use the CSV library, but there are wrappers around it.

For example, I use a wrapper of my own creation (csv_common.py) @ 
https://github.com/bschollnick/bas_Utilities 
.
The main benefit is that I can create variant CSV’s by just inheriting and 
redefining portions of it, very quickly.  

But, Panda’s and other tools are just as valid as well.

But I would really recommend playing with the CSV library first, and understand 
how it works, before using a wrapper or a totally different tool.  
Understanding the basics really simplifies the learning process, and gives you 
a foundation to build upon.

- Benjamin

> On Mar 16, 2021, at 8:23 AM, Gys  wrote:
> 
> On 3/12/21 11:28 AM, Johann Klammer wrote:
>> Specifically ones with quoted strings. I'll have whitespace in
>> there and possibly escaped quotechars.
>> maybe newlines too.
>> Which means that pyparsing commaSeparatedList.parseString(line) won't work.
>> I also like to edit them for visual alignment, so there'll
>> be whitespaces outside the strings(more than one)
>> ...therefore, csv.DictReader() won't work.
>> I'd like them read into a dict or something..
> 
> Hi Johann Klammer,
> I use Pandas for handling *.csv files
> 
> pandas documentation :
> 
> 
> 
> Hands on example :
> 
> 
> 
> -hth
> Gys
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New Python implementation

[Benjamin Schollnick]

>> that method was borrowed from or vice versa. Being a rich language has pro's
>> and cons. LISP only had cons.

Now, Now.  That’s certainly not correct.  

LISP does have a few Pros.  Namely Job security.  You’ll have a hard time 
replacing a experienced and professional LISP programmer.

- Benjamin



-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Managing plug-ins

[Benjamin Schollnick]

Did you ever find anything that met your requirements?

If not, I have a prototype that I need to build out some more…

https://github.com/bschollnick/PyPlugInMgr 


I use it for some home grown utilities, but it needs to be fleshed out some 
more…
If you’re interested feel free to take a look.

- Benjamin



> On Feb 23, 2020, at 5:45 PM, DL Neil via Python-list  
> wrote:
> 
> Please recommend a library which will manage plug-ins.
> 
> 
> (Regret that searching PyPi or using a web SE results in an overwhelming 
> number of 'false positives')
> 
> Not wanting to 'reinvent the wheel, have been looking for an 'approved' way 
> to manage a set of previously-prepared routines plus user-added functionality:
> - user selects particular type of analysis (cmdLN)
> - take that string and convert to a function/method
> - 'plug-in' code could be located within application
> - 'plug-in' could be 'included' from user
> - interface standardisation/checking not part of question
> - Python3
> 
> Application is statistical analysis. Users want to control aspects such as 
> data selection/inclusion policies, other data pre-processing/massaging, 
> distribution curve to be applied, and similar.
> NB not looking for stats-code, but to easily manage a range of plug-ins from 
> which users may select to suit their particular research objective.
> -- 
> Regards,
> =dn
> -- 
> https://mail.python.org/mailman/listinfo/python-list

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue37709] CSVReader ignores dialect.lineterminator

[Benjamin Schollnick]

Benjamin Schollnick  added the comment:

This is tested under python 3...

filename = "csv_Sample.csv"
from csv import DictReader
datafile = open(filename, 'r')
data = csv.DictReader(datafile, lineterminator = '\x07', delimiter='\x06')
print(next(data))
OrderedDict([('Field1', 'A'), ('Field2', 'B'), ('Field3', 'C'), ('Field4', 
'D'), ('Field5', 'E'), ('Field6', 'F'), ('Field7', 'G'), ('Field8', 'H'), 
('Field9', 'I'), ('Field10\x07', 'J\x07')])
print(ord(data.reader.dialect.lineterminator))

So it's untested under python 2, since I've stopped developing under Py2.  

I noticed the note in the CSV reader documentation, *AFTER* I diagnosed the 
issue with the CSV reader...  Which is why I opened the bug / feature 
enhancement request, since this is an very odd edge case.

I agree 90+% of all CSVs are going to be \n line terminated, but if we offer it 
for writing, we should offer it for reading.

The main emphasis here is this code will not working in the real world, eg.

filename = "csvFile.csv"
from csv import DictReader, DictWriter
import csv
with open(filename, mode='w') as output_file:
outcsv = csv.writer(output_file, delimiter=',', lineterminator=";")
outcsv.writerow(['John Cleese', 'CEO', 'March'])
outcsv.writerow(['Graham Chapman', 'CFO', 'November'])
outcsv.writerow(['Terry Jones', 'Animation', 'March'])
outcsv.writerow(['Eric Idle', 'Laugh Track', 'November'])
outcsv.writerow(['Michael Palin', 'Snake Wrangler', 'March'])

with open(filename, mode='r') as input_file:
csv_reader = csv.reader(input_file, delimiter=',', lineterminator=";")
for row in csv_reader:
print(row)

--
Added file: https://bugs.python.org/file48518/CSV_SAMPLE.CSV

___
Python tracker 
<https://bugs.python.org/issue37709>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37709] CSVReader ignores dialect.lineterminator

[Benjamin Schollnick]

New submission from Benjamin Schollnick :

I've run into a situation where the CSV input file is very unusual.  The 
Delimiter is "\x06" and the lineterminator is "\x07".

While I've written code to work around this, it would be significantly nicer if 
the CSV Reader code actually paid attention to the dialect's lineterminator 
value.

--
components: Library (Lib)
messages: 348681
nosy: Benjamin Schollnick
priority: normal
severity: normal
status: open
title: CSVReader ignores dialect.lineterminator
type: enhancement
versions: Python 3.7

___
Python tracker 
<https://bugs.python.org/issue37709>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

RAR functionality

[Benjamin Schollnick]

Folks,

Can anyone explain to me why RAR support isn’t available natively in python?

We have the zipfile and tarfile modules… 

Yes, there is rarfile and PaTools but they both rely on RAR being available to 
shell out to…  
What is it about RAR files that makes it unreasonable to handle in natively in 
python?
Licensing?  Bad rar file design?

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: pypi submission?

[Benjamin Schollnick]

 On Apr 6, 2015, at 6:47 PM, Dylan Evans dylanlee.evan...@gmail.com wrote:
 
 On 6 April 2015 at 23:21:02, Benjamin Schollnick (bscholln...@gmail.com) 
 wrote:
 And I’ve been messing around with different config files, etc, trying to 
 solve the problem  
 myself.
 Can anyone point out what I have done wrong?
 
 What upload command are you using? I see no files at 
 https://pypi.python.org/pypi/Directory_Caching/1.0.5, so pip search returns 
 an entry but there’s nothing to install.  Usually there’s a table with the 
 first heading ‘File’ which contains links to tarballs etc. 
 
 The command python setup.py sdist upload” should do it.

That’s what I have been doing…  

I started with using the guide from Peter Downs 
(http://peterdowns.com/posts/first-time-with-pypi.html 
http://peterdowns.com/posts/first-time-with-pypi.html).
That seemed to work fine, until I went to install from pip.  And I’ve been 
hacking at it for about a week or so, with no luck.  I haven’t found any 
troubleshooting guides for pypi.

Here’s the captures of the register, sdist upload, and a verbose pip install 
from pypitest.

My guess is that there is something odd in my setup.py file?  

It’s a merge between several different setup.y’s, I’ve been swapping parts out, 
trying to figure out what is causing it.  But from day 1, using the peter Downs 
setup.py, this problem existed.

Here’s the current version:

Directory Caching - A Caching library for Directories  Files

The Directory Caching library caches and manages directory and file listings 
data.
The library is being developed for the (Web) Gallery software.


classifiers = \
Development Status :: 4 - Beta
Intended Audience :: Developers
License :: OSI Approved :: MIT License
Programming Language :: Python
Topic :: Database
Topic :: Software Development :: Libraries :: Python Modules
Operating System :: OS Independent
Operating System :: MacOS :: MacOS X
Operating System :: Microsoft :: Windows
Operating System :: Unix
Programming Language :: Python
Programming Language :: Python :: 2.6
Programming Language :: Python :: 2.7

try:
from setuptools import setup, find_packages
except:
from disutils.core import setup, find_packages

__title__ = 'Directory_Caching'
__version__ = '1.10.1'
__author__ = 'Benjamin Schollnick'
__license__ = 'MIT'
__copyright__ = 'Copyright 2015 Benjamin Schollnick'

dependencies = ['natsort', 'scandir']

doclines = __doc__.split(\n)

#
#   Modelled after requests - 
https://github.com/kennethreitz/requests/blob/master/setup.py
#
setup(
name='Directory_Caching',
version=__version__,
description = doclines[0],
long_description = \n.join(doclines[2:]),
author=__author__,
author_email='benja...@schollnick.net',
url='https://github.com/bschollnick/Directory_Caching',
license=MIT,
maintainer='Benjamin Schollnick',
maintainer_email='benja...@schollnick.net',
packages=find_packages(),
package_dir={'directory_caching': 'directory_caching'},
include_package_data=True,
#platforms=[Any],
download_url = 
'https://github.com/bschollnick/Directory_Caching/tarball/1.05',
install_requires=dependencies,
keywords = ['caching', 'files', 'directories', 'scandir', 'naturalsort'],
classifiers=filter(None, classifiers.split(\n)),
)


nerv:directory_caching Benjamin$ python setup.py register -r pypitest
running register
running egg_info
writing requirements to Directory_Caching.egg-info/requires.txt
writing Directory_Caching.egg-info/PKG-INFO
writing top-level names to Directory_Caching.egg-info/top_level.txt
writing dependency_links to Directory_Caching.egg-info/dependency_links.txt
reading manifest file 'Directory_Caching.egg-info/SOURCES.txt'
writing manifest file 'Directory_Caching.egg-info/SOURCES.txt'
running check
Registering Directory_Caching to https://testpypi.python.org/pypi
Server response (200): OK


nerv:directory_caching Benjamin$ python setup.py sdist upload -r pypitest
running sdist
running egg_info
writing requirements to Directory_Caching.egg-info/requires.txt
writing Directory_Caching.egg-info/PKG-INFO
writing top-level names to Directory_Caching.egg-info/top_level.txt
writing dependency_links to Directory_Caching.egg-info/dependency_links.txt
reading manifest file 'Directory_Caching.egg-info/SOURCES.txt'
writing manifest file 'Directory_Caching.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, 
README.rst, README.txt

running check
creating Directory_Caching-1.10.1
creating Directory_Caching-1.10.1/Directory_Caching.egg-info
creating Directory_Caching-1.10.1/directory_caching
making hard links in Directory_Caching-1.10.1...
hard linking setup.cfg - Directory_Caching-1.10.1
hard linking setup.py - Directory_Caching-1.10.1
hard linking Directory_Caching.egg-info/PKG-INFO - 
Directory_Caching-1.10.1/Directory_Caching.egg-info

pypi submission?

[Benjamin Schollnick]

Folks,

I’m having issues with submitting to pypi.

I can register and upload my package.

nerv:~ Benjamin$ pip search directory_caching
Directory_Caching - A Caching library for Directories  Files
nerv:~ Benjamin$ 

but, if you try to install it using pip:

pip install directory_caching
Reading Profile from ~/dropbox/shellscripts/profile
nerv:~ Benjamin$ pip install directory_caching
Collecting directory-caching
  Could not find any downloads that satisfy the requirement directory-caching
  No distributions at all found for directory-caching

If you do a verbose:

Last login: Mon Apr  6 18:05:02 on ttys000
Reading Profile from ~/dropbox/shellscripts/profile
nerv:~ Benjamin$ pip -v install directory_caching
Collecting directory-caching
  Getting page https://pypi.python.org/simple/directory-caching/
  URLs to search for versions for directory-caching:
  * https://pypi.python.org/simple/directory-caching/
  Getting page https://pypi.python.org/simple/directory-caching/
  Analyzing links from page https://pypi.python.org/simple/directory-caching/
  Could not find any downloads that satisfy the requirement directory-caching
  Cleaning up...
  No distributions at all found for directory-caching
  Exception information:
  Traceback (most recent call last):
File /Library/Python/2.7/site-packages/pip/basecommand.py, line 232, in 
main
  status = self.run(options, args)
File /Library/Python/2.7/site-packages/pip/commands/install.py, line 339, 
in run
  requirement_set.prepare_files(finder)
File /Library/Python/2.7/site-packages/pip/req/req_set.py, line 333, in 
prepare_files
  upgrade=self.upgrade,
File /Library/Python/2.7/site-packages/pip/index.py, line 397, in 
find_requirement
  'No distributions at all found for %s' % req
  DistributionNotFound: No distributions at all found for directory-caching

The package is here, https://github.com/bschollnick/Directory_Caching 
https://github.com/bschollnick/Directory_Caching.

And I’ve been messing around with different config files, etc, trying to solve 
the problem myself.  
Can anyone point out what I have done wrong?

If a simpler package would be useful, then 
https://github.com/bschollnick/semantic_url 
https://github.com/bschollnick/semantic_url

I haven’t attempted to release semantic_url yet, simply because I’m trying to 
solve Directory_Caching first.
And yes, I know caching files  directory listings is a black voodoo art, and 
asking for trouble.

This attempts to validate three different ways, and I’ve been using it for 
quite a while without any issue.  It’s invaluable in the web gallery when you 
have a directory that can have 2-4K worth of files.

- Benjamin

-- 
https://mail.python.org/mailman/listinfo/python-list

Directory Caching, suggestions and comments?

[Benjamin Schollnick]

Folks,

I am going to be using this code as part of a web system, and I would love
any feedback, comments and criticism.

Just as a side note, I'm not completely PEP 8.  I know that, I use a
slightly laxer setting in pylint, but I'm working my way up to it...

I am using scandir from benhoyt to speed up the directory listings, and
data collection.

The module is here as well,
https://dl.dropboxusercontent.com/u/241415/misc/directory_caching_v1.py

I had considered using OrderedDicts, but I really didn't see how that would
help the system.

I'm not completely happy with the return_sort_* functions, since they
return two different tuples, one goal was to try to keep everything in the
dictionary, but I couldn't think of a better method.

So any suggestions are welcome.

 - Benjamin



Directory Caching system.

Used to cache  speed up directory listing.

Preqs -

Scandir - https://github.com/benhoyt/scandir

scandir is a module which provides a generator version of
os.listdir() that also exposes the extra file information the
operating system returns when you iterate a directory.

Generally 2-3 (or more) times faster than the standard library.
(It's quite noticeable!)

import os
import os.path
import re
from stat import ST_MODE, ST_INO, ST_DEV, ST_NLINK, ST_UID, ST_GID, \
ST_SIZE, ST_ATIME, ST_MTIME, ST_CTIME

import time
import scandir

plugin_name = dir_cache

#
class   CachedDirectory(object):

For example:

To be added shortly.


def __init__(self):
self.files_to_ignore = ['.ds_store', '.htaccess']
self.root_path = None
# This is the path in the OS that is being examined
#(e.g. /Volumes/Users/username/)
self.directory_cache = {}


def _scan_directory_list(self, scan_directory):

Scan the directory scan_directory, and save it to the
self.directory_cache dictionary.

Low Level function, intended to be used by the populate
function.

scan_directory = os.path.abspath(scan_directory)
directories = {}
files = {}
self.directory_cache[scan_directory.strip().lower()] = {}
self.directory_cache[scan_directory.strip().lower()][number_dirs]
= 0

self.directory_cache[scan_directory.strip().lower()][number_files] = 0
for x in scandir.scandir(scan_directory):
st = x.lstat()
data = {}
data[fq_filename] = os.path.realpath(scan_directory).lower()
+ \
os.sep+x.name.strip().lower()
data[parentdirectory] = os.sep.join(\
os.path.split(scan_directory)[0:-1])
data[st_mode] = st[ST_MODE]
data[st_inode] = st[ST_INO]
data[st_dev] = st[ST_DEV]
data[st_nlink] = st[ST_NLINK]
data[st_uid] = st[ST_UID]
data[st_gid] = st[ST_GID]
data[compressed] = st[ST_SIZE]
data[st_size] = st[ST_SIZE]   #10
data[st_atime] = st[ST_ATIME] #11
data[raw_st_mtime] = st[ST_MTIME] #12
data[st_mtime] = time.asctime(time.localtime(st[ST_MTIME]))
data[st_ctime] = st[ST_CTIME]
if not x.name.strip().lower() in self.files_to_ignore:
if x.is_dir():
self.directory_cache[scan_directory.strip().lower()]\
[number_dirs] += 1
data[archivefilename] = 
data[filename] = 
data[directoryname] = x.name.strip().lower()
data[dot_extension] = .dir
data[file_extension] = dir
directories[x.name.lower().strip()] = True
self._scan_directory_list(data[fq_filename])
data[number_files] = self.directory_cache\
[data[fq_filename]][number_files]
data[number_dirs] = self.directory_cache\
[data[fq_filename]][number_dirs]
directories[x.name.lower().strip()] = data
else:
self.directory_cache[scan_directory.strip().lower()]\
[number_files] += 1
data[archivefilename] = 
data[filename] = x.name.strip().lower()
data[directoryname] = scan_directory
data[dot_extension] = os.path.splitext\
(x.name)[1].lower()
data[file_extension] = os.path.splitext\
(x.name)[1][1:].lower()
files[x.name.lower().strip()] = data
self.directory_cache[scan_directory.strip().lower()][files] =
files
self.directory_cache[scan_directory.strip().lower()]\
[dirs] = directories
self.directory_cache[scan_directory.strip().lower()]\
 

Re: Cookie issue(cant fix it with anyhting)

[Benjamin Schollnick]

Nikos,

Hello i having the following code to try and retrieve the visitor's
 saved cookie form the browser.
 
 [CODE]
 # initialize cookie and retrieve cookie from clients browser try:
 cookie = cookies.SimpleCookie( os.environ['HTTP_COOKIE'] )
 cookieID = cookie['name'].value
 except:
 cookieID = 'visitor'

As it has been said before, change the except to be an explicit error check.  
The all purpose except is hiding an error condition from you.

Take a look at this:

http://www.jayconrod.com/posts/17/how-to-use-http-cookies-in-python

- Benjamin


-- 
https://mail.python.org/mailman/listinfo/python-list

Code suggestions?

[Benjamin Schollnick]

Folks,

I am re-writing an image gallery application that I wrote in Twisted.

As part of this rewrite, I am attempting to simplify the process of managing 
the directory listings.

Would anyone have any suggestions on streamlining or optimizing this code?

The raw version of this file, is available from dropbox.

https://dl.dropboxusercontent.com/u/241415/unified.py

If you are testing with RAR files, you'll need rarfile.py

https://dl.dropboxusercontent.com/u/241415/rarfile.py

If anyone knows a better built-in Python 2.66/2.75 solution, that would be 
helpful.  

I eventually expect to support caching, but I think that will be an application 
level feature, and not built-in to this module.

The entire idea is to unify (thus unified.py) behavior better files, 
directories, and archives.

Yes, there is differences in the data coming back, but the calling logic, is 
identical.

So far, this rewrite has tremendously simplified the code for the image 
gallery.  If there is any interest, I'm willing to github unified or the image 
gallery.  

- Benjamin

import os
import os.path
from stat import *
import time
class   Unified_Directory:

An Attempt to unify directories, and archives into one single 
storage package.

For example:

gallery_listings = unified.Unified_Directory ()

gallery_listings.populate_file_data_from_filesystem ( filepathname = 
directory_path)
print files: ,gallery_listings.files,/n/n
print subdirectories: 
,gallery_listings.subdirectories, /n/n
print Display Gallery for , 
gallery_listings.root_path



def __init__ ( self ):
self.rar_file_types = ['cbr', 'rar']
self.zip_file_types = ['cbz', 'zip']
self.archive_file_types = self.rar_file_types + 
self.zip_file_types

self.files_to_ignore= []#   File 
filter to ignore

self.root_path  = None  #   
This is the path in the OS that is being examined (e.g. 
/Volumes/Users/username/ )
self.subdirectories = []#   This is 
the subdirectories that exist in that path (e.g. Desktop, Music, Videos, etc)


self.files  = []
#   This is the list of files that are in the root_path.

#   Each file consists of a dictionary containing:

#

#   * Filename

#   * st_mode   - Unix chmod

#   * st_ino- inode number

#   * st_dev- Unix Device

#   * st_nlink  - hard link count

#   * st_uid- User/Owner ID

#   * st_gid- Group ID of Owner

#   * st_size   - File size

#   * st_atime  - Last Access time

#   * st_mtime  - Last Modified Time

#   * st_ctime  - Last Metadata change

#   * dot_extension - File extension with . 
prefix (lower case)

#   * file_extension- File Extension 
without . prefix (lower case)

Re: Why 'files.py' does not print the filenames into a table format?

[Benjamin Schollnick]

 Nick, at this point, you need to hire someone to do your work for you.
 
 The code is completely ready.
 Some detail is missing and its not printing the files as expected.

Then the code is not completely ready, it has bugs, and your trying to have the 
list debug and fix it for you for free.

 Irrelevant to my question i just noticed weird behavior about my 
 pelatologio.py script which can be seen here:
 
 http://superhost.gr/?show=stats
 
 The first 3 files are of my doing.
 All the rest are of someone else's that managed to append entries into my 
 counters database utilizing this code:
 
 
 
   try:
   #find the needed counter for the page URL
   cur.execute('''SELECT ID FROM counters WHERE url = %s''', page )
   data = cur.fetchone()#URL is unique, so should only be 
 one
   
   if not data:
   #first time for page; primary key is automatic, hit is 
 defaulted
   cur.execute('''INSERT INTO counters (url) VALUES 
 (%s)''', page )
   cID = cur.lastrowid#get the primary key value 
 of the new record
 ==
 
 Does someone want to state something?

Sure, whoever wrote that code is a fool.

http://xkcd.com/327/

img src=http://imgs.xkcd.com/comics/exploits_of_a_mom.png;

They didn't sanitize your database inputs.

My suggestion would be for you to stop trying to re-invent the wheel, and use 
COTS software.  

- Benjamin

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A few questiosn about encoding

[Benjamin Schollnick]

Nick,

 The only thing that i didn't understood is this line.
 First please tell me what is a byte value
 
 \x1b is a sequence you find inside strings (and byte strings, the
 b'...' format).
 
 \x1b is a character(ESC) represented in hex format
 
 b'\x1b' is a byte object that represents what?
 
 
  chr(27).encode('utf-8')
 b'\x1b'
 
  b'\x1b'.decode('utf-8')
 '\x1b'
 
 After decoding it gives the char ESC in hex format
 Shouldn't it result in value 27 which is the ordinal of ESC ?

I'm sorry are you not listening?

1b is a HEXADECIMAL Number.  As a so-called programmer, did you seriously not 
consider that?

Try this:

1) Open a Web browser
2) Go to Google.com
3) Type in Hex 1B 
4) Click on the first link
5) In the Hexadecimal column find 1B.

Or open your favorite calculator, and convert Hexadecimal 1B to Decimal (Base 
10).

- Benjamin


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Apache and suexec issue that wont let me run my python script

[Benjamin Schollnick]

On Jun 4, 2013, at 6:12 AM, Νικόλαος Κούρας nikos.gr...@gmail.com wrote:

 Checking the permissions of /var/log/httpd directory itself:
 
 ni...@superhost.gr [~/www/cgi-bin]# ls -ld /var/log/httpd/
 drwx-- 2 root root 4096 Jun  1 02:52 /var/log/httpd//
 
 Is that a problem?
 
 http != Apache ?

Yes, httpd is Apache, or at least part of Apache.

 i'm still confused about what is:
 '/var/log/httpd' and what is '/usr/local/Apache'
 
 Is seems like this is the same service runnign twice under different names.

Not really.  

Unix unlike some other OSes, separates your data from your applications.

That's one reason, when Apple designed Mac OS X, you can re-install Mac OS X 
over your current installation, and not lose any data.  Your user data is 
separate from the OS data.

The /Usr tree is considered read-only.  In theory nothing should write to that 
folder, unless you are installing Unix tools.  Please note, it's not read only 
in the OS, yes, it does require super user rights, but that tree is not read 
only.

The /Var tree is where the OS writes data to.  For example, Log files, 
temporary work files, etc.  

I hope this clears some of this up.

If I have made any mistakes here, please feel free to politely correct me |-)

- Benjamin

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: This mail never gets delivered. Any ideas why?

[Benjamin Schollnick]

Nagia,

 Thank you Thomas but that simple line as i have it now its capable of sending 
 mail successfully wven with greek letters two in subject or in message 
 variables.
 
 Now need to dad additional extra code for it to work.

Your code as it is, isn't working.  They just sent you several suggestions on 
why it might not be working, and your not even testing to see if it might work 
better, or assist in tracking down the problem?

Even if these suggestions were completely unfounded, are they more readable 
then your existing code?  Or more standards compliant?

If you wish to receive help, please at least try to work with the people that 
are trying to help you.

Otherwise, you'll end up with someone helping you that won't have your best 
interests in mind.

- Benjamin

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python Makefiles... are they possible?

[Benjamin Schollnick]

 One thing we do in our Makefiles is find . -name '*.pyc' | xargs rm.
 It avoids all sorts of nasty and hard to track down bugs (consider what
 happens if you move a .py file from one place in your source tree to
 another and leave the old .pyc behind).
 
 How often do you move files around in the source tree? Meanwhile, *every* 
 time you run make, you take a performance hit on every Python module in 
 your project, whether it has moved or not.
 
 Seems to me like a fairly heavy-handed response for something quite rare, 
 but I suppose that depends on how often you run make.

If the performance hit doesn't really matter.  

Then simply walk the build tree, compare time date stamps, anything that 
doesn't match up in the make directory, gets deleted.  Anything that has 
different Date Created / Date Modified time from the build tree match, get's 
deleted.

This way, we are preserving any files that should be identical.  But there 
should be some mechanism documented to forcibly clear the build cache.

- Benjamin

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: sorting 1172026 entries

[Benjamin Schollnick]

On May 6, 2012, at 11:57 AM, J. Mwebaze wrote:

 I have several lists with approx 1172026 entries. I have been trying to sort 
 the records, but have failed.. I tried lists.sort() i also trired sorted 
 python's inbuilt method. This has been running for weeks.
 
 Any one knows of method that can handle such lists. 

The issue there is the sheer size of the list.  I can't think of an algorithium 
that wouldn't have a problem with a list of that size.

Two suggestions

1) Is there no other way to organize this data, other than having it in a 
single list?  You can't organize it by, for example, zip code, area code, year, 
or something, and make multiple lists?  Reducing the size would speed the sort 
up.

2) Maybe consider a different storage method, for example, adding the data into 
a database?  And then connecting to the database via python?

- Benjamin


-- 
http://mail.python.org/mailman/listinfo/python-list

Need advice on Web / Database framework...

[Benjamin Schollnick]

Folks,

I need some advice on a python web  database framework to use...?

I have handcrafted a sqllite3 python script, that is a basic web application, 
interfacing with a sqlite3 database...

But I am concerned at the thought of handcrafting a administration interface, 
and so forth.

Are there any recommendations on a python toolkit / framework that could help 
deal with the realities of this?
The main issue is hand crafting all the forms and html web pages...  

I have done a little cheetah templating...  So that might be a partial 
solution, but I'm concerned with someone trying to craft a malicious payload in 
the fields, and so forth...  

I have thought about an out of the box solution, for example, using a wordpress 
install for the front-end, but I haven't been able to think of a good way to 
bridge this gap.

Would Zope be a good solution possibly?  Any suggestions would be appreciated...

- Ben

-- 
http://mail.python.org/mailman/listinfo/python-list

Small Problem with Weather Underground XML API...

[Benjamin Schollnick]

I am working scraping the Weather Underground using the XML interface...  

I am hoping to to add this into the pywapi, but that looks like it's been 
abandoned?  I haven't seen any updates in ages to it...

And I'm using the Weather Underground XML API 
(http://wiki.wunderground.com/index.php/API_-_XML)...  And it's working, except 
something is happening odd with the Forecast portion...

When parsed the Forecast, the Highs  Lows are the same value...

I don't see another approach to this though.  Weather Underground is presenting 
the same data structure for the forecast, which is why I am breaking it into a 
list...  I'm not the best expert at XML, but I believe that I have etree 
working fine...  But not necessarily the best way, Is there a better way to 
read this via etree?

The only limitation I have is the code has to be python 2.51, due to 
limitations in the Indigo framework...

The scan_node function scans the individual node, and works fine for the 
Weather forecast...  but due to the duplicate XML tags in the forecast XML 
interface, I had to manually break it out into a list...

But this doesn't explain the issue with the high's not being read properly...

Anyone?

WUND_WEATHER_URL= 
'http://api.wunderground.com/auto/wui/geo/WXCurrentObXML/index.xml?query=%s'
WUND_FORECAST_URL   = 
'http://api.wunderground.com/auto/wui/geo/ForecastXML/index.xml?query=%s'
WUND_PWS_WEATHER_URL = 
'http://api.wunderground.com/weatherstation/WXCurrentObXML.asp?ID=%s'

def scan_node ( data, node, ns_wund_data_structure):
for (category, attrs) in ns_wund_data_structure.iteritems():
if node.tag in attrs:
for attrsname in attrs:
if attrsname == node.tag:
if not(category in data.keys() ):
#
#   key not in dictionary, 
create subdictionary
#
data [category] = {}

if node.text  None:
data [category] 
[node.tag.strip()] = node.text.strip()
return data

def get_weather_from_wund(location_id, hl = ''):
url = WUND_WEATHER_URL % (location_id)
handler = urllib2.urlopen(url)
tree = parse ( handler)
handler.close()
weather_data = {}
elem = tree.getroot ()

ns_wund_data_structure = { 
'display_location': ('full', 'city', 'state', 'state_name', 
'country', 'zip', 'latitude', 'longitude', 'elevation'),
'current_observation': ('station_id', 'observation_time', 
'observation_time_rfc822', 'local_time', 'local_time_rfc822',
'local_epoch', 
'weather', 'temperature_string', 'temp_f', 'temp_c', 'relative_humidity',
'wind_string', 
'wind_dir', 'wind_degrees', 'wind_mpg', 'wind_gust', 'pressure_string',
'pressure_mb', 
'pressure_in', 'dewpoint_string', 'dewpoint_f', 'dewpoint_c', 

'heat_index_string', 'heat_index_f', 'heat_index_c', 'windchill_string', 
'windchill_f', 
'windchill_c', 
'visibility_mi', 'visibility_km', 'forceast_url','history_url',
'ob_url', 
'icon_url_base', 'icon_url_name', 'icon', 'forecast_url'),
'icons' : ('icon_set', 'icon_url', 
'icon_url_base', 'icon_url_name', 'icon')
}   

for category in ns_wund_data_structure:
weather_data[category] = {}

for node in elem.getchildren():
children = node.getchildren()
if children  []:
for subnode in children:
weather_data = scan_node( weather_data, 
subnode, ns_wund_data_structure)

else:
weather_data = scan_node ( weather_data, node, 
ns_wund_data_structure)
return weather_data

def walk_tree (root_node, data, dstructure):
for node in root_node.getchildren():
children = node.getchildren()
if children  []:
for subnode in children:
if subnode.getchildren()  []:
walk_tree (subnode, data, dstructure)
else:
data = scan_node ( data, subnode, 
dstructure)
else:

Has anyone seen any updated versions of pywapi? (Python Weather API)

[Benjamin Schollnick]

The Google Code site is at http://code.google.com/p/python-weather-api/

And it's powerful, but I don't see any updates since late 2010...  Does anyone 
know of a different pre-built API for accessing weather information?

- Benjamin

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Has anyone seen any updated versions of pywapi? (Python Weather API)

[Benjamin Schollnick]

Because there was several issues that suggested they were incorporating weather 
underground support, which I am having trouble with...

Can't seem to get the weather underground forecast feed to process correctly 
with their model.  I have hacked in element tree but it's not quite right...  
And I am bashing my head in frustration...

And that was the last check in.  I don't see any updates on the pywapi web site 
for quite a bit longer... I don't mind using a toolkit if it's supported, but 
if it's abandoned, well then I am asking for trouble down the road, assuming I 
use that toolkit.

   - Ben


--
Sent from my Mobile Communication Device.


On Jun 5, 2011, at 10:31 PM, Brian Curtin brian.cur...@gmail.com wrote:

 On Sun, Jun 5, 2011 at 19:27, Benjamin Schollnick 
 bscholln...@schollnick.net wrote:
 The Google Code site is at http://code.google.com/p/python-weather-api/
 
 And it's powerful, but I don't see any updates since late 2010...  Does 
 anyone know of a different pre-built API for accessing weather information?
  
 Why does it matter if it hasn't been modified in ~6 months? 
-- 
http://mail.python.org/mailman/listinfo/python-list

Perl conversion to python...

[Benjamin Schollnick]

Folks,

I'm having some issues here with pyserial  trying to translate a perl
script to python...  It's probably my inexperience with PySerial 
perl that is troubling me...

Can anyone assist?

I'm concerned, since I can't seem to receive the data in any reliable
manner..  I've tested multiple times, and only once received data...
So I suspect that my Transmit  receive code is faulty...

def xmit ( data, serialport ):
for x in data:
xmit_byte (x, serialport)
#   serialport.write ( binascii.unhexlify ( data ) )
#   for x in data:
#   print str(x).encode ('hex')
#   serialport.write ( x.encode('hex'))

def receive ( serialport ):
received = serialport.read (20)
print received, !

- Perl Code 
sub tx_command {
my $port = shift;
my $cmd = shift;

#   warn tx_command($cmd)\n;

my @cmd_bytes = split(/\s/, $cmd);

foreach my $byte (@cmd_bytes) {
$byte = pack('C', hex($byte));

$port - write($byte);
select(undef, undef, undef, 0.01);
}
}

# returns the rtt, or 0 if no response
sub rx_response {
my ($port, $address) = @_;

$port-read_char_time(0); # don't wait for each character
$port-read_const_time(5000); # timeout if we don't get what we're
looking for

my $buf = '';

my $t_start = time;

### accumulate one byte at a time until we see the substring we're
looking for

while (1) {
my ($count_in, $string_in) = $port-read(1);

if ($count_in == 0) {
#   warn TIMEOUT\n;
return 0;
}

$buf .= $string_in;

my $bufstring = packed_to_text($buf);

#warn bufstring: .$bufstring;

if ($bufstring =~/02 50 $address (.. .. ..) (..) (.. ..)/) {

my $powerlinc_addr = $1;
my $flags = $2;
my $command = $3;

#   warn got response!\n;

my $rtt = time() - $t_start;

return $rtt;
}

}
}

-- 
http://mail.python.org/mailman/listinfo/python-list

SSL via Proxy (URLLIB2) on Windows gives Unknown Protocol error?

[Benjamin Schollnick]

Folks,

With Windows XP, and Python v2.41 I am running into a problem

The following code gives me an unknown protocol error And I am not
sure how
to resolve it...

I have a API written for DocuShare for a non-SSL server, and I wanted
to update it
to support the SSL/S3 login

So here's the SSL test code, and I can't get the page to retrieve

import urllib2
proxy_handler = urllib2.ProxyHandler({'http':
'http://www.wb.xerox.com:8000',
  'https':
'http://www.wb.xerox.com:8000'})

request =
urllib2.Request(url=https://www.docushare.xerox.com;)#+server_url)
request.set_proxy( www.wb.xerox.com:8000, https)


stream = urllib2.urlopen( request )

print '\n'.join(stream.readlines())

sys.exit(5)

The error is:

urllib2.URLError: urlopen error (1, 'error:140770FC:SSL
routines:SSL23_GET_SERVER_HELLO:unknown protocol')

As far as I can tell the Python 2.41 MSI bundle includes the SSL code?

C:\develope\docusharepython
Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)]
on win32
Type help, copyright, credits or license for more information.
 import socket
 print socket.ssl
function ssl at 0x00BC95B0

Anyone have any suggestions?

 - Ben

-- 
http://mail.python.org/mailman/listinfo/python-list