Re: Error getting data from website

[Chris Angelico]

On Sat, Dec 7, 2019 at 1:21 PM DL Neil via Python-list
 wrote:
>
> On 7/12/19 1:51 PM, Chris Angelico wrote:
> > On Sat, Dec 7, 2019 at 11:46 AM Michael Torrie  wrote:
> >>
> >> On 12/6/19 5:31 PM, DL Neil via Python-list wrote:
> >>> If you read the HTML data that the REPL has happily splattered all over
> >>> your terminal's screen (scroll back) (NB "soup" is easier to read than
> >>> is "content"!) you will observe that what you saw in your web-browser is
> >>> not what Amazon served in response to the Python "requests.get()"!
> >>
> >> Sadly it's likely that Amazon's page is largely built from javascript.
> >> So scraping static html is probably not going to get you where you want
> >> to go.  There are heavier tools, such as Selenium that uses a real
> >> browser to grab a page, and the result of that you can parse and search
> >> perhaps.
> >
> > Or look for an API instead.
>
>
> Both +1
> However, Selenium is possibly less-manageable for a 'beginner'.
> (NB my poorly-based assumption of OP)
>
> Amazon's HTML-response actually says this/these, but I left it open as a
> (learning) exercise for the OP. They likely prefer the API approach,
> because it can be measured...
>

Yes, and because it's way WAY easier to guarantee API stability than
Selenium-based page parseability.

But even when there's no *actual* API, you can sometimes delve into
the page and find the actual useful content, perhaps as a big blob of
JSON inside a 

Re: Error getting data from website

[DL Neil via Python-list]

On 7/12/19 1:51 PM, Chris Angelico wrote:

On Sat, Dec 7, 2019 at 11:46 AM Michael Torrie  wrote:


On 12/6/19 5:31 PM, DL Neil via Python-list wrote:

If you read the HTML data that the REPL has happily splattered all over
your terminal's screen (scroll back) (NB "soup" is easier to read than
is "content"!) you will observe that what you saw in your web-browser is
not what Amazon served in response to the Python "requests.get()"!


Sadly it's likely that Amazon's page is largely built from javascript.
So scraping static html is probably not going to get you where you want
to go.  There are heavier tools, such as Selenium that uses a real
browser to grab a page, and the result of that you can parse and search
perhaps.


Or look for an API instead.



Both +1
However, Selenium is possibly less-manageable for a 'beginner'.
(NB my poorly-based assumption of OP)

Amazon's HTML-response actually says this/these, but I left it open as a 
(learning) exercise for the OP. They likely prefer the API approach, 
because it can be measured...


--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Error getting data from website

[Chris Angelico]

On Sat, Dec 7, 2019 at 11:46 AM Michael Torrie  wrote:
>
> On 12/6/19 5:31 PM, DL Neil via Python-list wrote:
> > If you read the HTML data that the REPL has happily splattered all over
> > your terminal's screen (scroll back) (NB "soup" is easier to read than
> > is "content"!) you will observe that what you saw in your web-browser is
> > not what Amazon served in response to the Python "requests.get()"!
>
> Sadly it's likely that Amazon's page is largely built from javascript.
> So scraping static html is probably not going to get you where you want
> to go.  There are heavier tools, such as Selenium that uses a real
> browser to grab a page, and the result of that you can parse and search
> perhaps.

Or look for an API instead.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Error getting data from website

[Michael Torrie]

On 12/6/19 5:31 PM, DL Neil via Python-list wrote:
> If you read the HTML data that the REPL has happily splattered all over 
> your terminal's screen (scroll back) (NB "soup" is easier to read than 
> is "content"!) you will observe that what you saw in your web-browser is 
> not what Amazon served in response to the Python "requests.get()"!

Sadly it's likely that Amazon's page is largely built from javascript.
So scraping static html is probably not going to get you where you want
to go.  There are heavier tools, such as Selenium that uses a real
browser to grab a page, and the result of that you can parse and search
perhaps.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Error getting data from website

[DL Neil via Python-list]

On 7/12/19 12:53 PM, Sam Paython wrote:

This is the code I am writing:
import requests
from bs4 import BeautifulSoup
request = requests.get("https://www.amazon.ca/dp/B07RZFQ6HC";)
content = request.content
soup = BeautifulSoup(content, "html.parser")
element = soup.find("span",{"id":"priceblock_dealprice"})
print(element.text.strip())

and this is the error I am getting:
C:\Users\Sam\PycharmProjects\untitled2\venv\Scripts\python.exe 
C:/Users/Sam/PycharmProjects/untitled2/src/app.py
Traceback (most recent call last):
   File "C:/Users/Sam/PycharmProjects/untitled2/src/app.py", line 9, in 
 print(element.text.strip())
AttributeError: 'NoneType' object has no attribute 'text'

Could someone please help?



The err.msg/stack-trace is your friend! The comment about "NoneType" 
means 'there's nothing there' (roughly!) to print().


The question then becomes: "why?" or "why not?"...

With a short piece of code like this, and (I am assuming) trying-out a 
library for the first time, may I recommend that you use the Python 
REPL, because it allows you to 'see' what's going-on behind the 
scenes/underneath the hood - and ultimately, reveals the problem.


From a Python terminal (cmd is appropriate to your PC's OpSys):

[dn@JrBrown ~]$ python3
Python 3.7.4 (default, Jul  9 2019, 16:48:28)
[GCC 8.3.1 20190223 (Red Hat 8.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> from bs4 import BeautifulSoup
>>> request = requests.get("https://www.amazon.ca/dp/B07RZFQ6HC";)
>>> request# notice how I'm asking to 'see' what happened

>>> content = request.content
>>> content# there is no need to enclose in print()!
b'\n

Error getting data from website

[Sam Paython]

Hi all,

This is the code I am writing:

import requests
from bs4 import BeautifulSoup


request = requests.get("https://www.amazon.ca/dp/B07RZFQ6HC";)
content = request.content
soup = BeautifulSoup(content, "html.parser")
element = soup.find("span",{"id":"priceblock_dealprice"})
print(element.text.strip())


and this is the error I am getting: 

C:\Users\Sam\PycharmProjects\untitled2\venv\Scripts\python.exe 
C:/Users/Sam/PycharmProjects/untitled2/src/app.py
Traceback (most recent call last):
  File "C:/Users/Sam/PycharmProjects/untitled2/src/app.py", line 9, in 
print(element.text.strip())
AttributeError: 'NoneType' object has no attribute 'text'

Could someone please help?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unicode filenames

[Terry Reedy]

On 12/6/2019 1:17 PM, Bob van der Poel wrote:

I have some files which came off the net with, I'm assuming, unicode
characters in the names. I have a very short program which takes the
filename and puts into an emacs buffer, and then lets me add information to
that new file (it's a poor man's DB).

Next, I can look up text in the file and open the saved filename.
Everything works great until I hit those darn unicode filenames.

Just to confuse me even more, the error seems to be coming from a bit of
tkinter code:
  if sresults.has_key(textAtCursor):
 bookname = os.path.expanduser(sresults[textAtCursor].strip())


'textAtCursor' does not appear in any 3.9 tkinter/*.py file


which generates

   UnicodeWarning: Unicode equal comparison failed to convert both arguments
to Unicode - interpreting them as being unequal  if
sresults.has_key(textAtCursor):

I really don't understand the business about "both arguments".


'sresults.has_key(textAtCursor)' will see if the hash value of 
textAtCursor matches the hash value of any key and then compare the 
strings.  'failed to convert' suggests to me that you are running 2.x 
and that one of the strings is bytes and the other unicode.



 Not sure how

to proceed here. Hoping for a guideline!

Thanks.





--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list

Re: Make warning an exception?

[DL Neil via Python-list]

On 7/12/19 9:58 AM, Israel Brewster wrote:

I was running some code and I saw this pop up in the console:

2019-12-06 11:53:54.087 Python[85524:39651849] WARNING: nextEventMatchingMask 
should only be called from the Main Thread! This will throw an exception in the 
future.

The only problem is, I have no idea what is generating that warning - I never 
call nextEventMatchingMask directly, so it must be getting called from one of 
the libraries I’m calling. Is there some way I can force python to throw an 
exception now, so my debugger can catch it and let me know where in my code the 
originating call is? I’ve tried stepping through the obvious options, with no 
luck so far.



We are able to "filter" errors, including turning warnings into 
full-bore errors. Of possible use: 
https://docs.python.org/3/library/warnings.html


--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Re: Make warning an exception?

[Rob Gaddi]

On 12/6/19 12:58 PM, Israel Brewster wrote:

I was running some code and I saw this pop up in the console:

2019-12-06 11:53:54.087 Python[85524:39651849] WARNING: nextEventMatchingMask 
should only be called from the Main Thread! This will throw an exception in the 
future.

The only problem is, I have no idea what is generating that warning - I never 
call nextEventMatchingMask directly, so it must be getting called from one of 
the libraries I’m calling. Is there some way I can force python to throw an 
exception now, so my debugger can catch it and let me know where in my code the 
originating call is? I’ve tried stepping through the obvious options, with no 
luck so far.

---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145



You need to set the warning filter to "error", which you can do either with 
warnings.simplefilter at the start of your program or by setting the 
PYTHONWARNINGS environment variable.


https://docs.python.org/3/library/warnings.html#the-warnings-filter

This the same project you're having PySide/threading problems on?
--
https://mail.python.org/mailman/listinfo/python-list

Make warning an exception?

[Israel Brewster]

I was running some code and I saw this pop up in the console:

2019-12-06 11:53:54.087 Python[85524:39651849] WARNING: nextEventMatchingMask 
should only be called from the Main Thread! This will throw an exception in the 
future.

The only problem is, I have no idea what is generating that warning - I never 
call nextEventMatchingMask directly, so it must be getting called from one of 
the libraries I’m calling. Is there some way I can force python to throw an 
exception now, so my debugger can catch it and let me know where in my code the 
originating call is? I’ve tried stepping through the obvious options, with no 
luck so far.

---
Israel Brewster
Software Engineer
Alaska Volcano Observatory 
Geophysical Institute - UAF 
2156 Koyukuk Drive 
Fairbanks AK 99775-7320
Work: 907-474-5172
cell:  907-328-9145

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Unicode filenames

[DL Neil via Python-list]

On 7/12/19 7:17 AM, Bob van der Poel wrote:

I have some files which came off the net with, I'm assuming, unicode
characters in the names. I have a very short program which takes the
filename and puts into an emacs buffer, and then lets me add information to
that new file (it's a poor man's DB).

Next, I can look up text in the file and open the saved filename.
Everything works great until I hit those darn unicode filenames.

Just to confuse me even more, the error seems to be coming from a bit of
tkinter code:
  if sresults.has_key(textAtCursor):
 bookname = os.path.expanduser(sresults[textAtCursor].strip())

which generates

   UnicodeWarning: Unicode equal comparison failed to convert both arguments
to Unicode - interpreting them as being unequal  if
sresults.has_key(textAtCursor):

I really don't understand the business about "both arguments". Not sure how
to proceed here. Hoping for a guideline!



(I'm guessing that) the "both arguments" relates to expanduser() because 
this is the first time that the fileNM has been identified to Python as 
anything more than a string of characters.


[a fileNM will be a string of characters, but a string of characters is 
not necessarily a (legal) fileNM!]


Further suggesting, that if you are using Python3 (cf 2), your analysis 
may be the wrong-way-around. Python3 treats strings as Unicode. However, 
there is, and certainly in the past, was, no requirement for OpSys and 
IOCS to encode in Unicode.


The problem (for me) came from MSFT's (for example) many variations of 
ISO-8859-n and that there are no clues as to which of these was used in 
naming the file, and thus many possibly 'translations' into Unicode.


You can start to address the issue by using Python's bytes (instead of 
strings), however that cold reality still intrudes.


Do you know the provenance of these files, eg they are in French and 
from an MS-Win machine? If so, you may be able to use decode() and 
encode(), but...


Still looking for trouble? Knowing a fileNM was in Spanish/Portuguese I 
was able to take the fileNM's individual Unicode characters/surrogates 
and subtract an applicable constant, so that accented letters fell 
'back' into the correct Unicode range. (this is extremely risky, and 
could quite easily make matters worse/more confusing).


I warn you that pursuing this matter involves disappearing down into a 
very deep 'rabbit hole', but YMMV!


WebRefs:
https://docs.python.org/3/howto/unicode.html
https://www.dictionary.com/e/slang/rabbit-hole/
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Unicode filenames

[Bob van der Poel]

I have some files which came off the net with, I'm assuming, unicode
characters in the names. I have a very short program which takes the
filename and puts into an emacs buffer, and then lets me add information to
that new file (it's a poor man's DB).

Next, I can look up text in the file and open the saved filename.
Everything works great until I hit those darn unicode filenames.

Just to confuse me even more, the error seems to be coming from a bit of
tkinter code:
 if sresults.has_key(textAtCursor):
bookname = os.path.expanduser(sresults[textAtCursor].strip())

which generates

  UnicodeWarning: Unicode equal comparison failed to convert both arguments
to Unicode - interpreting them as being unequal  if
sresults.has_key(textAtCursor):

I really don't understand the business about "both arguments". Not sure how
to proceed here. Hoping for a guideline!

Thanks.


-- 

 Listen to my FREE CD at http://www.mellowood.ca/music/cedars 
Bob van der Poel ** Wynndel, British Columbia, CANADA **
EMAIL: b...@mellowood.ca
WWW:   http://www.mellowood.ca
-- 
https://mail.python.org/mailman/listinfo/python-list

Zipapp can't find sqlite db

[Abdur-Rahmaan Janhangeer]

Greetings,

I'm using zipapp to include a gui + db

__main__.py
dbs/
file.db

When packaging, the db is there. When querying through sqlalchemy, it says
can't open db file. Help appreciated!

Yours,

Abdur-Rahmaan Janhangeer
pythonmembers.club  | github

Mauritius
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: IPC10 archive anywhere?

[Skip Montanaro]

> I've poked around a bit and have been unable to come up with an
> archive of the papers delivered at IPC10 (2001, I believe - pre-PyCon
> days). Might anyone have a link?

Found it a few minutes later:

https://legacy.python.org/workshops/

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

IPC10 archive anywhere?

[Skip Montanaro]

I've poked around a bit and have been unable to come up with an
archive of the papers delivered at IPC10 (2001, I believe - pre-PyCon
days). Might anyone have a link?

Thanks,

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list