Re: Best way to clean up list items?
On 5/2/2016 2:27 PM, Jussi Piitulainen wrote:
DFS writes:
On 5/2/2016 12:57 PM, Jussi Piitulainen wrote:
DFS writes:
Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n ']
Want: list1 = ['Item 1','Item 2']
. .
Funny-looking data you have.
I know - sadly, it's actual data:
from lxml import html
import requests
webpage =
"http://www.usdirectory.com/ypr.aspx?fromform=qsearch&qs=TN&wqhqn=2&qc=Nashville&rg=30&qhqn=restaurant&sb=zipdisc&ap=2";
page = requests.get(webpage)
tree = html.fromstring(page.content)
addr1 = tree.xpath('//span[@class="text3"]/text()')
print 'Addresses: ', addr1
I couldn't figure out a better way to extract it from the HTML (maybe
XML and DOM?)
I should have guessed :) But now I'm a bit worried about those spaces
inside your items. Can it happen that item text is split into strings in
the middle?
Meaning split by me, or comes 'malformed' from the data source?
Then the above sanitation does the wrong thing.
If someone has the right solution, I'm watching, too.
Here's the raw data as stored in the tree:
---
1st page
['\r\n', '\r\n1918 W End
Ave, Nashville, TN 37203', '\r\n
', '\r\n1806 Hayes St, Nashville,
TN 37203', '\r\n', '\r\n
1701 Broadway, Nashville, TN 37203', '\r\n', '\r\n
209 10th Ave S, Nashville, TN 37203', '\r\n
', '\r\n907 20th Ave S, Nashville, TN
37212', '\r\n', '\r\n911
20th Ave S, Nashville, TN 37212', '\r\n', '\r\n
1722 W End Ave, Nashville, TN 37203', '\r\n
', '\r\n1905 Hayes St,
Nashville, TN 37203', '\r\n
', '\r\n2000 W End Ave,
Nashville, TN 37203']
---
Next page
['\r\n', '\r\n120 19th
Ave N, Nashville, TN 37203', '\r\n
', '\r\n1719 W End Ave Ste 101,
Nashville, TN 37203', '\r\n
', '\r\n1922 W End Ave, Nashville, TN
37203', '\r\n', '\r\n
909 20th Ave S, Nashville, TN 37212', '\r\n
', '\r\n
1807 Church St, Nashville, TN 37203', '\r\n
', '\r\n1721 Church St, Nashville, TN 37203',
'\r\n', '\r\n718
Division St, Nashville, TN 37203', '\r\n', '\r\n
907 12th Ave S, Nashville, TN 37203', '\r\n
', '\r\n204 21st Ave S,
Nashville, TN 37203', '\r\n
', '\r\n1811 Division St, Nashville,
TN 37203', '\r\n', '\r\n
903 Gleaves St, Nashville, TN 37203', '\r\n', '\r\n
1720 W End Ave Ste 530, Nashville, TN 37203', '\r\n
', '\r\n
1200 Division St Ste 100-A, Nashville, TN 37203', '\r\n
', '\r\n
422 7th Ave S, Nashville, TN 37203', '\r\n',
'\r\n605 8th Ave S, Nashville, TN 37203']
and so on
---
I've checked a couple hundred addresses visually, and so far I've only
seen 2 formats:
1. '\r\n'
2. '\r\n address '
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best way to clean up list items?
DFS writes:
> On 5/2/2016 12:57 PM, Jussi Piitulainen wrote:
>> DFS writes:
>>
>>> Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n ']
>>> Want: list1 = ['Item 1','Item 2']
. .
>> Funny-looking data you have.
>
> I know - sadly, it's actual data:
>
>
> from lxml import html
> import requests
>
> webpage =
> "http://www.usdirectory.com/ypr.aspx?fromform=qsearch&qs=TN&wqhqn=2&qc=Nashville&rg=30&qhqn=restaurant&sb=zipdisc&ap=2";
>
> page = requests.get(webpage)
> tree = html.fromstring(page.content)
> addr1 = tree.xpath('//span[@class="text3"]/text()')
> print 'Addresses: ', addr1
>
>
> I couldn't figure out a better way to extract it from the HTML (maybe
> XML and DOM?)
I should have guessed :) But now I'm a bit worried about those spaces
inside your items. Can it happen that item text is split into strings in
the middle? Then the above sanitation does the wrong thing.
If someone has the right solution, I'm watching, too.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best way to clean up list items?
On Mon, May 2, 2016, at 11:09 AM, DFS wrote: > I'd prefer to get clean data in the first place, but I don't know a > better way to extract it from the HTML. Ah, right. I didn't know you were scraping HTML. Scraping HTML is rarely clean so you have to do a lot of cleanup. -- Stephen Hansen m e @ i x o k a i . i o -- https://mail.python.org/mailman/listinfo/python-list
Re: Best way to clean up list items?
On 5/2/2016 12:57 PM, Jussi Piitulainen wrote:
DFS writes:
Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n ']
Want: list1 = ['Item 1','Item 2']
I wrote this, which works fine, but maybe it can be tidier?
1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n
2. list3 = [t.strip(' ') for t in list2]#trim whitespace
3. list1 = filter(None, list3) #remove empty items
After each step:
1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n
2. list3 = ['Item 1','Item 2',''] #trim whitespace
3. list1 = ['Item 1','Item 2'] #remove empty items
Try filter(None, (t.strip() for t in list1)). The default.
Works and drops a line of code. Thx.
Funny-looking data you have.
I know - sadly, it's actual data:
from lxml import html
import requests
webpage =
"http://www.usdirectory.com/ypr.aspx?fromform=qsearch&qs=TN&wqhqn=2&qc=Nashville&rg=30&qhqn=restaurant&sb=zipdisc&ap=2";
page = requests.get(webpage)
tree = html.fromstring(page.content)
addr1 = tree.xpath('//span[@class="text3"]/text()')
print 'Addresses: ', addr1
I couldn't figure out a better way to extract it from the HTML (maybe
XML and DOM?)
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best way to clean up list items?
On 5/2/2016 1:25 PM, Stephen Hansen wrote:
On Mon, May 2, 2016, at 09:33 AM, DFS wrote:
Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n ']
I'm curious how you got to this point, it seems like you can solve the
problem in how this is generated.
from lxml import html
import requests
webpage =
"http://www.usdirectory.com/ypr.aspx?fromform=qsearch&qs=TN&wqhqn=2&qc=Nashville&rg=30&qhqn=restaurant&sb=zipdisc&ap=2";
page = requests.get(webpage)
tree = html.fromstring(page.content)
addr1 = tree.xpath('//span[@class="text3"]/text()')
print 'Addresses: ', addr1
I'd prefer to get clean data in the first place, but I don't know a
better way to extract it from the HTML.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best way to clean up list items?
DFS wrote:
> Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n ']
> Want: list1 = ['Item 1','Item 2']
>
>
> I wrote this, which works fine, but maybe it can be tidier?
>
> 1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n
> 2. list3 = [t.strip(' ') for t in list2]#trim whitespace
> 3. list1 = filter(None, list3) #remove empty items
>
>
> After each step:
>
> 1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n
> 2. list3 = ['Item 1','Item 2',''] #trim whitespace
> 3. list1 = ['Item 1','Item 2'] #remove empty items
>
>
> Thanks!
s.strip() strips all whitespace, so you can combine steps 1 and 2:
>>> items = ['\r\n Item 1 ',' Item 2 ','\r\n ']
>>> stripped = (s.strip() for s in items)
The (...) instead of [...] denote a generator expression, so the iteration
has not started yet. The final step uses a list comprehension instead of
filter():
>>> [s for s in stripped if s]
['Item 1', 'Item 2']
That way the same code works with both Python 2 and Python 3. Note that you
can iterate over the generator expression only once; if you try it again
you'll end empty-handed:
>>> [s for s in stripped if s]
[]
If you want to do it in one step here are two options that both involve some
duplicate work:
>>> [s.strip() for s in items if s and not s.isspace()]
['Item 1', 'Item 2']
>>> [s.strip() for s in items if s.strip()]
['Item 1', 'Item 2']
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best way to clean up list items?
On Mon, May 2, 2016, at 09:33 AM, DFS wrote: > Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n '] I'm curious how you got to this point, it seems like you can solve the problem in how this is generated. > Want: list1 = ['Item 1','Item 2'] That said: list1 = [t.strip() for t in list1 if t and not t.isspace()] -- Stephen Hansen m e @ i x o k a i . i o -- https://mail.python.org/mailman/listinfo/python-list
Re: Best way to clean up list items?
On May 2, 2016 10:03 AM, "Jussi Piitulainen"
wrote:
>
> DFS writes:
>
> > Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n ']
> > Want: list1 = ['Item 1','Item 2']
> >
> >
> > I wrote this, which works fine, but maybe it can be tidier?
> >
> > 1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n
> > 2. list3 = [t.strip(' ') for t in list2]#trim whitespace
> > 3. list1 = filter(None, list3) #remove empty items
> >
> > After each step:
> >
> > 1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n
> > 2. list3 = ['Item 1','Item 2',''] #trim whitespace
> > 3. list1 = ['Item 1','Item 2'] #remove empty items
You could also try compiled regex to remove unwanted characters.
Then loop through the list and do a replace for each item.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Best way to clean up list items?
DFS writes:
> Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n ']
> Want: list1 = ['Item 1','Item 2']
>
>
> I wrote this, which works fine, but maybe it can be tidier?
>
> 1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n
> 2. list3 = [t.strip(' ') for t in list2]#trim whitespace
> 3. list1 = filter(None, list3) #remove empty items
>
> After each step:
>
> 1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n
> 2. list3 = ['Item 1','Item 2',''] #trim whitespace
> 3. list1 = ['Item 1','Item 2'] #remove empty items
Try filter(None, (t.strip() for t in list1)). The default.
Funny-looking data you have.
--
https://mail.python.org/mailman/listinfo/python-list
Best way to clean up list items?
Have: list1 = ['\r\n Item 1 ',' Item 2 ','\r\n ']
Want: list1 = ['Item 1','Item 2']
I wrote this, which works fine, but maybe it can be tidier?
1. list2 = [t.replace("\r\n", "") for t in list1] #remove \r\n
2. list3 = [t.strip(' ') for t in list2]#trim whitespace
3. list1 = filter(None, list3) #remove empty items
After each step:
1. list2 = [' Item 1 ',' Item 2 ',' '] #remove \r\n
2. list3 = ['Item 1','Item 2',''] #trim whitespace
3. list1 = ['Item 1','Item 2'] #remove empty items
Thanks!
--
https://mail.python.org/mailman/listinfo/python-list
