Re: Python read text file columnwise
On 2019-01-15, Juris __ wrote:
> Hi!
>
> On 15/01/2019 17:04, Neil Cerutti wrote:
>> On 2019-01-11, shibashib...@gmail.com wrote:
>>> Hello
I'm very new in python. I have a file in the format:
2018-05-31 16:00:0028.90 81.77 4.3
2018-05-31 20:32:0028.17 84.89 4.1
2018-06-20 04:09:0027.36 88.01 4.8
2018-06-20 04:15:0027.31 87.09 4.7
2018-06-28 04.07:0027.87 84.91 5.0
2018-06-29 00.42:0032.20 104.61 4.8
>>>
>>> I would like to read this file in python column-wise.
>>>
>>> I tried this way but not working
>>>event_list = open('seismicity_R023E.txt',"r")
>>> info_event = read(event_list,'%s %s %f %f %f %f\n');
>>
>> If it's really tabular data in fixed-width columns you can read
>> it that way with Python.
>>
>> records = []
>> for line in file:
>> record = []
>> i = 0
>> for width in (30, 8, 7, 5): # approximations
>> item = line[i:i+width]
>> record.append(item)
>> i += width
>> records.append(record)
>>
>> This leaves them all strings, which in my experience is more
>> convenient in practice. You can convert as you go if you
>> want,though it won't look nice and simple any longer.
>>
>
> Perhaps even better approach is to use csv module from standard library:
>
> import csv
>
> csv_reader = csv.reader(file, dialect="excel-tab")
> for row in csv_reader:
> # do something with record data which is conveniently parsed to list
> print(row)
>
> ['2018-05-31', '16:00:00', '28.90', '81.77', '4.3']
> ...
> ['2018-06-29', '00.42:00', '32.20', '104.61', '4.8']
Yes, if applicable it is awesome!
--
Neil Cerutti
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python read text file columnwise
Hi!
On 15/01/2019 17:04, Neil Cerutti wrote:
> On 2019-01-11, shibashib...@gmail.com wrote:
>> Hello
>>>
>>> I'm very new in python. I have a file in the format:
>>>
>>> 2018-05-31 16:00:0028.90 81.77 4.3
>>> 2018-05-31 20:32:0028.17 84.89 4.1
>>> 2018-06-20 04:09:0027.36 88.01 4.8
>>> 2018-06-20 04:15:0027.31 87.09 4.7
>>> 2018-06-28 04.07:0027.87 84.91 5.0
>>> 2018-06-29 00.42:0032.20 104.61 4.8
>>
>> I would like to read this file in python column-wise.
>>
>> I tried this way but not working
>>event_list = open('seismicity_R023E.txt',"r")
>> info_event = read(event_list,'%s %s %f %f %f %f\n');
>
> If it's really tabular data in fixed-width columns you can read
> it that way with Python.
>
> records = []
> for line in file:
> record = []
> i = 0
> for width in (30, 8, 7, 5): # approximations
> item = line[i:i+width]
> record.append(item)
> i += width
> records.append(record)
>
> This leaves them all strings, which in my experience is more
> convenient in practice. You can convert as you go if you
> want,though it won't look nice and simple any longer.
>
Perhaps even better approach is to use csv module from standard library:
import csv
csv_reader = csv.reader(file, dialect="excel-tab")
for row in csv_reader:
# do something with record data which is conveniently parsed to list
print(row)
['2018-05-31', '16:00:00', '28.90', '81.77', '4.3']
...
['2018-06-29', '00.42:00', '32.20', '104.61', '4.8']
BR, Juris
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python read text file columnwise
On 2019-01-11, shibashib...@gmail.com wrote:
> Hello
>>
>> I'm very new in python. I have a file in the format:
>>
>> 2018-05-31 16:00:0028.90 81.77 4.3
>> 2018-05-31 20:32:0028.17 84.89 4.1
>> 2018-06-20 04:09:0027.36 88.01 4.8
>> 2018-06-20 04:15:0027.31 87.09 4.7
>> 2018-06-28 04.07:0027.87 84.91 5.0
>> 2018-06-29 00.42:0032.20 104.61 4.8
>
> I would like to read this file in python column-wise.
>
> I tried this way but not working
> event_list = open('seismicity_R023E.txt',"r")
> info_event = read(event_list,'%s %s %f %f %f %f\n');
If it's really tabular data in fixed-width columns you can read
it that way with Python.
records = []
for line in file:
record = []
i = 0
for width in (30, 8, 7, 5): # approximations
item = line[i:i+width]
record.append(item)
i += width
records.append(record)
This leaves them all strings, which in my experience is more
convenient in practice. You can convert as you go if you
want,though it won't look nice and simple any longer.
--
Neil Cerutti
--
https://mail.python.org/mailman/listinfo/python-list
RE: Python read text file columnwise
About the original question: If I were you, I would put the 3 numbers into a
list (or a tuple, if you don't need to modify them) and put this into a
dictionary. The key would be the date & time string.
Then, if you need to find a particular entry you can look it up by date and
time. But I suspect, since you want column access, you won't need to do that.
You can iterate through the entries in the dictionary easily and extract the
data from a column, or from all the columns, if that’s what you want.
for entry in mydict:
value = entry.datalist[0] # I hope I have the syntax correct
Now, what you do with value is up to you. I think personally rather than
building a list I would make a generator function. A generator uses a "yield"
statement to return a value, and it waits in that state. The next time you
call it it continues and returns the next value. Kind of useful when the
alternative is making and passing around huge lists.
--- Joseph S.
-Original Message-
From: DL Neil
Sent: Saturday, January 12, 2019 4:48 PM
To: python-list@python.org
Subject: Re: Python read text file columnwise
On 12/01/19 1:03 PM, Piet van Oostrum wrote:
> shibashib...@gmail.com writes:
>
>> Hello
>>>
>>> I'm very new in python. I have a file in the format:
>>>
>>> 2018-05-31 16:00:0028.90 81.77 4.3
>>> 2018-05-31 20:32:0028.17 84.89 4.1
>>> 2018-06-20 04:09:0027.36 88.01 4.8
>>> 2018-06-20 04:15:0027.31 87.09 4.7
>>> 2018-06-28 04.07:0027.87 84.91 5.0
>>> 2018-06-29 00.42:0032.20 104.61 4.8
>>
>> I would like to read this file in python column-wise.
>>
>> I tried this way but not working
>>event_list = open('seismicity_R023E.txt',"r")
>> info_event = read(event_list,'%s %s %f %f %f %f\n');
To the OP:
Python's standard I/O is based around data "streams". Whilst there is a concept
of "lines" and thus an end-of-line character, there is not the idea of a
record, in the sense of fixed-length fields and thus a defining and distinction
between data items based upon position.
Accordingly, whilst the formatting specification of strings and floats might
work for output, there is no equivalent for accepting input data.
Please re-read refs on file, read, readline, etc.
> Why would you think that this would work?
To the PO:
Because in languages/libraries built around fixed-length files this is
how one specifies the composition of fields making up a record - a data
structure which dates back to FORTRAN and Assembler on mainframes and
other magtape-era machines.
Whilst fixed-length records/files are, by definition, less flexible than
the more free-form data input Python accepts, they are more efficient
and faster in situations where the data (format) is entirely consistent
- such as the OP is describing!
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list
RE: Python read text file columnwise
-Original Message-
From: Avi Gross
Sent: Saturday, January 12, 2019 8:26 PM
To: 'DL Neil'
Subject: RE: Python read text file columnwise
I am not sure what the big deal is here. If the data is consistently
formatted you can read in a string per line and use offsets as in line[0:8]
and so on then call the right transformations to comvert them to dates and
so on. If it is delimited by something consistent like spaces or table or
commas, we have all kinds of solutions ranging from splitting the line on
the delimiter to using the kind of functionality that reads in such files
into a pandas DataFrame.
In the latter case, you get the columns already. In the former, there are
well known ways to extract the info such as:
[row[0] for row in listofrows]
And repeat for additional items.
Or am I missing something and there is no end of line and you need to read
in the entire file and split it into size N chunks first? Still fairly
straightforward.
-Original Message-
From: Python-list On
Behalf Of DL Neil
Sent: Saturday, January 12, 2019 4:48 PM
To: python-list@python.org
Subject: Re: Python read text file columnwise
On 12/01/19 1:03 PM, Piet van Oostrum wrote:
> shibashib...@gmail.com writes:
>
>> Hello
>>>
>>> I'm very new in python. I have a file in the format:
>>>
>>> 2018-05-31 16:00:0028.90 81.77 4.3
>>> 2018-05-31 20:32:0028.17 84.89 4.1
>>> 2018-06-20 04:09:0027.36 88.01 4.8
>>> 2018-06-20 04:15:0027.31 87.09 4.7
>>> 2018-06-28 04.07:0027.87 84.91 5.0
>>> 2018-06-29 00.42:0032.20 104.61 4.8
>>
>> I would like to read this file in python column-wise.
>>
>> I tried this way but not working
>>event_list = open('seismicity_R023E.txt',"r")
>> info_event = read(event_list,'%s %s %f %f %f %f\n');
To the OP:
Python's standard I/O is based around data "streams". Whilst there is a
concept of "lines" and thus an end-of-line character, there is not the idea
of a record, in the sense of fixed-length fields and thus a defining and
distinction between data items based upon position.
Accordingly, whilst the formatting specification of strings and floats might
work for output, there is no equivalent for accepting input data.
Please re-read refs on file, read, readline, etc.
> Why would you think that this would work?
To the PO:
Because in languages/libraries built around fixed-length files this is how
one specifies the composition of fields making up a record - a data
structure which dates back to FORTRAN and Assembler on mainframes and other
magtape-era machines.
Whilst fixed-length records/files are, by definition, less flexible than the
more free-form data input Python accepts, they are more efficient and faster
in situations where the data (format) is entirely consistent
- such as the OP is describing!
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python read text file columnwise
On 12/01/19 1:03 PM, Piet van Oostrum wrote:
shibashib...@gmail.com writes:
Hello
I'm very new in python. I have a file in the format:
2018-05-31 16:00:0028.90 81.77 4.3
2018-05-31 20:32:0028.17 84.89 4.1
2018-06-20 04:09:0027.36 88.01 4.8
2018-06-20 04:15:0027.31 87.09 4.7
2018-06-28 04.07:0027.87 84.91 5.0
2018-06-29 00.42:0032.20 104.61 4.8
I would like to read this file in python column-wise.
I tried this way but not working
event_list = open('seismicity_R023E.txt',"r")
info_event = read(event_list,'%s %s %f %f %f %f\n');
To the OP:
Python's standard I/O is based around data "streams". Whilst there is a
concept of "lines" and thus an end-of-line character, there is not the
idea of a record, in the sense of fixed-length fields and thus a
defining and distinction between data items based upon position.
Accordingly, whilst the formatting specification of strings and floats
might work for output, there is no equivalent for accepting input data.
Please re-read refs on file, read, readline, etc.
Why would you think that this would work?
To the PO:
Because in languages/libraries built around fixed-length files this is
how one specifies the composition of fields making up a record - a data
structure which dates back to FORTRAN and Assembler on mainframes and
other magtape-era machines.
Whilst fixed-length records/files are, by definition, less flexible than
the more free-form data input Python accepts, they are more efficient
and faster in situations where the data (format) is entirely consistent
- such as the OP is describing!
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list
Silent data corruption in pandas, was Re: Python read text file columnwise
Peter Otten wrote:
> shibashib...@gmail.com wrote:
>
>> Hello
>>>
>>> I'm very new in python. I have a file in the format:
>>>
>>> 2018-05-31 16:00:0028.90 81.77 4.3
>>> 2018-05-31 20:32:0028.17 84.89 4.1
>>> 2018-06-20 04:09:0027.36 88.01 4.8
>>> 2018-06-20 04:15:0027.31 87.09 4.7
>>> 2018-06-28 04.07:0027.87 84.91 5.0
>>> 2018-06-29 00.42:0032.20 104.61 4.8
>>
>> I would like to read this file in python column-wise.
> However, in the long term you may be better off with a tool like pandas:
>
import pandas
pandas.read_table(
> ... "seismicity_R023E.txt", sep=r"\s+",
> ... names=["date", "time", "foo", "bar", "baz"],
> ... parse_dates=[["date", "time"]]
> ... )
> date_timefoo bar baz
> 0 2018-05-31 16:00:00 28.90 81.77 4.3
> 1 2018-05-31 20:32:00 28.17 84.89 4.1
> 2 2018-06-20 04:09:00 27.36 88.01 4.8
> 3 2018-06-20 04:15:00 27.31 87.09 4.7
> 4 2018-06-28 04:00:00 27.87 84.91 5.0
> 5 2018-06-29 00:00:00 32.20 104.61 4.8
>
> [6 rows x 4 columns]
>
> It will be harder in the beginning, but if you work with tabular data
> regularly it will pay off.
After posting the above I noted that the malformed time in the last two rows
was silently botched. So I just spent an insane amount of time to try and
fix this from within pandas:
import datetime
import numpy
import pandas
def parse_datetime(dt):
return datetime.datetime.strptime(
dt.replace(".", ":"), "%Y-%m-%d %H:%M:%S"
)
def date_parser(dates, times):
return numpy.array([
parse_datetime(date + " " + time)
for date, time in zip(dates, times)
])
df = pandas.read_table(
"seismicity_R023E.txt", sep=r"\s+",
names=["date", "time", "foo", "bar", "baz"],
parse_dates=[["date", "time"]], date_parser=date_parser
)
print(df)
There's probably a better way as I am only a determined amateur...
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python read text file columnwise
shibashib...@gmail.com wrote:
> Hello
>>
>> I'm very new in python. I have a file in the format:
>>
>> 2018-05-31 16:00:0028.90 81.77 4.3
>> 2018-05-31 20:32:0028.17 84.89 4.1
>> 2018-06-20 04:09:0027.36 88.01 4.8
>> 2018-06-20 04:15:0027.31 87.09 4.7
>> 2018-06-28 04.07:0027.87 84.91 5.0
>> 2018-06-29 00.42:0032.20 104.61 4.8
>
> I would like to read this file in python column-wise.
>
> I tried this way but not working
> event_list = open('seismicity_R023E.txt',"r")
> info_event = read(event_list,'%s %s %f %f %f %f\n');
There is actually a library that implements a C-like scanf. You can install
it with
$ pip install scanf
After that:
$ cat read_table.py
from scanf import scanf
with open("seismicity_R023E.txt") as f:
for line in f:
print(
scanf("%s %s %f %f %f\n", line)
)
$ cat seismicity_R023E.txt
2018-05-31 16:00:0028.90 81.77 4.3
2018-05-31 20:32:0028.17 84.89 4.1
2018-06-20 04:09:0027.36 88.01 4.8
2018-06-20 04:15:0027.31 87.09 4.7
2018-06-28 04.07:0027.87 84.91 5.0
2018-06-29 00.42:0032.20 104.61 4.8
$ python read_table.py
('2018-05-31', '16:00:00', 28.9, 81.77, 4.3)
('2018-05-31', '20:32:00', 28.17, 84.89, 4.1)
('2018-06-20', '04:09:00', 27.36, 88.01, 4.8)
('2018-06-20', '04:15:00', 27.31, 87.09, 4.7)
('2018-06-28', '04.07:00', 27.87, 84.91, 5.0)
('2018-06-29', '00.42:00', 32.2, 104.61, 4.8)
$
However, in the long term you may be better off with a tool like pandas:
>>> import pandas
>>> pandas.read_table(
... "seismicity_R023E.txt", sep=r"\s+",
... names=["date", "time", "foo", "bar", "baz"],
... parse_dates=[["date", "time"]]
... )
date_timefoo bar baz
0 2018-05-31 16:00:00 28.90 81.77 4.3
1 2018-05-31 20:32:00 28.17 84.89 4.1
2 2018-06-20 04:09:00 27.36 88.01 4.8
3 2018-06-20 04:15:00 27.31 87.09 4.7
4 2018-06-28 04:00:00 27.87 84.91 5.0
5 2018-06-29 00:00:00 32.20 104.61 4.8
[6 rows x 4 columns]
>>>
It will be harder in the beginning, but if you work with tabular data
regularly it will pay off.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python read text file columnwise
On 11Jan2019 12:43, shibashib...@gmail.com wrote: I'm very new in python. I have a file in the format: 2018-05-31 16:00:0028.90 81.77 4.3 2018-05-31 20:32:0028.17 84.89 4.1 2018-06-20 04:09:0027.36 88.01 4.8 2018-06-20 04:15:0027.31 87.09 4.7 2018-06-28 04.07:0027.87 84.91 5.0 2018-06-29 00.42:0032.20 104.61 4.8 It is unclear what delimits the columns, but it looks like whitespace: tabs and/or spaces. You could read the file a line at a time and call .split() on each line to get the nonwhitespace fields, which would seem to correspond to the columns above. That gets you an list of strings; then you could convert them for processing as required. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: Python read text file columnwise
shibashib...@gmail.com writes:
> Hello
>>
>> I'm very new in python. I have a file in the format:
>>
>> 2018-05-31 16:00:0028.90 81.77 4.3
>> 2018-05-31 20:32:0028.17 84.89 4.1
>> 2018-06-20 04:09:0027.36 88.01 4.8
>> 2018-06-20 04:15:0027.31 87.09 4.7
>> 2018-06-28 04.07:0027.87 84.91 5.0
>> 2018-06-29 00.42:0032.20 104.61 4.8
>
> I would like to read this file in python column-wise.
>
> I tried this way but not working
> event_list = open('seismicity_R023E.txt',"r")
> info_event = read(event_list,'%s %s %f %f %f %f\n');
Why would you think that this would work?
See https://docs.python.org/3/library/csv.html
Something like:
#!/usr/bin/env python3
import csv
with open('testcsv.csv', newline='') as csvfile:
reader = csv.reader(csvfile, delimiter='\t')
for row in reader:
for i in range(2, 5):
row[i] = float(row[i])
print(row)
You could convert the first two columns to datetime format if you wish.
--
Piet van Oostrum
WWW: http://piet.vanoostrum.org/
PGP key: [8DAE142BE17999C4]
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python read text file columnwise
On Fri, 11 Jan 2019, shibashib...@gmail.com wrote: I'm very new in python. I have a file in the format: 2018-05-31 16:00:0028.90 81.77 4.3 2018-05-31 20:32:0028.17 84.89 4.1 2018-06-20 04:09:0027.36 88.01 4.8 2018-06-20 04:15:0027.31 87.09 4.7 2018-06-28 04.07:0027.87 84.91 5.0 2018-06-29 00.42:0032.20 104.61 4.8 So? What do you want to do with it? Are the fields fixed length? Tab-separated? Space-separated? Regardless of what you want to do, replace all whitespace with commas. Rich -- https://mail.python.org/mailman/listinfo/python-list
Re: Python read text file columnwise
Hello
>
> I'm very new in python. I have a file in the format:
>
> 2018-05-3116:00:0028.90 81.77 4.3
> 2018-05-3120:32:0028.17 84.89 4.1
> 2018-06-2004:09:0027.36 88.01 4.8
> 2018-06-2004:15:0027.31 87.09 4.7
> 2018-06-2804.07:0027.87 84.91 5.0
> 2018-06-2900.42:0032.20 104.61 4.8
I would like to read this file in python column-wise.
I tried this way but not working
event_list = open('seismicity_R023E.txt',"r")
info_event = read(event_list,'%s %s %f %f %f %f\n');
--
https://mail.python.org/mailman/listinfo/python-list
Python read text file columnwise
Hello, I'm very new in python. I have a file in the format: 2018-05-31 16:00:0028.90 81.77 4.3 2018-05-31 20:32:0028.17 84.89 4.1 2018-06-20 04:09:0027.36 88.01 4.8 2018-06-20 04:15:0027.31 87.09 4.7 2018-06-28 04.07:0027.87 84.91 5.0 2018-06-29 00.42:0032.20 104.61 4.8 -- https://mail.python.org/mailman/listinfo/python-list
