Re: fseek In Compressed Files
On Tuesday, February 4, 2014 2:27:38 AM UTC+5:30, Dave Angel wrote:
Ayushi Dalmia ayushidalmia2...@gmail.com Wrote in message:
On Thursday, January 30, 2014 4:20:26 PM UTC+5:30, Ayushi Dalmia wrote:
Hello,
I need to randomly access a bzip2 or gzip file. How can I set the offset
for a line and later retreive the line from the file using the offset.
Pointers in this direction will help.
This is what I have done:
import bz2
import sys
from random import randint
index={}
data=[]
f=open('temp.txt','r')
for line in f:
data.append(line)
filename='temp1.txt.bz2'
with bz2.BZ2File(filename, 'wb', compresslevel=9) as f:
f.writelines(data)
prevsize=0
list1=[]
offset={}
with bz2.BZ2File(filename, 'rb') as f:
for line in f:
words=line.strip().split(' ')
list1.append(words[0])
offset[words[0]]= prevsize
prevsize = sys.getsizeof(line)+prevsize
sys.getsizeof looks at internal size of a python object, and is
totally unrelated to a size on disk of a text line. len () might
come closer, unless you're on Windows. You really should be using
tell to define the offsets for later seek. In text mode any other
calculation is not legal, ie undefined.
data=[]
count=0
with bz2.BZ2File(filename, 'rb') as f:
while count20:
y=randint(1,25)
print y
print offset[str(y)]
count+=1
f.seek(int(offset[str(y)]))
x= f.readline()
data.append(x)
f=open('b.txt','w')
f.write(''.join(data))
f.close()
where temp.txt is the posting list file which is first written in a
compressed format and then read later.
I thought you were starting with a compressed file. If you're
being given an uncompressed file, just deal with it directly.
I am trying to build the index for the entire wikipedia dump which needs to
be done in a space and time optimised way. The temp.txt is as follows:
1 456 t0b3c0i0e0:784 t0b2c0i0e0:801 t0b2c0i0e0
2 221 t0b1c0i0e0:774 t0b1c0i0e0:801 t0b2c0i0e0
3 455 t0b7c0i0e0:456 t0b1c0i0e0:459 t0b2c0i0e0:669 t0b10c11i3e0:673
t0b1c0i0e0:678 t0b2c0i1e0:854 t0b1c0i0e0
4 410 t0b4c0i0e0:553 t0b1c0i0e0:609 t0b1c0i0e0
5 90 t0b1c0i0e0
So every line begins with its line number in ascii form? If true,
the dict above called offsets should just be a list.
Maybe you should just quote the entire assignment. You're
probably adding way too much complication to it.
--
DaveA
Hey! I am new here. Sorry about the incorrect posts. Didn't understand the
protocol then.
Although, I have the uncompressed text, I cannot start right away with them
--
https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
30.01.14 18:21, Peter Otten написав(ла): Do you know an efficient way to implement random access for a bzip2 or gzip file? See dictzip and BGZF. Unfortunately Python stdlib doesn't support them. -- https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
Ayushi Dalmia ayushidalmia2...@gmail.com Wrote in message:
On Thursday, January 30, 2014 4:20:26 PM UTC+5:30, Ayushi Dalmia wrote:
Hello,
I need to randomly access a bzip2 or gzip file. How can I set the offset for
a line and later retreive the line from the file using the offset. Pointers
in this direction will help.
This is what I have done:
import bz2
import sys
from random import randint
index={}
data=[]
f=open('temp.txt','r')
for line in f:
data.append(line)
filename='temp1.txt.bz2'
with bz2.BZ2File(filename, 'wb', compresslevel=9) as f:
f.writelines(data)
prevsize=0
list1=[]
offset={}
with bz2.BZ2File(filename, 'rb') as f:
for line in f:
words=line.strip().split(' ')
list1.append(words[0])
offset[words[0]]= prevsize
prevsize = sys.getsizeof(line)+prevsize
sys.getsizeof looks at internal size of a python object, and is
totally unrelated to a size on disk of a text line. len () might
come closer, unless you're on Windows. You really should be using
tell to define the offsets for later seek. In text mode any other
calculation is not legal, ie undefined.
data=[]
count=0
with bz2.BZ2File(filename, 'rb') as f:
while count20:
y=randint(1,25)
print y
print offset[str(y)]
count+=1
f.seek(int(offset[str(y)]))
x= f.readline()
data.append(x)
f=open('b.txt','w')
f.write(''.join(data))
f.close()
where temp.txt is the posting list file which is first written in a
compressed format and then read later.
I thought you were starting with a compressed file. If you're
being given an uncompressed file, just deal with it directly.
I am trying to build the index for the entire wikipedia dump which needs to be
done in a space and time optimised way. The temp.txt is as follows:
1 456 t0b3c0i0e0:784 t0b2c0i0e0:801 t0b2c0i0e0
2 221 t0b1c0i0e0:774 t0b1c0i0e0:801 t0b2c0i0e0
3 455 t0b7c0i0e0:456 t0b1c0i0e0:459 t0b2c0i0e0:669 t0b10c11i3e0:673
t0b1c0i0e0:678 t0b2c0i1e0:854 t0b1c0i0e0
4 410 t0b4c0i0e0:553 t0b1c0i0e0:609 t0b1c0i0e0
5 90 t0b1c0i0e0
So every line begins with its line number in ascii form? If true,
the dict above called offsets should just be a list.
Maybe you should just quote the entire assignment. You're
probably adding way too much complication to it.
--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
Ayushi Dalmia ayushidalmia2...@gmail.com Wrote in message: The size of this file will be 10 GB. The version of Python I am using is 2.7.2. Yes, performance is an important issue. Then the only viable option is to extract the entire file and write it to a temp location. Perhaps as you extract it, you could also build a list of offsets, so the seeking by line number can be efficient. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
On Thursday, January 30, 2014 9:51:28 PM UTC+5:30, Peter Otten wrote: Serhiy Storchaka wrote: 30.01.14 13:28, Peter Otten написав(ла): Ayushi Dalmia wrote: I need to randomly access a bzip2 or gzip file. How can I set the offset for a line and later retreive the line from the file using the offset. Pointers in this direction will help. with gzip.open(filename) as f: f.seek(some_pos) print(f.readline()) f.seek(some_pos) print(f.readline()) seems to work as expected. Can you tell a bit more about your usecase (if it isn't covered by that basic example)? I don't recommend to seek backward in compressed file. This is very inefficient operation. Do you know an efficient way to implement random access for a bzip2 or gzip file? Nothing that I know of. -- https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
On Friday, January 31, 2014 12:16:59 AM UTC+5:30, Dave Angel wrote: Ayushi Dalmia ayushidalmia2...@gmail.com Wrote in message: On Thursday, January 30, 2014 4:20:26 PM UTC+5:30, Ayushi Dalmia wrote: Hello, I need to randomly access a bzip2 or gzip file. How can I set the offset for a line and later retreive the line from the file using the offset. Pointers in this direction will help. We are not allowed to use databases! I need to do this without database. Why do you reply to your own message? Makes it hard for people to make sense of your post. Have you any answers to earlier questions? How big is this file, what python version, do you care about performance, code you've tried, ... -- DaveA The size of this file will be 10 GB. The version of Python I am using is 2.7.2. Yes, performance is an important issue. -- https://mail.python.org/mailman/listinfo/python-list
fseek In Compressed Files
Hello, I need to randomly access a bzip2 or gzip file. How can I set the offset for a line and later retreive the line from the file using the offset. Pointers in this direction will help. -- https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
Ayushi Dalmia wrote: I need to randomly access a bzip2 or gzip file. How can I set the offset for a line and later retreive the line from the file using the offset. Pointers in this direction will help. with gzip.open(filename) as f: f.seek(some_pos) print(f.readline()) f.seek(some_pos) print(f.readline()) seems to work as expected. Can you tell a bit more about your usecase (if it isn't covered by that basic example)? -- https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
On Thursday, January 30, 2014 4:20:26 PM UTC+5:30, Ayushi Dalmia wrote:
Hello,
I need to randomly access a bzip2 or gzip file. How can I set the offset for
a line and later retreive the line from the file using the offset. Pointers
in this direction will help.
This is what I have done:
import bz2
import sys
from random import randint
index={}
data=[]
f=open('temp.txt','r')
for line in f:
data.append(line)
filename='temp1.txt.bz2'
with bz2.BZ2File(filename, 'wb', compresslevel=9) as f:
f.writelines(data)
prevsize=0
list1=[]
offset={}
with bz2.BZ2File(filename, 'rb') as f:
for line in f:
words=line.strip().split(' ')
list1.append(words[0])
offset[words[0]]= prevsize
prevsize = sys.getsizeof(line)+prevsize
data=[]
count=0
with bz2.BZ2File(filename, 'rb') as f:
while count20:
y=randint(1,25)
print y
print offset[str(y)]
count+=1
f.seek(int(offset[str(y)]))
x= f.readline()
data.append(x)
f=open('b.txt','w')
f.write(''.join(data))
f.close()
where temp.txt is the posting list file which is first written in a compressed
format and then read later. I am trying to build the index for the entire
wikipedia dump which needs to be done in a space and time optimised way. The
temp.txt is as follows:
1 456 t0b3c0i0e0:784 t0b2c0i0e0:801 t0b2c0i0e0
2 221 t0b1c0i0e0:774 t0b1c0i0e0:801 t0b2c0i0e0
3 455 t0b7c0i0e0:456 t0b1c0i0e0:459 t0b2c0i0e0:669 t0b10c11i3e0:673
t0b1c0i0e0:678 t0b2c0i1e0:854 t0b1c0i0e0
4 410 t0b4c0i0e0:553 t0b1c0i0e0:609 t0b1c0i0e0
5 90 t0b1c0i0e0
6 727 t0b2c0i0e0
7 431 t0b2c0i1e0
8 532 t0b1c0i0e0:652 t0b1c0i0e0:727 t0b2c0i0e0
9 378 t0b1c0i0e0
10 666 t0b2c0i0e0
11 405 t0b1c0i0e0
12 702 t0b1c0i0e0
13 755 t0b1c0i0e0
14 781 t0b1c0i0e0
15 593 t0b1c0i0e0
16 725 t0b1c0i0e0
17 989 t0b2c0i1e0
18 221 t0b1c0i0e0:402 t0b1c0i0e0:842 t0b1c0i0e0
19 405 t0b1c0i0e0
20 200 t0b1c0i0e0:300 t0b1c0i0e0:398 t0b1c0i0e0:649 t0b1c0i0e0
21 66 t0b1c0i0e0
22 30 t0b1c0i0e0
23 126 t0b1c0i0e0:895 t0b1c0i0e0
24 355 t0b1c0i0e0:374 t0b1c0i0e0:378 t0b1c0i0e0:431 t0b3c0i0e0:482
t0b1c0i0e0:546 t0b3c0i0e0:578 t0b1c0i0e0
25 198 t0b1c0i0e0
--
https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
On Fri, Jan 31, 2014 at 12:34 AM, Ayushi Dalmia ayushidalmia2...@gmail.com wrote: where temp.txt is the posting list file which is first written in a compressed format and then read later. Unless you specify otherwise, a compressed file is likely to have sub-byte boundaries. It might not be possible to seek to a specific line. What you could do, though, is explicitly compress each line, then write out separately-compressed blocks. You can then seek to any one that you want, read it, and decompress it. But at this point, you're probably going to do better with a database; PostgreSQL, for instance, will automatically compress any content that it believes it's worthwhile to compress (as long as it's in a VARCHAR field or similar and the table hasn't been configured to prevent that, yada yada). All you have to do is tell Postgres to store this, retrieve that, and it'll worry about the details of compression and decompression. As an added benefit, you can divide the text up and let it do the hard work of indexing, filtering, sorting, etc. I suspect you'll find that deploying a database is a much more efficient use of your development time than recreating all of that. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
30.01.14 13:28, Peter Otten написав(ла): Ayushi Dalmia wrote: I need to randomly access a bzip2 or gzip file. How can I set the offset for a line and later retreive the line from the file using the offset. Pointers in this direction will help. with gzip.open(filename) as f: f.seek(some_pos) print(f.readline()) f.seek(some_pos) print(f.readline()) seems to work as expected. Can you tell a bit more about your usecase (if it isn't covered by that basic example)? I don't recommend to seek backward in compressed file. This is very inefficient operation. -- https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
On Thursday, January 30, 2014 4:20:26 PM UTC+5:30, Ayushi Dalmia wrote: Hello, I need to randomly access a bzip2 or gzip file. How can I set the offset for a line and later retreive the line from the file using the offset. Pointers in this direction will help. We are not allowed to use databases! I need to do this without database. -- https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
Serhiy Storchaka wrote: 30.01.14 13:28, Peter Otten написав(ла): Ayushi Dalmia wrote: I need to randomly access a bzip2 or gzip file. How can I set the offset for a line and later retreive the line from the file using the offset. Pointers in this direction will help. with gzip.open(filename) as f: f.seek(some_pos) print(f.readline()) f.seek(some_pos) print(f.readline()) seems to work as expected. Can you tell a bit more about your usecase (if it isn't covered by that basic example)? I don't recommend to seek backward in compressed file. This is very inefficient operation. Do you know an efficient way to implement random access for a bzip2 or gzip file? -- https://mail.python.org/mailman/listinfo/python-list
Re: fseek In Compressed Files
Ayushi Dalmia ayushidalmia2...@gmail.com Wrote in message: On Thursday, January 30, 2014 4:20:26 PM UTC+5:30, Ayushi Dalmia wrote: Hello, I need to randomly access a bzip2 or gzip file. How can I set the offset for a line and later retreive the line from the file using the offset. Pointers in this direction will help. We are not allowed to use databases! I need to do this without database. Why do you reply to your own message? Makes it hard for people to make sense of your post. Have you any answers to earlier questions? How big is this file, what python version, do you care about performance, code you've tried, ... -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
