Re: [PHP] How to jump to line number in large file
Thanks for the heads up on fgetc() incrementing by one. I hadn't actually tested that code yet, I was using the original fseek($handle,$pos). strpos would be ideal but it needs to work on a string and not a file - I don't want to load a 100Mb file into memory if I don't have to. Perhaps I should test how quick the fgets() and ftell() method is because at least it loads in one line at a time. Does anybody know any other ways to go about the problem? Many thanks, Steve Greg Bowser [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] fseek($handle, 1, SEEK_CUR); // or fseek($handle, $pos) $t = fgetc($handle); This probably won't help you, but given a quick glance, it looks like you're incrementing the file pointer by 2 positions on each iteration of your while loop. The fgetc() function returns the character at the current position and increments the file pointer by 1. I haven't tried this, but perhaps using strpos would be faster? Use strpos to seek to find the first \n, then use the offset parameter to seek to the second, and so on, until strpos() returns false. --GREG -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to jump to line number in large file
Richard Heyes [EMAIL PROTECTED] wrote in message
news:[EMAIL PROTECTED]
Thanks for the heads up on fgetc() incrementing by one. I hadn't actually
tested that code yet, I was using the original fseek($handle,$pos).
strpos would be ideal but it needs to work on a string and not a file - I
don't want to load a 100Mb file into memory if I don't have to. Perhaps I
should test how quick the fgets() and ftell() method is because at least
it loads in one line at a time.
Does anybody know any other ways to go about the problem?
Haven't read the rest of the thread, and so going by the subject alone,
fgets() finishes when it encounters a newline, so you can use this
wondrous fact to seek to a specific line:
?php
$fp = fopen('filename', 'r');
$num = 18; // Desired line number
for ($i=0; $i$num; $i++)
$line = fgets($fp);
echo $line;
?
It works because fgets() stops when it encounters a newline (\n). So it's
just a case of counting the calls to fgets().
fgets() would work but as I'm constantly jumping around a 500,000 line file
I thought it was better to maintain a cache of line number positions.
As a final update to anybody following:
- Taking away the unnecessary fseek() made the script execute in 63 seconds
- Using a buffer system, (reading in 1Mb of the text file at a time and then
looping through the string in memory) made the script execute in 36 seconds.
Huge improvement, but...
- Porting the code to C++, doing a shell_exec and reading the results back
in to PHP, took less than 2 seconds.
As fgetc() etc are all effectively C wrappers I was quite surprised at the
speed increase
Best wishes,
Steve
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] How to jump to line number in large file
So the only variation on a theme that I didn't test is the one that performs
the best by an order of magnitude... nice. Many thanks for your time
everyone.
Robert Cummings [EMAIL PROTECTED] wrote in message
news:[EMAIL PROTECTED]
On Sat, 2008-04-05 at 19:09 +0100, Steve McGill wrote:
Richard Heyes [EMAIL PROTECTED] wrote in message
news:[EMAIL PROTECTED]
Thanks for the heads up on fgetc() incrementing by one. I hadn't
actually
tested that code yet, I was using the original fseek($handle,$pos).
strpos would be ideal but it needs to work on a string and not a
file - I
don't want to load a 100Mb file into memory if I don't have to.
Perhaps I
should test how quick the fgets() and ftell() method is because at
least
it loads in one line at a time.
Does anybody know any other ways to go about the problem?
Haven't read the rest of the thread, and so going by the subject alone,
fgets() finishes when it encounters a newline, so you can use this
wondrous fact to seek to a specific line:
?php
$fp = fopen('filename', 'r');
$num = 18; // Desired line number
for ($i=0; $i$num; $i++)
$line = fgets($fp);
echo $line;
?
It works because fgets() stops when it encounters a newline (\n). So
it's
just a case of counting the calls to fgets().
fgets() would work but as I'm constantly jumping around a 500,000 line
file
I thought it was better to maintain a cache of line number positions.
As a final update to anybody following:
- Taking away the unnecessary fseek() made the script execute in 63
seconds
- Using a buffer system, (reading in 1Mb of the text file at a time and
then
looping through the string in memory) made the script execute in 36
seconds.
Huge improvement, but...
- Porting the code to C++, doing a shell_exec and reading the results
back
in to PHP, took less than 2 seconds.
As fgetc() etc are all effectively C wrappers I was quite surprised at
the
speed increase
It really depends on how you write your code... I ran the following
script on a 150 meg text log file containing 1905883 lines in 4 seconds
(note that it performs caching). Here's the script:
?php
$path = $argv[1];
if( ($fPtr = fopen( $path, 'r' )) === false )
{
echo Couldn't open for reading: $path\n;
exit();
}
$line = 1;
$lines[$line] = 0;
while( fgets( $fPtr ) !== false )
{
$lines[++$line] = ftell( $fPtr );
}
fclose( $fPtr );
?
Here's the run times on several iterations (Athlon 2400+):
real0m4.065s
user0m3.488s
sys 0m0.464s
real0m4.005s
user0m3.464s
sys 0m0.436s
real0m5.816s
user0m3.336s
sys 0m0.536s
real0m3.994s
user0m3.384s
sys 0m0.504s
real0m4.069s
user0m3.512s
sys 0m0.444s
real0m4.009s
user0m3.344s
sys 0m0.552s
Cheers,
Rob.
--
http://www.interjinn.com
Application and Templating Framework for PHP
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] How to jump to line number in large file
Hello,
I have code where it is appropriate for me to jump to certain lines in a
text file and edit that particular line. I need to keep a record of the byte
position of every line so that I can fseek to it.
I need to do in-line editing so I don't want to write the entire file out
again etc. I don't want to read the entire file into memory using file().
Finally each line has an unpredicatable line length as it is generated by a
3rd party program so I can't do some maths to work out the correct byte
position.
I am using code like this to keep a record of all the byte positions:
$line_positions = array(1=0);
$line_number = 2;
while(!feof($handle)) {
fseek($handle, 1, SEEK_CUR); // or fseek($handle, $pos)
$t = fgetc($handle);
if($t == \n) {
$line_positions[$line_number] = $pos + 1;
$line_number++;
}
$pos++;
}
This behaves fine with smaller textfiles but not very well with 100Mb files,
etc.
I've commented out assigning the array in the loop and that doesn't really
help, it seems it's the fseek fgetc etc which takes forever.
Can anybody point me in the right direction? Editplus opens up the 100Mb
file in a couple of seconds and can then jump to a line number
instantaneously, whereas my code takes a couple of minutes to loop through
the file. I'm assuming this isn't the difference between PHP C but rather
that I'm doing something stupid :-)
Many thanks in advance for your time,
Steve
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Unnecessary if statement? Programming technique
Hi everyone
Quick question:
If I have such a loop:
?
for($i=0;$i100;$i++) {
if($i==100) {
// do something special for this occurence
}
// do something standard
}
?
In this case it seems such a waste that the if() statement is done 99
times when it's not needed. Is there any obvious trick that I am missing?
I'm not sure how taxing a simple if() statement is on a server, maybe it's
negligible, or is it something to worry about?
Something which I'd prefer NOT to do:
?
for($i=0;$i100;$i++) {
// do something standard
}
// do something special for $i = 100
for($i=101;$i100;$i++) {
// do something standard
}
?
as I would have have to either keep two copies of the code or write a
function just for this purpose, which hardly seems worth it.
Thanks to anyone who takes the time to think about my question and/or
respond.
Best wishes,
Steve McGill
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] HTTP User Authentication Problems
Hi Rahul,
Try doing a print_r($_SERVER) to see if the variables are being set.
If they aren't, take a look at the phpinfo(); - chances are your host is
using a CGI binary version of PHP to use together with a security wrapper
like suPHP or phpSuExec, which might throw away the variables that you
require.
Steve
Rahul S. Johari [EMAIL PROTECTED] schreef in bericht
news:[EMAIL PROTECTED]
Ave,
I¹m trying to run this very simple HTTP user authentication script:
?php
if (!isset($_SERVER['PHP_AUTH_USER'])) {
header('WWW-Authenticate: Basic realm=Private');
header('HTTP/1.0 401 Unauthorized');
echo 'Unauthorized';
exit;
} else {
if(($_SERVER['PHP_AUTH_USER']==try)
($_SERVER['PHP_AUTH_PW']==try)) {
?
HTML
HEAD/HEAD
BODY
My Stuff Goes Here!
/BODY
/HTML
?
}
else {
echo Access Denied;
}
}
?
The problem is, this script runs perfectly fine on my localhost machine at
home, however it¹s not working on my web server (purchased webhosting). When
I open this page on my web site, it does bring up the popup box asking for
Username Password the problem is, even if I type the correct user/pass,
it won¹t accept it. It keeps asking me for user/pass again and again and
finally brings up the ³Unauthorized² text on the page.
Why won¹t it allow the correct user/pass to login? I don¹t understand.
The only difference between the PHP on my localhost and the PHP on my
webhost server is that my local machine is running PHP 5 and the webhost
server is running PHP 4.4.1
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Unnecessary if statement? Programming technique
Hi,
David is right about the unwanted side-effect. Thanks for the idea though.
Unfortunately the 'greater problem' is not so great, I've just been doing
this for a while now and find myself programming loops like these so often
and I've never got round to testing if a simple IF statement is a major
drain on the CPU. Somehow I doubt it.
I got this reply from someone direct to my mail address, which seems to sum
it up:
--
In truth you are not evaluating the whole if block just the condition
and since its such a simple condition I can't see how it would be at
all taxing on the server. In your specific case I can't think of a
better way to do it either.
--
I'll try and think of a better example:
?
$bool = true; // this is set dynamically and not known in advance
while(true) {
if($bool) { // this condition tested in every single loop
// do first code
} else {
// do second code
}
}
?
and I am wondering if the compiler is smart enough to turn this into:
?
$bool = true; // this is set dynamically and not known in advance
if($bool) { // this condition only tested once
while(true) {
// do first code
}
} else {
while(true) {
// do second code
}
}
?
I realise this might be hard to follow without giving specific examples and
code.
In this case, the coding style of the 2nd example seems far better, but
sometimes the 2 blocks of code are practically identical and it's a
programmer's nightmare to have the blocks of code in 2 places and to
remember to keep them both updated.
I'm also assuming that using function calls is also much slower than
evaluating a very simple IF statement.
Thanks for your interest.
Best wishes,
Steve
David Grant [EMAIL PROTECTED] schreef in bericht
news:[EMAIL PROTECTED]
Jared Williams wrote:
Why not
for ($i = 0; $i 100/100; ++$i)
This involves dividing 100 by 100 for each iteration of the loop.
It would be better to test against 1.
There is also the unwanted side-effect of executing the code on each
hundredth iteration, which is unwanted (as far as I understand the
problem). :)
It would be interesting if Steve could divulge the greater problem that
he is seeking a solution to.
Cheers,
David
--
David Grant
http://www.grant.org.uk/
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] PHP + Javascript, immediate database update
Hi everyone, I have a webform which my users are expecting to act like a Windows program, they only need to check the box and it is automatically written to the database. So I'd like to use a combination of javascript, like this, but it isn't quite elegent enough: - Tick the box. - Javascript opens up a popup window, SQL query is made, popup window closes again Or a second method: - Register an onexit() function, and auto-submit the form. The only problem there is to remember to redirect them to their intended destination, after having submitted the form. (i.e. they pressed back, forward, or entered in another URL). Can anybody point me in the right direction? Many thanks, Steve -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
[PHP] Most efficient way of fetching 1,000 records from MySQL ?
Hello,
In my script I have generated an array of 1,000 userIDs for example. Now for
I need to fetch each user's record and I am wondering what is the best way
to go about it:
1) 1000 sql queries
// code is sort of pseudo for clarity
foreach($users as $userID) {
$user = sql(select * from users where userID = '$userID');
// bla bla
}
2) 1 query
$users = sql(select * from users where userID='1' or userID='2' or
userID='5' or userID='10');
I imagine the 2nd one would be a bit of a nightmare for MySQL to parse, if
it gets too long?
Or am I missing a more efficient 3rd / 4th option?
Many thanks in advance for your help.
Steve McGill
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
Re: [PHP] Most efficient way of fetching 1,000 records from MySQL ?
I think you'll find that your first approach is not only more correct,
it's the only manageable one.
No, it will be very slow. The biggest overhead is in transfering data to
and from sql server. It's always better to get the results in one sql
query.
Use this aproach:
$users =
sql('select * from users where userID IN ('. implode(', ',$users) .')');
That is precisely what I wanted, many thanks for your help guys.
Steve
--
PHP General Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php
