Hi, I hope someone can shed light on this problem. I am trying to crawl a particular site, and am getting strange results. I had crawled it successfully in the past but lately I only am able to crawl about 1400 of the 8000 pages. I am constantly getting (as reported in a verbose log file): HTTP request sent, awaiting response... 500 Internal Server Error 11:04:18 ERROR 500: Internal Server Error.
This error happens intermitently on some pages during the first 1400 pages, i.e. http://www.ryland.com/find-your-new-home/29-northern-kentucky/1115-claiborne/11777-shenandoah.html But then this is what happens in the log file, and subsequently ALL files receive the error 500 (as shown in a non-verbose log file): WARNING: Certificate verification error for www.ryland.com: unable to get local issuer certificate 16:58:03 URL:https://www.ryland.com/home/contact-us/29-1361-community-and-floor-plan-information.html [257903/257903] -> "files/www.ryland.com/home/contact-us/29-1361-community-and-floor-plan-information.html" [1] 16:58:19 URL:http://www.ryland.com/home/29-1034-contact-us.html [115640/115640] -> "files/www.ryland.com/home/29-1034-contact-us.html" [1] http://www.ryland.com/find-your-new-home/29-northern-kentucky/[EMAIL PROTECTED]/driving-directions.html: 16:58:23 ERROR 500: Internal Server Error. http://www.ryland.com/find-your-new-home/29-northern-kentucky/1034-french-quarter-orleans/11256-summit.html: 16:58:27 ERROR 500: Internal Server Error. Now if I was to call wget only on the page that failed with ERROR 500, it would crawl just fine. Here are settings I am using: wget www.ryland.com -o LogRyland3.txt -t 5 --random-wait -v I have run wget numerous times on this site, and I receive ERROR 500 on different pages before it reaches the point where all pages fail from then on, as shown above. And I have appended the INI file which has more settings (below). Again, crawling this site used to work fine. And crawling failed pages works when I crawl them individually. Any help would be GREATLY appreciated! ----INI FILE STARTS HERE----------- # Rewrote the wgetrc / wget.ini file from scratch, based on the manual for version 1.9 logfile = log.txt tries = 1 timeout = 30 wait = 1 randomwait = on quota = 5000m restrict_file_names = windows add_hostdir = on span_hosts = off dir_prefix = files cache = off recursive = on use_proxy = off robots = off verbose = off keep-session-cookies = on save-cookies = sw_cookies.txt check_certificate = off #reclevel = 7 reject = GIF,jpg,JPG,jpeg,JPEG,bmp,BMP,pdf,PDF,css,CSS,js,JS,mpeg,MPEG,mov,MOV,avi,AVI,wmv,WMV,doc,DOC,ppt,PPT,csv,CSV,xls,XLS,txt,TXT,png,PNG,ra,RA,ram,RAM,tif,TIF,zip,ZIP,rar,RAR,class,CLASS,swf,SWF,pl,xml,XML,mp3,MP3,sid,SID,ivr,IVR,psd,PSD,rft,RTF,dwf,DWF,abk,acl,acm,acp,act,acv,ad,adb,add,adm,adp,adr,af2,af3,afm,ai,aif,alb,all,ams,anc,ani,ans,api,apr,aps,arc,arj,art,asa,asc,asd,asf,asm,ast,asx,att,avi,awd,b4,bak,bas,bat,bfc,bg,bi,bif,bin,bk,bks,bm1,bmk,bmp,brx,bs1,bsp,btm,cab,cal,cas,cat,cb,ccb,ccf,cch,ccm,cda,cdf,cdi,cdr,cdt,cdx,cel,cfb,cfg,cgm,ch,chk,chp,cil,cim,cin,ck1,ck2,ck3,ck4,ck5,ck6,cla,clp,cls,cmd,cmf,cmp,cmv,cnf,cnm,cnq,cnt,cob,cod,com,cpd,cpe,cpi,cpl,cpp,cpr,cpt,cpx,crd,crp,crt,csc,csp,css,csv,ct,ctl,cue,cur,cut,cv,cwk,cws,cxx,dat,dbf,dbx,dcr,dcs,dcx,ddf,def,der,dib,dic,dif,dir,diz,dlg,dll,dmf,dmg,doc,dot,dpr,drv,drw,dsg,dsm,dsp,dsq,dsw,dwg,dxf,emf,enc,eps,er1,erx,evy,ewl,exe,f77,f90,far,fav,fax,fh3,fif,fit,flc,fli,flt,fmb,fmt,fmx,fog,fon,for,fot,fp,fp1,fp3,fpx,frm,frx,gal,gcp,ged,gem,gen,gfc,gfi,gfx,gid,gif,gim,gix,gna,gnx,gra,grd,grp,gt2,gtk,gwx,gwz,gz,hed,hel,hex,hgl,hlp,hog,hpj,hpp,hqx,hst,ht,htx,ica,icb,icm,ico,idd,idq,iff,igf,iif,ima,img,inc,inf,ini,inp,ins,iso,isp,isu,it,iw,jar,jav,jbf,jff,jif,jmp,jn1,jpe,jpg,js,jtf,kdc,kfx,kye,lbm,ldb,leg,lha,lib,lis,log,lpd,lrc,lst,lwo,lwp,lzh,lzs,m3d,mad,maf,mak,mam,map,maq,mar,mas,mat,max,maz,mb1,mcc,mcs,mcw,mda,mdb,mde,mdl,mdn,mdw,mdz,med,mer,met,mi,mic,mid,mmf,mmm,mod,mov,mp3,mpe,mpg,mpp,msg,msi,msn,msp,mtm,mus,mvb,mwp,nap,ncb,nsf,nst,ntf,obd,obj,obz,ocx,ofn,oft,okt,olb,ole,opt,or2,or3,org,p10,p65,pab,pak,pal,pat,pbk,pbm,pcd,pcl,pcs,pct,pcx,pdf,pdq,pfa,pfb,pfc,pfm,pgl,pgm,pic,pif,pig,pin,pix,pj,pkg,pl,plt,pm5,pm6,png,pnt,pot,pp4,ppa,ppm,pps,ppt,pre,prf,prn,prs,prz,ps,psd,pst,ptm,pub,pwd,pwz,pxl,qad,qbw,qdt,qlb,qry,qt,qtm,qxd,ra,ram,ras,raw,rc,rec,reg,res,rft,rle,rm,rmi,rov,rpt,rtf,rtm,s3m,sam,sav,sc2,scc,scd,sch,scn,scp,scr,sct,sdl,sdr,sdt,sea,sep,shb,shg,shs,shw,sit,slk,snd,sqc,sqr,sty,svx,sys,t2t,tar,taz,tex,tga,tgz,the,thn,tif,tig,tlb,tmp,top,trm,trn,ttf,twf,tww,tx8,txt,udf,ult,url,use,vbp,vbw,vbx,vcf,vda,vi,vlb,voc,vp,vsd,vst,vsw,vxd,wad,wav,wb1,wb2,wbk,wbl,wcm,wdb,web,wgp,wid,wil,wiz,wk1,wk3,wk4,wks,wlf,wll,wmf,wow,wp,wp4,wp5,wp6,wpd,wpg,wps,wpt,wpw,wq1,wq2,wri,wrl,ws1,ws2,ws3,ws4,ws5,ws6,ws7,wsd,wvl,xar,xla,xlb,xlc,xld,xlk,xlm,xls,xlt,xlv,xlw,xm,xr1,xy3,xy4,xyp,xyw,yal,ybk,zip,zoo, ABK,ACL,ACM,ACP,ACT,ACV,AD,ADB,ADD,ADM,ADP,ADR,AF2,AF3,AFM,AI,AIF,ALB,ALL,AMS,ANC,ANI,ANS,API,APR,APS,ARC,ARJ,ART,ASA,ASC,ASD,ASF,ASM,AST,ASX,ATT,AVI,AWD,B4,BAK,BAS,BAT,BFC,BG,BI,BIF,BIN,BK,BKS,BM1,BMK,BMP,BRX,BS1,BSP,BTM,CAB,CAL,CAS,CAT,CB,CCB,CCF,CCH,CCM,CDA,CDF,CDI,CDR,CDT,CDX,CEL,CFB,CFG,CGM,CH,CHK,CHP,CIL,CIM,CIN,CK1,CK2,CK3,CK4,CK5,CK6,CLA,CLP,CLS,CMD,CMF,CMP,CMV,CNF,CNM,CNQ,CNT,COB,COD,COM,CPD,CPE,CPI,CPL,CPP,CPR,CPT,CPX,CRD,CRP,CRT,CSC,CSP,CSS,CSV,CT,CTL,CUE,CUR,CUT,CV,CWK,CWS,CXX,DAT,DBF,DBX,DCR,DCS,DCX,DDF,DEF,DER,DIB,DIC,DIF,DIR,DIZ,DLG,DLL,DMF,DMG,DOC,DOT,DPR,DRV,DRW,DSG,DSM,DSP,DSQ,DSW,DWG,DXF,EMF,ENC,EPS,ER1,ERX,EVY,EWL,EXE,F77,F90,FAR,FAV,FAX,FH3,FIF,FIT,FLC,FLI,FLT,FMB,FMT,FMX,FOG,FON,FOR,FOT,FP,FP1,FP3,FPX,FRM,FRX,GAL,GCP,GED,GEM,GEN,GFC,GFI,GFX,GID,GIF,GIM,GIX,GNA,GNX,GRA,GRD,GRP,GT2,GTK,GWX,GWZ,GZ,HED,HEL,HEX,HGL,HLP,HOG,HPJ,HPP,HQX,HST,HT,HTX,ICA,ICB,ICM,ICO,IDD,IDQ,IFF,IGF,IIF,IMA,IMG,INC,INF,INI,INP,INS,ISO,ISP,ISU,IT,IW,JAR,JAV,JBF,JFF,JIF,JMP,JN1,JPE,JPG,JS,JTF,KDC,KFX,KYE,LBM,LDB,LEG,LHA,LIB,LIS,LOG,LPD,LRC,LST,LWO,LWP,LZH,LZS,M3D,MAD,MAF,MAK,MAM,MAP,MAQ,MAR,MAS,MAT,MAX,MAZ,MB1,MCC,MCS,MCW,MDA,MDB,MDE,MDL,MDN,MDW,MDZ,MED,MER,MET,MI,MIC,MID,MMF,MMM,MOD,MOV,MP3,MPE,MPG,MPP,MSG,MSI,MSN,MSP,MTM,MUS,MVB,MWP,NAP,NCB,NSF,NST,NTF,OBD,OBJ,OBZ,OCX,OFN,OFT,OKT,OLB,OLE,OPT,OR2,OR3,ORG,P10,P65,PAB,PAK,PAL,PAT,PBK,PBM,PCD,PCL,PCS,PCT,PCX,PDF,PDQ,PFA,PFB,PFC,PFM,PGL,PGM,PIC,PIF,PIG,PIN,PIX,PJ,PKG,PL,PLT,PM5,PM6,PNG,PNT,POT,PP4,PPA,PPM,PPS,PPT,PRE,PRF,PRN,PRS,PRZ,PS,PSD,PST,PTM,PUB,PWD,PWZ,PXL,QAD,QBW,QDT,QLB,QRY,QT,QTM,QXD,RA,RAM,RAS,RAW,RC,REC,REG,RES,RFT,RLE,RM,RMI,ROV,RPT,RTF,RTM,S3M,SAM,SAV,SC2,SCC,SCD,SCH,SCN,SCP,SCR,SCT,SDL,SDR,SDT,SEA,SEP,SHB,SHG,SHS,SHW,SIT,SLK,SND,SQC,SQR,STM,STY,SVX,SYS,T2T,TAR,TAZ,TEX,TGA,TGZ,THE,THN,TIF,TIG,TLB,TMP,TOP,TRM,TRN,TTF,TWF,TWW,TX8,TXT,UDF,ULT,URL,USE,VBP,VBW,VBX,VCF,VDA,VI,VLB,VOC,VP,VSD,VST,VSW,VXD,WAD,WAV,WB1,WB2,WBK,WBL,WCM,WDB,WEB,WGP,WID,WIL,WIZ,WK1,WK3,WK4,WKS,WLF,WLL,WMF,WOW,WP,WP4,WP5,WP6,WPD,WPG,WPS,WPT,WPW,WQ1,WQ2,WRI,WRL,WS1,WS2,WS3,WS4,WS5,WS6,WS7,WSD,WVL,XAR,XLA,XLB,XLC,XLD,XLK,XLM,XLS,XLT,XLV,XLW,XM,XR1,XY3,XY4,XYP,XYW,YAL,YBK,ZIP,ZOO, Abk,Acl,Acm,Acp,Act,Acv,Ad,Adb,Add,Adm,Adp,Adr,Af2,Af3,Afm,Ai,Aif,Alb,All,Ams,Anc,Ani,Ans,Api,Apr,Aps,Arc,Arj,Art,Asa,Asc,Asd,Asf,Asm,Ast,Asx,Att,Avi,Awd,B4,Bak,Bas,Bat,Bfc,Bg,Bi,Bif,Bin,Bk,Bks,Bm1,Bmk,Bmp,Brx,Bs1,Bsp,Btm,Cab,Cal,Cas,Cat,Cb,Ccb,Ccf,Cch,Ccm,Cda,Cdf,Cdi,Cdr,Cdt,Cdx,Cel,Cfb,Cfg,Cgm,Ch,Chk,Chp,Cil,Cim,Cin,Ck1,Ck2,Ck3,Ck4,Ck5,Ck6,Cla,Clp,Cls,Cmd,Cmf,Cmp,Cmv,Cnf,Cnm,Cnq,Cnt,Cob,Cod,Com,Cpd,Cpe,Cpi,Cpl,Cpp,Cpr,Cpt,Cpx,Crd,Crp,Crt,Csc,Csp,Csv,Ct,Ctl,Cue,Cur,Cut,Cv,Cwk,Cws,Cxx,Dat,Dbf,Dbx,Dcr,Dcs,Dcx,Ddf,Def,Der,Dib,Dic,Dif,Dir,Diz,Dlg,Dll,Dmf,Dmg,Doc,Dot,Dpr,Drv,Drw,Dsg,Dsm,Dsp,Dsq,Dsw,Dwg,Dxf,Emf,Enc,Eps,Er1,Erx,Evy,Ewl,Exe,F77,F90,Far,Fav,Fax,Fh3,Fif,Fit,Flc,Fli,Flt,Fmb,Fmt,Fmx,Fog,Fon,For,Fot,Fp,Fp1,Fp3,Fpx,Frm,Frx,Gal,Gcp,Ged,Gem,Gen,Gfc,Gfi,Gfx,Gid,Gif,Gim,Gix,Gna,Gnx,Gra,Grd,Grp,Gt2,Gtk,Gwx,Gwz,Gz,Hed,Hel,Hex,Hgl,Hlp,Hog,Hpj,Hpp,Hqx,Hst,Ht,Htx,Ica,Icb,Icm,Ico,Idd,Idq,Iff,Igf,Iif,Ima,Img,Inc,Inf,Ini,Inp,Ins,Iso,Isp,Isu,It,Iw,Jar,Jav,Jbf,Jff,Jif,Jmp,Jn1,Jpe,Jpg,Js,Jtf,Kdc,Kfx,Kye,Lbm,Ldb,Leg,Lha,Lib,Lis,Log,Lpd,Lrc,Lst,Lwo,Lwp,Lzh,Lzs,M3D,Mad,Maf,Mak,Mam,Map,Maq,Mar,Mas,Mat,Max,Maz,Mb1,Mcc,Mcs,Mcw,Mda,Mdb,Mde,Mdl,Mdn,Mdw,Mdz,Med,Mer,Met,Mi,Mic,Mid,Mmf,Mmm,Mod,Mov,Mp3,Mpe,Mpg,Mpp,Msg,Msi,Msn,Msp,Mtm,Mus,Mvb,Mwp,Nap,Ncb,Nsf,Nst,Ntf,Obd,Obj,Obz,Ocx,Ofn,Oft,Okt,Olb,Ole,Opt,Or2,Or3,Org,P10,P65,Pab,Pak,Pal,Pat,Pbk,Pbm,Pcd,Pcl,Pcs,Pct,Pcx,Pdf,Pdq,Pfa,Pfb,Pfc,Pfm,Pgl,Pgm,Pic,Pif,Pig,Pin,Pix,Pj,Pkg,Pl,Plt,Pm5,Pm6,Png,Pnt,Pot,Pp4,Ppa,Ppm,Pps,Ppt,Pre,Prf,Prn,Prs,Prz,Ps,Psd,Pst,Ptm,Pub,Pwd,Pwz,Pxl,Qad,Qbw,Qdt,Qlb,Qry,Qt,Qtm,Qxd,Ra,Ram,Ras,Raw,Rc,Rec,Reg,Res,Rft,Rle,Rm,Rmi,Rov,Rpt,Rtf,Rtm,S3M,Sam,Sav,Sc2,Scc,Scd,Sch,Scn,Scp,Scr,Sct,Sdl,Sdr,Sdt,Sea,Sep,Shb,Shg,Shs,Shw,Sit,Slk,Snd,Sqc,Sqr,Stm,Sty,Svx,Sys,T2T,Tar,Taz,Tex,Tga,Tgz,The,Thn,Tif,Tig,Tlb,Tmp,Top,Trm,Trn,Ttf,Twf,Tww,Tx8,Txt,Udf,Ult,Url,Use,Vbp,Vbw,Vbx,Vcf,Vda,Vi,Vlb,Voc,Vp,Vsd,Vst,Vsw,Vxd,Wad,Wav,Wb1,Wb2,Wbk,Wbl,Wcm,Wdb,Web,Wgp,Wid,Wil,Wiz,Wk1,Wk3,Wk4,Wks,Wlf,Wll,Wmf,Wow,Wp,Wp4,Wp5,Wp6,Wpd,Wpg,Wps,Wpt,Wpw,Wq1,Wq2,Wri,Wrl,Ws1,Ws2,Ws3,Ws4,Ws5,Ws6,Ws7,Wsd,Wvl,Xar,Xla,Xlb,Xlc,Xld,Xlk,Xlm,Xls,Xlt,Xlv,Xlw,Xm,Xr1,Xy3,Xy4,Xyp,Xyw,Yal,Ybk,Zip,Zoo,"*forum*","*Forum*","*messageboard*","*Messageboard*","*Viewforum*","*viewforum*","*discussion*","*Discussion*" --------- INI FILE ENDS HERE------------ Thanks. Yoav
