New topic: Slow file processing?
<http://forums.realsoftware.com/viewtopic.php?t=47068> Page 1 of 1 [ 1 post ] Previous topic | Next topic Author Message Zack.H Post subject: Slow file processing?Posted: Mon Feb 25, 2013 2:17 pm Joined: Sun Jun 15, 2008 1:50 am Posts: 69 So I've written some code which works fine on a file with about 5/6 lines haven't been able to test it on anything larger till now, and now that I've tested it on a much larger file it seems like the processing time is taking wayyyy longer then I had expected. Does MD5Digest slow down code a lot?, Is it possible to use threads in Web Apps so I can split the file into chunks and process each chunk in it's own thread? Any other suggestions on how I can optimize this because the amount of time it's taking to do it's processing is like 2mins per 1,000 lines or such if not longer. As far as what the application is for it's a freelance project ( I know some of you are gonna wonder about the email and such in there, the company wants it for their newsletter system and they have about 20,000 subscribers they want to email through it so processing at 1k every 2-10mins is just horrid for what they're looking for.. Previous to adding in the MD5Digest code for unique IDs it seemed to be a lot faster but then again I also had no reason to re-write the file at that point because I had no data to append to it.) Dim F As FolderItem = GetFolderItem("./Lists/"+File) Dim T As TextInputStream Dim Data As String Dim Lines() As String Dim Emails() As String Dim dbFile as FolderItem Dim ImageHeaders As String Dim ImageUrl As String Dim ImageId As String Dim db as RealSQLDatabase db=New RealSQLdatabase dbFile = GetFolderItem("database.rsd") db.DatabaseFile=dbFile if db.Connect() Then T = T.Open( F ) Data = T.ReadAll() Lines = Data.Split(EndOfLine.Windows) For Each Row as String In Lines Dim Columns() As String Columns = Row.Split(",") for each Column As String In Columns If( Column.InStr("@") > 0 ) Then Emails.Append(Column) end next Next for i as integer = 0 to Images.Ubound() ImageHeaders = ImageHeaders+"Img"+str(i)+"," next Dim O As FolderItem = GetFolderItem("./Lists/"+File+"_Out.csv") Dim Tout As TextOutputStream Tout = Tout.Create(O) Tout.WriteLine("clickid,unsubid,openid,"+ImageHeaders+"*To,") Dim Lists() As String For each Email As String in Emails ImageId = "" Dim Md5ClickId as Md5Digest = new Md5Digest Dim Md5UnsubId As Md5Digest = new Md5Digest Dim Md5OpenId As Md5Digest = new Md5Digest Dim R As Random = new Random R.RandomizeSeed() for each Image As String in Images Dim Md5ImgId As Md5Digest = new Md5Digest Md5ImgId.Process(Email+str(R.InRange(0,9000))+Image) ImageId = ImageId + EncodeHex(Md5ImgId.Value).Uppercase+"," db.SQLExecute("INSERT INTO urls VALUES('"+EncodeHex(Md5ImgId.Value).Uppercase+"','"+Image+"','"+Email+"','img','')") if db.error = true Then MsgBox db.ErrorMessage else db.commit end next MD5ClickId.Process(Email+str(R.InRange(0,9000))+ClickUrl) Md5UnsubId.Process(Email+str(R.InRange(0,9000))+UnsubUrl) Md5OpenId.Process(Email+str(R.InRange(0,9000))) if RandomizeLists.Value = True Then Lists.Append(EncodeHex(Md5ClickId.Value).UpperCase+","+EncodeHex(Md5UnsubId.Value).Uppercase+","+EncodeHex(Md5OpenId.Value).Uppercase+","+ImageId+Email+",") else Tout.WriteLine(EncodeHex(Md5ClickId.Value).UpperCase+","+EncodeHex(Md5UnsubId.Value).Uppercase+","+EncodeHex(Md5OpenId.Value).Uppercase+","+ImageId+Email+",") end db.SQLExecute("INSERT INTO urls VALUES('"+EncodeHex(Md5ClickId.Value).Uppercase+"','"+ClickUrl+"','"+Email+"','click','')") if db.Error = True Then MsgBox db.ErrorMessage else db.commit end db.SQLExecute("INSERT INTO urls VALUES('"+EncodeHex(Md5UnsubId.Value).Uppercase+"','"+UnsubUrl+"','"+Email+"','unsub','')") if db.Error = True Then MsgBox db.ErrorMessage else db.commit end db.SQLExecute("INSERT INTO urls VALUES('"+EncodeHex(Md5OpenId.Value).Uppercase+"','open','"+Email+"','open',' ')") if db.Error = True Then MsgBox db.ErrorMessage else db.commit end Next if RandomizeLists.Value = True Then Lists.Shuffle() for each List As String in Lists Tout.WriteLine(List) next end end Top Display posts from previous: All posts1 day7 days2 weeks1 month3 months6 months1 year Sort by AuthorPost timeSubject AscendingDescending Page 1 of 1 [ 1 post ] -- Over 1500 classes with 29000 functions in one REALbasic plug-in collection. The Monkeybread Software Realbasic Plugin v9.3. http://www.monkeybreadsoftware.de/realbasic/plugins.shtml [email protected]
