Ah, so the only way to avoid scanning large files, is to create a filelist beforehand, then remove the entires that are too large? And the same goes for mime types too, I guess?
Den torsdag den 17. august 2017 kl. 12.55.04 UTC+2 skrev Wesley Shields: > > Ok, so it isn’t a question of incorrect matching like you originally said. > It is really a question of searching a file even if it doesn’t pass the > global rules. This is a misunderstanding I sometimes see. All strings are > collected and searched for in a single pass, then global rules are > evaluated. > > — WXS > > On Thu, Aug 17, 2017 at 6:49 AM necrophcodr <[email protected] > <javascript:>> wrote: > >> Alright, I've solved the issue: >> >> Albeit this is synthetic, running >> >> ``` >> for f in $(seq 0 1000000); do printf "\n\n\n\n\n\n\n\n\n\n" >> text.txt; >> done >> ``` >> >> And then >> >> ``` >> yara inc.yar . >> ``` >> >> In the directory with the yara files, yields, on my test system: >> >> ./misc.yar(9): warning: $newline is slowing down scanning (critical!) >> fsL ./inc.yar >> fsL ./global.yar >> fsL ./misc.yar >> error scanning ./text.txt: internal error: 30 >> >> >> And while using the newline scan is not a great idea, clearly the file is >> still being scanned in one way or another, in spite of the global rule. >> >> >> Den torsdag den 17. august 2017 kl. 12.24.11 UTC+2 skrev necrophcodr: >>> >>> I'm afraid I cannot post the exact files. I'll create a working >>> environment that replicates all the variables required, and I'll post it >>> here when I've gotten this done. >>> >>> Den onsdag den 16. august 2017 kl. 16.31.35 UTC+2 skrev Wesley Shields: >>>> >>>> I still can not replicate your problem. >>>> >>>> A couple of things to note however: >>>> >>>> "internal error: 30" is because there are too many matches, which >>>> happens when a single string matches too many times. It has nothing to do >>>> with file size like you guessed. >>>> >>>> Your "newline_one" rule is marked as private so it should never be >>>> reported. >>>> >>>> At this point I can not replicate your problem so I'm curious if you >>>> could zip up the files you're using to do this and post them somewhere for >>>> me to see them exactly? I would need your exact YARA rules and the file >>>> you >>>> are scanning. >>>> >>>> -- WXS >>>> >>>> > On Aug 16, 2017, at 5:51 AM, necrophcodr <[email protected]> >>>> wrote: >>>> > >>>> > Alright, so I've returned with a result: >>>> > >>>> > If I have `~/inc.yar` with the following content: >>>> > >>>> > ``` >>>> > include "./global.yar" >>>> > include "./misc.yar" >>>> > ``` >>>> > >>>> > And the content of these files respectively: >>>> > >>>> > ``` >>>> > global rule fsL { condition: filesize < 8MB } >>>> > ``` >>>> > >>>> > And >>>> > >>>> > ``` >>>> > private rule newline_one { >>>> > meta: >>>> > description = "Files that contain one newline" >>>> > author = "Steffen Rytter Postas" >>>> > >>>> > strings: >>>> > $newline = "\n" >>>> > >>>> > condition: >>>> > ( #newline == 1 ) >>>> > } >>>> > ``` >>>> > >>>> > Then the issue prevails. >>>> > >>>> > Note that this requires an actually large file that contains >>>> newlines. Doing `dd if=/dev/zero bs=4M count=250 of=file.bin` and scanning >>>> that won't yield usable results. >>>> > >>>> > Den onsdag den 16. august 2017 kl. 11.43.17 UTC+2 skrev necrophcodr: >>>> > Hi Wesley, >>>> > >>>> > Sorry for the late reply, vacations and all. >>>> > >>>> > So first and foremost: >>>> > >>>> > `yara -v` >>>> > yara 3.5.0 >>>> > >>>> > The files getting scanned are reporting ` internal error: 30` which >>>> I'm reading to be due to files being too large. These files are often >>>> larger than 500MB too, well above the 8MB margin. >>>> > >>>> > I've attempted to replicate it using my own instructions, coupled >>>> with your misc.yar, and the result is that it works just fine. >>>> > >>>> > So I'm guessing the issue is with my own setup, and I'll continue >>>> evaluating the specifics and return with a response when I've found the >>>> culprit. >>>> > >>>> > Den mandag den 7. august 2017 kl. 16.06.59 UTC+2 skrev Wesley >>>> Shields: >>>> > I can't replicate this behavior using 3.5.0 or latest master. >>>> > >>>> > wxs@wxs-mbp yara % cat foo >>>> > include "./global.yar" >>>> > include "./misc.yar" >>>> > wxs@wxs-mbp yara % cat global.yar >>>> > global rule fileSizeLimit { condition: filesize < 1KB } >>>> > wxs@wxs-mbp yara % cat misc.yar >>>> > rule foo { condition: true } >>>> > wxs@wxs-mbp yara % ls -l /bin/ls >>>> > -rwxr-xr-x 1 root wheel 38624 Jul 15 00:29 /bin/ls* >>>> > wxs@wxs-mbp yara % ./yara foo /bin/ls >>>> > wxs@wxs-mbp yara % >>>> > >>>> > When you say regardless of file size are you sure you're above the >>>> 8MB? Keep in mind that 8MB is 8 * 1048576, which is 8388608. >>>> > >>>> > -- WXS >>>> > >>>> > > On Jul 28, 2017, at 7:01 AM, necrophcodr <[email protected]> >>>> wrote: >>>> > > >>>> > > So I've got quite a few rules, but it all comes down to this: >>>> > > >>>> > > include "./rules/global.yar" >>>> > > include "./rules/misc.yar" >>>> > > >>>> > > >>>> > > The global.yar file contains >>>> > > >>>> > > global rule fileSizeLimit { condition: filesize < 8MB } >>>> > > >>>> > > >>>> > > Any rule defined in rules/misc.yar are matched regardless of file >>>> size, but this is not what I intend. What am I doing wrong here? >>>> > > >>>> > > If this is not the right place to post, that's alright, feel free >>>> to slap me on the wrist and direct me to the correct location. >>>> > > >>>> > > edit: >>>> > > >>>> > > I should mention this is using Yara 3.5.0. I don't have a chance to >>>> upgrade this within the week. >>>> > > >>>> > > >>>> > > -- >>>> > > You received this message because you are subscribed to the Google >>>> Groups "YARA" group. >>>> > > To unsubscribe from this group and stop receiving emails from it, >>>> send an email to [email protected]. >>>> > > For more options, visit https://groups.google.com/d/optout. >>>> > >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "YARA" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "YARA" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
