On May 14, 2015, at 10:56 AM, Andrey Repin <anrdae...@yandex.ru> wrote: > > Greetings, Vince Rice! > >> uname says "CYGWIN_NT-6.1 machinename 1.7.35(0.287/5/3) 2015-03-04 12:07 >> i686 Cygwin”. >> I’m running grep 2.21.2, which cygcheck -c says is OK. > >> Does Cygwin’s grep support Unicode files? The output from a SQL Server SQL >> Agent job is a Unicode file, i.e. if you look at it in a hex editor every >> other character is 00 because each character is taking up two bytes. The >> filename itself is fine, it’s the contents that is Unicode. I can’t get grep >> to work on it, either with or without -a. > >> This may not be a Cygwin-specific question, but I haven’t been able to find >> anything after several Google searches, including the archives, and neither >> --help nor the man page for grep references Unicode. > >> By default I have neither LC_ALL nor LC_COLLATE set. > >> A pointer to a better search or a website that explains this would be >> great, or if it can’t currently be done, that’s OK, too. > > grep only treat files as text if they are matching current locale. > Check `locale` output to see your current settings.
First, to the other responder(s), running it through iconv with a from of UTF16 and a to of UTF8 did work. Thanks for the pointer. (I’ve never had to deal with anything but ANSI files, so I didn’t know about iconv. And I guessed on the UTF8, given what I found below.) locale run from a cmd.exe session says that everything is “C.UTF-8”, while locale run from mintty says that everything is en_US.UTF-8. A “which” in both cases shows that the locale being run is cygwin’s, so I assume mintty does something slightly differently than the normal console? I don’t even know if there’s a difference. (Have I mentioned I don’t know anything about all of this?) From cmd.exe: LANG= LC_CTYPE="C.UTF-8" LC_NUMERIC="C.UTF-8" LC_TIME="C.UTF-8" LC_COLLATE="C.UTF-8" LC_MONETARY="C.UTF-8" LC_MESSAGES="C.UTF-8" LC_ALL= From mintty LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_ALL= Now, pardon my continued ignorance, but which of those variables needs to be set to UTF16 in order for grep to work? And I assume it (they?) should be set to en_US.UTF-16? Thanks to everyone for your help. I think you’ve all confirmed this isn’t cygwin-specific, but I couldn’t find anything even searching generically (“grep unicode” and now “grep utf16”). I did finally find an external reference to iconv, but if grep is supposed to be handle this natively, I haven’t been able to find much on how to do it. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
