Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
On 10/3/2016 2:39 AM, Mark Geisert wrote: Tony Kelman wrote: I see what you've been saying about different insider builds of Windows 10 but I don't think anybody here is beating up on these releases. Win10 anniversary with a single cygwin64 installation is sufficiently difficult for me. Unscheduled Microsoft update reboots, inability to bootstrap gcc (where it works on win8.1)... -- Tim Prince -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
> No, that looks like I'd expect it to. You need to look at the process > map when you have fork problems, since the problem seems to be > intermittent. How so? Something like for i in `seq 1000`; do cat /proc/self/maps > procselfmaps_`date +%s.%N`.txt; done & before running the problematic command? It does take a variable amount of time to trigger, and sometimes exhibits as a complete freeze instead of an error. What should I be looking for here? > Have you tried to play around with the virus scanner settings? I already have a Windows Defender exclusion for my Cygwin installs. Turning off real-time protection temporarily didn't help. > One sure-fire way to make sure your 32-bit E: installation is not causing > havoc > with your 64-bit C: installation is to temporarily rename E:\cygwin32 to > E:\somethingelse and retry your build. This should not be necessary but I > don't > know what else to suggest. Good idea. Tried it, didn't help. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
Tony Kelman wrote: You've got two different cygwin1.dll somewhere on your PATH. For more information, I have one cygwin64 installation on C:, one cygwin32 installation on E:, and they're never both on my path at the same time. I'm not sure where that orphan registry entry that was showing up in cygcheck.out came from, but I removed it and it made no difference. With only one Cygwin installation on C: I can only come up with wild guesses. Are you somehow alternating between the C: and E: Cygwin versions, or running them simultaneously from separate windows, or accidentally running 32-bit Cygwin commands from a 64-bit Cygwin terminal window? Unusual stuff like that? One sure-fire way to make sure your 32-bit E: installation is not causing havoc with your 64-bit C: installation is to temporarily rename E:\cygwin32 to E:\somethingelse and retry your build. This should not be necessary but I don't know what else to suggest. I have in the past run 32-bit and 64-bit Cygwin installations simultaneously on the same machine, in order to build 32-bit and 64-bit Cygwin DLLs at the same time. Never had a problem. But that was Windows 7 so probably not applicable. This is absolutely due to a change in Windows itself. 3 updates in a row now, 14926, 14931, and now 14936, it's 100% reproducible just from updating Windows. Downgrading to build 14915 gets everything back working again, but that build is now expired so the only way for me to get back to a supported build of Windows where cygwin works properly is by totally reinstalling Windows. I haven't done that yet, but if we can't identify the problem and a solution soon, I will have to. I see what you've been saying about different insider builds of Windows 10 but I don't think anybody here is beating up on these releases. ..mark -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
Tony Kelman writes: >> Something occupies the heap area for Cygwin, based on the low address. >> What does /proc/self/maps tell you? > > $ cat /proc/self/maps […] > > Hopefully you know what you're looking at there. Anything meaningful in that? No, that looks like I'd expect it to. You need to look at the process map when you have fork problems, since the problem seems to be intermittent. Have you tried to play around with the virus scanner settings? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
>> Still a problem in 14936. Folks, this could be very bad. Anyone at all >> testing the insider builds, or are we going to be blindsided when an >> update goes out to everyone that breaks cygwin? > > How about you start with a sane PATH that doesn#t contain all the > Windows stuff? Set a system variable CYGWIN_NOWINPATH=true and try > again. Didn't help. BTW, that remains undocumented, ref https://cygwin.com/ml/cygwin/2014-08/msg00418.html C:\cygwin64\bin>set CYGWIN_NOWINPATH=true C:\cygwin64\bin>sh -l Tony@LAPTOP-O230JCFF ~ $ cd github/curl Tony@LAPTOP-O230JCFF ~/github/curl $ rm -rf build && mkdir build && cd build Tony@LAPTOP-O230JCFF ~/github/curl/build $ cmake .. && make -j4 && echo ok -- The C compiler identification is GNU 5.4.0 CMake Warning at /usr/share/cmake-3.6.2/Modules/Platform/CYGWIN.cmake:15 (message): CMake no longer defines WIN32 on Cygwin! (1) If you are just trying to build this project, ignore this warning or quiet it by setting CMAKE_LEGACY_CYGWIN_WIN32=0 in your environment or in the CMake cache. If later configuration or build errors occur then this project may have been written under the assumption that Cygwin is WIN32. In that case, set CMAKE_LEGACY_CYGWIN_WIN32=1 instead. (2) If you are developing this project, add the line set(CMAKE_LEGACY_CYGWIN_WIN32 0) # Remove when CMake >= 2.8.4 is required at the top of your top-level CMakeLists.txt file or set the minimum required version of CMake to 2.8.4 or higher. Then teach your project to build on Cygwin without WIN32. Call Stack (most recent call first): /usr/share/cmake-3.6.2/Modules/CMakeSystemSpecificInformation.cmake:36 (include) CMakeLists.txt:47 (project) -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done CMake Warning at CMakeLists.txt:49 (message): the curl cmake build system is poorly maintained. Be aware -- curl version=[7.51.0-DEV] -- Performing Test HAVE_SOCKADDR_IN6_SIN6_ADDR -- Performing Test HAVE_SOCKADDR_IN6_SIN6_ADDR - Success -- Performing Test HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID -- Performing Test HAVE_SOCKADDR_IN6_SIN6_SCOPE_ID - Success Found *nroff option: -- -man -- Looking for dlopen in dl; -- Looking for dlopen in dl; - found -- Looking for connect in socket;dl -- Looking for connect in socket;dl - not found -- Looking for gethostbyname in c -- Looking for gethostbyname in c - found -- Looking for gethostname -- Looking for gethostname - found 0 [main] cmake 1280 child_info_fork::abort: C:\cygwin64\bin\cygintl-8.dll: Loaded to different address: parent(0x3E368) != child(0xC) [1]+ Stopped(SIGSTOP)cmake .. > Something occupies the heap area for Cygwin, based on the low address. > What does /proc/self/maps tell you? $ cat /proc/self/maps 0001-0002 rw-s : 0 [win heap 1 default shared] 0002-00021000 rw-s : 0 0003-00048000 r--s : 0 0005-00054000 r--s : 0 0006-00061000 r--s : 0 0007-00072000 rw-p : 0 0009-00099000 rw-p : 0 [win heap 0 default grow] 00099000-0019 ===p 9000 : 0 [win heap 0 default grow] 0019-00191000 rw-p : 0 [win heap 0 default grow] 00191000-001C2000 ===p 1000 : 0 [win heap 0 default grow] 0020-00352000 ===p : 0 00352000-00353000 rw-p 00152000 : 0 [peb] 00353000-00355000 rw-p 00153000 : 0 [teb (tid 7204)] 00355000-00357000 rw-p 00155000 : 0 [teb (tid 7552)] 00357000-00359000 rw-p 00157000 : 0 [teb (tid 7648)] 00359000-0035B000 rw-p 00159000 : 0 [teb (tid 10468)] 0035B000-0035D000 rw-p 0015B000 : 0 [teb (tid 3876)] 0035D000-0040 ===p 0015D000 : 0 0060-006C1000 r--s C225:5FBC 1407374885978955 /cygdrive/c/Windows/System32/locale.nls 006D-008CB000 ===p : 0 [stack (tid 7552)] 008CB000-008CE000 rw-g 001FB000 : 0 [stack (tid 7552)] 008CE000-008D rw-p 001FE000 : 0 [stack (tid 7552)] 008D-00ACB000 ===p : 0 [stack (tid 7648)] 00ACB000-00ACE000 rw-g 001FB000 : 0 [stack (tid 7648)] 00ACE000-00AD rw-p 001FE000 : 0 [stack (tid 7648)] 00AD-00CCB000 ===p : 0 [stack (tid 10468)] 00CCB000-00CCE000 rw-g 001FB000 : 0 [stack (tid 10468)] 00CCE000-00CD rw-p 001FE000 : 0 [stack
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
Tony Kelman writes: >> Could you paste a complete sample of the error message so we can >> determine where in the Cygwin code it's coming from? > > Still a problem in 14936. Folks, this could be very bad. Anyone at all > testing the insider builds, or are we going to be blindsided when an > update goes out to everyone that breaks cygwin? How about you start with a sane PATH that doesn#t contain all the Windows stuff? Set a system variable CYGWIN_NOWINPATH=true and try again. > Here's one: > > 1 [main] cp (6432) C:\cygwin64\bin\cp.exe: *** fatal error - cygheap > base mismatch detected - 0x180302408/0xD92408. > This problem is probably due to using incompatible versions of the > cygwin DLL. Something occupies the heap area for Cygwin, based on the low address. What does /proc/self/maps tell you? > And another: > > 0 [main] cmake 10384 child_info_fork::abort: > C:\cygwin64\bin\cygintl-8.dll: Loaded to different address: > parent(0x3E368) != child(0x19) Again either an address collision or some BLODA intercepting the DLL, check the memory maps to see what might be going on. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Waldorf MIDI Implementation & additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
> You've got two different cygwin1.dll somewhere on your PATH. For more information, I have one cygwin64 installation on C:, one cygwin32 installation on E:, and they're never both on my path at the same time. I'm not sure where that orphan registry entry that was showing up in cygcheck.out came from, but I removed it and it made no difference. This is absolutely due to a change in Windows itself. 3 updates in a row now, 14926, 14931, and now 14936, it's 100% reproducible just from updating Windows. Downgrading to build 14915 gets everything back working again, but that build is now expired so the only way for me to get back to a supported build of Windows where cygwin works properly is by totally reinstalling Windows. I haven't done that yet, but if we can't identify the problem and a solution soon, I will have to. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
> You've got two different cygwin1.dll somewhere on your PATH. As your builds > proceed, they likely start out using the correct cygwin1.dll but sometimes > load > the wrong cygwin1.dll along the way. The error message tells you how to solve > the problem. But I usually use this method: > - open a Command Prompt running as Administrator > - cd c:\ or wherever the root of your Cygwin installation drive is > - dir /S cygwin1.dll > You might see multiple listings of the same cygwin1.dll due to the crazy WoW > support but these will all have the same date and size. You want to find the > cygwin1.dll that has a different date (probably older) and size. That's the > problematic one; rename it or delete it. And if not? C:\>dir /S cygwin1.dll Volume in drive C is Windows Volume Serial Number is C225-5FBC Directory of C:\cygwin64\bin 08/31/2016 05:32 AM 3,306,962 cygwin1.dll 1 File(s) 3,306,962 bytes Total Files Listed: 1 File(s) 3,306,962 bytes 0 Dir(s) 58,480,676,864 bytes free C:\> -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
Tony Kelman wrote: Could you paste a complete sample of the error message so we can determine where in the Cygwin code it's coming from? Still a problem in 14936. Folks, this could be very bad. Anyone at all testing the insider builds, or are we going to be blindsided when an update goes out to everyone that breaks cygwin? Here's one: 1 [main] cp (6432) C:\cygwin64\bin\cp.exe: *** fatal error - cygheap base mismatch detected - 0x180302408/0xD92408. This problem is probably due to using incompatible versions of the cygwin DLL. Search for cygwin1.dll using the Windows Start->Find/Search facility and delete all but the most recent version. The most recent version *should* reside in x:\cygwin\bin, where 'x' is the drive on which you have installed the cygwin distribution. Rebooting is also suggested if you are unable to find another cygwin DLL. OK, you're seeing two different problems. Solving the one above is critical. It might help solve the other problem shown below; we'll have to see. You've got two different cygwin1.dll somewhere on your PATH. As your builds proceed, they likely start out using the correct cygwin1.dll but sometimes load the wrong cygwin1.dll along the way. The error message tells you how to solve the problem. But I usually use this method: - open a Command Prompt running as Administrator - cd c:\ or wherever the root of your Cygwin installation drive is - dir /S cygwin1.dll You might see multiple listings of the same cygwin1.dll due to the crazy WoW support but these will all have the same date and size. You want to find the cygwin1.dll that has a different date (probably older) and size. That's the problematic one; rename it or delete it. Then try running your build again. And another: 0 [main] cmake 10384 child_info_fork::abort: C:\cygwin64\bin\cygintl-8.dll: Loaded to different address: parent(0x3E368) != child(0x19) This is a garden-variety fork() failure due to the child having something in its address space at 0x3E368 such that Cygwin can't load cygintl-8.dll at that address where the parent has it. But since you've apparently got two Cygwin installations on your PATH, maybe that's complicating things. See if fixing the first problem above makes this problem go away too. ..mark -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
> You're absolutely sure your rebasing operations are being done without > any Cygwin processes running, even background cygserver, sshd, etc? Pretty sure. I don't have any cygwin processes set up to start at login, and I was rebooting several times here. > Could you paste a complete sample of the error message so we can > determine where in the Cygwin code it's coming from? I'll have to see if I can reproduce in a VM, and hope they release a stable build soon that doesn't expire on me or exhibit this problem. Is anyone else testing insider builds? -Tony -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
Tony Kelman wrote: I've tried manually rebasing, setting rebase-trigger, rerunning latest setup, rebooting, etc. During big parallel makes none of the above helped with getting rid of intermittent heap mismatch errors during fork after I upgraded to Windows 10 Insider build 14926. I had been on the Insider fast ring for a few months Anybody? Just checked 14931 and it has the same problem. I'll be switching back the slow ring as soon as it lets me, but I'm about to start getting "build will expire" popups. You're absolutely sure your rebasing operations are being done without any Cygwin processes running, even background cygserver, sshd, etc? Could you paste a complete sample of the error message so we can determine where in the Cygwin code it's coming from? ..mark -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Advice for debugging heap mismatches? (Win10 Insider build 14926)
> I've tried manually rebasing, setting rebase-trigger, rerunning latest > setup, rebooting, etc. During big parallel makes none of the above helped > with getting rid of intermittent heap mismatch errors during fork after I > upgraded to Windows 10 Insider build 14926. I had been on the Insider fast > ring for a few months Anybody? Just checked 14931 and it has the same problem. I'll be switching back the slow ring as soon as it lets me, but I'm about to start getting "build will expire" popups. -Tony -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
