Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Heh, it's Christmas time and you're hacking? I'm having a hard time deciding if I should worry or just say way to go! ;-) On the other hand, I should rather shut up. After all, here I am reading and responding to email... Cheers, Richard In message [EMAIL PROTECTED] on Mon, 24 Dec 2007 16:30:23 -0500, Stephen Leake [EMAIL PROTECTED] said: stephen_leake I started another branch net.venge.monotone.win32_pipes_2, to try to stephen_leake get named pipes working on Win32, and Unix pipes on Unix. stephen_leake netxx_pipe.hh has an extensive comment discussing the rationale; stephen_leake basically, it's the only way to support 'mtn sync ssh:' and 'mtn sync stephen_leake file:', and detect connection closures. stephen_leake stephen_leake However, I realized that the current Cygwin mtn works quite nicely for stephen_leake 'mtn sync ssh:' and 'mtn sync file:' (ie, Cygwin emulates 'select' far stephen_leake better than the current mtn PipeCompatibleStream does), so I'll just stephen_leake use that for a while. stephen_leake stephen_leake I may come back to this later. -- Richard Levitte [EMAIL PROTECTED] http://richard.levitte.org/ When I became a man I put away childish things, including the fear of childishness and the desire to be very grown up. -- C.S. Lewis ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
I've got 'mtn sync file:' working on Win32. I modified netsync slightly so that call_server terminates when the 'bye' protocol is finished, rather than waiting for the socket to disconnect. In addition, 'mtn sync ssh:RedHat machine' from the RedHat machine to itself works fine. However, 'mtn sync ssh:RedHat machine' from Win32 does _not_ work, because ssh doesn't do 'recv/send' socket calls on win32; it does 'read/write', which don't work with sockets. I need ssh access to a central server; that's just how it is :(. I'll look into using ssh to replicate ports; maybe 'mtn sync local replicated port' will work. Or maybe go back to getting Win32 pipes to work, now that I know a lot more about how netsync works. -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
I've got this almost working. The remaining problem is that the local mtn doesn't detect that the spawned server mtn closes its socket. Your proof of concept code doesn't demonstrate this, either. I've checked in a unit test pipe:spawn_stdio that shows the problem. One way around this is to change netsync to exit when it gets the bye 2 message from the server, rather than waiting for the server to close the connection. -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Progress; I've checked in unit tests in netxx_pipes.cc that show that StdioStream, StdioProbe, and SpawnedStream all work properly on Win32 and Unix. Tested on Win32 MinGW and RedHat. However, the lua test netsync_over_pipes still hangs on Win32 and Unix, so there's more work to do. -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Matthew Gregan [EMAIL PROTECTED] writes: Ok. Can you send me that code? I'm at the point of implementing some simple standalone debug code. Maybe I could start with yours. Attached. Try not to be too shocked, it's a fearsome trainwreck of write-only test code. Looks like the secret is to call GetStdHandle instead of just using STDIN_FILENO. At least, that gets past the select on non-socket problem; now I have other problems. Making progress ... -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
At 2007-12-11T19:44:54-0500, Stephen Leake wrote:
Hmm. If we want Windows to be an ssh server, it must handle this case.
Well, not quite this case. mtn serve --stdio on a console guarantees that
the stdio handles are console handles, and we know we can't treat them like
sockets.
That is, the client does:
mtn sync ssh:my.windows.box.org
then on 'my.window.box.org', ssh runs the command
mtn serve --sdtio
and stdio for mtn is _not_ a socket. Actually, I have no idea what
mechanism ssh uses to spawn mtn in this case. But if we advertise
--stdio, it aught to handle the non-socket case.
Hmm. I guess I should look in the ssh source to see what it does here;
it might be instructive.
...but ssh might be different. I don't know what it does, either. Did you
look into this yet?
Or we can just say mtn can't be an ssh server on Windows. That would
work for me, but it would be disappointing.
It'd be nice to have the server side working with ssh, but I think having
the client side will be an 80%+ solution--ssh servers on Windows seem to be
fairly uncommon in my experience.
Ok. Can you send me that code? I'm at the point of implementing some
simple standalone debug code. Maybe I could start with yours.
Attached. Try not to be too shocked, it's a fearsome trainwreck of
write-only test code.
Cheers,
-mjg
--
Matthew Gregan |/
/|[EMAIL PROTECTED]
/* test ncm's socketpair for win32 */
#define WIN32_LEAN_AND_MEAN
#include io.h
#include windows.h
#include winsock2.h
#include assert.h
#include stdio.h
#include stdlib.h
typedef LONG NTSTATUS;
typedef struct _IO_STATUS_BLOCK {
union {
NTSTATUS Status;
PVOID Pointer;
};
ULONG_PTR Information;
} IO_STATUS_BLOCK, *PIO_STATUS_BLOCK;
typedef enum _FILE_INFORMATION_CLASS {// Query Set File/Directory
FileModeInformation= 16 // Y Y F
} FILE_INFORMATION_CLASS, *PFILE_INFORMATION_CLASS;
typedef struct _FILE_MODE_INFORMATION { // Information Class 16
ULONG Mode;
} FILE_MODE_INFORMATION, *PFILE_MODE_INFORMATION;
// NtQueryInformationFile retrieves information about a file object.
NTSTATUS
NTAPI
NtQueryInformationFile(
IN HANDLE FileHandle,
OUT PIO_STATUS_BLOCK IoStatusBlock,
OUT PVOID FileInformation,
IN ULONG FileInformationLength,
IN FILE_INFORMATION_CLASS FileInformationClass
);
int dumb_socketpair(SOCKET socks[2], int make_overlapped);
HANDLE hChild;
BOOL WINAPI
ctrl_c_handler(DWORD dwCtrlType)
{
fprintf(stderr, terminate %d\n, TerminateProcess(hChild, 0));
fflush(stderr);
return TRUE;
}
static void
fatal(const char * msg)
{
fprintf(stderr, %s (%d)\n, msg, WSAGetLastError());
exit(EXIT_FAILURE);
}
int
main()
{
WSADATA wsadata;
assert(WSAStartup(MAKEWORD(2, 2), wsadata) == 0);
SOCKET pair[2];
assert(dumb_socketpair(pair, 0) == 0);
FILE_MODE_INFORMATION info;
IO_STATUS_BLOCK io;
NTSTATUS status;
status = NtQueryInformationFile((HANDLE) pair[0], io, info, sizeof(info), FileModeInformation);
fprintf(stderr, pair[0] overlapped? %s\n, info.Mode == 0x20 ? no : yes);
status = NtQueryInformationFile((HANDLE) pair[1], io, info, sizeof(info), FileModeInformation);
fprintf(stderr, pair[1] overlapped? %s\n, info.Mode == 0x20 ? no : yes);
status = NtQueryInformationFile(GetStdHandle(STD_INPUT_HANDLE), io, info, sizeof(info), FileModeInformation);
fprintf(stderr, stdin overlapped? %s\n, info.Mode == 0x20 ? no : yes);
status = NtQueryInformationFile((HANDLE) _get_osfhandle(_fileno(stdin)), io, info, sizeof(info), FileModeInformation);
fprintf(stderr, stdin overlapped? %s\n, info.Mode == 0x20 ? no : yes);
#if 1
static char const cat[] = e:\\build\\src\\sockio\\a.exe;
#else
static char const cat[] = d:\\msys\\1.0\\bin\\ssh.exe; //d:\\msys\\1.0\\bin\\cat.exe; //e:\\build\\src\\sockio\\echoer.exe;
static char const args[][2] = { /*d:\\msys\\1.0\\bin\\ssh.exe,*/ {brak}, {NULL} };
#endif
STARTUPINFO startup;
PROCESS_INFORMATION procinfo;
memset(startup, 0, sizeof(startup));
startup.cb = sizeof(startup);
// should dup handle before using for in/out/err so child can legally
// close one without losing the others
HANDLE hin, hout, herr;
assert(DuplicateHandle(GetCurrentProcess(), (HANDLE) pair[0], GetCurrentProcess(), hin, 0, TRUE, DUPLICATE_SAME_ACCESS) != 0);
assert(DuplicateHandle(GetCurrentProcess(), (HANDLE) pair[0], GetCurrentProcess(), hout, 0, TRUE, DUPLICATE_SAME_ACCESS) != 0);
assert(DuplicateHandle(GetCurrentProcess(), (HANDLE) pair[0], GetCurrentProcess(), herr, 0, TRUE, DUPLICATE_SAME_ACCESS) != 0);
CloseHandle((HANDLE) pair[0]);
startup.hStdInput = hin;
startup.hStdOutput = hout;
startup.hStdError = herr;
startup.hStdError = GetStdHandle(STD_ERROR_HANDLE);
startup.dwFlags = STARTF_USESTDHANDLES;
assert(CreateProcess(cat, (char *) cat, NULL, NULL, TRUE, 0, NULL, NULL, startup, procinfo) != 0);
fprintf(stderr, %d %d\n, (int)
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
At 2007-12-09T16:32:42-0500, Stephen Leake wrote: mtn: fatal: std::runtime_error: network error: select(2): An operation was attempted on something that is not a socket. (10038) Okay so far, you'd expect to see this when the stdio handles are not sockets. The same occurs when 'mtn --stdio' is spawned with a socket as stdio, as you can see in tester.log. So... we need to work out why this is happening, because it should work. I haven't had time to look over your changes in any depth yet, but I will have some time in the weekend, so if you're still stuck on this I'll take a crack at it as well. So I think we are back to the fundamental problem; how do we do non-blocking IO on Win32 stdio? I have a working standalone proof of concept for this approach already, so I know we can make it work with monotone. Cheers, -mjg -- Matthew Gregan |/ /|[EMAIL PROTECTED] ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Stephen Leake [EMAIL PROTECTED] writes:
I've created a branch n.v.m.experimental.win32_pipes; it has no
changes in it yet.
I've checked in a first attempt.
It compiles on Win32, but fails the relevant tests:
unit tests:
59 pipe_spawned FAIL (line 1)
lua tests:
116 clone_-b_no_dir FAIL (line 8)
117 clone_creates__MTN_logFAIL (line 16)
118 clone_creates_right__MTN_options FAIL (line 20)
119 clone_validates_target_directory FAIL (line 9)
120 clone_warning_with_multiple_heads FAIL (line 19)
306 netsync_over_pipesFAIL (line 9)
I also have access to a RedHat box, so I'll work on the Unix version
as well.
I split PipeStream into SpawnedStream and StdioStream; that makes it
clearer what is going on. There is a comment in netxx_pipe.hh that
explains the rationale. The code looks much cleaner, but it isn't
working. I haven't started debugging it yet.
Once it's working, we can refactor stuff into win32/ and unix/, and
rename netxx_pipe.{hh|cc}.
--
-- Stephe
___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Stephen Leake [EMAIL PROTECTED] writes: Stephen Leake [EMAIL PROTECTED] writes: I've checked in a first attempt. I've fixed a few things, and commited them, but now I've hit a wall. The commit messages are confused; I've been chasing a bug in the Emacs DVC interface to monotone; I think I finally fixed it. On Win32, first run this to setup a test environment; you'll have to kill it with ^c: cd c:/Gnu/monotone.experimental.win32_pipes/ ./run_lua_tests netsync_over_pipes Then this: cd c:/Gnu/monotone.experimental.win32_pipes/tester_dir/tests/netsync_over_pipes /Gnu/monotone.experimental.win32_pipes/mtn.exe --quiet --db test2.db serve --stdio --no-transport-auth gives this error: mtn: fatal: std::runtime_error: network error: select(2): An operation was attempted on something that is not a socket. (10038) The same occurs when 'mtn --stdio' is spawned with a socket as stdio, as you can see in tester.log. So I think we are back to the fundamental problem; how do we do non-blocking IO on Win32 stdio? Or maybe I've just done something wrong. I'll try this on RedHat, just to get another data point. -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Stephen Leake [EMAIL PROTECTED] writes: I'll try this on RedHat, just to get another data point. I fixed a couple compilation errors. unit test pipe:spawned passes, but netsync_over_pipes hangs. -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Matthew Gregan [EMAIL PROTECTED] writes: At 2007-12-04T19:33:35-0500, Stephen Leake wrote: Currently, all code outside netxx_pipe.cc uses netxx_pipe.cc. So I don't see what we are gaining by replacing Unix pipes by TCP sockets. Simplicity. See Zack's summary below. I'm not clear what you mean by cross platform. To me, that means not needing '#ifdef OS'. Clearly, netxx_pipe.cc needs '#ifdef OS'. netxx_pipe.cc is mostly either WIN32 or Unix; almost none of it is shared. The current netxx_pipe.cc is a bit of an exception to the rest of the code in monotone (except maybe tester.cc, but that's special). In general, when we need to use some platform-specific API, we wrap it up and put appropriate implementations in unix/ and win32/. The user of the wrapped API #includes platform.hh and can write general cross-platform code and expect that the wrapped call will behave the same across platforms. Right. Also, we're not fixing it, we're reworking the code with the intention of making it better. Ah, ok. I was just trying to fix the Win32 implementation, changing as little as possible. If it turns out that we're really better off using pipes on Unix and socket pairs on Win32, fine, but I'd rather that we tried the solution that resulted in the least, simplest code first. At 2007-12-04T16:44:25-0800, Zack Weinberg wrote: Unless I am very confused, the idea is to be able to use the current Unix-specific code (replacing two calls to pipe() with one call to socketpair()) on both Unix and Windows; and the benefit of this is that the low-level Windows API for asynchronous I/O has fundamentally different semantics from the low-level Unix API, and the rest of netxx expects the Unix semantics, which is why we are getting hangs. Right, that's the plan. So we are replacing _all_ pipes in Netxx::PipeStream with sockets, and changing _all_ code that deals with Netxx::PipeStream to expect sockets. I would not describe that as using the current Unix-specific code, but I think I understand. I'll see if I can make that work. There are places in netsync that expect a Netxx::PipeStream to have either two Unix pipes or one Windows named pipe; those will change to just expect one socket. That will make the code cleaner. I gather that the point of PipeStream is to let the netsync code communicate via one object, that internally is either a normal socket (when started as 'mtn serve'), or stdin/stdout (when started as 'mtn serve --stdio'). I've created a branch n.v.m.experimental.win32_pipes; it has no changes in it yet. -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Matthew Gregan [EMAIL PROTECTED] writes: At 2007-12-03T19:04:28-0500, Stephen Leake wrote: Can you point me to a URL for socketpair.c? I could look at implementing it as well. http://cantrip.org/socketpair.c This is not under the GPL license, but it may be compatible. We should probably ask Nathan Myers if we can release it under GPL -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Matthew Gregan [EMAIL PROTECTED] writes: At 2007-12-03T06:59:26-0500, Stephen Leake wrote: I forgot to mention that this seems to be a bug in the pipe implementation on Win32. Yeah, the Win32 pipe code is pretty buggy. Rather than trying to debug it, I think the best solution is to toss it out, import Nathan Myers's Win32 socketpair.c into our tree, and replace the existing pipe code with a cross platform implementation that uses socketpair(). I've looked at this some. I think we need to change the parts of netxx_pipe.cc that deal with WIN32 pipes to deal with sockets instead. Note that dumb_socketpair is _not_ actually cross-platform; it has the same #ifdef WIN32 style that netxx_pipe.cc currently has, and the WIN32 version has an extra parameter that is important. Apparently 'socketpair' is implemented on some/most unix platforms, but not on Win32? I don't think there's any reason to change the non-WIN32 parts of netxx_pipe.cc. Although we should split netxx_pipe.cc into win32/netxx_pipe.cc, unix/netxx_pipe.cc. So it would be cleaner to copy the dumb_socketpair implementation idea, directly in netxx_pipe.cc, rather than using dumb_socketpair as is. That could work around the license issue as well. I assume this should be done on a branch; n.v.m.experimental.win32_pipes ? -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
At 2007-12-04T06:52:46-0500, Stephen Leake wrote: I've looked at this some. I think we need to change the parts of netxx_pipe.cc that deal with WIN32 pipes to deal with sockets instead. Yep. It'll end up looking very much like the existing code for non-Windows platforms. Note that dumb_socketpair is _not_ actually cross-platform; it has the same #ifdef WIN32 style that netxx_pipe.cc currently has, and the WIN32 version has an extra parameter that is important. Apparently 'socketpair' is implemented on some/most unix platforms, but not on Win32? Right, that code is an implementation of socketpair() for Windows because it is not provided by the Windows socket API. Cross platform refers to the rest of the code--all platforms can use socketpair() and almost all of the rest of the code can be shared between platforms (except for process creation, but we probably already have sufficient wrappers for these in the $platform/ directories already). I don't think there's any reason to change the non-WIN32 parts of netxx_pipe.cc. Although we should split netxx_pipe.cc into win32/netxx_pipe.cc, unix/netxx_pipe.cc. We try to keep as much code as possible cross platform. I'd prefer to keep the code shared in the top level netxx_pipe.cc and just have any OS-specific wrappers pushed down into the $platform/ directories. In this case, that would be the dumb_socketpair code and possibly some extension to our existing process creation code. So it would be cleaner to copy the dumb_socketpair implementation idea, directly in netxx_pipe.cc, rather than using dumb_socketpair as is. That could work around the license issue as well. I'd prefer to ask Nathan about the license and then include the file directly. The Win32 specific code can go in win32/socketpair.c. The unix/ implementation can just call the real socketpair() directly like the !WIN32 case in socketpair.c does. I assume this should be done on a branch; n.v.m.experimental.win32_pipes ? It can be--branches are free, after all. Cheers, -mjg -- Matthew Gregan |/ /|[EMAIL PROTECTED] ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
On Dec 4, 2007 12:11 PM, Matthew Gregan [EMAIL PROTECTED] wrote: Note that dumb_socketpair is _not_ actually cross-platform; it has the same #ifdef WIN32 style that netxx_pipe.cc currently has, and the WIN32 version has an extra parameter that is important. Apparently 'socketpair' is implemented on some/most unix platforms, but not on Win32? Right, that code is an implementation of socketpair() for Windows because it is not provided by the Windows socket API. Cross platform refers to the rest of the code--all platforms can use socketpair() and almost all of the rest of the code can be shared between platforms (except for process creation, but we probably already have sufficient wrappers for these in the $platform/ directories already). There is one wrinkle - if I understand that code correctly, it's necessary to call closesocket() instead of close() on the fds returned from that socketpair implementation. I guess we could just #define closesocket(n) close(n) on Unix... zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
At 2007-12-04T15:18:18-0800, Zack Weinberg wrote: On Dec 4, 2007 12:11 PM, Matthew Gregan [EMAIL PROTECTED] wrote: Right, that code is an implementation of socketpair() for Windows because it is not provided by the Windows socket API. Cross platform refers to the rest of the code--all platforms can use socketpair() and almost all of the rest of the code can be shared between platforms (except for process creation, but we probably already have sufficient wrappers for these in the $platform/ directories already). There is one wrinkle - if I understand that code correctly, it's necessary to call closesocket() instead of close() on the fds returned from that socketpair implementation. I guess we could just #define closesocket(n) close(n) on Unix... Yeah, true. The problem is that there is no close() on Windows (there's _close() for CRT file descriptors, closesocket() for WinSock sockets, and CloseHandle() for most other things). It shouldn't be a big deal to abstract away--either with a #define or by adding a closesocket() into our platform code. Cheers, -mjg -- Matthew Gregan |/ /|[EMAIL PROTECTED] ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
On Dec 4, 2007 3:29 PM, Matthew Gregan [EMAIL PROTECTED] wrote: Yeah, true. The problem is that there is no close() on Windows (there's _close() for CRT file descriptors, closesocket() for WinSock sockets, and CloseHandle() for most other things). It shouldn't be a big deal to abstract away--either with a #define or by adding a closesocket() into our platform code. As long as you've got a plan, I'm good. (Well, I have this irrational dislike of using anything that's not a bare system call for this kind of code -- to the extent you *can* get at bare system calls on Windows, I guess I mean a depressingly ill-defined subset of the kernel32.dll interfaces -- but it sounds like going up to Winsock will actually work better in this case, so I should just deal.) zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Matthew Gregan [EMAIL PROTECTED] writes: Note that dumb_socketpair is _not_ actually cross-platform; it has the same #ifdef WIN32 style that netxx_pipe.cc currently has, and the WIN32 version has an extra parameter that is important. Apparently 'socketpair' is implemented on some/most unix platforms, but not on Win32? Right, that code is an implementation of socketpair() for Windows because it is not provided by the Windows socket API. Cross platform refers to the rest of the code--all platforms can use socketpair() and almost all of the rest of the code can be shared between platforms (except for process creation, but we probably already have sufficient wrappers for these in the $platform/ directories already). Currently, all code outside netxx_pipe.cc uses netxx_pipe.cc. So I don't see what we are gaining by replacing Unix pipes by TCP sockets. I don't think there's any reason to change the non-WIN32 parts of netxx_pipe.cc. Although we should split netxx_pipe.cc into win32/netxx_pipe.cc, unix/netxx_pipe.cc. We try to keep as much code as possible cross platform. I'm not clear what you mean by cross platform. To me, that means not needing '#ifdef OS'. Clearly, netxx_pipe.cc needs '#ifdef OS'. I'd prefer to keep the code shared in the top level netxx_pipe.cc netxx_pipe.cc is mostly either WIN32 or Unix; almost none of it is shared. and just have any OS-specific wrappers pushed down into the $platform/ directories. In this case, that would be the dumb_socketpair code and possibly some extension to our existing process creation code. So it would be cleaner to copy the dumb_socketpair implementation idea, directly in netxx_pipe.cc, rather than using dumb_socketpair as is. That could work around the license issue as well. I'd prefer to ask Nathan about the license and then include the file directly. The Win32 specific code can go in win32/socketpair.c. The Win32 specific code implementing sockets is less complex than the current WIN32 specific code in netxx_pipe.cc. So the minimal change to the current system is to just replace that. The unix/ implementation can just call the real socketpair() directly like the !WIN32 case in socketpair.c does. Why should we change the unix part of netxx_pipe.cc? It's not broken; let's not fix it. -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
On Dec 4, 2007 4:33 PM, Stephen Leake [EMAIL PROTECTED] wrote: Note that dumb_socketpair is _not_ actually cross-platform; it has the same #ifdef WIN32 style that netxx_pipe.cc currently has, and the WIN32 version has an extra parameter that is important. Apparently 'socketpair' is implemented on some/most unix platforms, but not on Win32? Right, that code is an implementation of socketpair() for Windows because it is not provided by the Windows socket API. Cross platform refers to the rest of the code--all platforms can use socketpair() and almost all of the rest of the code can be shared between platforms (except for process creation, but we probably already have sufficient wrappers for these in the $platform/ directories already). Currently, all code outside netxx_pipe.cc uses netxx_pipe.cc. So I don't see what we are gaining by replacing Unix pipes by TCP sockets. Unless I am very confused, the idea is to be able to use the current Unix-specific code (replacing two calls to pipe() with one call to socketpair()) on both Unix and Windows; and the benefit of this is that the low-level Windows API for asynchronous I/O has fundamentally different semantics from the low-level Unix API, and the rest of netxx expects the Unix semantics, which is why we are getting hangs. ... wait, are we actually going to be able to do async I/O with ncm's fake socketpair()? I do not fully understand the significance of the make_overlapped flag, but I see comments in netxx_pipe.cc averring that async I/O only works on Windows in overlapped mode, and comments in socketpair.c saying the socket is not suitable for use as a child process stdio handle if overlapped mode is on. zw ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
At 2007-12-04T19:33:35-0500, Stephen Leake wrote:
Currently, all code outside netxx_pipe.cc uses netxx_pipe.cc. So I don't
see what we are gaining by replacing Unix pipes by TCP sockets.
Simplicity. See Zack's summary below.
I'm not clear what you mean by cross platform.
To me, that means not needing '#ifdef OS'. Clearly, netxx_pipe.cc
needs '#ifdef OS'.
netxx_pipe.cc is mostly either WIN32 or Unix; almost none of it is shared.
The current netxx_pipe.cc is a bit of an exception to the rest of the code
in monotone (except maybe tester.cc, but that's special). In general, when
we need to use some platform-specific API, we wrap it up and put appropriate
implementations in unix/ and win32/. The user of the wrapped API #includes
platform.hh and can write general cross-platform code and expect that the
wrapped call will behave the same across platforms.
So the idea in this case is to put a socketpair() implementation into unix/
and win32 and netxx_pipe.cc becomes OS agnostic.
Why should we change the unix part of netxx_pipe.cc? It's not broken;
let's not fix it.
Simplicity. See Zack's summary below.
Also, we're not fixing it, we're reworking the code with the intention of
making it better. If it turns out that we're really better off using pipes
on Unix and socket pairs on Win32, fine, but I'd rather that we tried the
solution that resulted in the least, simplest code first.
At 2007-12-04T16:44:25-0800, Zack Weinberg wrote:
Unless I am very confused, the idea is to be able to use the current
Unix-specific code (replacing two calls to pipe() with one call to
socketpair()) on both Unix and Windows; and the benefit of this is that
the low-level Windows API for asynchronous I/O has fundamentally different
semantics from the low-level Unix API, and the rest of netxx expects the
Unix semantics, which is why we are getting hangs.
Right, that's the plan.
... wait, are we actually going to be able to do async I/O with ncm's fake
socketpair()? I do not fully understand the significance of the
make_overlapped flag, but I see comments in netxx_pipe.cc averring that
async I/O only works on Windows in overlapped mode, and comments in
socketpair.c saying the socket is not suitable for use as a child process
stdio handle if overlapped mode is on.
Yes, calling select() on these non-overlapped sockets still works fine. The
flag for dumb_socketpair() is to do with Win32-specific overlapped I/O
model. Passing 1 would get you the standard behaviour from WinSock. In our
case, we need to pass 0 to ensure one of the sockets is not created in
overlapped mode. This is the socket we pass to the child process as a stdio
handle.
Overlapped I/O is also asynchronous, but you need to use a different
programming model and APIs to take advantage of it (e.g. you need to use
WSASend() with a WSAOVERLAPPED structure, plain send() is non-overlapped no
matter what mode the socket was created in).
The problem I ran into when first working on this, after much hair-tearing,
is that a process expects its stdin/stdout/stderr handles to be
non-overlapped. The process calls functions like {Read,Write}File() on the
stdio handles as if they are non-overlapped. These functions notice the
handle has overlapped mode set and try to work in overlapped mode even
though the caller isn't expecting them to. Chaos ensues.
Cheers,
-mjg
--
Matthew Gregan |/
/|[EMAIL PROTECTED]
___
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Stephen Leake [EMAIL PROTECTED] writes: I have a reproducible hang on Win32/MinGW during a sync file:. I've saved copies of the two databases. The command is: mtn --debug --db /Gnu/monotone/tests/temp_2/gds-local-1.db sync \ file:/Gnu/monotone/tests/temp_2/gds-remote-1.db I forgot to mention that this seems to be a bug in the pipe implementation on Win32. I can work around the problem by using TCP sockets: mtn --db /Gnu/monotone/tests/temp_2/gds-remote-1.db serve mtn sync local ip address -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
At 2007-12-03T06:59:26-0500, Stephen Leake wrote: I forgot to mention that this seems to be a bug in the pipe implementation on Win32. Yeah, the Win32 pipe code is pretty buggy. Rather than trying to debug it, I think the best solution is to toss it out, import Nathan Myers's Win32 socketpair.c into our tree, and replace the existing pipe code with a cross platform implementation that uses socketpair(). I started working on doing exactly this a while back but got busy with other stuff. I'll try to find some time to work on it again soon. Cheers, -mjg -- Matthew Gregan |/ /|[EMAIL PROTECTED] ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
Matthew Gregan [EMAIL PROTECTED] writes: At 2007-12-03T06:59:26-0500, Stephen Leake wrote: I forgot to mention that this seems to be a bug in the pipe implementation on Win32. Yeah, the Win32 pipe code is pretty buggy. Rather than trying to debug it, I think the best solution is to toss it out, import Nathan Myers's Win32 socketpair.c into our tree, and replace the existing pipe code with a cross platform implementation that uses socketpair(). I started working on doing exactly this a while back but got busy with other stuff. I'll try to find some time to work on it again soon. I had a similar thought; use TCP sockets instead of Win32 pipes. Can you point me to a URL for socketpair.c? I could look at implementing it as well. Although I think using sockets would open up a security hole; file: runs the server with --no-transport-auth. So for a brief time an external machine could attach to the server. -- -- Stephe ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
On Mon, Dec 03, 2007 at 07:04:28PM -0500, Stephen Leake wrote: Although I think using sockets would open up a security hole; file: runs the server with --no-transport-auth. So for a brief time an external machine could attach to the server. Surely win32 sockets can be bound to loopback, so only local processes can connect? That doesn't *fully* close the hole, but it helps. What would fully close the hole would be to generate a random nonce, and send it over the socket as the first thing we do -- since we hold both ends of the socket, we know what random nonce to expect, but no-one else would. This could be fully hidden within the socketpair code (i.e., it would transmit and receive the nonce to check the connection before returning the sockets). -- Nathaniel -- Eternity is very long, especially towards the end. -- Woody Allen ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
At 2007-12-03T19:04:28-0500, Stephen Leake wrote: Can you point me to a URL for socketpair.c? I could look at implementing it as well. http://cantrip.org/socketpair.c Although I think using sockets would open up a security hole; file: runs the server with --no-transport-auth. So for a brief time an external machine could attach to the server. I don't think so. The listener is bound to localhost and expects exactly one connection. The port number is ephemeral. The other end of the socket is set up immediately. Worst case, an attacker can guess the ephemeral port number and connect to it, but it will just cause socketpair() to return an error because its own attempt to connect to the listening socket will fail. Cheers, -mjg -- Matthew Gregan |/ /|[EMAIL PROTECTED] ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
Re: [Monotone-devel] hang on Win32/MinGW with sync file:
On Tue, Dec 04, 2007 at 01:48:57PM +1300, Matthew Gregan wrote: I don't think so. The listener is bound to localhost and expects exactly one connection. The port number is ephemeral. The other end of the socket is set up immediately. Worst case, an attacker can guess the ephemeral port number and connect to it, but it will just cause socketpair() to return an error because its own attempt to connect to the listening socket will fail. Oh yeah, that works too... -- Nathaniel -- If you can explain how you do something, then you're very very bad at it. -- John Hopfield ___ Monotone-devel mailing list Monotone-devel@nongnu.org http://lists.nongnu.org/mailman/listinfo/monotone-devel
