mesos git commit: Improved Windows isolators with `struct Info` abstraction.

2017-12-04 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 6f3c74a4a -> 6a98530a5 Improved Windows isolators with `struct Info` abstraction. Replaced the two maps of `ContainerId -> pid` and `ContainerId -> limit` with a mapping of `ContainerId -> struct Info { pid, limit }`. This abstraction corre

[2/4] mesos git commit: Set `BUILD_FLAGS` flag in CMake.

2017-12-05 Thread andschwa
target_compile_definitions( - mesos PUBLIC - USE_CMAKE_BUILD_CONFIG - BUILD_JAVA_JVM_LIBRARY="${JAVA_JVM_LIBRARY}" - - # TODO(andschwa): (MESOS-5455) `BUILD_FLAGS` is currently a placeholder value. - BUILD_FLAGS="") +target_compile_definitions(mesos PUBLIC USE_CMAKE_BUILD_

[4/4] mesos git commit: Moved Java build code to `java/CMakeLists.txt`.

2017-12-05 Thread andschwa
ckage -DEPENDS ${MESOS_JAR_SRC} ${JAVA_PROTOBUF_SRC} -WORKING_DIRECTORY java) - - add_custom_target(mesos-jar DEPENDS ${MESOS_JAR}) - - # TODO(andschwa): Build the Example JAR. - set(EXAMPLE_JAR_SRC -examples/java/TestExceptionFramework.java -examples/java/TestExecutor.java

[1/4] mesos git commit: Windows: Documented additional step for running JVM code.

2017-12-05 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 75ebbbe9f -> d72d60924 Windows: Documented additional step for running JVM code. The error message returned when the JVM dependencies cannot be found is difficult to understand, especially for those unfamiliar with runtime library loading o

[3/4] mesos git commit: Fixed CMake binary dependencies.

2017-12-05 Thread andschwa
Fixed CMake binary dependencies. This resolves MESOS-8035 so that building just the `mesos-agent`, etc. target should correctly build its runtime dependencies (such as the containerizer, executor, etc.). Review: https://reviews.apache.org/r/63342 Project: http://git-wip-us.apache.org/repos/asf/

[2/3] mesos git commit: Renamed CMake `curl` target to `libcurl`.

2017-12-06 Thread andschwa
Renamed CMake `curl` target to `libcurl`. This target was renamed where it is imported, and so this change reflects that. Review: https://reviews.apache.org/r/64103/ Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/9e9de115 T

[3/3] mesos git commit: Added `curl` executable to Windows build.

2017-12-06 Thread andschwa
Added `curl` executable to Windows build. Some Mesos health checks use the `curl` executable, which is a required component on Linux, but also shipped on the system. On Windows, `curl.exe` is not normally available. However, we already build `curl` for other components which link to `libcurl`. Par

[1/3] mesos git commit: Windows: Enabled HTTP health checks.

2017-12-06 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master f031e031f -> 20a877747 Windows: Enabled HTTP health checks. This required an additional dependency on `curl` for Windows, which is an imported target that is deployed next to the other executables by CMake. With it in place, the HTTP health

[02/11] mesos git commit: Windows: Modifed `os::write` to write binary files.

2017-12-08 Thread andschwa
Windows: Modifed `os::write` to write binary files. By default, `write` on Windows will use Windows-style CRLF `\r\n` line endings, regardless of the input data, which results in unexpected behavior. Passing the `O_BINARY` flag stops Windows from changing line endings from the original data as it

[11/11] mesos git commit: Included `stout/os/ftruncate.hpp` to fix build failure.

2017-12-08 Thread andschwa
Included `stout/os/ftruncate.hpp` to fix build failure. The `stout/os.hpp` file for Windows does not include `ftruncate.hpp`, but it was used so it needs to be included. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/2fed8cc

[08/11] mesos git commit: Ported `mesos-fetcher` to Windows.

2017-12-08 Thread andschwa
Ported `mesos-fetcher` to Windows. Tests for tar, gzip, and such won't be working on Windows for the time being. Thoughts are to provide this capability to the Fetcher in a cross-platform manner via a programmatic code library rather than Linux-specific command line tools (tar, gzip, etc). See MES

[05/11] mesos git commit: Enabled HDFS compilation and associated tests.

2017-12-08 Thread andschwa
Enabled HDFS compilation and associated tests. Note that HDFS tests are disabled for Windows due to dependence on 'sh' shell. Be aware: HDFS hasn't been tested, although it should work. We will formally add support for HDFS at a later date, see MESOS-5460. Adding it now makes porting other code e

[09/11] mesos git commit: Fixed bug where we didn't check non-localhost URIs were absolute.

2017-12-08 Thread andschwa
artsWith(path, FILE_URI_LOCALHOST)) { -path = path.substr(FILE_URI_LOCALHOST.size()); -fileUri = true; + if (!fileUri && strings::contains(uri, "://")) { +return None(); } - path = path::from_uri(path); + // TODO(andschwa): Fix `path::from_uri` to remove hostname component

[07/11] mesos git commit: Added `uri::FILE_PREFIX` constant for "file://".

2017-12-08 Thread andschwa
Added `uri::FILE_PREFIX` constant for "file://". Instead of hard-coding the string multiple times. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/245af401 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/245af401 Diff

[01/11] mesos git commit: Added `os::copyfile(src, dst)` to stout.

2017-12-08 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 2ab1a3194 -> 2fed8cc85 Added `os::copyfile(src, dst)` to stout. The `os::copyfile()` method provides a cross-platform API to copy a file from one location to another. The POSIX implementation shells out to `cp`, and the Windows implementati

[10/11] mesos git commit: Enabled `DefaultExecutorTest.TaskWithFileURI` on Windows.

2017-12-08 Thread andschwa
Enabled `DefaultExecutorTest.TaskWithFileURI` on Windows. This fixes the use of `file:// + path` with `uri::from_path()`, and replaces a Bash test with the PowerShell equivalent on Windows. Review: https://reviews.apache.org/r/63253 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Com

[04/11] mesos git commit: Eliminated `os::shell()` calls from HDFS for Windows.

2017-12-08 Thread andschwa
Eliminated `os::shell()` calls from HDFS for Windows. On Windows, `os::shell()` is explicitly deleted, so reworked the code to use `process::subprocess()` instead. Review: https://reviews.apache.org/r/60626 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apa

[06/11] mesos git commit: Replaced manual URI creation with `uri::from_path()`.

2017-12-08 Thread andschwa
Replaced manual URI creation with `uri::from_path()`. Especially on Windows, this ensures that the URI is well-formed (uses forward slash as a path separator, and idempotently prefixes with `file://`). On Windows, the path separator is a backslash, and so manually creating a URI by doing `"file://

[03/11] mesos git commit: Added `path::from_uri` and `uri::from_path` to stout.

2017-12-08 Thread andschwa
Added `path::from_uri` and `uri::from_path` to stout. The former converts a well-formed URI to a path, fixing the path separator for Windows. The latter reverses the process, and produces a well-formed URI from a path, keeping in mind that the URI path separator is always a forward slash, and so

mesos git commit: Windows: Enabled remaining `FlagsTests`.

2017-12-11 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master e7d1db246 -> b780d873b Windows: Enabled remaining `FlagsTests`. This resolve MESOS-3437 trivially because MESOS-5880 and MESOS-5937 have since been resolved. This replaces review 63239 because the final iteration was not right. Review: ht

mesos git commit: Windows: Fixed `os::stat::size()`.

2017-12-11 Thread andschwa
n Bytes of a given file system entry. When -// applied to a symbolic link with `follow` set to -// `DO_NOT_FOLLOW_SYMLINK`, this will return the length of the entry -// name (strlen). -// -// TODO(andschwa): Replace `::_stat`. See MESOS-8275. +// Returns the size in Bytes of a given file system entr

mesos git commit: Updated Windows documentation.

2017-12-11 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master adf4fa3f2 -> 3dc93d6fd Updated Windows documentation. Removed outdated notes about administrative privileges being necessary due to symlinks, which was resovled in MESOS-7370. Added note about minimum supported Windows versions, including c

mesos git commit: Windows: Added legacy support for admin-only symlinks.

2017-12-14 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master ca6729315 -> 4bf96153a Windows: Added legacy support for admin-only symlinks. Some users are regrettably stuck on versions of Windows before this feature was added. If the `ALLOW_UNPRIVILEGED` flag is unsupported, the creation of the symlin

[1/6] mesos git commit: Windows: Patched protobuf to fix warnings.

2017-12-15 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 071e8cfbe -> 3eaab8a2e Windows: Patched protobuf to fix warnings. The patch was generated by diffing the following commit against tag v3.5.0, this was merged upstream in https://github.com/google/protobuf/pull/4000: commit 80809271a Author

[3/6] mesos git commit: Windows: Silenced known deprecation warning from Google Test.

2017-12-15 Thread andschwa
Windows: Silenced known deprecation warning from Google Test. We previously fixed the Google Test build by defining `_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING` when building Google Test, but by not also adding it to the interface, it generates a warning for every test file including it. This is v

[4/6] mesos git commit: Windows: Fixed type in job object `pid_list` struct.

2017-12-15 Thread andschwa
Windows: Fixed type in job object `pid_list` struct. This type was erroneously declared as `ULONG_PTR` because the documentation states it as such. This is true if the array is dynamically allocated, then the struct ends with a pointer to the first element of the array. However, we avoid this allo

[5/6] mesos git commit: Included `` where used.

2017-12-15 Thread andschwa
Included `` where used. This was not caught previously because the pre-compiled header included implicitly. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/bd7b1f32 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/bd7b

[2/6] mesos git commit: Windows: Stopped compiling files with no source code.

2017-12-15 Thread andschwa
Windows: Stopped compiling files with no source code. These two `utils.cpp` files have no code on Windows due to the source being wrapped in `#ifdef __linux__`, and so they generate warnings as the compiled objects defined no new symbols. Since they are unused, removing them from the Windows build

[6/6] mesos git commit: Windows: Removed `/debug:fastlink` linker flag.

2017-12-15 Thread andschwa
Windows: Removed `/debug:fastlink` linker flag. With Visual Studio 2017, `fastlink` is now the default, so this is superfluous. Morever, this particular CMake variable also passes the flag to `lib.exe` when generating a static library, and this emits a warning because `lib.exe` does not accept thi

mesos git commit: Made OfferOperationStatusUpdateManager tests pass on Windows.

2017-12-15 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 3eaab8a2e -> a4b1134e4 Made OfferOperationStatusUpdateManager tests pass on Windows. Review: https://reviews.apache.org/r/64522/ Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mes

[2/2] mesos git commit: Windows: Fixed `os::open()` to always use `O_BINARY`.

2017-12-19 Thread andschwa
Windows: Fixed `os::open()` to always use `O_BINARY`. Previously, we had been manually adding the `O_BINARY` flag as we encountered bugs due to the Windows default behavior of performing line-ending translation. This was error prone. Given this precedent, it seems safe to assume that all our exis

[1/2] mesos git commit: Windows: Removed manual use of `O_BINARY`.

2017-12-19 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 0cc636b2d -> 45669854d Windows: Removed manual use of `O_BINARY`. This is now superfluous as it is added in stout. Review: https://reviews.apache.org/r/64690 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-w

mesos git commit: Windows: Deleted unused and unnecessary OS version functions.

2017-12-19 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 45669854d -> b7c218fda Windows: Deleted unused and unnecessary OS version functions. The Windows API `GetVersionEx` was deprecated in Windows 8. Using it in the `os.hpp` header caused hundreds of warnings that it was deprecated to be emitte

[1/3] mesos git commit: Removed executable bit from website files.

2017-12-19 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master b7c218fda -> c222453e2 Removed executable bit from website files. These CSS, PNG, JA, MD, and ERB files are not executables, and so should not have the executable bit set. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit:

[2/3] mesos git commit: Removed executable bit from text file.

2017-12-19 Thread andschwa
Removed executable bit from text file. `CMakeLists.txt` should not be marked executable. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/24f67027 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/24f67027 Diff: http://g

[3/3] mesos git commit: Removed executable bit from text files.

2017-12-19 Thread andschwa
Removed executable bit from text files. `CMakeLists.txt` and `docker.cpp` should not be marked executable. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/5601070a Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/56010

mesos git commit: Windows: Updated cURL to version 7.57.0.

2017-12-19 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master c222453e2 -> 884226e3b Windows: Updated cURL to version 7.57.0. The cURL source and build have improved significantly since we pulled in the previous versoin (7.43.0). Most notably, the new version now supports SChannel (native Windows cryp

mesos git commit: Windows: Added `/EHsc` to `CMAKE_CXX_FLAGS`.

2017-12-19 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 4feb36706 -> a3febec7e Windows: Added `/EHsc` to `CMAKE_CXX_FLAGS`. After upgrading Visual Studio to 15.5.2, warning C4530 started being emitted for practically every file. Since we use the STL (which uses exceptions), apparently we need to

[03/10] mesos git commit: Fixed conversion warnings.

2018-01-12 Thread andschwa
Fixed conversion warnings. Fixed conversion warnings in `cram_md5` authentication. The potential for overflow here is exceedingly unlikely, and we have been ignoring these warnings to date without problem. Fixed conversion warning in `files.cpp`. The signature accepts an `off_t` type, but we calc

[06/10] mesos git commit: Fixed conversion warnings in tests.

2018-01-12 Thread andschwa
Fixed conversion warnings in tests. Stringified a `size_t` for `http::Request.headers` value. The types in `metrics_tests.cpp` were defined as `double`, and so should be compared with `EXPECT_DOUBLE_EQ`, not `EXPECT_FLOAT_EQ`. Review: https://reviews.apache.org/r/64733 Project: http://git-wip-

[01/10] mesos git commit: Changed `Sorter::count()` to return `size_t` instead of `int`.

2018-01-12 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master ccc5051a1 -> 93ab49409 Changed `Sorter::count()` to return `size_t` instead of `int`. Also applies to `DRFSorter::count()`. The implementations return `hashmap::count()` which is a `size_t`. Converting `size_t` to `int` implicitly generates

[10/10] mesos git commit: Windows: Fixed memory leak.

2018-01-12 Thread andschwa
Windows: Fixed memory leak. The environment strings variable block is allocated by the system, and must be manually freed (according to the docs). Review: https://reviews.apache.org/r/64961 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf

[09/10] mesos git commit: Windows: Explicitly state source and destination path for extract.

2018-01-12 Thread andschwa
Windows: Explicitly state source and destination path for extract. Makes it more clear that it is in the correct order. Review: https://reviews.apache.org/r/64962 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/93ab4940 Tree

[04/10] mesos git commit: Fixed conversion warnings in tests.

2018-01-12 Thread andschwa
Fixed conversion warnings in tests. The signature of `flags.load` takes an `int`, but `arraySize` (correctly) returns a `size_t`. We don't want to change `flags.load` to take a `size_t` because its signature is `int argc, char** argv`, and so we may be sending it `int` in other places. As all the

[07/10] mesos git commit: Ended `IOSwitchboard::_prepare` with `UNREACHABLE`.

2018-01-12 Thread andschwa
Ended `IOSwitchboard::_prepare` with `UNREACHABLE`. All paths should `return`, but the compiler wasn't able to determine this. Review: https://reviews.apache.org/r/64731 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/9db041

[08/10] mesos git commit: Fixed conversion warnings.

2018-01-12 Thread andschwa
Fixed conversion warnings. Just casting to the used type. Review: https://reviews.apache.org/r/64734 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/c79412cd Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/c79412cd D

[02/10] mesos git commit: Templated operator overloads of `Duration`.

2018-01-12 Thread andschwa
Templated operator overloads of `Duration`. Without this change, every use of `Duration * 2` throws a conversion warning because we multiply `Duration` by `int` and `size_t` without casting to `double`, and the operator overloads were all accepting only doubles. By templating them on `typename T`

[05/10] mesos git commit: Fixed conversion warnings in tests.

2018-01-12 Thread andschwa
Fixed conversion warnings in tests. Fixed `double -> float` conversions. While the Protobuf type and literal type are both `double`, the `FLOAT_EQ` tests were being used instead of `DOUBLE_EQ`, causing a conversion warning. Fixed `size_t` to `int` conversion warnings by casting. Fixed warning fo

[3/3] mesos git commit: Windows: Fixed docker executor `PATH` variable.

2018-01-16 Thread andschwa
/docker.cpp @@ -43,6 +43,7 @@ #include #include +#include #include "common/status_utils.hpp" @@ -1465,6 +1466,24 @@ Future DockerContainerizerProcess::launchExecutorProcess( if (environment.count("PATH") == 0) { environment["PATH"] = os::host_default_p

[1/3] mesos git commit: Fixed use of `os::which`.

2018-01-16 Thread andschwa
will run\n" @@ -252,11 +253,11 @@ public: bool disable(const ::testing::TestInfo* test) const { -return matches(test, "NVIDIA_GPU_") && !exists; +return matches(test, "NVIDIA_GPU_") && nvidiaGpuError; } private: - bool exists; + bool nvid

[2/3] mesos git commit: Ported `os::which` to Windows.

2018-01-16 Thread andschwa
Ported `os::which` to Windows. Because `os::which` still lived in `posix/os.hpp`, it was refactored into its own `os/which.hpp` (which the respective `os/posix/which.hpp` and `os/windows/which.hpp`. Consumers of this will need additionally include the new header, instead of just `os.hpp`. The dif

[1/3] mesos git commit: Windows: Fixed docker executor `PATH` variable.

2018-01-16 Thread andschwa
("PATH") == 0) { environment["PATH"] = os::host_default_path(); + +// TODO(andschwa): We will consider removing the `#ifdef` in future, as +// other platforms may benefit from being pointed to the same `docker` in +// both Agent and Executor (there is a cha

[2/3] mesos git commit: Fixed use of `os::which`.

2018-01-16 Thread andschwa
fo* test) const { -return matches(test, "NVIDIA_GPU_") && !exists; +return matches(test, "NVIDIA_GPU_") && nvidiaGpuError; } private: - bool exists; + bool nvidiaGpuError; }; @@ -396,6

[3/3] mesos git commit: Ported `os::which` to Windows.

2018-01-16 Thread andschwa
Ported `os::which` to Windows. Because `os::which` still lived in `posix/os.hpp`, it was refactored into its own `os/which.hpp` (which the respective `os/posix/which.hpp` and `os/windows/which.hpp`. Consumers of this will need additionally include the new header, instead of just `os.hpp`. The dif

[2/2] mesos git commit: Added Developer Guide to documentation.

2018-01-16 Thread andschwa
Added Developer Guide to documentation. This guide was started with the intent to capture "tribal knowledge" about Mesos development. I've added several general practices and patterns I've had explained to me by various Mesos developers, and also added a Windows section with design patterns I had

[1/2] mesos git commit: Updated C++ Style Guide.

2018-01-16 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master e34ed9602 -> 9b1fda56a Updated C++ Style Guide. Added three missing style notes: * Use of `::` for global namespace * Put `template ` on own line * Prefer `->foo()` to `.get().foo()` Review: https://reviews.apache.org/r/65114 Project: h

[3/3] mesos git commit: Windows: Fixed mock signal values in stout.

2018-01-17 Thread andschwa
Windows: Fixed mock signal values in stout. Removed `SIGSTOP` and `SIGCONT` on Windows, since they are meaningless, and never unused. Also, fixed the WEXITSTATUS macro to cast the exit code instead of bit-masking it, since Windows exit codes are 32 bit unsigned ints. Review: https://reviews.apach

[1/3] mesos git commit: Windows: Updated networking documentation.

2018-01-17 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 7d8ae37b0 -> 8964137f6 Windows: Updated networking documentation. The networking docs now describe how the Docker network modes in the `Network` enum work on Windows, since the enum only has Linux network modes. Review: https://reviews.apa

[2/3] mesos git commit: Windows: Mapped the Docker network info types.

2018-01-17 Thread andschwa
Windows: Mapped the Docker network info types. The Network enum in DockerInfo is specific to Linux containers. `HOST` doesn't exist on Windows and `BRIDGE` is `NAT` on Windows. The current default docker network setting was always `HOST`, which broke the Windows docker executor. Now, if a specific

mesos git commit: Fixed build break on Windows.

2018-04-05 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 240a25856 -> 887dfaf28 Fixed build break on Windows. We do not use `make` on Windows, so this `FATAL_ERROR` when it's not found broke the build. Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.o

[2/4] mesos git commit: Linked execinfo in Glog build on FreeBSD.

2018-04-05 Thread andschwa
(andschwa): Build with CMake instead when glog is updated. set(GLOG_CONFIG_CMD ${GLOG_ROOT}/src/../configure --with-pic GTEST_CONFIG=no --prefix=${GLOG_ROOT}-build) + if (FREEBSD) +list(APPEND GLOG_CONFIG_CMD LDFLAGS=-lexecinfo) + endif () set(GLOG_BUILD_CMD ${MAKE_PROGRAM}) set

[3/4] mesos git commit: Linked subversion in stout build on FreeBSD.

2018-04-05 Thread andschwa
Linked subversion in stout build on FreeBSD. Review: https://reviews.apache.org/r/66387/ Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/279e487f Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/279e487f Diff: http://g

[1/4] mesos git commit: Linked libm in ZooKeeper build on FreeBSD.

2018-04-05 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 887dfaf28 -> 6c77580d4 Linked libm in ZooKeeper build on FreeBSD. Review: https://reviews.apache.org/r/66384/ Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/2a503da1

[4/4] mesos git commit: Imported sasl2 target on non-Windows platforms.

2018-04-05 Thread andschwa
Imported sasl2 target on non-Windows platforms. Review: https://reviews.apache.org/r/66392/ Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/6c77580d Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/6c77580d Diff: http:

[2/3] mesos git commit: Windows: Included used `jobobject.hpp` stout header in libprocess.

2018-04-05 Thread andschwa
Windows: Included used `jobobject.hpp` stout header in libprocess. The job object functions were refactored upstream from `windows/os.hpp` to `os/windows/jobobject.hpp`. Also removed `windows/os.hpp` because `os.hpp` includes it for us. Review: https://reviews.apache.org/r/66421/ Project: http

[1/3] mesos git commit: Windows: Extracted job object code into `os/windows/jobobject.hpp`.

2018-04-05 Thread andschwa
processes.insert(process.get()); +} + } + + return processes; +} + + +inline Try> get_job_processes(pid_t pid) +{ + // TODO(andschwa): Overload open_job to use pid. + Try job_handle = os::open_job(JOB_OBJECT_QUERY, false, pid); + if (job_handle.isError()) { +return Error(job_handle.

[3/3] mesos git commit: Windows: Included used `jobobject.hpp` stout header in Mesos.

2018-04-05 Thread andschwa
Windows: Included used `jobobject.hpp` stout header in Mesos. The job object functions were refactored upstream from `windows/os.hpp` to `os/windows/jobobject.hpp`. Review: https://reviews.apache.org/r/66422/ Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.a

mesos git commit: Windows: Changed `SLEEP_COMMAND` to use `ping`.

2018-04-10 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 88f5629e5 -> df89829de Windows: Changed `SLEEP_COMMAND` to use `ping`. We have gone back to `ping` as the `sleep` command on Windows, because PowerShell is too slow to startup, and can crash when the machine is overloaded. This time, howeve

mesos git commit: Made FreeBSD default to non-GNU ld.

2018-04-12 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master bffa9f3c5 -> 3fb36f158 Made FreeBSD default to non-GNU ld. This patch fixes linking on FreeBSD (when building with Clang) by adding the necessary linker flags to use either LLD or GOLD (whichever is found first) instead of GNU LD (BFD), whi

mesos git commit: Fixed Windows build by guarding Unix socket code.

2018-04-16 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master bd688e4cf -> b1d2c98c3 Fixed Windows build by guarding Unix socket code. We kept the `csi/paths.cpp` file in the build on Windows because `slave/paths.cpp` now depends on it, and the only "broken" part is the use of Unix sockets in `getEndp

mesos git commit: Changed call from `os::shell()` to `os::system()`.

2018-04-18 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master ca21ca820 -> 896aee76c Changed call from `os::shell()` to `os::system()`. The function `os::shell` is not implemented on Windows, making it necessary to either hide all calls behind a preprocessor macro or to avoid the function. Since we do

mesos git commit: Windows: Fixed Ninja in Release configuration.

2018-04-18 Thread andschwa
exe IMPORTED_LOCATION_RELEASE ${CURL_ROOT}-build/src/Release/curl.exe) else () +# This is for single-configuration generators such as Ninja. +if (CMAKE_BUILD_TYPE MATCHES Debug) + set(CURL_SUFFIX "-d") +endif () + set_target_properties( libcurl PROPERTIES - # TODO(a

mesos git commit: Made 3rdparty jemalloc only build when enabled.

2018-04-19 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master 1e30b3e35 -> 2246b8ab4 Made 3rdparty jemalloc only build when enabled. When declaring a library with `add_library()`, the library is added to the list of default targets, such that `make all` (and equivalents) will build it, even if nothing

mesos git commit: Windows: Fixed build due to uses of `flags.switch_user`.

2018-04-20 Thread andschwa
ent gracefully drops tasks when // a scheduler launches as a user that is not present on the agent. -TEST_F_TEMP_DISABLED_ON_WINDOWS( -SlaveTest, ROOT_RunTaskWithCommandInfoWithInvalidUser) +// +// TODO(andschwa): Enable after `flags.switch_user` is added. +TEST_F(SlaveT

mesos git commit: Windows: Documented GNU patch's error if line endings are wrong.

2018-04-25 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master a1c6a7a3c -> ac4d52d89 Windows: Documented GNU patch's error if line endings are wrong. While the step necessary to prevent the error was documented, it was easy to skip, and moreover, the error message from GNU patch is entirely unhelpful

[16/31] mesos git commit: Windows: Refactored `subprocess_windows.cpp` to use `os::open()`.

2018-05-01 Thread andschwa
Windows: Refactored `subprocess_windows.cpp` to use `os::open()`. Previously, `os::open()` used the CRT function `_wopen()`, and so this file was written to use the `CreateFile()` API directly. Now that `os::open()` uses the Windows API, all this duplicate code can be deleted in favor of using the

[20/31] mesos git commit: Windows: Made `protobuf::write()` use CRT file descriptor explicitly.

2018-05-01 Thread andschwa
Windows: Made `protobuf::write()` use CRT file descriptor explicitly. This is another edge case where a third-party library (protobuf) requires a CRT integer file descriptor. Thus we duplicate the `int_fd` and then explicitly allocate via `crt()`, which requires that we also manually close it via

[19/31] mesos git commit: Windows: Removed `FD_CRT` from `WindowsFD` abstraction.

2018-05-01 Thread andschwa
tout/os/windows/dup.hpp index 54b78b1..af98054 100644 --- a/3rdparty/stout/include/stout/os/windows/dup.hpp +++ b/3rdparty/stout/include/stout/os/windows/dup.hpp @@ -25,16 +25,7 @@ namespace os { inline Try dup(const int_fd& fd) { switch (fd.type()) { -// TODO(andschwa): Remove this

[12/31] mesos git commit: Windows: Deleted dead code from `process::internal` namespace.

2018-05-01 Thread andschwa
Windows: Deleted dead code from `process::internal` namespace. The deleted code was purely self-referential. Review: https://reviews.apache.org/r/66429 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/37e28446 Tree: http://gi

[02/31] mesos git commit: Replaced `int` and `HANDLE` types with `int_fd`.

2018-05-01 Thread andschwa
Replaced `int` and `HANDLE` types with `int_fd`. Review: https://reviews.apache.org/r/66835 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/d4895888 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/d4895888 Diff: http:

[17/31] mesos git commit: Added overloads for `int_fd` to `os::stat::isdir()` and `size()`.

2018-05-01 Thread andschwa
:string& path) @@ -84,6 +95,14 @@ inline bool isdir( } +// TODO(andschwa): Share logic with other overload. +inline bool isdir(const int_fd fd) +{ + Try s = internal::stat(fd); + return s.isSome() && S_ISDIR(s->st_mode); +} + + inline bool isfile( const std::st

[08/31] mesos git commit: Windows: More constness in stout.

2018-05-01 Thread andschwa
Windows: More constness in stout. Also small fixes such as `reserve` over an allocation, and a bad name `si` instead of `info`. Review: https://reviews.apache.org/r/66426 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/9e93d

[13/31] mesos git commit: Windows: Fixed `os::dup()` to use `DuplicateHandle()`.

2018-05-01 Thread andschwa
(fd.type()) { -case WindowsFD::FD_CRT: -case WindowsFD::FD_HANDLE: { - // TODO(andschwa): Replace this with `::DuplicateHandle` after figuring - // out how to make it compatible with handles to stdin/stdout/stderr, as - // well as defining sane inheritance semantics. +

[14/31] mesos git commit: Windows: Fixed `os::read()` to use `ReadFile()`.

2018-05-01 Thread andschwa
l::windows::longpath(path); + // NOTE: The `wchar_t` constructor of `ifstream` is an MSVC + // extension. + // + // TODO(andschwa): This might need `io_base::binary` like other + // streams on Windows. + std::ifstream file(longpath.data()); + if (!file.is_open()) { +return E

[24/31] mesos git commit: Fixed `Subprocess::ChildHook::CHDIR()` to use `os::chdir()`.

2018-05-01 Thread andschwa
Fixed `Subprocess::ChildHook::CHDIR()` to use `os::chdir()`. This needed to use the Stout API so that the correct Windows implementation is used, as `::chdir` is part of the CRT. Also included used but not included `stout/os/*` headers. Review: https://reviews.apache.org/r/66443 Project: http:

[26/31] mesos git commit: Windows: Cleaned up included CRT headers.

2018-05-01 Thread andschwa
Windows: Cleaned up included CRT headers. The set `errno` value in `os::kill()` is never checked (especially on Windows), so `_set_errno()` and thus `errno.h` were removed. The `fcntl.h` is used only to provide `O_CREAT` etc., and so belonged in `open.hpp`, not `windows.hpp`. The remaining heade

[30/31] mesos git commit: Fixed `mesos-tcp-connect` to use `net::socket`.

2018-05-01 Thread andschwa
Fixed `mesos-tcp-connect` to use `net::socket`. Use the stout wrapper instead of `::socket` so we have built-in error checking (and don't have to worry about types). Review: https://reviews.apache.org/r/66836 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.a

[11/31] mesos git commit: Windows: Fixed `os::ftruncate()` to use `FileEndOfFileInfo`.

2018-05-01 Thread andschwa
Windows: Fixed `os::ftruncate()` to use `FileEndOfFileInfo`. This previously used the CRT API `_chsize_s()`, which required a CRT integer file descriptor. Instead, we can achieve the same behavior by calling `SetFileInformationByHandle(FileEndOfFileInfo)`. This is significantly easier than using `

[01/31] mesos git commit: Added `SubprocessTest.PipeLargeOutput`.

2018-05-01 Thread andschwa
Repository: mesos Updated Branches: refs/heads/master d4a903a4a -> 8e2d6d296 Added `SubprocessTest.PipeLargeOutput`. This new test checks exercises our `Subprocess::PIPE()` logic more thoroughly by specifically piping enough data such that a single memory page is insufficient to hold it. This

[15/31] mesos git commit: Windows: Fixed `os::write()` to use `WriteFile()`.

2018-05-01 Thread andschwa
type()) { -case WindowsFD::FD_CRT: -case WindowsFD::FD_HANDLE: { +// TODO(andschwa): Remove this when `FD_CRT` is removed, MESOS-8675. +case WindowsFD::FD_CRT: { return ::_write(fd.crt(), data, static_cast(size)); } +case WindowsFD::FD_HANDLE: { + DWORD b

[28/31] mesos git commit: Windows: Ported more unit tests from `os_tests.cpp`.

2018-05-01 Thread andschwa
Windows: Ported more unit tests from `os_tests.cpp`. Fixed `os::sleep()` to return an invalid parameter error if given a negative value. Fixed tests around the `cloexec` and `nonblock` stubs. Extended the `bootid` test to use `std::chrono` to assert the boot id (which is the system boot time) is

[25/31] mesos git commit: Windows: Deleted `stout/os/windows/signals.hpp`.

2018-05-01 Thread andschwa
Windows: Deleted `stout/os/windows/signals.hpp`. This file had never been included before, so it didn't compile. Because everything in this file will remain unimplemented, we chose to delete the Windows version of the file entirely. The POSIX implementation is left intact in `stout/os/posix/signal

[10/31] mesos git commit: Windows: Fixed `os::lseek()` to use `SetFilePointerEx()`.

2018-05-01 Thread andschwa
are + // 0, 1, 2, the same as `FILE_BEGIN`, `FILE_CURRENT`, and + // `FILE_END`. Thus we don't need to map them, and they can be + // casted to a `DWORD` safely. + + LARGE_INTEGER offset_; + offset_.QuadPart = offset; + + LARGE_INTEGER new_offset; + + // TODO(andschwa): This may ne

[09/31] mesos git commit: Split `stout/os/lseek.hpp` into Windows and POSIX files.

2018-05-01 Thread andschwa
Split `stout/os/lseek.hpp` into Windows and POSIX files. Review: https://reviews.apache.org/r/66427 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/e7d4b3a7 Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/e7d4b3a7 Dif

[31/31] mesos git commit: Windows: Specialized `flags::parse`.

2018-05-01 Thread andschwa
Windows: Specialized `flags::parse`. Review: https://reviews.apache.org/r/66834 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/0b38be5f Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/0b38be5f Diff: http://git-wip-us

[06/31] mesos git commit: Split `stout/os/open.hpp` into Windows and POSIX files.

2018-05-01 Thread andschwa
Split `stout/os/open.hpp` into Windows and POSIX files. The logic remained the same, just with the Windows code removed from the POSIX code, and vice versa. Review: https://reviews.apache.org/r/66423 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org

[29/31] mesos git commit: Windows: Ported the rest of the `SubprocessTest` suite.

2018-05-01 Thread andschwa
{ TestFlags flags; @@ -684,8 +726,13 @@ TEST_F(SubprocessTest, Flags) string out = path::join(os::getcwd(), "stdout"); Try s = subprocess( +#ifdef __WINDOWS__ + os::Shell::name, + {os::Shell::arg0, os::Shell::arg1, "echo"}, +#else "/bin/echo",

[18/31] mesos git commit: Removed use of `fstat()` from `http.cpp` and `http_proxy.cpp`.

2018-05-01 Thread andschwa
Removed use of `fstat()` from `http.cpp` and `http_proxy.cpp`. The functions `os::stat::size()` and `os::stat::isdir()` are now overloaded for an `int_fd` type, using `fstat()` on POSIX, and the equivalent functions with a `HANDLE` on Windows. This allowed us to remove the use of `::fstat()`, whic

[03/31] mesos git commit: Added `FsTest.Open` to cover `os::open()`.

2018-05-01 Thread andschwa
Added `FsTest.Open` to cover `os::open()`. Review: https://reviews.apache.org/r/66641 Project: http://git-wip-us.apache.org/repos/asf/mesos/repo Commit: http://git-wip-us.apache.org/repos/asf/mesos/commit/942d495b Tree: http://git-wip-us.apache.org/repos/asf/mesos/tree/942d495b Diff: http://git-

[07/31] mesos git commit: Windows: Replaced `WindowsFD` with `int_fd` typedef.

2018-05-01 Thread andschwa
44 --- a/3rdparty/stout/include/stout/os/windows/shell.hpp +++ b/3rdparty/stout/include/stout/os/windows/shell.hpp @@ -30,7 +30,7 @@ #include #include -#include +#include #include @@ -241,7 +241,7 @@ inline Try create_process( const std::vector& argv, const Option&

  1   2   3   4   >