[cmake-developers] [PATCH v5] For consoles output on Windows use our own std::streambuf

2016-07-27 Thread Dāvis Mosāns 
Currently Microsoft's C++ libraries implementation of std::cout/cerr
can't output Unicode characters but only ASCII or ANSI if locale is set
so we implement and use our own ConsoleBuf which can output Unicode
characters to console and it doesn't matter what locale or console's
codepage is set.
---
 CMakeLists.txt   |   1 +
 Source/cmakemain.cxx |   6 +
 Source/kwsys/CMakeLists.txt  |  17 +-
 Source/kwsys/ConsoleBuf.hxx.in   | 318 ++
 Source/kwsys/testConsoleBuf.cxx  | 424 +++
 Source/kwsys/testConsoleBuf.hxx  |  25 +++
 Source/kwsys/testConsoleBufChild.cxx |  59 +
 bootstrap|   1 +
 8 files changed, 850 insertions(+), 1 deletion(-)
 create mode 100644 Source/kwsys/ConsoleBuf.hxx.in
 create mode 100644 Source/kwsys/testConsoleBuf.cxx
 create mode 100644 Source/kwsys/testConsoleBuf.hxx
 create mode 100644 Source/kwsys/testConsoleBufChild.cxx

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ae5990e..792b5a5 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -258,6 +258,7 @@ macro (CMAKE_BUILD_UTILITIES)
   set(KWSYS_USE_MD5 1)
   set(KWSYS_USE_Process 1)
   set(KWSYS_USE_CommandLineArguments 1)
+  set(KWSYS_USE_ConsoleBuf 1)
   set(KWSYS_HEADER_ROOT ${CMake_BINARY_DIR}/Source)
   set(KWSYS_INSTALL_DOC_DIR "${CMAKE_DOC_DIR}")
   add_subdirectory(Source/kwsys)
diff --git a/Source/cmakemain.cxx b/Source/cmakemain.cxx
index 521a5bf..8b10d50 100644
--- a/Source/cmakemain.cxx
+++ b/Source/cmakemain.cxx
@@ -26,6 +26,7 @@
 #include "cmake.h"
 #include "cmcmd.h"
 #include 
+#include 
 
 #ifdef CMAKE_BUILD_WITH_CMAKE
 static const char* cmDocumentationName[][2] = {
@@ -153,6 +154,11 @@ static void cmakemainProgressCallback(const char* m, float 
prog,
 
 int main(int ac, char const* const* av)
 {
+#if defined(_WIN32)
+  // Replace streambuf so we can output Unicode to console
+  cmsys::ConsoleBuf::Manager consoleOut(std::cout);
+  cmsys::ConsoleBuf::Manager consoleErr(std::cerr, true);
+#endif
   cmsys::Encoding::CommandLineArguments args =
 cmsys::Encoding::CommandLineArguments::Main(ac, av);
   ac = args.argc();
diff --git a/Source/kwsys/CMakeLists.txt b/Source/kwsys/CMakeLists.txt
index 5fce50f..a211425 100644
--- a/Source/kwsys/CMakeLists.txt
+++ b/Source/kwsys/CMakeLists.txt
@@ -123,6 +123,7 @@ IF(KWSYS_STANDALONE OR CMake_SOURCE_DIR)
   SET(KWSYS_USE_FStream 1)
   SET(KWSYS_USE_String 1)
   SET(KWSYS_USE_SystemInformation 1)
+  SET(KWSYS_USE_ConsoleBuf 1)
 ENDIF()
 
 # Enforce component dependencies.
@@ -154,6 +155,9 @@ ENDIF()
 IF(KWSYS_USE_FStream)
   SET(KWSYS_USE_Encoding 1)
 ENDIF()
+IF(KWSYS_USE_ConsoleBuf)
+  SET(KWSYS_USE_Encoding 1)
+ENDIF()
 
 # Setup the large file support default.
 IF(KWSYS_LFS_DISABLE)
@@ -673,7 +677,7 @@ SET(KWSYS_HXX_FILES Configure String ProcessOutput
 # Add selected C++ classes.
 SET(cppclasses
   Directory DynamicLoader Encoding Glob RegularExpression SystemTools
-  CommandLineArguments IOStream FStream SystemInformation
+  CommandLineArguments IOStream FStream SystemInformation ConsoleBuf
   )
 FOREACH(cpp ${cppclasses})
   IF(KWSYS_USE_${cpp})
@@ -926,6 +930,17 @@ IF(KWSYS_STANDALONE OR CMake_SOURCE_DIR)
 testFStream
 )
 ENDIF()
+IF(KWSYS_USE_ConsoleBuf)
+  IF("${CMAKE_CXX_COMPILER_ID}" STREQUAL "MSVC")
+add_compile_options(/utf-8)
+  ENDIF()
+  ADD_EXECUTABLE(testConsoleBufChild testConsoleBufChild.cxx)
+  SET_PROPERTY(TARGET testConsoleBufChild PROPERTY LABELS 
${KWSYS_LABELS_EXE})
+  TARGET_LINK_LIBRARIES(testConsoleBufChild ${KWSYS_NAMESPACE})
+  SET(KWSYS_CXX_TESTS ${KWSYS_CXX_TESTS}
+testConsoleBuf
+)
+ENDIF()
 IF(KWSYS_USE_SystemInformation)
   SET(KWSYS_CXX_TESTS ${KWSYS_CXX_TESTS} testSystemInformation)
 ENDIF()
diff --git a/Source/kwsys/ConsoleBuf.hxx.in b/Source/kwsys/ConsoleBuf.hxx.in
new file mode 100644
index 000..ed95bb0
--- /dev/null
+++ b/Source/kwsys/ConsoleBuf.hxx.in
@@ -0,0 +1,318 @@
+/*
+  KWSys - Kitware System Library
+  Copyright 2000-2016 Kitware, Inc., Insight Software Consortium
+
+  Distributed under the OSI-approved BSD License (the "License");
+  see accompanying file Copyright.txt for details.
+
+  This software is distributed WITHOUT ANY WARRANTY; without even the
+  implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+  See the License for more information.
+*/
+#ifndef @KWSYS_NAMESPACE@_ConsoleBuf_hxx
+#define @KWSYS_NAMESPACE@_ConsoleBuf_hxx
+
+#include <@KWSYS_NAMESPACE@/Configure.hxx>
+#include <@KWSYS_NAMESPACE@/Encoding.hxx>
+#include 
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#if defined(_WIN32)
+#  include 
+#  if __cplusplus >= 201103L
+#include 
+#  endif
+#endif
+
+namespace @KWSYS_NAMESPACE@
+{
+#if

Re: [cmake-developers] [PATCH v4 4/4] For Windows encode process output to internally used encoding

2016-07-21 Thread Dāvis Mosāns 
2016-07-21 20:46 GMT+03:00 Brad King <brad.k...@kitware.com>:
> On 07/21/2016 01:36 PM, Dāvis Mosāns wrote:
>> Anyway I improved this in places where it was easy, but in some places it's
>> more complicated...
>>
>> For example
>>
>>while ((p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) 
>> {
>>  // Put the output in the right place.
>>  if (p == cmsysProcess_Pipe_STDOUT && !output_quiet) {
>>if (output_variable.empty()) {
>>  cmSystemTools::Stdout(data, length);
>>
>> Here we output buffer immediately.
>>
>>  while ((out || err) &&
>>   (p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) 
>> {
>>  if (out && p == cmsysProcess_Pipe_STDOUT) {
>>if (!out->Process(data, length)) {
>
> In such cases the data need to be piped through a buffered decoder
> that can keep partial fragments around between updates.
>
> Does MultiByteToWideChar or some other API have a way to detect
> such boundaries?

As far as I know in WinAPI there isn't any such function.
With MultiByteToWideChar such partial char would be replaced with ? (U+003F)
or � (U+FFFD).

We would need to use some library or implement this ourselves.
In WinAPI there's CharPrevExA and IsDBCSLeadByteEx (or GetCPInfo) which we can
use and implement this easily for 1-2 byte code pages but it doesn't work for
code pages where character can be more than 2 bytes, eg. UTF-8. Those would
need to be handled separately.
Also could check if last character is ? and try again with one byte less.


Using EnumSystemCodePages and GetCPInfoEx I collected info about all supported
code pages on my Windows 10

https://paste.kde.org/pthwqdbxv/rjwgwd/raw

Code pages where MaxCharSize is more than 1 and UseLeadByte is No need special
handling for those depending on that particular encoding.

>> +  bool DecodeText(std::string raw, std::string& decoded)
>> +  {
>> +bool success = true;
>> +decoded = raw;
>> +#if defined(_WIN32)
>> +if (raw.size() > 0 && codepage != defaultCodepage) {
>> +  success = false;
>> +  const int wlength = MultiByteToWideChar(codepage, 0, raw.c_str(), 
>> int(raw.size()), NULL, 0);
>
> Why do we need new calls to MultiByteToWideChar instead of
> having clients just directly use kwsysEncoding_mbstowcs?
>

Because from WaitForData we're getting data and length, and I assume that data
might not be null-terminated but kwsysEncoding_mbstowcs expects source to be
null-terminated and doesn't accept length.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH v6] For Windows encode process output to internally used encoding

2016-08-13 Thread Dāvis Mosāns 
Typically Windows applications (eg. MSVC compiler) use current console's
codepage for output to pipes so we need to encode that to internally used
encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE).
---
 Source/CMakeLists.txt  |   2 +
 Source/ProcessOutput.hxx.in| 160 +
 Source/cmExecProgramCommand.cxx|   3 +
 Source/cmExecuteProcessCommand.cxx |  11 ++-
 Source/cmProcessTools.cxx  |   9 ++-
 Source/cmSystemTools.cxx   |  11 ++-
 bootstrap  |   3 +
 7 files changed, 193 insertions(+), 6 deletions(-)
 create mode 100644 Source/ProcessOutput.hxx.in

diff --git a/Source/CMakeLists.txt b/Source/CMakeLists.txt
index a790994..eb51683 100644
--- a/Source/CMakeLists.txt
+++ b/Source/CMakeLists.txt
@@ -419,6 +419,8 @@ foreach(command_file
 endforeach()
 configure_file(cmCommands.cxx.in ${CMAKE_CURRENT_BINARY_DIR}/cmCommands.cxx 
@ONLY)
 
+configure_file(ProcessOutput.hxx.in 
${CMAKE_CURRENT_BINARY_DIR}/ProcessOutput.hxx)
+
 # Kdevelop only works on UNIX and not windows
 if(UNIX)
   set(SRCS ${SRCS} cmGlobalKdevelopGenerator.cxx)
diff --git a/Source/ProcessOutput.hxx.in b/Source/ProcessOutput.hxx.in
new file mode 100644
index 000..ea7b881
--- /dev/null
+++ b/Source/ProcessOutput.hxx.in
@@ -0,0 +1,160 @@
+/*
+  CMake - Cross Platform Makefile Generator
+  Copyright 2000-2016 Kitware, Inc., Insight Software Consortium
+
+  Distributed under the OSI-approved BSD License (the "License");
+  see accompanying file Copyright.txt for details.
+
+  This software is distributed WITHOUT ANY WARRANTY; without even the
+  implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+  See the License for more information.
+*/
+#ifndef ProcessOutput_hxx
+#define ProcessOutput_hxx
+
+#include 
+#include 
+#if defined(_WIN32)
+#  ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN 1
+#  endif
+#  include 
+#endif
+
+  class ProcessOutput
+  {
+public:
+#if defined(_WIN32)
+  static const UINT defaultCodepage = @KWSYS_ENCODING_DEFAULT_CODEPAGE@;
+#endif
+  // must match to KWSYSPE_PIPE_BUFFER_SIZE
+  ProcessOutput(unsigned int maxSize = 1024)
+  {
+#if defined(_WIN32)
+bufferSize = maxSize;
+codepage = GetConsoleCP();
+if (!codepage) {
+  codepage = GetACP();
+}
+#else
+static_cast(maxSize);
+#endif
+  }
+
+  ~ProcessOutput()
+  {
+  }
+
+  bool DecodeText(std::string raw, std::string& decoded, size_t id = 0)
+  {
+bool success = true;
+decoded = raw;
+#if defined(_WIN32)
+if (id > 0) {
+  if (rawparts.size() < id) {
+rawparts.reserve(id);
+while (rawparts.size() < id) rawparts.push_back(std::string());
+  }
+  raw = rawparts[id - 1] + raw;
+  rawparts[id - 1].clear();
+  decoded = raw;
+}
+if (raw.size() > 0 && codepage != defaultCodepage) {
+  success = false;
+  CPINFOEXW cpinfo;
+  if (id > 0 && raw.size() == bufferSize && GetCPInfoExW(codepage, 0, 
) == 1 && cpinfo.MaxCharSize > 1) {
+if (cpinfo.MaxCharSize == 2 && cpinfo.LeadByte[0] != 0) {
+  LPSTR prevChar = CharPrevExA(codepage, raw.c_str(), raw.c_str() 
+ raw.size(), 0);
+  bool isLeadByte = (*(prevChar + 1) == 0) && 
IsDBCSLeadByteEx(codepage, *prevChar);
+  if (isLeadByte) {
+rawparts[id - 1] += *(raw.end() - 1);
+raw.resize(raw.size() - 1);
+  }
+  success = DoDecodeText(raw, decoded, NULL);
+} else {
+  bool restoreDecoded = false;
+  std::string firstDecoded = decoded;
+  wchar_t lastChar = 0;
+  for (UINT i = 0; i < cpinfo.MaxCharSize; i++) {
+success = DoDecodeText(raw, decoded, );
+if (success && lastChar != 0) {
+  if (i == 0) {
+firstDecoded = decoded;
+  }
+  if (lastChar == cpinfo.UnicodeDefaultChar) {
+restoreDecoded = true;
+rawparts[id - 1] = *(raw.end() - 1) + rawparts[id - 1];
+raw.resize(raw.size() - 1);
+  } else {
+restoreDecoded = false;
+break;
+  }
+} else {
+  break;
+}
+  }
+  if (restoreDecoded) {
+decoded = firstDecoded;
+rawparts[id - 1].clear();
+  }
+}
+  } else {
+success = DoDecodeText(raw, decoded, NULL);
+  }
+}
+#else
+static_cast(id);
+#endif
+return success;
+  }
+
+  bool

[cmake-developers] [PATCH v4] Add MinGW support for FStream

2016-07-12 Thread Dāvis Mosāns 
std::basic_filebuf::open(const wchar_t *) isn't part of C++ standard
and it's only present for MSVC but it isn't present in libstdc++ (MinGW)
so we implement this functionality using GNU libstdc++ stdio_filebuf
extension and _wfopen function.
---
 CMakeLists.txt  |  14 +++
 Source/kwsys/CMakeLists.txt |   8 ++
 Source/kwsys/FStream.hxx.in | 235 +++-
 3 files changed, 187 insertions(+), 70 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 792b5a5..b53c6b1 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -273,6 +273,20 @@ macro (CMAKE_BUILD_UTILITIES)
 CMAKE_SET_TARGET_FOLDER(${KWSYS_NAMESPACE}TestSharedForward 
"${kwsys_folder}")
   endif()
 
+  IF(KWSYS_USE_SystemTools)
+SET(KWSYS_USE_Directory 1)
+SET(KWSYS_USE_FStream 1)
+SET(KWSYS_USE_Encoding 1)
+  ENDIF()
+
+  IF(KWSYS_USE_FStream)
+INCLUDE(CheckIncludeFileCXX)
+CHECK_INCLUDE_FILE_CXX(ext/stdio_filebuf.h HAVE_EXT_STDIO_FILEBUF_H)
+IF(HAVE_EXT_STDIO_FILEBUF_H)
+  add_definitions(-DHAVE_EXT_STDIO_FILEBUF_H=1)
+ENDIF()
+  ENDIF()
+
   #-
   # Setup third-party libraries.
   # Everything in the tree should be able to include files from the
diff --git a/Source/kwsys/CMakeLists.txt b/Source/kwsys/CMakeLists.txt
index 33a97e6..02ba2db 100644
--- a/Source/kwsys/CMakeLists.txt
+++ b/Source/kwsys/CMakeLists.txt
@@ -636,6 +636,14 @@ IF(KWSYS_USE_SystemInformation)
   ENDIF()
 ENDIF()
 
+IF(KWSYS_USE_FStream)
+  INCLUDE(CheckIncludeFileCXX)
+  CHECK_INCLUDE_FILE_CXX(ext/stdio_filebuf.h HAVE_EXT_STDIO_FILEBUF_H)
+  IF(HAVE_EXT_STDIO_FILEBUF_H)
+add_definitions(-DHAVE_EXT_STDIO_FILEBUF_H=1)
+  ENDIF()
+ENDIF()
+
 #-
 # Choose a directory for the generated headers.
 IF(NOT KWSYS_HEADER_ROOT)
diff --git a/Source/kwsys/FStream.hxx.in b/Source/kwsys/FStream.hxx.in
index 681e4d8..eb911c9 100644
--- a/Source/kwsys/FStream.hxx.in
+++ b/Source/kwsys/FStream.hxx.in
@@ -14,152 +14,247 @@
 
 #include <@KWSYS_NAMESPACE@/Encoding.hxx>
 #include 
+#if defined(_WIN32)
+#  if !defined(_MSC_VER) && defined(HAVE_EXT_STDIO_FILEBUF_H)
+#include 
+#  elif !defined(_MSC_VER) || _MSC_VER < 1400
+#pragma message("WARNING: Opening non-ASCII files might fail!")
+#  endif
+#endif
 
 namespace @KWSYS_NAMESPACE@
 {
-#if defined(_MSC_VER) && _MSC_VER >= 1400
+#if defined(_WIN32) && (defined(_MSC_VER) || defined(HAVE_EXT_STDIO_FILEBUF_H))
 # if defined(_NOEXCEPT)
 #  define @KWSYS_NAMESPACE@_FStream_NOEXCEPT _NOEXCEPT
 # else
 #  define @KWSYS_NAMESPACE@_FStream_NOEXCEPT
 # endif
+
+#if defined(_MSC_VER)
+
   template
   class basic_filebuf : public std::basic_filebuf
   {
+# if _MSC_VER >= 1400
 public:
   typedef std::basic_filebuf my_base_type;
   basic_filebuf *open(char const *s,std::ios_base::openmode mode)
   {
+const std::wstring wstr = Encoding::ToWide(s);
 return static_cast(
-  my_base_type::open(Encoding::ToWide(s).c_str(), mode)
+  my_base_type::open(wstr.c_str(), mode)
   );
   }
+# endif
+  };
+
+#else
+
+  inline std::wstring getcmode(const std::ios_base::openmode mode) {
+std::wstring cmode;
+bool plus = false;
+if (mode & std::ios_base::app) {
+  cmode += L"a";
+  plus = mode & std::ios_base::in ? true : false;
+} else if (mode & std::ios_base::trunc ||
+(mode & std::ios_base::out && (mode & std::ios_base::in) == 
0)) {
+  cmode += L"w";
+  plus = mode & std::ios_base::in ? true : false;
+} else {
+  cmode += L"r";
+  plus = mode & std::ios_base::out ? true : false;
+}
+if (plus) {
+  cmode += L"+";
+}
+if (mode & std::ios_base::binary) {
+  cmode += L"b";
+} else {
+  cmode += L"t";
+}
+return cmode;
   };
 
+#endif
+
   template >
-  class basic_ifstream : public std::basic_istream
+  class basic_efilebuf
   {
+public:
+#if defined(_MSC_VER)
+  typedef basic_filebuf internal_buffer_type;
+#else
+  typedef __gnu_cxx::stdio_filebuf internal_buffer_type;
+#endif
+
+  basic_efilebuf() : file_(0)
+  {
+buf_ = 0;
+  }
+
+  bool _open(char const *file_name,std::ios_base::openmode mode)
+  {
+if (is_open() || file_) {
+  return false;
+}
+#if defined(_MSC_VER)
+const bool success = buf_->open(file_name,mode) != 0;
+#else
+const std::wstring wstr = Encoding::ToWide(file_name);
+bool success = false;
+std::wstring cmode = getcmode(mode);
+file_ = _wfopen(wstr.c_str(), cmode.c_str());
+if (file_) {
+  if (buf_) {
+delete buf_;
+  }
+  buf_ = new internal_buffer_type(file_, mode);
+  success = true;
+}
+#endif
+

[cmake-developers] [PATCH v5 1/2] Refactor FStream (basic_ifstream and basic_ofstream classes)

2016-07-16 Thread Dāvis Mosāns 
Use common basic_efilebuf for both to reduce code duplication.
---
 Source/kwsys/FStream.hxx.in | 164 +---
 1 file changed, 95 insertions(+), 69 deletions(-)

diff --git a/Source/kwsys/FStream.hxx.in b/Source/kwsys/FStream.hxx.in
index 681e4d8..248b0cb 100644
--- a/Source/kwsys/FStream.hxx.in
+++ b/Source/kwsys/FStream.hxx.in
@@ -30,136 +30,162 @@ namespace @KWSYS_NAMESPACE@
   typedef std::basic_filebuf my_base_type;
   basic_filebuf *open(char const *s,std::ios_base::openmode mode)
   {
+const std::wstring wstr = Encoding::ToWide(s);
 return static_cast(
-  my_base_type::open(Encoding::ToWide(s).c_str(), mode)
+  my_base_type::open(wstr.c_str(), mode)
   );
   }
   };
 
   template >
-  class basic_ifstream : public std::basic_istream
+  class basic_efilebuf
   {
+public:
+  typedef basic_filebuf internal_buffer_type;
+
+  basic_efilebuf()
+  {
+buf_ = 0;
+  }
+
+  bool _open(char const *file_name,std::ios_base::openmode mode)
+  {
+if (is_open()) {
+  return false;
+}
+const bool success = buf_->open(file_name,mode) != 0;
+return success;
+  }
+
+  bool is_open()
+  {
+if (!buf_) {
+  return false;
+}
+return buf_->is_open();
+  }
+
+  bool is_open() const
+  {
+if (!buf_) {
+  return false;
+}
+return buf_->is_open();
+  }
+
+  bool _close()
+  {
+bool success = false;
+if (buf_) {
+  success = buf_->close() != 0;
+}
+return success;
+  }
+
+  static void _set_state(bool success, std::basic_ios 
*ios)
+  {
+if (!success) {
+  ios->setstate(std::ios_base::failbit);
+} else {
+  ios->clear();
+}
+  }
+
+  ~basic_efilebuf()
+ {
+   if (buf_) {
+ delete buf_;
+   }
+ }
+
+protected:
+  internal_buffer_type* buf_;
+  };
+
+template >
+class basic_ifstream : public std::basic_istream,
+   public basic_efilebuf
+{
+  using basic_efilebuf::is_open;
+
   public:
-typedef basic_filebuf internal_buffer_type;
+typedef typename basic_efilebuf::internal_buffer_type 
internal_buffer_type;
 typedef std::basic_istream internal_stream_type;
 
 basic_ifstream() : internal_stream_type(new internal_buffer_type())
 {
-  buf_ = static_cast(internal_stream_type::rdbuf());
+  this->buf_ = static_cast(internal_stream_type::rdbuf());
 }
 explicit basic_ifstream(char const *file_name,
 std::ios_base::openmode mode = std::ios_base::in)
   : internal_stream_type(new internal_buffer_type())
 {
-  buf_ = static_cast(internal_stream_type::rdbuf());
+  this->buf_ = static_cast(internal_stream_type::rdbuf());
   open(file_name,mode);
 }
+
 void open(char const *file_name,std::ios_base::openmode mode = 
std::ios_base::in)
 {
-  if(!buf_->open(file_name,mode | std::ios_base::in))
-{
-this->setstate(std::ios_base::failbit);
-}
-  else
-{
-this->clear();
-}
-}
-bool is_open()
-{
-  return buf_->is_open();
-}
-bool is_open() const
-{
-  return buf_->is_open();
+  mode = mode | std::ios_base::in;
+  this->_set_state(this->_open(file_name, mode), this);
 }
+
 void close()
 {
-  if(!buf_->close())
-{
-this->setstate(std::ios_base::failbit);
-}
-  else
-  {
-this->clear();
-  }
+  this->_set_state(this->_close(), this);
 }
 
 internal_buffer_type *rdbuf() const
 {
-  return buf_;
+  return this->buf_;
 }
 
 ~basic_ifstream() @KWSYS_NAMESPACE@_FStream_NOEXCEPT
 {
-  buf_->close();
-  delete buf_;
+  close();
 }
-
-  private:
-internal_buffer_type* buf_;
 };
 
 template >
-class basic_ofstream : public std::basic_ostream
+class basic_ofstream : public std::basic_ostream,
+   public basic_efilebuf
 {
+  using basic_efilebuf::is_open;
+
   public:
-  typedef basic_filebuf internal_buffer_type;
+  typedef typename basic_efilebuf::internal_buffer_type 
internal_buffer_type;
   typedef std::basic_ostream internal_stream_type;
 
   basic_ofstream() : internal_stream_type(new internal_buffer_type())
   {
-  buf_ = static_cast(internal_stream_type::rdbuf());
+this->buf_ = static_cast(internal_stream_type::rdbuf());
   }
   explicit basic_ofstream(char const *file_name,std::ios_base::openmode mode =

[cmake-developers] [PATCH v5 2/2] Add MinGW support for FStream

2016-07-16 Thread Dāvis Mosāns 
std::basic_filebuf::open(const wchar_t *) isn't part of C++ standard
and it's only present for MSVC but it isn't present in libstdc++ (MinGW)
so we implement this functionality using GNU libstdc++ stdio_filebuf
extension and _wfopen function.
---
 Source/kwsys/CMakeLists.txt|  5 ++
 Source/kwsys/Configure.hxx.in  |  7 ++-
 Source/kwsys/FStream.hxx.in| 86 ++
 Source/kwsys/kwsysPlatformTestsCXX.cxx |  6 +++
 bootstrap  | 11 +
 5 files changed, 105 insertions(+), 10 deletions(-)

diff --git a/Source/kwsys/CMakeLists.txt b/Source/kwsys/CMakeLists.txt
index 33a97e6..297a7bd 100644
--- a/Source/kwsys/CMakeLists.txt
+++ b/Source/kwsys/CMakeLists.txt
@@ -636,6 +636,11 @@ IF(KWSYS_USE_SystemInformation)
   ENDIF()
 ENDIF()
 
+IF(KWSYS_USE_FStream)
+  KWSYS_PLATFORM_CXX_TEST(KWSYS_CXX_HAS_EXT_STDIO_FILEBUF_H
+"Checking whether  is available" DIRECT)
+ENDIF()
+
 #-
 # Choose a directory for the generated headers.
 IF(NOT KWSYS_HEADER_ROOT)
diff --git a/Source/kwsys/Configure.hxx.in b/Source/kwsys/Configure.hxx.in
index ff8e49d..4ce680d 100644
--- a/Source/kwsys/Configure.hxx.in
+++ b/Source/kwsys/Configure.hxx.in
@@ -17,6 +17,8 @@
 
 /* Whether wstring is available.  */
 #define @KWSYS_NAMESPACE@_STL_HAS_WSTRING @KWSYS_STL_HAS_WSTRING@
+/* Whether  is available. */
+#define @KWSYS_NAMESPACE@_CXX_HAS_EXT_STDIO_FILEBUF_H 
@KWSYS_CXX_HAS_EXT_STDIO_FILEBUF_H@
 
 /* If building a C++ file in kwsys itself, give the source file
access to the macros without a configured namespace.  */
@@ -24,8 +26,9 @@
 # if !@KWSYS_NAMESPACE@_NAME_IS_KWSYS
 #  define kwsys @KWSYS_NAMESPACE@
 # endif
-# define KWSYS_NAME_IS_KWSYS@KWSYS_NAMESPACE@_NAME_IS_KWSYS
-# define KWSYS_STL_HAS_WSTRING  @KWSYS_NAMESPACE@_STL_HAS_WSTRING
+# define KWSYS_NAME_IS_KWSYS   @KWSYS_NAMESPACE@_NAME_IS_KWSYS
+# define KWSYS_STL_HAS_WSTRING @KWSYS_NAMESPACE@_STL_HAS_WSTRING
+# define KWSYS_CXX_HAS_EXT_STDIO_FILEBUF_H 
@KWSYS_NAMESPACE@_CXX_HAS_EXT_STDIO_FILEBUF_H
 #endif
 
 #endif
diff --git a/Source/kwsys/FStream.hxx.in b/Source/kwsys/FStream.hxx.in
index 248b0cb..c13f91d 100644
--- a/Source/kwsys/FStream.hxx.in
+++ b/Source/kwsys/FStream.hxx.in
@@ -12,20 +12,32 @@
 #ifndef @KWSYS_NAMESPACE@_FStream_hxx
 #define @KWSYS_NAMESPACE@_FStream_hxx
 
+#include <@KWSYS_NAMESPACE@/Configure.hxx>
 #include <@KWSYS_NAMESPACE@/Encoding.hxx>
 #include 
+#if defined(_WIN32)
+#  if !defined(_MSC_VER) && @KWSYS_NAMESPACE@_CXX_HAS_EXT_STDIO_FILEBUF_H
+#include 
+#  elif !defined(_MSC_VER) || _MSC_VER < 1400
+#pragma message("WARNING: Opening non-ASCII files might fail!")
+#  endif
+#endif
 
 namespace @KWSYS_NAMESPACE@
 {
-#if defined(_MSC_VER) && _MSC_VER >= 1400
+#if defined(_WIN32) && (defined(_MSC_VER) || 
@KWSYS_NAMESPACE@_CXX_HAS_EXT_STDIO_FILEBUF_H)
 # if defined(_NOEXCEPT)
 #  define @KWSYS_NAMESPACE@_FStream_NOEXCEPT _NOEXCEPT
 # else
 #  define @KWSYS_NAMESPACE@_FStream_NOEXCEPT
 # endif
+
+#if defined(_MSC_VER)
+
   template
   class basic_filebuf : public std::basic_filebuf
   {
+# if _MSC_VER >= 1400
 public:
   typedef std::basic_filebuf my_base_type;
   basic_filebuf *open(char const *s,std::ios_base::openmode mode)
@@ -35,25 +47,73 @@ namespace @KWSYS_NAMESPACE@
   my_base_type::open(wstr.c_str(), mode)
   );
   }
+# endif
+  };
+
+#else
+
+  inline std::wstring getcmode(const std::ios_base::openmode mode) {
+std::wstring cmode;
+bool plus = false;
+if (mode & std::ios_base::app) {
+  cmode += L"a";
+  plus = mode & std::ios_base::in ? true : false;
+} else if (mode & std::ios_base::trunc ||
+(mode & std::ios_base::out && (mode & std::ios_base::in) == 
0)) {
+  cmode += L"w";
+  plus = mode & std::ios_base::in ? true : false;
+} else {
+  cmode += L"r";
+  plus = mode & std::ios_base::out ? true : false;
+}
+if (plus) {
+  cmode += L"+";
+}
+if (mode & std::ios_base::binary) {
+  cmode += L"b";
+} else {
+  cmode += L"t";
+}
+return cmode;
   };
 
+#endif
+
   template >
   class basic_efilebuf
   {
 public:
+#if defined(_MSC_VER)
   typedef basic_filebuf internal_buffer_type;
+#else
+  typedef __gnu_cxx::stdio_filebuf internal_buffer_type;
+#endif
 
-  basic_efilebuf()
+  basic_efilebuf() : file_(0)
   {
 buf_ = 0;
   }
 
   bool _open(char const *file_name,std::ios_base::openmode mode)
   {
-if (is_open()) {
+if (is_open() || file_) {
   return false;
 }
+#if defined(_MSC_VER)
 const bool success = buf_->open(file_name,mode) != 0;
+#else
+const std::wstring wstr = Encoding::ToWide(file_name);
+bool success = false;
+

Re: [cmake-developers] [PATCH v4 fixed 1/4] On Windows use correct encoding for SystemTools::GetEnv

2016-07-08 Thread Dāvis Mosāns 
2016-07-08 16:04 GMT+03:00 Brad King <brad.k...@kitware.com>:
> On 07/07/2016 06:32 PM, Dāvis Mosāns wrote:
>> On Windows getenv (and putenv) uses ANSI codepage so it needs to be encoded
>> to internally used encoding (eg. UTF-8). Here we use _wgetenv (and _wputenv)
>> instead and encode that.
>>
>> Also add SystemTools::HasEnv function.
>
> Thanks.  KWSys is maintained externally to CMake so I've ported
> this patch over for integration there first.  I split it into
> two commits for review and testing here:
>
>  http://review.source.kitware.com/21318
>  http://review.source.kitware.com/21319
>
> Once integrated there I can update it in CMake and then move
> on to more of your patches.
>

Great! Thanks!
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH v3 5/7] For consoles output on Windows use our own std::streambuf

2016-07-06 Thread Dāvis Mosāns 
Currently Microsoft's C++ libraries implementation of std::cout/cerr
can't output Unicode characters but only ASCII or ANSI if locale is set
so we implement and use our own ConsoleBuf which can output Unicode
characters to console and it doesn't matter what locale or console's
codepage is set.
---
 CMakeLists.txt |   1 +
 Source/cmakemain.cxx   |  28 +
 Source/kwsys/CMakeLists.txt|   6 +-
 Source/kwsys/ConsoleBuf.hxx.in | 131 +
 bootstrap  |   1 +
 5 files changed, 166 insertions(+), 1 deletion(-)
 create mode 100644 Source/kwsys/ConsoleBuf.hxx.in

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ae5990e..792b5a5 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -258,6 +258,7 @@ macro (CMAKE_BUILD_UTILITIES)
   set(KWSYS_USE_MD5 1)
   set(KWSYS_USE_Process 1)
   set(KWSYS_USE_CommandLineArguments 1)
+  set(KWSYS_USE_ConsoleBuf 1)
   set(KWSYS_HEADER_ROOT ${CMake_BINARY_DIR}/Source)
   set(KWSYS_INSTALL_DOC_DIR "${CMAKE_DOC_DIR}")
   add_subdirectory(Source/kwsys)
diff --git a/Source/cmakemain.cxx b/Source/cmakemain.cxx
index 521a5bf..2002e4b 100644
--- a/Source/cmakemain.cxx
+++ b/Source/cmakemain.cxx
@@ -26,6 +26,7 @@
 #include "cmake.h"
 #include "cmcmd.h"
 #include 
+#include 
 
 #ifdef CMAKE_BUILD_WITH_CMAKE
 static const char* cmDocumentationName[][2] = {
@@ -153,6 +154,23 @@ static void cmakemainProgressCallback(const char* m, float 
prog,
 
 int main(int ac, char const* const* av)
 {
+#if defined(_WIN32)
+  // Replace streambuf so we can output Unicode to console
+  cmsys::ConsoleBuf *cbufio = CM_NULLPTR;
+  cmsys::ConsoleBuf *cbuferr = CM_NULLPTR;
+  std::streambuf *coutbuf = CM_NULLPTR;
+  std::streambuf *cerrbuf = CM_NULLPTR;
+  try {
+cbufio = new cmsys::ConsoleBuf();
+coutbuf = std::cout.rdbuf(cbufio);
+cbuferr = new cmsys::ConsoleBuf(true);
+cerrbuf = std::cerr.rdbuf(cbuferr);
+  } catch (const std::system_error& ex) {
+std::cerr << "Failed to create ConsoleBuf!" << std::endl
+  << "Error code: " << ex.code() << std::endl
+  << ex.what() << std::endl;
+  };
+#endif
   cmsys::Encoding::CommandLineArguments args =
 cmsys::Encoding::CommandLineArguments::Main(ac, av);
   ac = args.argc();
@@ -171,6 +189,16 @@ int main(int ac, char const* const* av)
 #ifdef CMAKE_BUILD_WITH_CMAKE
   cmDynamicLoader::FlushCache();
 #endif
+#if defined(_WIN32)
+  if (cbufio) {
+delete cbufio;
+std::cout.rdbuf(coutbuf);
+  }
+  if (cbuferr) {
+delete cbuferr;
+std::cerr.rdbuf(cerrbuf);
+  }
+#endif
   return ret;
 }
 
diff --git a/Source/kwsys/CMakeLists.txt b/Source/kwsys/CMakeLists.txt
index 65203c0..33a97e6 100644
--- a/Source/kwsys/CMakeLists.txt
+++ b/Source/kwsys/CMakeLists.txt
@@ -123,6 +123,7 @@ IF(KWSYS_STANDALONE OR CMake_SOURCE_DIR)
   SET(KWSYS_USE_FStream 1)
   SET(KWSYS_USE_String 1)
   SET(KWSYS_USE_SystemInformation 1)
+  SET(KWSYS_USE_ConsoleBuf 1)
 ENDIF()
 
 # Enforce component dependencies.
@@ -154,6 +155,9 @@ ENDIF()
 IF(KWSYS_USE_FStream)
   SET(KWSYS_USE_Encoding 1)
 ENDIF()
+IF(KWSYS_USE_ConsoleBuf)
+  SET(KWSYS_USE_Encoding 1)
+ENDIF()
 
 # Setup the large file support default.
 IF(KWSYS_LFS_DISABLE)
@@ -668,7 +672,7 @@ SET(KWSYS_HXX_FILES Configure String
 # Add selected C++ classes.
 SET(cppclasses
   Directory DynamicLoader Encoding Glob RegularExpression SystemTools
-  CommandLineArguments IOStream FStream SystemInformation
+  CommandLineArguments IOStream FStream SystemInformation ConsoleBuf
   )
 FOREACH(cpp ${cppclasses})
   IF(KWSYS_USE_${cpp})
diff --git a/Source/kwsys/ConsoleBuf.hxx.in b/Source/kwsys/ConsoleBuf.hxx.in
new file mode 100644
index 000..90adc03
--- /dev/null
+++ b/Source/kwsys/ConsoleBuf.hxx.in
@@ -0,0 +1,131 @@
+/*
+  KWSys - Kitware System Library
+  Copyright 2000-2016 Kitware, Inc., Insight Software Consortium
+
+  Distributed under the OSI-approved BSD License (the "License");
+  see accompanying file Copyright.txt for details.
+
+  This software is distributed WITHOUT ANY WARRANTY; without even the
+  implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+  See the License for more information.
+*/
+#ifndef @KWSYS_NAMESPACE@_ConsoleBuf_hxx
+#define @KWSYS_NAMESPACE@_ConsoleBuf_hxx
+
+#include <@KWSYS_NAMESPACE@/Configure.hxx>
+#include <@KWSYS_NAMESPACE@/Encoding.hxx>
+#include 
+
+namespace @KWSYS_NAMESPACE@
+{
+#if defined(_WIN32)
+
+  template>
+  class @KWSYS_NAMESPACE@_EXPORT BasicConsoleBuf : public 
std::basic_streambuf {
+public:
+  typedef typename Traits::int_type int_type;
+  typedef typename Traits::char_type char_type;
+
+  BasicConsoleBuf(const bool err = false) {
+m_hInput = ::GetStdHandle(STD_INPUT_HANDLE);
+if (m_hInput == INVALID_HANDLE_VALUE) {
+  throw

[cmake-developers] [PATCH v3 1/7] On Windows use correct encoding for SystemTools::GetEnv

2016-07-06 Thread Dāvis Mosāns 
On Windows getenv (and putenv) uses ANSI codepage so it needs to be encoded
to internally used encoding (eg. UTF-8). Here we use _wgetenv (and _wputenv)
instead and encode that.

Also add SystemTools::HasEnv function.
---
 Source/kwsys/SystemTools.cxx| 56 ++---
 Source/kwsys/SystemTools.hxx.in |  2 ++
 2 files changed, 49 insertions(+), 9 deletions(-)

diff --git a/Source/kwsys/SystemTools.cxx b/Source/kwsys/SystemTools.cxx
index c6e668d..2b1db4d 100644
--- a/Source/kwsys/SystemTools.cxx
+++ b/Source/kwsys/SystemTools.cxx
@@ -468,16 +468,21 @@ const char* SystemTools::GetEnv(const std::string& key)
 
 bool SystemTools::GetEnv(const char* key, std::string& result)
 {
+#if defined(_WIN32)
+  const std::wstring wkey = Encoding::ToWide(key);
+  const wchar_t* wv = _wgetenv(wkey.c_str());
+  if (wv) {
+result = Encoding::ToNarrow(wv);
+return true;
+  }
+#else
   const char* v = getenv(key);
-  if(v)
-{
+  if (v) {
 result = v;
 return true;
-}
-  else
-{
-return false;
-}
+  }
+#endif
+  return false;
 }
 
 bool SystemTools::GetEnv(const std::string& key, std::string& result)
@@ -485,6 +490,22 @@ bool SystemTools::GetEnv(const std::string& key, 
std::string& result)
   return SystemTools::GetEnv(key.c_str(), result);
 }
 
+bool SystemTools::HasEnv(const char* key)
+{
+#if defined(_WIN32)
+  const std::wstring wkey = Encoding::ToWide(key);
+  const wchar_t* v = _wgetenv(wkey.c_str());
+#else
+  const char* v = getenv(key);
+#endif
+  return v != 0;
+}
+
+bool SystemTools::HasEnv(const std::string& key)
+{
+  return SystemTools::HasEnv(key.c_str());
+}
+
 //
 
 #if defined(__CYGWIN__) || defined(__GLIBC__)
@@ -533,13 +554,25 @@ static int kwsysUnPutEnv(const std::string& env)
 # ifdef KWSYS_PUTENV_EMPTY
   buf[len] = '=';
   buf[len+1] = 0;
-  if(putenv(buf) < 0)
+#if defined(_WIN32)
+  const std::wstring wbuf = Encoding::ToWide(buf);
+  const int r = _wputenv(wbuf.c_str());
+#else
+  const int r = putenv(buf);
+#endif
+  if(r < 0)
 {
 err = errno;
 }
 # else
   buf[len] = 0;
-  if(putenv(buf) < 0 && errno != EINVAL)
+#if defined(_WIN32)
+  const std::wstring wbuf = Encoding::ToWide(buf);
+  const int r = _wputenv(wbuf.c_str());
+#else
+  const int r = putenv(buf);
+#endif
+  if(r < 0 && errno != EINVAL)
 {
 err = errno;
 }
@@ -679,7 +712,12 @@ public:
 static_cast(oldEnv);
 char* newEnv = strdup(env);
 this->insert(newEnv);
+#if defined(_WIN32)
+const std::wstring wEnv = Encoding::ToWide(newEnv);
+return _wputenv(wEnv.c_str()) == 0;
+#else
 return putenv(newEnv) == 0;
+#endif
 }
   bool UnPut(const char* env)
 {
diff --git a/Source/kwsys/SystemTools.hxx.in b/Source/kwsys/SystemTools.hxx.in
index bba5a5c..c9b18b7 100644
--- a/Source/kwsys/SystemTools.hxx.in
+++ b/Source/kwsys/SystemTools.hxx.in
@@ -843,6 +843,8 @@ public:
   static const char* GetEnv(const std::string& key);
   static bool GetEnv(const char* key, std::string& result);
   static bool GetEnv(const std::string& key, std::string& result);
+  static bool HasEnv(const char* key);
+  static bool HasEnv(const std::string& key);
 
   /** Put a string into the environment
   of the form var=value */
-- 
2.9.0

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH v3 7/7] Add MinGW support for FStream

2016-07-06 Thread Dāvis Mosāns 
std::basic_filebuf::open(const wchar_t *) isn't part of C++ standard
and it's only present for MSVC but it's not present in libstdc++ (MinGW)
so we implement this functionality using GNU stdio_filebuf extension and
_wfopen function.
---
 Source/kwsys/FStream.hxx.in | 117 +---
 1 file changed, 109 insertions(+), 8 deletions(-)

diff --git a/Source/kwsys/FStream.hxx.in b/Source/kwsys/FStream.hxx.in
index 681e4d8..f39f95a 100644
--- a/Source/kwsys/FStream.hxx.in
+++ b/Source/kwsys/FStream.hxx.in
@@ -14,33 +14,76 @@
 
 #include <@KWSYS_NAMESPACE@/Encoding.hxx>
 #include 
+#if defined(_WIN32) && !defined(_MSC_VER)
+#include 
+#endif
 
 namespace @KWSYS_NAMESPACE@
 {
-#if defined(_MSC_VER) && _MSC_VER >= 1400
+#if defined(_WIN32)
 # if defined(_NOEXCEPT)
 #  define @KWSYS_NAMESPACE@_FStream_NOEXCEPT _NOEXCEPT
 # else
 #  define @KWSYS_NAMESPACE@_FStream_NOEXCEPT
 # endif
+
+#if defined(_MSC_VER)
+
   template
   class basic_filebuf : public std::basic_filebuf
   {
+# if _MSC_VER >= 1400
 public:
   typedef std::basic_filebuf my_base_type;
   basic_filebuf *open(char const *s,std::ios_base::openmode mode)
   {
+const std::wstring wstr = Encoding::ToWide(s);
 return static_cast(
-  my_base_type::open(Encoding::ToWide(s).c_str(), mode)
+  my_base_type::open(wstr.c_str(), mode)
   );
   }
+# else
+#  pragma message("Warning! Opening non-ASCII files might fail!")
+# endif
+
   };
 
+#else
+
+  inline const std::wstring getcmode(const std::ios_base::openmode mode) {
+std::wstring cmode;
+bool plus = false;
+if (mode & std::ios_base::app) {
+  cmode += L"a";
+  plus = mode & std::ios_base::in ? true : false;
+} else if (mode & std::ios_base::trunc || mode & std::ios_base::out) {
+  cmode += L"w";
+  plus = mode & std::ios_base::in ? true : false;
+} else {
+  cmode += L"r";
+}
+if (mode & std::ios_base::binary) {
+  cmode += L"b";
+} else {
+  cmode += L"t";
+}
+if (plus) {
+  cmode += L"+";
+}
+return cmode;
+  };
+
+#endif
+
   template >
   class basic_ifstream : public std::basic_istream
   {
   public:
+#if defined(_MSC_VER)
 typedef basic_filebuf internal_buffer_type;
+#else
+typedef __gnu_cxx::stdio_filebuf internal_buffer_type;
+#endif
 typedef std::basic_istream internal_stream_type;
 
 basic_ifstream() : internal_stream_type(new internal_buffer_type())
@@ -56,7 +99,24 @@ namespace @KWSYS_NAMESPACE@
 }
 void open(char const *file_name,std::ios_base::openmode mode = 
std::ios_base::in)
 {
-  if(!buf_->open(file_name,mode | std::ios_base::in))
+  mode = mode | std::ios_base::in;
+#if defined(_MSC_VER)
+  const bool success = buf_->open(file_name,mode) != 0;
+#else
+  const std::wstring wstr = Encoding::ToWide(file_name);
+  bool success = false;
+  std::wstring cmode = getcmode(mode);
+  file_ = _wfopen(wstr.c_str(), cmode.c_str());
+  if (file_) {
+if (buf_) {
+  delete buf_;
+}
+buf_ = new internal_buffer_type(file_, mode);
+this->set_rdbuf(buf_);
+success = true;
+  }
+#endif
+  if(!success)
 {
 this->setstate(std::ios_base::failbit);
 }
@@ -75,7 +135,14 @@ namespace @KWSYS_NAMESPACE@
 }
 void close()
 {
-  if(!buf_->close())
+  bool success = buf_->close() != 0;
+#if !defined(_MSC_VER)
+  if (file_) {
+success = fclose(file_) == 0 ? success : false;
+file_ = 0;
+  }
+#endif
+  if(!success)
 {
 this->setstate(std::ios_base::failbit);
 }
@@ -92,19 +159,26 @@ namespace @KWSYS_NAMESPACE@
 
 ~basic_ifstream() @KWSYS_NAMESPACE@_FStream_NOEXCEPT
 {
-  buf_->close();
+  close();
   delete buf_;
 }
 
   private:
 internal_buffer_type* buf_;
+#if !defined(_MSC_VER)
+FILE *file_ = 0;
+#endif
 };
 
 template >
 class basic_ofstream : public std::basic_ostream
 {
   public:
+#if defined(_MSC_VER)
   typedef basic_filebuf internal_buffer_type;
+#else
+  typedef __gnu_cxx::stdio_filebuf internal_buffer_type;
+#endif
   typedef std::basic_ostream internal_stream_type;
 
   basic_ofstream() : internal_stream_type(new internal_buffer_type())
@@ -119,7 +193,24 @@ class basic_ofstream : public 
std::basic_ostream
   }
   void open(char const *file_name,std::ios_base::openmode mode = 
std::ios_base::out)
   {
-if(!buf_->open(file_name,mode | std::ios_base::out))
+mode = mode | std::ios_base::out;
+#if defined(_MSC_VER)
+const bool success = buf_->open(file_name,mode) != 0;
+#else
+const std::wstring wstr = Encoding::ToWide(file_name);
+bool success = false;
+std::wstring cmode =

[cmake-developers] [PATCH v3 2/7] Deprecate const char* SystemTools::GetEnv function

2016-07-06 Thread Dāvis Mosāns 
On Windows this function returns environment variable encoded
in ANSI codepage which might not match internally used encoding.
---
 Source/kwsys/SystemTools.hxx.in | 14 --
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/Source/kwsys/SystemTools.hxx.in b/Source/kwsys/SystemTools.hxx.in
index c9b18b7..51fb206 100644
--- a/Source/kwsys/SystemTools.hxx.in
+++ b/Source/kwsys/SystemTools.hxx.in
@@ -26,6 +26,16 @@
 # include  // For access permissions for use with access()
 #endif
 
+#if __cplusplus >= 201402L
+# define DEPRECATED [[deprecated]]
+#elif defined(__GNUC__)
+# define DEPRECATED __attribute__ ((deprecated))
+#elif defined(_MSC_VER)
+# define DEPRECATED __declspec(deprecated)
+#else
+# define DEPRECATED
+#endif
+
 // Required for va_list
 #include 
 // Required for FILE*
@@ -839,8 +849,8 @@ public:
   /**
* Read an environment variable
*/
-  static const char* GetEnv(const char* key);
-  static const char* GetEnv(const std::string& key);
+  DEPRECATED static const char* GetEnv(const char* key);
+  DEPRECATED static const char* GetEnv(const std::string& key);
   static bool GetEnv(const char* key, std::string& result);
   static bool GetEnv(const std::string& key, std::string& result);
   static bool HasEnv(const char* key);
-- 
2.9.0

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH v3 6/7] Use Windows version of Directory::Load for MinGW too

2016-07-06 Thread Dāvis Mosāns 
Otherwise it would use POSIX functions which works only for ASCII paths.
---
 Source/kwsys/Directory.cxx | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Source/kwsys/Directory.cxx b/Source/kwsys/Directory.cxx
index c549792..659c559 100644
--- a/Source/kwsys/Directory.cxx
+++ b/Source/kwsys/Directory.cxx
@@ -86,7 +86,7 @@ void Directory::Clear()
 
 // First microsoft compilers
 
-#if defined(_MSC_VER) || defined(__WATCOMC__)
+#if defined(_WIN32) || defined(__WATCOMC__)
 #include 
 #include 
 #include 
-- 
2.9.0

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH v3 3/7] Use SystemTools::GetEnv and HasEnv functions

2016-07-06 Thread Dāvis Mosāns 
---
 Source/CPack/cmCPackGenerator.cxx   |  8 +++---
 Source/CTest/cmCTestCoverageHandler.cxx | 12 -
 Source/CTest/cmCTestCurl.cxx| 27 ++--
 Source/CTest/cmCTestMultiProcessHandler.cxx |  8 +++---
 Source/cmBuildCommand.cxx   | 25 +--
 Source/cmCLocaleEnvironmentScope.cxx|  5 ++--
 Source/cmCTest.cxx  | 11 +
 Source/cmCommandArgumentParserHelper.cxx|  8 +++---
 Source/cmConditionEvaluator.cxx |  2 +-
 Source/cmExportCommand.cxx  |  5 ++--
 Source/cmExtraEclipseCDT4Generator.cxx  |  9 ---
 Source/cmFileCommand.cxx| 11 +
 Source/cmFindPackageCommand.cxx |  4 +--
 Source/cmGlobalVisualStudio7Generator.cxx   |  6 ++---
 Source/cmMakefile.cxx   |  5 +++-
 Source/cmNinjaTargetGenerator.cxx   |  2 +-
 Source/cmQtAutoGenerators.cxx   |  2 +-
 Source/cmSetCommand.cxx |  7 +++---
 Source/cmState.cxx  |  5 ++--
 Source/cmSystemTools.cxx|  6 ++---
 Source/cmTimestamp.cxx  |  7 +++---
 Source/cmUtils.hxx  | 26 
 Source/cmake.cxx| 21 +---
 Source/cmcmd.cxx| 16 +---
 Source/kwsys/SystemInformation.cxx  | 20 +++
 Source/kwsys/SystemTools.cxx| 38 +++--
 Source/kwsys/testSystemTools.cxx|  9 ---
 27 files changed, 156 insertions(+), 149 deletions(-)
 create mode 100644 Source/cmUtils.hxx

diff --git a/Source/CPack/cmCPackGenerator.cxx 
b/Source/CPack/cmCPackGenerator.cxx
index df8bb0f..76609e1 100644
--- a/Source/CPack/cmCPackGenerator.cxx
+++ b/Source/CPack/cmCPackGenerator.cxx
@@ -1074,11 +1074,11 @@ const char* cmCPackGenerator::GetInstallPath()
 return this->InstallPath.c_str();
   }
 #if defined(_WIN32) && !defined(__CYGWIN__)
-  const char* prgfiles = cmsys::SystemTools::GetEnv("ProgramFiles");
-  const char* sysDrive = cmsys::SystemTools::GetEnv("SystemDrive");
-  if (prgfiles) {
+  std::string prgfiles;
+  std::string sysDrive;
+  if (cmsys::SystemTools::GetEnv("ProgramFiles", prgfiles)) {
 this->InstallPath = prgfiles;
-  } else if (sysDrive) {
+  } else if (cmsys::SystemTools::GetEnv("SystemDrive", sysDrive)) {
 this->InstallPath = sysDrive;
 this->InstallPath += "/Program Files";
   } else {
diff --git a/Source/CTest/cmCTestCoverageHandler.cxx 
b/Source/CTest/cmCTestCoverageHandler.cxx
index 7102533..9410a52 100644
--- a/Source/CTest/cmCTestCoverageHandler.cxx
+++ b/Source/CTest/cmCTestCoverageHandler.cxx
@@ -727,10 +727,7 @@ int cmCTestCoverageHandler::HandleCoberturaCoverage(
   // if it doesn't exist or is empty, assume the
   // binary directory is used.
   std::string coverageXMLFile;
-  const char* covDir = cmSystemTools::GetEnv("COBERTURADIR");
-  if (covDir && strlen(covDir) != 0) {
-coverageXMLFile = std::string(covDir);
-  } else {
+  if (!cmSystemTools::GetEnv("COBERTURADIR", coverageXMLFile) || 
coverageXMLFile.empty()) {
 coverageXMLFile = this->CTest->GetBinaryDir();
   }
   // build the find file string with the directory from above
@@ -791,7 +788,8 @@ struct cmCTestCoverageHandlerLocale
 {
   cmCTestCoverageHandlerLocale()
   {
-if (const char* l = cmSystemTools::GetEnv("LC_ALL")) {
+std::string l;
+if (cmSystemTools::GetEnv("LC_ALL", l)) {
   lc_all = l;
 }
 if (lc_all != "C") {
@@ -2121,8 +2119,8 @@ int cmCTestCoverageHandler::RunBullseyeSourceSummary(
 int cmCTestCoverageHandler::HandleBullseyeCoverage(
   cmCTestCoverageHandlerContainer* cont)
 {
-  const char* covfile = cmSystemTools::GetEnv("COVFILE");
-  if (!covfile || strlen(covfile) == 0) {
+  std::string covfile;
+  if (!cmSystemTools::GetEnv("COVFILE", covfile) || covfile.empty()) {
 cmCTestOptionalLog(this->CTest, HANDLER_VERBOSE_OUTPUT,
" COVFILE environment variable not found, not running "
" bullseye\n",
diff --git a/Source/CTest/cmCTestCurl.cxx b/Source/CTest/cmCTestCurl.cxx
index 6b8e5b5..b335e32 100644
--- a/Source/CTest/cmCTestCurl.cxx
+++ b/Source/CTest/cmCTestCurl.cxx
@@ -219,16 +219,18 @@ bool cmCTestCurl::HttpRequest(std::string const& url,
 
 void cmCTestCurl::SetProxyType()
 {
-  if (cmSystemTools::GetEnv("HTTP_PROXY")) {
-this->HTTPProxy = cmSystemTools::GetEnv("HTTP_PROXY");
-if (cmSystemTools::GetEnv("HTTP_PROXY_PORT")) {
+  this->HTTPProxy = "";
+  // this is the default
+  this->HTTPProxyType = CURLPROXY_HTTP;
+  this->HTTPProxyAuth = "";
+  if (cmSystemTools::GetEnv("HTTP_PROXY", this->HTTPProxy)) {
+std::string port;
+if (cmSystemTools::GetEnv("HTTP_PROXY_PORT", port)) {
   this->HTTPProxy += ":";
-  this->HTTPProxy += cmSystemTools::GetEnv("HTTP_PROXY_PORT");
+

[cmake-developers] [PATCH v3 4/7] For Windows encode process output to internally used encoding

2016-07-06 Thread Dāvis Mosāns 
Typically Windows applications (eg. MSVC compiler) use current console's
codepage for output to pipes so we need to encode that to internally used
encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE).
---
 Source/kwsys/CMakeLists.txt |  2 ++
 Source/kwsys/ProcessWin32.c | 25 -
 2 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/Source/kwsys/CMakeLists.txt b/Source/kwsys/CMakeLists.txt
index 39b03b3..65203c0 100644
--- a/Source/kwsys/CMakeLists.txt
+++ b/Source/kwsys/CMakeLists.txt
@@ -708,6 +708,8 @@ IF(KWSYS_USE_Process)
   IF(NOT UNIX)
 # Use the Windows implementation.
 SET(KWSYS_C_SRCS ${KWSYS_C_SRCS} ProcessWin32.c)
+SET_PROPERTY(SOURCE ProcessWin32.c APPEND PROPERTY COMPILE_DEFINITIONS
+  KWSYS_ENCODING_DEFAULT_CODEPAGE=${KWSYS_ENCODING_DEFAULT_CODEPAGE})
   ELSE()
 # Use the UNIX implementation.
 SET(KWSYS_C_SRCS ${KWSYS_C_SRCS} ProcessUNIX.c)
diff --git a/Source/kwsys/ProcessWin32.c b/Source/kwsys/ProcessWin32.c
index 2b93e69..153dc0b 100644
--- a/Source/kwsys/ProcessWin32.c
+++ b/Source/kwsys/ProcessWin32.c
@@ -181,7 +181,7 @@ struct kwsysProcessPipeData_s
   /* - Data managed per call to Execute - */
 
   /* Buffer for data read in this pipe's thread.  */
-  char DataBuffer[KWSYSPE_PIPE_BUFFER_SIZE];
+  char DataBuffer[KWSYSPE_PIPE_BUFFER_SIZE*2];
 
   /* The length of the data stored in the buffer.  */
   DWORD DataLength;
@@ -319,6 +319,9 @@ struct kwsysProcess_s
   /* Own handles for the child's ends of the pipes in the parent process.
  Used temporarily during process creation.  */
   HANDLE PipeChildStd[3];
+
+  /* Console's active codepage */
+  UINT codepage;
 };
 
 /*--*/
@@ -1626,6 +1629,21 @@ void kwsysProcessPipeThreadReadPipe(kwsysProcess* cp, 
kwsysProcessPipeData* td)
   KWSYSPE_DEBUG((stderr, "read closed %d\n", td->Index));
   }
 
+if (td->DataLength > 0) {
+  if (cp->codepage != KWSYS_ENCODING_DEFAULT_CODEPAGE) {
+const int wlength = MultiByteToWideChar(cp->codepage, 0, 
td->DataBuffer, td->DataLength, NULL, 0);
+wchar_t* wdata = malloc(wlength * sizeof(wchar_t));
+int r = MultiByteToWideChar(cp->codepage, 0, td->DataBuffer, 
td->DataLength, wdata, wlength);
+if (r > 0) {
+  r = WideCharToMultiByte(KWSYS_ENCODING_DEFAULT_CODEPAGE, 0, wdata, 
wlength, td->DataBuffer, KWSYSPE_PIPE_BUFFER_SIZE * 2, NULL, NULL);
+  if (r > 0) {
+td->DataLength = r;
+  }
+}
+free(wdata);
+  }
+}
+
 KWSYSPE_DEBUG((stderr, "read %d\n", td->Index));
 
 /* Wait for our turn to be handled by the main thread.  */
@@ -1761,6 +1779,11 @@ int kwsysProcessInitialize(kwsysProcess* cp)
 }
   }
 
+  cp->codepage = GetConsoleCP();
+  if (!cp->codepage) {
+cp->codepage = GetACP();
+  }
+
   return 1;
 }
 
-- 
2.9.0

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] [PATCH v3 5/7] For consoles output on Windows use our own std::streambuf

2016-07-06 Thread Dāvis Mosāns 
2016-07-07 1:36 GMT+03:00 Mike Gelfand <mike...@mikedld.com>:
> On 07/07/2016 12:42 AM, clin...@elemtech.com wrote:
>> From what I remember, WriteConsoleW doesn't support redirection to a
>> file or pipe.  I don't see an alternative in the patch for the case
>> where stdout is not attached to the console.
> The SO thread I pointed at actually contained a check for whether
> corresponding stream is linked to terminal or not. Replacing stream
> buffers unconditionally is not right.
>

Sorry, I missed this... Will fix.

2016-07-07 2:21 GMT+03:00 Mike Gelfand <mike...@mikedld.com>:
> Just a small nore: as I failed to mention this before, the approach
> taken in this patch will lead to messages to be printed to stdout and
> stderr in CMake's internal encoding (UTF-8?) when ConsoleBuf is not
> used, i.e. 1) on non-Windows and 2) on Windows with one or another
> stream redirected to file/pipe (considering corresponding check is in
> place). Seems to be the case anyway now, but maybe something to think about.
>

I know this, it always have been the case and with this patch I fix only
output to console because currently it would output internal encoding
which is unreadable for non-ASCII because console expects different codepage.

As for encoding used for files it doesn't matter because we use same one
for both writing and reading. It would matter if other applications would
want to read it. And because it's set as UTF-8 it should work fine.

> On 07/06/2016 10:12 PM, Dāvis Mosāns wrote:
>> --- a/Source/cmakemain.cxx
>> +++ b/Source/cmakemain.cxx
>> @@ -171,6 +189,16 @@ int main(int ac, char const* const* av)
>>  #ifdef CMAKE_BUILD_WITH_CMAKE
>>cmDynamicLoader::FlushCache();
>>  #endif
>> +#if defined(_WIN32)
>> +  if (cbufio) {
>> +delete cbufio;
>> +std::cout.rdbuf(coutbuf);
>> +  }
>> +  if (cbuferr) {
>> +delete cbuferr;
>> +std::cerr.rdbuf(cerrbuf);
>> +  }
>> +#endif
>>return ret;
>>  }
>>
> If exception was thrown in the beginning of main (as it seems you expect
> one there), `coutbuf` and/or `cerrbuf` could be nullptr, and passing
> nullptr here will fail.

coutbuf can be null only if

cbufio = new cmsys::ConsoleBuf();

threw an exception in which case cbufio will be null and so it won't be set
here. And same for cerrbuf, it will be set if cbuferr is set in which case it
means there weren't any exception. While I think it's not needed, I can
add check for it and set rdbuf only when not null.

>
>> +this->setg((char_type *)m_ibuffer.data(), (char_type 
>> *)m_ibuffer.data(), (char_type *)m_ibuffer.data() + m_ibuffer.size());
>> +this->setp((char_type *)m_obuffer.data(), (char_type 
>> *)m_obuffer.data() + m_obuffer.size());
> This pattern over and over again. Needs to be put into separate member
> function, otherwise some day someone will mix "i" and "o" and wonder why
> doesn't it work.
>
> Also, I don't see that many calls to set(g|p) in that SO snipped, are
> they all really needed?
>

There aren't really that many calls, we set both once in constructor,
both once in sync and setg twice (differently) in underflow, because
two cases:

"The function may update gptr, egptr and eback pointers to define the
location of newly loaded data (if any). On failure, the function ensures
that either gptr() == nullptr or gptr() == egptr"

from http://en.cppreference.com/w/cpp/io/basic_streambuf/underflow
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] [PATCH v3 7/7] Add MinGW support for FStream

2016-07-06 Thread Dāvis Mosāns 
2016-07-07 3:51 GMT+03:00 Mike Gelfand <mike...@mikedld.com>:
> On 07/07/2016 03:33 AM, Dāvis Mosāns wrote:
>> 2016-07-07 1:22 GMT+03:00 Mike Gelfand <mike...@mikedld.com>:
>>>> @@ -92,19 +159,26 @@ namespace @KWSYS_NAMESPACE@
>>>> [snip]
>>>>private:
>>>>  internal_buffer_type* buf_;
>>>> +#if !defined(_MSC_VER)
>>>> +FILE *file_ = 0;
>>>> +#endif
>>>>  };
>>>>
>>>>  template >
>>> In-place member initialization... As Brad wrote earlier, using C++11 in
>>> KWSys is not advised.
>>>
>> This was already there but only used for _MSC_VER >= 1400
>> Maybe could just guard this with  __cplusplus >= 201103L and not support
>> Unicode for older, it would be much easier...
>
> Apologies, don't quite get your response. Maybe the quote was too big; I
> surely meant to point to "FILE *file_ = 0;" line, not the template
> declaration or something else.


Oh sorry, I misunderstood you earlier and yeah I see, will fix.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH v4] For consoles output on Windows use our own std::streambuf

2016-07-09 Thread Dāvis Mosāns 
Currently Microsoft's C++ libraries implementation of std::cout/cerr
can't output Unicode characters but only ASCII or ANSI if locale is set
so we implement and use our own ConsoleBuf which can output Unicode
characters to console and it doesn't matter what locale or console's
codepage is set.
---
 CMakeLists.txt |   1 +
 Source/cmakemain.cxx   |  27 
 Source/kwsys/CMakeLists.txt|   6 +-
 Source/kwsys/ConsoleBuf.hxx.in | 282 +
 bootstrap  |   1 +
 5 files changed, 316 insertions(+), 1 deletion(-)
 create mode 100644 Source/kwsys/ConsoleBuf.hxx.in

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ae5990e..792b5a5 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -258,6 +258,7 @@ macro (CMAKE_BUILD_UTILITIES)
   set(KWSYS_USE_MD5 1)
   set(KWSYS_USE_Process 1)
   set(KWSYS_USE_CommandLineArguments 1)
+  set(KWSYS_USE_ConsoleBuf 1)
   set(KWSYS_HEADER_ROOT ${CMake_BINARY_DIR}/Source)
   set(KWSYS_INSTALL_DOC_DIR "${CMAKE_DOC_DIR}")
   add_subdirectory(Source/kwsys)
diff --git a/Source/cmakemain.cxx b/Source/cmakemain.cxx
index 521a5bf..95eb154 100644
--- a/Source/cmakemain.cxx
+++ b/Source/cmakemain.cxx
@@ -26,6 +26,7 @@
 #include "cmake.h"
 #include "cmcmd.h"
 #include 
+#include 
 
 #ifdef CMAKE_BUILD_WITH_CMAKE
 static const char* cmDocumentationName[][2] = {
@@ -153,6 +154,22 @@ static void cmakemainProgressCallback(const char* m, float 
prog,
 
 int main(int ac, char const* const* av)
 {
+#if defined(_WIN32)
+  // Replace streambuf so we can output Unicode to console
+  cmsys::ConsoleBuf *cbufio = CM_NULLPTR;
+  cmsys::ConsoleBuf *cbuferr = CM_NULLPTR;
+  std::streambuf *coutbuf = std::cout.rdbuf();
+  std::streambuf *cerrbuf = std::cerr.rdbuf();
+  try {
+cbufio = new cmsys::ConsoleBuf();
+coutbuf = std::cout.rdbuf(cbufio);
+cbuferr = new cmsys::ConsoleBuf(true);
+cerrbuf = std::cerr.rdbuf(cbuferr);
+  } catch (const std::runtime_error& ex) {
+std::cerr << "Failed to create ConsoleBuf!" << std::endl
+  << ex.what() << std::endl;
+  };
+#endif
   cmsys::Encoding::CommandLineArguments args =
 cmsys::Encoding::CommandLineArguments::Main(ac, av);
   ac = args.argc();
@@ -171,6 +188,16 @@ int main(int ac, char const* const* av)
 #ifdef CMAKE_BUILD_WITH_CMAKE
   cmDynamicLoader::FlushCache();
 #endif
+#if defined(_WIN32)
+  if (cbufio) {
+delete cbufio;
+std::cout.rdbuf(coutbuf);
+  }
+  if (cbuferr) {
+delete cbuferr;
+std::cerr.rdbuf(cerrbuf);
+  }
+#endif
   return ret;
 }
 
diff --git a/Source/kwsys/CMakeLists.txt b/Source/kwsys/CMakeLists.txt
index 65203c0..33a97e6 100644
--- a/Source/kwsys/CMakeLists.txt
+++ b/Source/kwsys/CMakeLists.txt
@@ -123,6 +123,7 @@ IF(KWSYS_STANDALONE OR CMake_SOURCE_DIR)
   SET(KWSYS_USE_FStream 1)
   SET(KWSYS_USE_String 1)
   SET(KWSYS_USE_SystemInformation 1)
+  SET(KWSYS_USE_ConsoleBuf 1)
 ENDIF()
 
 # Enforce component dependencies.
@@ -154,6 +155,9 @@ ENDIF()
 IF(KWSYS_USE_FStream)
   SET(KWSYS_USE_Encoding 1)
 ENDIF()
+IF(KWSYS_USE_ConsoleBuf)
+  SET(KWSYS_USE_Encoding 1)
+ENDIF()
 
 # Setup the large file support default.
 IF(KWSYS_LFS_DISABLE)
@@ -668,7 +672,7 @@ SET(KWSYS_HXX_FILES Configure String
 # Add selected C++ classes.
 SET(cppclasses
   Directory DynamicLoader Encoding Glob RegularExpression SystemTools
-  CommandLineArguments IOStream FStream SystemInformation
+  CommandLineArguments IOStream FStream SystemInformation ConsoleBuf
   )
 FOREACH(cpp ${cppclasses})
   IF(KWSYS_USE_${cpp})
diff --git a/Source/kwsys/ConsoleBuf.hxx.in b/Source/kwsys/ConsoleBuf.hxx.in
new file mode 100644
index 000..4ee37dd
--- /dev/null
+++ b/Source/kwsys/ConsoleBuf.hxx.in
@@ -0,0 +1,282 @@
+/*
+  KWSys - Kitware System Library
+  Copyright 2000-2016 Kitware, Inc., Insight Software Consortium
+
+  Distributed under the OSI-approved BSD License (the "License");
+  see accompanying file Copyright.txt for details.
+
+  This software is distributed WITHOUT ANY WARRANTY; without even the
+  implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+  See the License for more information.
+*/
+#ifndef @KWSYS_NAMESPACE@_ConsoleBuf_hxx
+#define @KWSYS_NAMESPACE@_ConsoleBuf_hxx
+
+#include <@KWSYS_NAMESPACE@/Configure.hxx>
+#include <@KWSYS_NAMESPACE@/Encoding.hxx>
+#include 
+#include 
+#include 
+#include 
+#include 
+
+#if defined(_WIN32)
+#  include 
+#  if __cplusplus >= 201103L
+#include 
+#  endif
+#endif
+
+namespace @KWSYS_NAMESPACE@
+{
+#if defined(_WIN32)
+
+  template >
+  class @KWSYS_NAMESPACE@_EXPORT BasicConsoleBuf : public 
std::basic_streambuf {
+public:
+  typedef typename Traits::int_type int_type;
+  typedef typename Traits::char_type char_type;
+
+  BasicConsoleBuf(const bool err = false) :
+

[cmake-developers] [PATCH v4 1/4] On Windows use correct encoding for SystemTools::GetEnv

2016-07-07 Thread Dāvis Mosāns 
On Windows getenv (and putenv) uses ANSI codepage so it needs to be encoded
to internally used encoding (eg. UTF-8). Here we use _wgetenv (and _wputenv)
instead and encode that.

Also add SystemTools::HasEnv function.
---
 Source/kwsys/SystemTools.cxx| 78 +++--
 Source/kwsys/SystemTools.hxx.in |  4 +++
 2 files changed, 72 insertions(+), 10 deletions(-)

diff --git a/Source/kwsys/SystemTools.cxx b/Source/kwsys/SystemTools.cxx
index c6e668d..fc1d756 100644
--- a/Source/kwsys/SystemTools.cxx
+++ b/Source/kwsys/SystemTools.cxx
@@ -406,6 +406,9 @@ struct SystemToolsPathCaseCmp
 class SystemToolsPathCaseMap:
   public std::map {};
+
+class SystemToolsEnvMap :
+public std::map {};
 #endif
 
 // adds the elements of the env variable path to the arg passed in
@@ -458,7 +461,21 @@ void SystemTools::GetPath(std::vector& path, 
const char* env)
 
 const char* SystemTools::GetEnv(const char* key)
 {
-  return getenv(key);
+  const char *v;
+#if defined(_WIN32)
+  const std::string env = SystemTools::GetEnv(key);
+  const std::string skey = key;
+  SystemToolsEnvMap::iterator i = EnvMap->find(skey);
+  if (i != EnvMap->end()) {
+i->second = env;
+  } else {
+i = EnvMap->insert(SystemToolsEnvMap::value_type(skey, env)).first;
+  }
+  v = i->second.c_str();
+#else
+  v = getenv(key);
+#endif
+  return v;
 }
 
 const char* SystemTools::GetEnv(const std::string& key)
@@ -468,16 +485,21 @@ const char* SystemTools::GetEnv(const std::string& key)
 
 bool SystemTools::GetEnv(const char* key, std::string& result)
 {
+#if defined(_WIN32)
+  const std::wstring wkey = Encoding::ToWide(key);
+  const wchar_t* wv = _wgetenv(wkey.c_str());
+  if (wv) {
+result = Encoding::ToNarrow(wv);
+return true;
+  }
+#else
   const char* v = getenv(key);
-  if(v)
-{
+  if (v) {
 result = v;
 return true;
-}
-  else
-{
-return false;
-}
+  }
+#endif
+  return false;
 }
 
 bool SystemTools::GetEnv(const std::string& key, std::string& result)
@@ -485,6 +507,22 @@ bool SystemTools::GetEnv(const std::string& key, 
std::string& result)
   return SystemTools::GetEnv(key.c_str(), result);
 }
 
+bool SystemTools::HasEnv(const char* key)
+{
+#if defined(_WIN32)
+  const std::wstring wkey = Encoding::ToWide(key);
+  const wchar_t* v = _wgetenv(wkey.c_str());
+#else
+  const char* v = getenv(key);
+#endif
+  return v != 0;
+}
+
+bool SystemTools::HasEnv(const std::string& key)
+{
+  return SystemTools::HasEnv(key.c_str());
+}
+
 //
 
 #if defined(__CYGWIN__) || defined(__GLIBC__)
@@ -533,13 +571,25 @@ static int kwsysUnPutEnv(const std::string& env)
 # ifdef KWSYS_PUTENV_EMPTY
   buf[len] = '=';
   buf[len+1] = 0;
-  if(putenv(buf) < 0)
+#if defined(_WIN32)
+  const std::wstring wbuf = Encoding::ToWide(buf);
+  const int r = _wputenv(wbuf.c_str());
+#else
+  const int r = putenv(buf);
+#endif
+  if(r < 0)
 {
 err = errno;
 }
 # else
   buf[len] = 0;
-  if(putenv(buf) < 0 && errno != EINVAL)
+#if defined(_WIN32)
+  const std::wstring wbuf = Encoding::ToWide(buf);
+  const int r = _wputenv(wbuf.c_str());
+#else
+  const int r = putenv(buf);
+#endif
+  if(r < 0 && errno != EINVAL)
 {
 err = errno;
 }
@@ -679,7 +729,12 @@ public:
 static_cast(oldEnv);
 char* newEnv = strdup(env);
 this->insert(newEnv);
+#if defined(_WIN32)
+const std::wstring wEnv = Encoding::ToWide(newEnv);
+return _wputenv(wEnv.c_str()) == 0;
+#else
 return putenv(newEnv) == 0;
+#endif
 }
   bool UnPut(const char* env)
 {
@@ -5371,6 +5426,7 @@ static unsigned int SystemToolsManagerCount;
 SystemToolsTranslationMap *SystemTools::TranslationMap;
 #ifdef _WIN32
 SystemToolsPathCaseMap *SystemTools::PathCaseMap;
+SystemToolsEnvMap *SystemTools::EnvMap;
 #endif
 #ifdef __CYGWIN__
 SystemToolsTranslationMap *SystemTools::Cyg2Win32Map;
@@ -5421,6 +5477,7 @@ void SystemTools::ClassInitialize()
   SystemTools::TranslationMap = new SystemToolsTranslationMap;
 #ifdef _WIN32
   SystemTools::PathCaseMap = new SystemToolsPathCaseMap;
+  SystemTools::EnvMap = new SystemToolsEnvMap;
 #endif
 #ifdef __CYGWIN__
   SystemTools::Cyg2Win32Map = new SystemToolsTranslationMap;
@@ -5480,6 +5537,7 @@ void SystemTools::ClassFinalize()
   delete SystemTools::TranslationMap;
 #ifdef _WIN32
   delete SystemTools::PathCaseMap;
+  delete SystemTools::EnvMap;
 #endif
 #ifdef __CYGWIN__
   delete SystemTools::Cyg2Win32Map;
diff --git a/Source/kwsys/SystemTools.hxx.in b/Source/kwsys/SystemTools.hxx.in
index bba5a5c..8f01e75 100644
--- a/Source/kwsys/SystemTools.hxx.in
+++ b/Source/kwsys/SystemTools.hxx.in
@@ -53,6 +53,7 @@ namespace @KWSYS_NAMESPACE@
 
 class SystemToolsTranslationMap;
 class SystemToolsPathCaseMap;
+class SystemToolsEnvMap;
 
 /** \class SystemToolsManager
  * \brief Use to make sure

[cmake-developers] [PATCH v4 4/4] For Windows encode process output to internally used encoding

2016-07-07 Thread Dāvis Mosāns 
Typically Windows applications (eg. MSVC compiler) use current console's
codepage for output to pipes so we need to encode that to internally used
encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE).
---
 Source/cmExecProgramCommand.cxx|  1 +
 Source/cmExecuteProcessCommand.cxx |  1 +
 Source/cmProcessTools.cxx  |  2 ++
 Source/cmSystemTools.cxx   |  3 +++
 Source/kwsys/CMakeLists.txt|  2 ++
 Source/kwsys/Process.h.in  | 13 +
 Source/kwsys/ProcessUNIX.c |  6 ++
 Source/kwsys/ProcessWin32.c| 33 -
 Source/kwsys/SystemInformation.cxx |  1 +
 9 files changed, 61 insertions(+), 1 deletion(-)

diff --git a/Source/cmExecProgramCommand.cxx b/Source/cmExecProgramCommand.cxx
index 58bbc31..92b89fc 100644
--- a/Source/cmExecProgramCommand.cxx
+++ b/Source/cmExecProgramCommand.cxx
@@ -220,6 +220,7 @@ bool cmExecProgramCommand::RunCommand(const char* command, 
std::string& output,
   char* data;
   int p;
   while ((p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) {
+cmsysProcess_DecodeTextOutput(cp, , );
 if (p == cmsysProcess_Pipe_STDOUT || p == cmsysProcess_Pipe_STDERR) {
   if (verbose) {
 cmSystemTools::Stdout(data, length);
diff --git a/Source/cmExecuteProcessCommand.cxx 
b/Source/cmExecuteProcessCommand.cxx
index d97b25f..b13fb2e 100644
--- a/Source/cmExecuteProcessCommand.cxx
+++ b/Source/cmExecuteProcessCommand.cxx
@@ -229,6 +229,7 @@ bool 
cmExecuteProcessCommand::InitialPass(std::vector const& args,
   char* data;
   int p;
   while ((p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) {
+cmsysProcess_DecodeTextOutput(cp, , );
 // Put the output in the right place.
 if (p == cmsysProcess_Pipe_STDOUT && !output_quiet) {
   if (output_variable.empty()) {
diff --git a/Source/cmProcessTools.cxx b/Source/cmProcessTools.cxx
index 34b8df2..8ab8070 100644
--- a/Source/cmProcessTools.cxx
+++ b/Source/cmProcessTools.cxx
@@ -23,10 +23,12 @@ void cmProcessTools::RunProcess(struct cmsysProcess_s* cp, 
OutputParser* out,
   while ((out || err) &&
  (p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) {
 if (out && p == cmsysProcess_Pipe_STDOUT) {
+  cmsysProcess_DecodeTextOutput(cp, , );
   if (!out->Process(data, length)) {
 out = CM_NULLPTR;
   }
 } else if (err && p == cmsysProcess_Pipe_STDERR) {
+  cmsysProcess_DecodeTextOutput(cp, , );
   if (!err->Process(data, length)) {
 err = CM_NULLPTR;
   }
diff --git a/Source/cmSystemTools.cxx b/Source/cmSystemTools.cxx
index 9740ef7..aeb5471 100644
--- a/Source/cmSystemTools.cxx
+++ b/Source/cmSystemTools.cxx
@@ -616,6 +616,7 @@ bool 
cmSystemTools::RunSingleCommand(std::vector const& command,
   (captureStdOut || captureStdErr || outputflag != OUTPUT_NONE)) {
 while ((pipe = cmsysProcess_WaitForData(cp, , , CM_NULLPTR)) >
0) {
+  cmsysProcess_DecodeTextOutput(cp, , );
   // Translate NULL characters in the output into valid text.
   // Visual Studio 7 puts these characters in the output of its
   // build process.
@@ -1689,11 +1690,13 @@ int cmSystemTools::WaitForLine(cmsysProcess* process, 
std::string& line,
   return pipe;
 } else if (pipe == cmsysProcess_Pipe_STDOUT) {
   // Append to the stdout buffer.
+  cmsysProcess_DecodeTextOutput(process, , );
   std::vector::size_type size = out.size();
   out.insert(out.end(), data, data + length);
   outiter = out.begin() + size;
 } else if (pipe == cmsysProcess_Pipe_STDERR) {
   // Append to the stderr buffer.
+  cmsysProcess_DecodeTextOutput(process, , );
   std::vector::size_type size = err.size();
   err.insert(err.end(), data, data + length);
   erriter = err.begin() + size;
diff --git a/Source/kwsys/CMakeLists.txt b/Source/kwsys/CMakeLists.txt
index 39b03b3..65203c0 100644
--- a/Source/kwsys/CMakeLists.txt
+++ b/Source/kwsys/CMakeLists.txt
@@ -708,6 +708,8 @@ IF(KWSYS_USE_Process)
   IF(NOT UNIX)
 # Use the Windows implementation.
 SET(KWSYS_C_SRCS ${KWSYS_C_SRCS} ProcessWin32.c)
+SET_PROPERTY(SOURCE ProcessWin32.c APPEND PROPERTY COMPILE_DEFINITIONS
+  KWSYS_ENCODING_DEFAULT_CODEPAGE=${KWSYS_ENCODING_DEFAULT_CODEPAGE})
   ELSE()
 # Use the UNIX implementation.
 SET(KWSYS_C_SRCS ${KWSYS_C_SRCS} ProcessUNIX.c)
diff --git a/Source/kwsys/Process.h.in b/Source/kwsys/Process.h.in
index 96563a2..0c79b47 100644
--- a/Source/kwsys/Process.h.in
+++ b/Source/kwsys/Process.h.in
@@ -67,6 +67,7 @@
 # define kwsysProcess_Execute   kwsys_ns(Process_Execute)
 # define kwsysProcess_Disownkwsys_ns(Process_Disown)
 # define kwsysProcess_WaitForData   kwsys_ns(Process_WaitForData)
+# define kwsysProcess_DecodeTextOutput  
kwsys_ns(Process_DecodeTextOutput)
 # define kwsysProcess_Pipes_e   kwsys_ns(Process_Pipes_e)
 # define kwsysProcess_Pipe_None

[cmake-developers] [PATCH v4 3/4] Use SystemTools::GetEnv and HasEnv functions

2016-07-07 Thread Dāvis Mosāns 
---
 Source/CPack/cmCPackGenerator.cxx   |  8 +++---
 Source/CTest/cmCTestCoverageHandler.cxx | 12 -
 Source/CTest/cmCTestCurl.cxx| 27 ++--
 Source/CTest/cmCTestMultiProcessHandler.cxx |  8 +++---
 Source/cmBuildCommand.cxx   | 25 +--
 Source/cmCLocaleEnvironmentScope.cxx|  5 ++--
 Source/cmCTest.cxx  | 11 +
 Source/cmCommandArgumentParserHelper.cxx|  8 +++---
 Source/cmConditionEvaluator.cxx |  2 +-
 Source/cmExportCommand.cxx  |  5 ++--
 Source/cmExtraEclipseCDT4Generator.cxx  |  9 ---
 Source/cmFileCommand.cxx| 11 +
 Source/cmFindPackageCommand.cxx |  4 +--
 Source/cmGlobalVisualStudio7Generator.cxx   |  6 ++---
 Source/cmMakefile.cxx   |  5 +++-
 Source/cmNinjaTargetGenerator.cxx   |  2 +-
 Source/cmQtAutoGenerators.cxx   |  2 +-
 Source/cmSetCommand.cxx |  7 +++---
 Source/cmState.cxx  |  5 ++--
 Source/cmSystemTools.cxx|  6 ++---
 Source/cmTimestamp.cxx  |  7 +++---
 Source/cmUtils.hxx  | 26 
 Source/cmake.cxx| 21 +---
 Source/cmcmd.cxx| 16 +---
 Source/kwsys/SystemInformation.cxx  | 20 +++
 Source/kwsys/SystemTools.cxx| 38 +++--
 Source/kwsys/testSystemTools.cxx|  9 ---
 27 files changed, 156 insertions(+), 149 deletions(-)
 create mode 100644 Source/cmUtils.hxx

diff --git a/Source/CPack/cmCPackGenerator.cxx 
b/Source/CPack/cmCPackGenerator.cxx
index df8bb0f..76609e1 100644
--- a/Source/CPack/cmCPackGenerator.cxx
+++ b/Source/CPack/cmCPackGenerator.cxx
@@ -1074,11 +1074,11 @@ const char* cmCPackGenerator::GetInstallPath()
 return this->InstallPath.c_str();
   }
 #if defined(_WIN32) && !defined(__CYGWIN__)
-  const char* prgfiles = cmsys::SystemTools::GetEnv("ProgramFiles");
-  const char* sysDrive = cmsys::SystemTools::GetEnv("SystemDrive");
-  if (prgfiles) {
+  std::string prgfiles;
+  std::string sysDrive;
+  if (cmsys::SystemTools::GetEnv("ProgramFiles", prgfiles)) {
 this->InstallPath = prgfiles;
-  } else if (sysDrive) {
+  } else if (cmsys::SystemTools::GetEnv("SystemDrive", sysDrive)) {
 this->InstallPath = sysDrive;
 this->InstallPath += "/Program Files";
   } else {
diff --git a/Source/CTest/cmCTestCoverageHandler.cxx 
b/Source/CTest/cmCTestCoverageHandler.cxx
index 7102533..9410a52 100644
--- a/Source/CTest/cmCTestCoverageHandler.cxx
+++ b/Source/CTest/cmCTestCoverageHandler.cxx
@@ -727,10 +727,7 @@ int cmCTestCoverageHandler::HandleCoberturaCoverage(
   // if it doesn't exist or is empty, assume the
   // binary directory is used.
   std::string coverageXMLFile;
-  const char* covDir = cmSystemTools::GetEnv("COBERTURADIR");
-  if (covDir && strlen(covDir) != 0) {
-coverageXMLFile = std::string(covDir);
-  } else {
+  if (!cmSystemTools::GetEnv("COBERTURADIR", coverageXMLFile) || 
coverageXMLFile.empty()) {
 coverageXMLFile = this->CTest->GetBinaryDir();
   }
   // build the find file string with the directory from above
@@ -791,7 +788,8 @@ struct cmCTestCoverageHandlerLocale
 {
   cmCTestCoverageHandlerLocale()
   {
-if (const char* l = cmSystemTools::GetEnv("LC_ALL")) {
+std::string l;
+if (cmSystemTools::GetEnv("LC_ALL", l)) {
   lc_all = l;
 }
 if (lc_all != "C") {
@@ -2121,8 +2119,8 @@ int cmCTestCoverageHandler::RunBullseyeSourceSummary(
 int cmCTestCoverageHandler::HandleBullseyeCoverage(
   cmCTestCoverageHandlerContainer* cont)
 {
-  const char* covfile = cmSystemTools::GetEnv("COVFILE");
-  if (!covfile || strlen(covfile) == 0) {
+  std::string covfile;
+  if (!cmSystemTools::GetEnv("COVFILE", covfile) || covfile.empty()) {
 cmCTestOptionalLog(this->CTest, HANDLER_VERBOSE_OUTPUT,
" COVFILE environment variable not found, not running "
" bullseye\n",
diff --git a/Source/CTest/cmCTestCurl.cxx b/Source/CTest/cmCTestCurl.cxx
index 6b8e5b5..b335e32 100644
--- a/Source/CTest/cmCTestCurl.cxx
+++ b/Source/CTest/cmCTestCurl.cxx
@@ -219,16 +219,18 @@ bool cmCTestCurl::HttpRequest(std::string const& url,
 
 void cmCTestCurl::SetProxyType()
 {
-  if (cmSystemTools::GetEnv("HTTP_PROXY")) {
-this->HTTPProxy = cmSystemTools::GetEnv("HTTP_PROXY");
-if (cmSystemTools::GetEnv("HTTP_PROXY_PORT")) {
+  this->HTTPProxy = "";
+  // this is the default
+  this->HTTPProxyType = CURLPROXY_HTTP;
+  this->HTTPProxyAuth = "";
+  if (cmSystemTools::GetEnv("HTTP_PROXY", this->HTTPProxy)) {
+std::string port;
+if (cmSystemTools::GetEnv("HTTP_PROXY_PORT", port)) {
   this->HTTPProxy += ":";
-  this->HTTPProxy += cmSystemTools::GetEnv("HTTP_PROXY_PORT");
+

[cmake-developers] [PATCH v4 fixed 1/4] On Windows use correct encoding for SystemTools::GetEnv

2016-07-07 Thread Dāvis Mosāns 
On Windows getenv (and putenv) uses ANSI codepage so it needs to be encoded
to internally used encoding (eg. UTF-8). Here we use _wgetenv (and _wputenv)
instead and encode that.

Also add SystemTools::HasEnv function.
---
 Source/kwsys/SystemTools.cxx| 80 +++--
 Source/kwsys/SystemTools.hxx.in |  4 +++
 2 files changed, 74 insertions(+), 10 deletions(-)

diff --git a/Source/kwsys/SystemTools.cxx b/Source/kwsys/SystemTools.cxx
index c6e668d..5bcf0d0 100644
--- a/Source/kwsys/SystemTools.cxx
+++ b/Source/kwsys/SystemTools.cxx
@@ -406,6 +406,9 @@ struct SystemToolsPathCaseCmp
 class SystemToolsPathCaseMap:
   public std::map {};
+
+class SystemToolsEnvMap :
+public std::map {};
 #endif
 
 // adds the elements of the env variable path to the arg passed in
@@ -458,7 +461,23 @@ void SystemTools::GetPath(std::vector& path, 
const char* env)
 
 const char* SystemTools::GetEnv(const char* key)
 {
-  return getenv(key);
+  const char *v = 0;
+#if defined(_WIN32)
+  std::string env;
+  if (SystemTools::GetEnv(key, env)) {
+const std::string skey = key;
+SystemToolsEnvMap::iterator i = EnvMap->find(skey);
+if (i != EnvMap->end()) {
+  i->second = env;
+} else {
+  i = EnvMap->insert(SystemToolsEnvMap::value_type(skey, env)).first;
+}
+v = i->second.c_str();
+  }
+#else
+  v = getenv(key);
+#endif
+  return v;
 }
 
 const char* SystemTools::GetEnv(const std::string& key)
@@ -468,16 +487,21 @@ const char* SystemTools::GetEnv(const std::string& key)
 
 bool SystemTools::GetEnv(const char* key, std::string& result)
 {
+#if defined(_WIN32)
+  const std::wstring wkey = Encoding::ToWide(key);
+  const wchar_t* wv = _wgetenv(wkey.c_str());
+  if (wv) {
+result = Encoding::ToNarrow(wv);
+return true;
+  }
+#else
   const char* v = getenv(key);
-  if(v)
-{
+  if (v) {
 result = v;
 return true;
-}
-  else
-{
-return false;
-}
+  }
+#endif
+  return false;
 }
 
 bool SystemTools::GetEnv(const std::string& key, std::string& result)
@@ -485,6 +509,22 @@ bool SystemTools::GetEnv(const std::string& key, 
std::string& result)
   return SystemTools::GetEnv(key.c_str(), result);
 }
 
+bool SystemTools::HasEnv(const char* key)
+{
+#if defined(_WIN32)
+  const std::wstring wkey = Encoding::ToWide(key);
+  const wchar_t* v = _wgetenv(wkey.c_str());
+#else
+  const char* v = getenv(key);
+#endif
+  return v != 0;
+}
+
+bool SystemTools::HasEnv(const std::string& key)
+{
+  return SystemTools::HasEnv(key.c_str());
+}
+
 //
 
 #if defined(__CYGWIN__) || defined(__GLIBC__)
@@ -533,13 +573,25 @@ static int kwsysUnPutEnv(const std::string& env)
 # ifdef KWSYS_PUTENV_EMPTY
   buf[len] = '=';
   buf[len+1] = 0;
-  if(putenv(buf) < 0)
+#if defined(_WIN32)
+  const std::wstring wbuf = Encoding::ToWide(buf);
+  const int r = _wputenv(wbuf.c_str());
+#else
+  const int r = putenv(buf);
+#endif
+  if(r < 0)
 {
 err = errno;
 }
 # else
   buf[len] = 0;
-  if(putenv(buf) < 0 && errno != EINVAL)
+#if defined(_WIN32)
+  const std::wstring wbuf = Encoding::ToWide(buf);
+  const int r = _wputenv(wbuf.c_str());
+#else
+  const int r = putenv(buf);
+#endif
+  if(r < 0 && errno != EINVAL)
 {
 err = errno;
 }
@@ -679,7 +731,12 @@ public:
 static_cast(oldEnv);
 char* newEnv = strdup(env);
 this->insert(newEnv);
+#if defined(_WIN32)
+const std::wstring wEnv = Encoding::ToWide(newEnv);
+return _wputenv(wEnv.c_str()) == 0;
+#else
 return putenv(newEnv) == 0;
+#endif
 }
   bool UnPut(const char* env)
 {
@@ -5371,6 +5428,7 @@ static unsigned int SystemToolsManagerCount;
 SystemToolsTranslationMap *SystemTools::TranslationMap;
 #ifdef _WIN32
 SystemToolsPathCaseMap *SystemTools::PathCaseMap;
+SystemToolsEnvMap *SystemTools::EnvMap;
 #endif
 #ifdef __CYGWIN__
 SystemToolsTranslationMap *SystemTools::Cyg2Win32Map;
@@ -5421,6 +5479,7 @@ void SystemTools::ClassInitialize()
   SystemTools::TranslationMap = new SystemToolsTranslationMap;
 #ifdef _WIN32
   SystemTools::PathCaseMap = new SystemToolsPathCaseMap;
+  SystemTools::EnvMap = new SystemToolsEnvMap;
 #endif
 #ifdef __CYGWIN__
   SystemTools::Cyg2Win32Map = new SystemToolsTranslationMap;
@@ -5480,6 +5539,7 @@ void SystemTools::ClassFinalize()
   delete SystemTools::TranslationMap;
 #ifdef _WIN32
   delete SystemTools::PathCaseMap;
+  delete SystemTools::EnvMap;
 #endif
 #ifdef __CYGWIN__
   delete SystemTools::Cyg2Win32Map;
diff --git a/Source/kwsys/SystemTools.hxx.in b/Source/kwsys/SystemTools.hxx.in
index bba5a5c..8f01e75 100644
--- a/Source/kwsys/SystemTools.hxx.in
+++ b/Source/kwsys/SystemTools.hxx.in
@@ -53,6 +53,7 @@ namespace @KWSYS_NAMESPACE@
 
 class SystemToolsTranslationMap;
 class SystemToolsPathCaseMap;
+class SystemToolsEnvMap;
 
 /** \class SystemToolsManager
  *

[cmake-developers] [PATCH v4 2/4] Deprecate const char* SystemTools::GetEnv function

2016-07-07 Thread Dāvis Mosāns 
---
 Source/kwsys/SystemTools.hxx.in | 14 --
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/Source/kwsys/SystemTools.hxx.in b/Source/kwsys/SystemTools.hxx.in
index 8f01e75..f6fc282 100644
--- a/Source/kwsys/SystemTools.hxx.in
+++ b/Source/kwsys/SystemTools.hxx.in
@@ -26,6 +26,16 @@
 # include  // For access permissions for use with access()
 #endif
 
+#if __cplusplus >= 201402L
+# define DEPRECATED [[deprecated]]
+#elif defined(__GNUC__)
+# define DEPRECATED __attribute__ ((deprecated))
+#elif defined(_MSC_VER)
+# define DEPRECATED __declspec(deprecated)
+#else
+# define DEPRECATED
+#endif
+
 // Required for va_list
 #include 
 // Required for FILE*
@@ -840,8 +850,8 @@ public:
   /**
* Read an environment variable
*/
-  static const char* GetEnv(const char* key);
-  static const char* GetEnv(const std::string& key);
+  DEPRECATED static const char* GetEnv(const char* key);
+  DEPRECATED static const char* GetEnv(const std::string& key);
   static bool GetEnv(const char* key, std::string& result);
   static bool GetEnv(const std::string& key, std::string& result);
   static bool HasEnv(const char* key);
-- 
2.9.0

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] [PATCH v4 4/4] For Windows encode process output to internally used encoding

2016-08-02 Thread Dāvis Mosāns 
2016-08-02 20:11 GMT+03:00 Brad King :
>
> How are we to know the encoding being produced by the child?
>

There isn't any reliable way to detect it, we just have to know what particular
application uses. Also there aren't any standard or API to determine it but
generally most applications use console's code page. Of course not all, it
could be OEM, ANSI or something else.

If application uses console's code page for pipes then app.exe | echo will
output correctly in cmd and piping in other application which also uses
it will work.

Here our issue is that when CMake checks for compiler path it gets output
from MSVC and it will give this path in console's codepage which will be
incorrectly interpreted as UTF-8 so we need to decode it first.

If there will some other application which uses different encoding then it will
need separate handling for that.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH v5] For Windows encode process output to internally used encoding

2016-07-21 Thread Dāvis Mosāns 
Typically Windows applications (eg. MSVC compiler) use current console's
codepage for output to pipes so we need to encode that to internally used
encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE).
---
 Source/cmExecProgramCommand.cxx|  3 ++
 Source/cmExecuteProcessCommand.cxx | 11 -
 Source/cmProcessTools.cxx  |  9 +++-
 Source/cmSystemTools.cxx   | 11 -
 Source/kwsys/CMakeLists.txt|  2 +-
 Source/kwsys/ProcessOutput.hxx.in  | 94 ++
 Source/kwsys/SystemInformation.cxx |  4 ++
 bootstrap  |  1 +
 8 files changed, 128 insertions(+), 7 deletions(-)
 create mode 100644 Source/kwsys/ProcessOutput.hxx.in

diff --git a/Source/cmExecProgramCommand.cxx b/Source/cmExecProgramCommand.cxx
index 58bbc31..217cda1 100644
--- a/Source/cmExecProgramCommand.cxx
+++ b/Source/cmExecProgramCommand.cxx
@@ -14,6 +14,7 @@
 #include "cmSystemTools.h"
 
 #include 
+#include 
 
 // cmExecProgramCommand
 bool cmExecProgramCommand::InitialPass(std::vector const& args,
@@ -219,6 +220,7 @@ bool cmExecProgramCommand::RunCommand(const char* command, 
std::string& output,
   int length;
   char* data;
   int p;
+  cmsys::ProcessOutput processOutput;
   while ((p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) {
 if (p == cmsysProcess_Pipe_STDOUT || p == cmsysProcess_Pipe_STDERR) {
   if (verbose) {
@@ -230,6 +232,7 @@ bool cmExecProgramCommand::RunCommand(const char* command, 
std::string& output,
 
   // All output has been read.  Wait for the process to exit.
   cmsysProcess_WaitForExit(cp, CM_NULLPTR);
+  processOutput.DecodeText(output, output);
 
   // Check the result of running the process.
   std::string msg;
diff --git a/Source/cmExecuteProcessCommand.cxx 
b/Source/cmExecuteProcessCommand.cxx
index d97b25f..bdefa8b 100644
--- a/Source/cmExecuteProcessCommand.cxx
+++ b/Source/cmExecuteProcessCommand.cxx
@@ -14,6 +14,7 @@
 #include "cmSystemTools.h"
 
 #include 
+#include 
 
 #include  /* isspace */
 
@@ -228,17 +229,21 @@ bool 
cmExecuteProcessCommand::InitialPass(std::vector const& args,
   int length;
   char* data;
   int p;
+  cmsys::ProcessOutput processOutput;
+  std::string strdata;
   while ((p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) {
 // Put the output in the right place.
 if (p == cmsysProcess_Pipe_STDOUT && !output_quiet) {
   if (output_variable.empty()) {
-cmSystemTools::Stdout(data, length);
+processOutput.DecodeText(data, length, strdata);
+cmSystemTools::Stdout(strdata.c_str(), strdata.size());
   } else {
 cmExecuteProcessCommandAppend(tempOutput, data, length);
   }
 } else if (p == cmsysProcess_Pipe_STDERR && !error_quiet) {
   if (error_variable.empty()) {
-cmSystemTools::Stderr(data, length);
+processOutput.DecodeText(data, length, strdata);
+cmSystemTools::Stderr(strdata.c_str(), strdata.size());
   } else {
 cmExecuteProcessCommandAppend(tempError, data, length);
   }
@@ -247,6 +252,8 @@ bool 
cmExecuteProcessCommand::InitialPass(std::vector const& args,
 
   // All output has been read.  Wait for the process to exit.
   cmsysProcess_WaitForExit(cp, CM_NULLPTR);
+  processOutput.DecodeText(tempOutput, tempOutput);
+  processOutput.DecodeText(tempError, tempError);
 
   // Fix the text in the output strings.
   cmExecuteProcessCommandFixText(tempOutput, output_strip_trailing_whitespace);
diff --git a/Source/cmProcessTools.cxx b/Source/cmProcessTools.cxx
index 34b8df2..fdfa276 100644
--- a/Source/cmProcessTools.cxx
+++ b/Source/cmProcessTools.cxx
@@ -12,6 +12,7 @@
 #include "cmProcessTools.h"
 
 #include 
+#include 
 
 void cmProcessTools::RunProcess(struct cmsysProcess_s* cp, OutputParser* out,
 OutputParser* err)
@@ -20,14 +21,18 @@ void cmProcessTools::RunProcess(struct cmsysProcess_s* cp, 
OutputParser* out,
   char* data = CM_NULLPTR;
   int length = 0;
   int p;
+  cmsys::ProcessOutput processOutput;
+  std::string strdata;
   while ((out || err) &&
  (p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) {
 if (out && p == cmsysProcess_Pipe_STDOUT) {
-  if (!out->Process(data, length)) {
+  processOutput.DecodeText(data, length, strdata);
+  if (!out->Process(strdata.c_str(), int(strdata.size( {
 out = CM_NULLPTR;
   }
 } else if (err && p == cmsysProcess_Pipe_STDERR) {
-  if (!err->Process(data, length)) {
+  processOutput.DecodeText(data, length, strdata);
+  if (!err->Process(strdata.c_str(), int(strdata.size( {
 err = CM_NULLPTR;
   }
 }
diff --git a/Source/cmSystemTools.cxx b/Source/cmSystemTools.cxx
index 9740ef7..ca92646 100644
--- a/Source/cmSystemTools.cxx
+++ b/Source/cmSystemTools.cxx
@@ -26,6 +26,7 @@
 #include 
 #include 
 #include 
+#include 
 #if defined(CMAKE_BUILD_WITH_CMAKE)
 #include "cmArchiveWrite.h"
 #include "cmLocale.h"
@@ -612,6 +613,8 @@ bool

Re: [cmake-developers] [PATCH v4 4/4] For Windows encode process output to internally used encoding

2016-07-21 Thread Dāvis Mosāns 
2016-07-18 17:04 GMT+03:00 Brad King <brad.k...@kitware.com>:
> On 07/07/2016 05:54 PM, Dāvis Mosāns wrote:
>> Typically Windows applications (eg. MSVC compiler) use current console's
>> codepage for output to pipes so we need to encode that to internally used
>> encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE).
> [snip]
>>while ((p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) 
>> {
>> +cmsysProcess_DecodeTextOutput(cp, , );
> [snip]
>>while ((p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) 
>> {
>> +cmsysProcess_DecodeTextOutput(cp, , );
>
> Unfortunately I don't think that pattern will work reliably because
> a multi-byte character could be split across a buffering boundary
> and therefore not decoded correctly.  We may need the consuming
> contexts to collect the whole output before converting.
>

That's true, but only for MBCS code pages, it will work fine for SBCS and DBCS
(if buffer size is even). I think this issue would appear very rarely.
Anyway I improved this in places where it was easy, but in some places it's
more complicated...

For example

   while ((p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) {
 // Put the output in the right place.
 if (p == cmsysProcess_Pipe_STDOUT && !output_quiet) {
   if (output_variable.empty()) {
 cmSystemTools::Stdout(data, length);

Here we output buffer immediately.

 while ((out || err) &&
  (p = cmsysProcess_WaitForData(cp, , , CM_NULLPTR), p)) {
 if (out && p == cmsysProcess_Pipe_STDOUT) {
   if (!out->Process(data, length)) {

and also here we process buffer immediately.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH] Improve encoding handling on Windows

2016-06-30 Thread Dāvis Mosāns 
On Windows getenv uses ANSI codepage so it needs to be encoded to
internally used encoding (eg. UTF-8). Here we use _wgetenv instead
and encode that.

Also typically Windows applications (eg. MSVC compiler) use current
console's codepage for output to pipes so we need to encode that
to internally used encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE).

Next, when we're outputing to console need to use wide functions.

This change allows that compilers such as MSVC on Windows can be
installed in non-ASCII path and will work correctly for all
console's codepages which supports that path's characters.
---
 Source/cmBuildCommand.cxx|  4 ++--
 Source/cmCommandArgumentParserHelper.cxx |  4 ++--
 Source/cmExtraEclipseCDT4Generator.cxx   |  2 +-
 Source/cmSetCommand.cxx  |  2 +-
 Source/cmSystemTools.cxx |  5 +
 Source/cmake.cxx |  6 +++---
 Source/cmakemain.cxx | 29 ++---
 Source/kwsys/CMakeLists.txt  |  2 ++
 Source/kwsys/Directory.cxx   |  2 +-
 Source/kwsys/FStream.hxx.in  | 19 +--
 Source/kwsys/ProcessWin32.c  | 21 -
 Source/kwsys/SystemInformation.cxx   |  8 +---
 Source/kwsys/SystemTools.cxx | 28 
 13 files changed, 105 insertions(+), 27 deletions(-)

diff --git a/Source/cmBuildCommand.cxx b/Source/cmBuildCommand.cxx
index fb143a2..16771cc 100644
--- a/Source/cmBuildCommand.cxx
+++ b/Source/cmBuildCommand.cxx
@@ -77,7 +77,7 @@ bool cmBuildCommand::MainSignature(std::vector 
const& args)
   // as the original 2-arg build_command signature:
   //
   if (!configuration || !*configuration) {
-configuration = getenv("CMAKE_CONFIG_TYPE");
+configuration = cmSystemTools::GetEnv("CMAKE_CONFIG_TYPE");
   }
   if (!configuration || !*configuration) {
 configuration = "Release";
@@ -109,7 +109,7 @@ bool 
cmBuildCommand::TwoArgsSignature(std::vector const& args)
   const char* cacheValue = this->Makefile->GetDefinition(define);
 
   std::string configType = "Release";
-  const char* cfg = getenv("CMAKE_CONFIG_TYPE");
+  const char* cfg = cmSystemTools::GetEnv("CMAKE_CONFIG_TYPE");
   if (cfg && *cfg) {
 configType = cfg;
   }
diff --git a/Source/cmCommandArgumentParserHelper.cxx 
b/Source/cmCommandArgumentParserHelper.cxx
index 294117c..5c615c4 100644
--- a/Source/cmCommandArgumentParserHelper.cxx
+++ b/Source/cmCommandArgumentParserHelper.cxx
@@ -71,12 +71,12 @@ char* 
cmCommandArgumentParserHelper::ExpandSpecialVariable(const char* key,
 return this->EmptyVariable;
   }
   if (strcmp(key, "ENV") == 0) {
-char* ptr = getenv(var);
+const char* ptr = cmSystemTools::GetEnv(var);
 if (ptr) {
   if (this->EscapeQuotes) {
 return this->AddString(cmSystemTools::EscapeQuotes(ptr));
   } else {
-return ptr;
+return (char *)ptr;
   }
 }
 return this->EmptyVariable;
diff --git a/Source/cmExtraEclipseCDT4Generator.cxx 
b/Source/cmExtraEclipseCDT4Generator.cxx
index 16cb082..6c9e9a1 100644
--- a/Source/cmExtraEclipseCDT4Generator.cxx
+++ b/Source/cmExtraEclipseCDT4Generator.cxx
@@ -208,7 +208,7 @@ void cmExtraEclipseCDT4Generator::AddEnvVar(std::ostream& 
out,
   // get the variables from the environment and from the cache and then
   // figure out which one to use:
 
-  const char* envVarValue = getenv(envVar);
+  const char* envVarValue = cmSystemTools::GetEnv(envVar);
 
   std::string cacheEntryName = "CMAKE_ECLIPSE_ENVVAR_";
   cacheEntryName += envVar;
diff --git a/Source/cmSetCommand.cxx b/Source/cmSetCommand.cxx
index 1484368..c0f8ab6 100644
--- a/Source/cmSetCommand.cxx
+++ b/Source/cmSetCommand.cxx
@@ -31,7 +31,7 @@ bool cmSetCommand::InitialPass(std::vector 
const& args,
 putEnvArg += "=";
 
 // what is the current value if any
-const char* currValue = getenv(varName);
+const char* currValue = cmSystemTools::GetEnv(varName);
 delete[] varName;
 
 // will it be set to something, then set it
diff --git a/Source/cmSystemTools.cxx b/Source/cmSystemTools.cxx
index 2d463f3..d8a1437 100644
--- a/Source/cmSystemTools.cxx
+++ b/Source/cmSystemTools.cxx
@@ -333,7 +333,12 @@ void cmSystemTools::Message(const char* m1, const char* 
title)
  s_MessageCallbackClientData);
 return;
   } else {
+#if defined(_WIN32)
+std::wstring wm1 = cmsys::Encoding::ToWide(m1);
+std::wcerr << wm1 << std::endl << std::flush;
+#else
 std::cerr << m1 << std::endl << std::flush;
+#endif
   }
 }
 
diff --git a/Source/cmake.cxx b/Source/cmake.cxx
index c597605..94ecd81 100644
--- a/Source/cmake.cxx
+++ b/Source/cmake.cxx
@@ -955,8 +955,8 @@ void cmake::SetGlobalGenerator(cmGlobalGenerator* gg)
   cmSystemTools::SetForceUnixPaths(this->GlobalGenerator->GetForceUnixPaths());
 
   // Save the environment variables CXX and CC
-  const char* cxx = getenv("CXX");
-  const char* cc =

Re: [cmake-developers] [PATCH] Improve encoding handling on Windows

2016-07-01 Thread Dāvis Mosāns 
2016-07-01 15:25 GMT+03:00 Daniel Pfeifer <dan...@pfeifer-mail.de>:

> Hi Dāvis,
>
> On Fri, Jul 1, 2016 at 4:18 AM, Dāvis Mosāns <davis...@gmail.com> wrote:
> > On Windows getenv uses ANSI codepage so it needs to be encoded to
> > internally used encoding (eg. UTF-8). Here we use _wgetenv instead
> > and encode that.
>
> Your change to the SystemTools::GetEnv function introduces memory
> leaks, since you return a pointer to a new[]-ed array.
> It seems impossible to refactor SystemTools::GetEnv to use _wgetenv
> and provide interface compatability.
> You should probably introduce a function that returns std::string and
> uses GetEnvironmentVariableW internally.
>
>
I know about memory leak, I intentionally left it this way for first version
of patch as there are multiple ways how to implement it properly.

POSIX's getenv doesn't need to be freed, but here for Windows since we need
to encode it to internal encoding we'll be returning a new pointer.

I see two ways:

1. add SystemTools::FreeEnv which delete's if on Windows but does nothing
  otherwise;
2. change GetEnv to return std::unique_ptr which will be
  automatically deleted once out of scope and we still can check if there
  wasn't such env if it's empty.

To me 2nd option seems best.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH v2] Improve encoding handling on Windows

2016-07-02 Thread Dāvis Mosāns 
On Windows getenv (and putenv) uses ANSI codepage so it needs to
be encoded to internally used encoding (eg. UTF-8). Here we use
_wgetenv (and _wputenv) instead and encode that.

Also typically Windows applications (eg. MSVC compiler) use current
console's codepage for output to pipes so we need to encode that
to internally used encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE).

Next, when we're outputing to console need to use wide functions.

This change allows that compilers such as MSVC on Windows can be
installed in non-ASCII path and will work correctly for all
console's codepages which supports that path's characters.
---
 Source/CPack/cmCPackGenerator.cxx   |  8 +--
 Source/CTest/cmCTestCoverageHandler.cxx | 12 ++--
 Source/CTest/cmCTestCurl.cxx| 21 +++
 Source/CTest/cmCTestMultiProcessHandler.cxx |  8 +--
 Source/cmBuildCommand.cxx   | 23 +++
 Source/cmCLocaleEnvironmentScope.cxx|  4 +-
 Source/cmCTest.cxx  | 11 ++--
 Source/cmCommandArgumentParserHelper.cxx|  8 +--
 Source/cmConditionEvaluator.cxx |  2 +-
 Source/cmExportCommand.cxx  |  5 +-
 Source/cmExtraEclipseCDT4Generator.cxx  |  9 +--
 Source/cmFileCommand.cxx| 12 ++--
 Source/cmFindPackageCommand.cxx |  4 +-
 Source/cmGlobalVisualStudio7Generator.cxx   |  6 +-
 Source/cmMakefile.cxx   |  5 +-
 Source/cmNinjaTargetGenerator.cxx   |  2 +-
 Source/cmQtAutoGenerators.cxx   |  2 +-
 Source/cmSetCommand.cxx |  6 +-
 Source/cmState.cxx  |  5 +-
 Source/cmSystemTools.cxx| 11 +++-
 Source/cmTimestamp.cxx  |  7 +--
 Source/cmake.cxx| 16 ++---
 Source/cmakemain.cxx| 38 ++-
 Source/cmcmd.cxx| 12 ++--
 Source/kwsys/CMakeLists.txt |  2 +
 Source/kwsys/Directory.cxx  |  2 +-
 Source/kwsys/FStream.hxx.in | 19 +-
 Source/kwsys/ProcessWin32.c | 25 +++-
 Source/kwsys/SystemInformation.cxx  | 20 +++---
 Source/kwsys/SystemTools.cxx| 98 +++--
 Source/kwsys/SystemTools.hxx.in |  4 +-
 Source/kwsys/testSystemTools.cxx|  9 +--
 32 files changed, 260 insertions(+), 156 deletions(-)

diff --git a/Source/CPack/cmCPackGenerator.cxx 
b/Source/CPack/cmCPackGenerator.cxx
index df8bb0f..76609e1 100644
--- a/Source/CPack/cmCPackGenerator.cxx
+++ b/Source/CPack/cmCPackGenerator.cxx
@@ -1074,11 +1074,11 @@ const char* cmCPackGenerator::GetInstallPath()
 return this->InstallPath.c_str();
   }
 #if defined(_WIN32) && !defined(__CYGWIN__)
-  const char* prgfiles = cmsys::SystemTools::GetEnv("ProgramFiles");
-  const char* sysDrive = cmsys::SystemTools::GetEnv("SystemDrive");
-  if (prgfiles) {
+  std::string prgfiles;
+  std::string sysDrive;
+  if (cmsys::SystemTools::GetEnv("ProgramFiles", prgfiles)) {
 this->InstallPath = prgfiles;
-  } else if (sysDrive) {
+  } else if (cmsys::SystemTools::GetEnv("SystemDrive", sysDrive)) {
 this->InstallPath = sysDrive;
 this->InstallPath += "/Program Files";
   } else {
diff --git a/Source/CTest/cmCTestCoverageHandler.cxx 
b/Source/CTest/cmCTestCoverageHandler.cxx
index 7102533..9410a52 100644
--- a/Source/CTest/cmCTestCoverageHandler.cxx
+++ b/Source/CTest/cmCTestCoverageHandler.cxx
@@ -727,10 +727,7 @@ int cmCTestCoverageHandler::HandleCoberturaCoverage(
   // if it doesn't exist or is empty, assume the
   // binary directory is used.
   std::string coverageXMLFile;
-  const char* covDir = cmSystemTools::GetEnv("COBERTURADIR");
-  if (covDir && strlen(covDir) != 0) {
-coverageXMLFile = std::string(covDir);
-  } else {
+  if (!cmSystemTools::GetEnv("COBERTURADIR", coverageXMLFile) || 
coverageXMLFile.empty()) {
 coverageXMLFile = this->CTest->GetBinaryDir();
   }
   // build the find file string with the directory from above
@@ -791,7 +788,8 @@ struct cmCTestCoverageHandlerLocale
 {
   cmCTestCoverageHandlerLocale()
   {
-if (const char* l = cmSystemTools::GetEnv("LC_ALL")) {
+std::string l;
+if (cmSystemTools::GetEnv("LC_ALL", l)) {
   lc_all = l;
 }
 if (lc_all != "C") {
@@ -2121,8 +2119,8 @@ int cmCTestCoverageHandler::RunBullseyeSourceSummary(
 int cmCTestCoverageHandler::HandleBullseyeCoverage(
   cmCTestCoverageHandlerContainer* cont)
 {
-  const char* covfile = cmSystemTools::GetEnv("COVFILE");
-  if (!covfile || strlen(covfile) == 0) {
+  std::string covfile;
+  if (!cmSystemTools::GetEnv("COVFILE", covfile) || covfile.empty()) {
 cmCTestOptionalLog(this->CTest, HANDLER_VERBOSE_OUTPUT,
" COVFILE environment variable not found, not running "
" bullseye\n",
diff --git a/Source/CTest/cmCTestCurl.cxx b/Source/CTest/cmCTestCurl.cxx
index

Re: [cmake-developers] [PATCH] Improve encoding handling on Windows

2016-07-02 Thread Dāvis Mosāns 
2016-07-01 17:41 GMT+03:00 Mike Gelfand :

>
> Since you already have "bool SystemTools::GetEnv(const char* key,
> std::string& result)", another option would be to use it everywhere and
> maybe introduce something like "bool SystemTools::HasEnv(const char*
> key)" for those several cases where you only need to check the existence.
>
>
I implemented this, it's actually really nice API to work with.


2016-07-02 1:54 GMT+03:00 :

>
> ...
>
> > +#if defined(_WIN32)
> > +  // Kinda hack, with MSVC (and MinGW) for some reason std::wcout
> > +  // (and all other std::w*) fails once it encounters non-ASCII
> > +  // string unless locale is set.
> > +  // Note that even with this, seems it can't output characters
> > +  // which aren't present in ANSI codepage no matter what encoding
> > +  // is used for console.
> > +  // Also once any character outside of ANSI codepage is tried to
> > +  // be outputed then after there anymore won't be output from
> > +  // any of std::w* functions.
> > +  _wsetlocale(LC_ALL, L"");
> > +#endif
>
> The _wsetlocale() may be a problem.
>
> See:
> https://gitlab.kitware.com/cmake/cmake/commit/87be2e142
> https://cmake.org/Bug/view.php?id=16099
>
>
>
Indeed, good catch, thanks, I didn't thought about this. But even now
most these locale aware functions like tolower/toupper and others are
wrong because internally we use UTF-8 and there 1 character can take
more than 1 byte so these functions won't work correctly for some strings
even if we don't set any locale.
Now here we actually set it only on Windows because there just isn't any
other way. Without setting locale we get only ASCII support and can't
output even ANSI characters. With locale we can atleast output ANSI
characters.
Currently Microsoft C++ library doesn't support UTF-8/UTF-16 locales.
Only way to output Unicode would be implement our own std::wcout which
would use wide WinAPI to write to console.

Anyway quick fix is to always use English locale then number parsing
will be expected and set user's codepage.

  std::wstring locale = L"English_United States.";
  locale += std::to_wstring(GetACP());
  _wsetlocale(LC_ALL, locale.c_str());


Of course proper Unicode support will be needed some day, but for now
this is still an improvement.


> @@ -30,8 +30,23 @@ namespace @KWSYS_NAMESPACE@
> >   typedef std::basic_filebuf my_base_type;
> >   basic_filebuf *open(char const *s,std::ios_base::openmode mode)
> >   {
> > +std::wstring wstr = Encoding::ToWide(s);
> > +const wchar_t *ws = wstr.c_str();
> > +#if defined(_MSC_VER) && _MSC_VER >= 1400
> > +const wchar_t *ss = ws;
> > +#else
> > +const char *ss = 0;
> > +size_t length = std::wcstombs(0, ws, 0);
> > +if (length != size_t(-1)) {
> > +  std::vector str(length + 1);
> > +  ss = str.data();
> > +  std::wcstombs((char *)ss, ws, str.size());
> > +} else {
> > +  ss = s;
> > +}
> > +#endif
> > return static_cast(
> > -  my_base_type::open(Encoding::ToWide(s).c_str(), mode)
> > +  my_base_type::open(ss, mode)
> >   );
>
> Yes.  It makes sense to convert from utf-8 to code page when we are not
> using the wide apis.
> This would at least give you support for the current code page, beyond
> ascii.   Beyond that, the wide functions should be used.
> Which compiler are you trying to support here, which doesn't give a wide
> open()?
>
>
Only MSVC have ofstream::open(const wchar_t *) so for MinGW need to use
ofstream::open(const char *) or use wofstream::open(const wchar_t *) which
would
require quite big changes.


> >   }
> >   };
> > diff --git a/Source/kwsys/ProcessWin32.c b/Source/kwsys/ProcessWin32.c
> > index 2b93e69..208e725 100644
> > --- a/Source/kwsys/ProcessWin32.c
> > +++ b/Source/kwsys/ProcessWin32.c
> > @@ -181,7 +181,7 @@ struct kwsysProcessPipeData_s
> >   /* - Data managed per call to Execute - */
> >
> >   /* Buffer for data read in this pipe's thread.  */
> > -  char DataBuffer[KWSYSPE_PIPE_BUFFER_SIZE];
> > +  char DataBuffer[KWSYSPE_PIPE_BUFFER_SIZE*2];
>
> This "*2" assumes all characters cmake sees will fit in the Basic
> Multilingual Plane.
> Are we OK assuming that?
> If the conversion from a code page to utf-16 results in surrogate pairs,
> there may not be enough space.
>
>
Not really, it assumes character string from ANSI code page encoded
to internal encoding (UTF-8) will not take more than 2x space. Which I
think is a pretty good guess, because here we're processing output
from a process and so output will be paths and texts usually in
ASCII with not many characters which would require more than 2 bytes.
And it will still work if say 1/4th uses 4 bytes and 1 byte for rest 3/4th.

1024 * 1/4 * 4 bytes + 1024 * 3/4 * 1 byte (1792) < 1024 * 2 (2048)

But of course can increase it to 4x, only I think it's very

Re: [cmake-developers] [PATCH v2] Improve encoding handling on Windows

2016-07-03 Thread Dāvis Mosāns 
2016-07-03 13:04 GMT+03:00 Mike Gelfand :

> For std::c(in|out|err) vs. std::wc(in|out|err), here's an implementation
> of std::streambuf which we currently use in our projects:
> http://stackoverflow.com/a/21728856/583456. You could either try using
> it as is or use it as a base for your own implementation; in any case,
> it's better than forcing people to use one or another stream based on
> current OS.
>
> Not sure what currently accepted CMake coding style dictates, but
> several variables could be made const (e.g. `envVarSet` in
> `cmExtraEclipseCDT4Generator::AddEnvVar`).
>
> Apart from notes above and a few indentation issues which may be fixed
> during merge, below are some more observations.
>
>
Huge thanks for review! Will fix mentioned issues in next version of patch.
Also I'll implement this solution with std::streambuf as it's much better
way
and it's actually not that much work I thought it would be.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [PATCH v7] For Windows encode process output to internally used encoding

2016-08-15 Thread Dāvis Mosāns 
Typically Windows applications (eg. MSVC compiler) use current console's
codepage for output to pipes so we need to encode that to internally used
encoding (KWSYS_ENCODING_DEFAULT_CODEPAGE).
---
 Source/CMakeLists.txt  |   6 ++
 Source/ProcessOutput.cxx   | 141 +
 Source/ProcessOutput.hxx   |  39 ++
 Source/cmExecProgramCommand.cxx|   3 +
 Source/cmExecuteProcessCommand.cxx |  11 ++-
 Source/cmProcessTools.cxx  |   9 ++-
 Source/cmSystemTools.cxx   |  11 ++-
 bootstrap  |   5 +-
 8 files changed, 218 insertions(+), 7 deletions(-)
 create mode 100644 Source/ProcessOutput.cxx
 create mode 100644 Source/ProcessOutput.hxx

diff --git a/Source/CMakeLists.txt b/Source/CMakeLists.txt
index cdc8fb1..46dd471 100644
--- a/Source/CMakeLists.txt
+++ b/Source/CMakeLists.txt
@@ -373,8 +373,14 @@ set(SRCS
   cm_sha2.c
   cm_utf8.h
   cm_utf8.c
+
+  ProcessOutput.cxx
+  ProcessOutput.hxx
   )
 
+SET_PROPERTY(SOURCE ProcessOutput.cxx APPEND PROPERTY COMPILE_DEFINITIONS
+  KWSYS_ENCODING_DEFAULT_CODEPAGE=${KWSYS_ENCODING_DEFAULT_CODEPAGE})
+
 set(COMMAND_INCLUDES "#include \"cmTargetPropCommandBase.cxx\"\n")
 list(APPEND SRCS cmTargetPropCommandBase.cxx)
 set_property(SOURCE cmTargetPropCommandBase.cxx PROPERTY HEADER_FILE_ONLY ON)
diff --git a/Source/ProcessOutput.cxx b/Source/ProcessOutput.cxx
new file mode 100644
index 000..6c66087
--- /dev/null
+++ b/Source/ProcessOutput.cxx
@@ -0,0 +1,141 @@
+/*
+  CMake - Cross Platform Makefile Generator
+  Copyright 2000-2016 Kitware, Inc., Insight Software Consortium
+
+  Distributed under the OSI-approved BSD License (the "License");
+  see accompanying file Copyright.txt for details.
+
+  This software is distributed WITHOUT ANY WARRANTY; without even the
+  implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+  See the License for more information.
+*/
+
+#include "ProcessOutput.hxx"
+
+#if defined(_WIN32)
+# include 
+unsigned int ProcessOutput::defaultCodepage = KWSYS_ENCODING_DEFAULT_CODEPAGE;
+#endif
+
+ProcessOutput::ProcessOutput(unsigned int maxSize)
+{
+#if defined(_WIN32)
+  bufferSize = maxSize;
+  codepage = GetConsoleCP();
+  if (!codepage) {
+codepage = GetACP();
+  }
+#else
+  static_cast(maxSize);
+#endif
+}
+
+ProcessOutput::~ProcessOutput()
+{
+}
+
+bool ProcessOutput::DecodeText(std::string raw, std::string& decoded, size_t 
id)
+{
+  bool success = true;
+  decoded = raw;
+#if defined(_WIN32)
+  if (id > 0) {
+if (rawparts.size() < id) {
+  rawparts.reserve(id);
+  while (rawparts.size() < id) rawparts.push_back(std::string());
+}
+raw = rawparts[id - 1] + raw;
+rawparts[id - 1].clear();
+decoded = raw;
+  }
+  if (raw.size() > 0 && codepage != defaultCodepage) {
+success = false;
+CPINFOEXW cpinfo;
+if (id > 0 && raw.size() == bufferSize && GetCPInfoExW(codepage, 0, 
) == 1 && cpinfo.MaxCharSize > 1) {
+  if (cpinfo.MaxCharSize == 2 && cpinfo.LeadByte[0] != 0) {
+LPSTR prevChar = CharPrevExA(codepage, raw.c_str(), raw.c_str() + 
raw.size(), 0);
+bool isLeadByte = (*(prevChar + 1) == 0) && IsDBCSLeadByteEx(codepage, 
*prevChar);
+if (isLeadByte) {
+  rawparts[id - 1] += *(raw.end() - 1);
+  raw.resize(raw.size() - 1);
+}
+success = DoDecodeText(raw, decoded, NULL);
+  } else {
+bool restoreDecoded = false;
+std::string firstDecoded = decoded;
+wchar_t lastChar = 0;
+for (UINT i = 0; i < cpinfo.MaxCharSize; i++) {
+  success = DoDecodeText(raw, decoded, );
+  if (success && lastChar != 0) {
+if (i == 0) {
+  firstDecoded = decoded;
+}
+if (lastChar == cpinfo.UnicodeDefaultChar) {
+  restoreDecoded = true;
+  rawparts[id - 1] = *(raw.end() - 1) + rawparts[id - 1];
+  raw.resize(raw.size() - 1);
+} else {
+  restoreDecoded = false;
+  break;
+}
+  } else {
+break;
+  }
+}
+if (restoreDecoded) {
+  decoded = firstDecoded;
+  rawparts[id - 1].clear();
+}
+  }
+} else {
+  success = DoDecodeText(raw, decoded, NULL);
+}
+  }
+#else
+  static_cast(id);
+#endif
+  return success;
+}
+
+bool ProcessOutput::DecodeText(const char* data, size_t length, std::string& 
decoded, size_t id)
+{
+  return DecodeText(std::string(data, length), decoded, id);
+}
+
+bool ProcessOutput::DecodeText(std::vector raw, std::vector& 
decoded, size_t id)
+{
+  std::string str;
+  const bool success = DecodeText(std::string(raw.begin(), raw.end()), str, 
id);
+  decoded.assign(str.begin(), str.end());
+  return success;
+}
+
+#if

Re: [cmake-developers] [PATCH v7] For Windows encode process output to internally used encoding

2016-10-30 Thread Dāvis Mosāns 
2016-08-17 16:47 GMT+03:00 Brad King :
> I squashed in one warning fix:
>
>  Windows: Encode child process output to internally-used encoding
>  https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=12924660
>
> However, then I had to revert the change from `next` because it
> causes the CTest.UpdateGIT test to fail on Windows machines.
> I think the problem is that we run Git with a `-z` option to
> produce binary output.  In such cases we should not do any
> encoding conversions.  cmProcessTools and RunSingleCommand
> will need to gain options for this.

While git's `-z` flag caused test to fail it wasn't because of binary
output (we're still working with text data like commit messages and
file names) but it showed a bug that we would truncate output till
first null byte. `-z` option specifies that git will use null bytes as
separators between entries and it's independent from used text
encoding. We still need to decode git's output (which contains null
bytes) to our internal encoding.

> Also I noticed that if DecodeText buffers partial characters we
> may need a finalize step later to finish them off.  Otherwise
> invalid byte sequences may be dropped if they appear at the end.
>
> Please fetch the above version and revise it as needed.
>

I've fixed it and submitted MR to
https://gitlab.kitware.com/cmake/cmake/merge_requests/221
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] [Discussion] Add python support for CMakeLists

2017-01-13 Thread Dāvis Mosāns 
2017-01-12 11:48 GMT+02:00 Charles Huet :
[...]
> Lua is the language that should be used, since it is easy to embed on all
> the platforms CMake supports. All arguments about language X being better
> than language Y do not matter, since all other languages do not answer to
> the constraints that are CMake's, which are, as I understand them:
> - being embeddable so as to not depend on the user to install something else
> - support for lots of various platforms (e.g. ARM)
>

Lua isn't only option, there is also [1] mruby (embedded Ruby)
compiles to C library which can be statically linked. It also is very
customizable, you can compile only those features which you need.


[1] https://github.com/mruby/mruby
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers