[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user amaya382 closed the pull request at: https://github.com/apache/incubator-hivemall/pull/80 ---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/80#discussion_r118157439
--- Diff: Makefile ---
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+DOCKCROSS_SCRIPT := 'dockcross.bash'
+XGBOOST_REPO := 'https://github.com/maropu/xgboost.git'
+XGBOOST_BRANCH := 'xgboost_v0.60_with_portable_binaries'
+HIVEMALL_HOME := "$(shell pwd)"
+HIVEMALL_OUT := "${HIVEMALL_HOME}/target"
+XGBOOST_OUT := "${HIVEMALL_OUT}/xgboost"
+HIVEMALL_LIB_DIR := "${HIVEMALL_HOME}/xgboost/src/main/resources/lib"
+CANDIDATES := 'linux-arm64' 'linux-armv6' 'linux-armv7' 'linux-ppc64le'
'linux-x64' 'linux-x86' 'windows-x64' 'windows-x86'
--- End diff --
supported platform of hivemall depends on Hadoop.
target os is linux, windows, mac osx.
main target platform is x86, x64.
mac osx support is only used for local testing though.
https://github.com/xerial/snappy-java/blob/master/Makefile#L167
https://github.com/multiarch/crossbuild
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user maropu commented on a diff in the pull request: https://github.com/apache/incubator-hivemall/pull/80#discussion_r117690004 --- Diff: bin/build_xgboost.sh --- @@ -1,87 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -# xgboost requires g++-4.6 or higher (https://github.com/dmlc/xgboost/blob/master/doc/build.md), -# so we need to first check if the requirement is satisfied. -COMPILER_REQUIRED_VERSION="4.6" -COMPILER_VERSION=`g++ --version 2> /dev/null` - -# Check if GNU g++ installed -if [ $? = 127 ]; then --- End diff -- Ah, ok. But I think we keep a script to build `xgboost` on native environments in terms of CPU optimization ( I think it'd be better to follow the same approach with `snappy-java`). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user amaya382 commented on a diff in the pull request: https://github.com/apache/incubator-hivemall/pull/80#discussion_r117666401 --- Diff: bin/build_xgboost.sh --- @@ -1,87 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -# xgboost requires g++-4.6 or higher (https://github.com/dmlc/xgboost/blob/master/doc/build.md), -# so we need to first check if the requirement is satisfied. -COMPILER_REQUIRED_VERSION="4.6" -COMPILER_VERSION=`g++ --version 2> /dev/null` - -# Check if GNU g++ installed -if [ $? = 127 ]; then --- End diff -- this script uses `gcc` automatically on dockcross, never call `clang` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user myui commented on a diff in the pull request: https://github.com/apache/incubator-hivemall/pull/80#discussion_r117665985 --- Diff: bin/build_xgboost.sh --- @@ -1,87 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -# xgboost requires g++-4.6 or higher (https://github.com/dmlc/xgboost/blob/master/doc/build.md), -# so we need to first check if the requirement is satisfied. -COMPILER_REQUIRED_VERSION="4.6" -COMPILER_VERSION=`g++ --version 2> /dev/null` - -# Check if GNU g++ installed -if [ $? = 127 ]; then --- End diff -- @maropu no need to support `clang` anymore. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user maropu commented on a diff in the pull request: https://github.com/apache/incubator-hivemall/pull/80#discussion_r117665868 --- Diff: bin/build_xgboost.sh --- @@ -1,87 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -# xgboost requires g++-4.6 or higher (https://github.com/dmlc/xgboost/blob/master/doc/build.md), -# so we need to first check if the requirement is satisfied. -COMPILER_REQUIRED_VERSION="4.6" -COMPILER_VERSION=`g++ --version 2> /dev/null` - -# Check if GNU g++ installed -if [ $? = 127 ]; then --- End diff -- We'd be better off printing explicit error messages when `clang` used. Could you? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user amaya382 commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/80#discussion_r117665318
--- Diff: xgboost/src/main/java/hivemall/xgboost/NativeLibLoader.java ---
@@ -54,15 +55,47 @@ private static boolean hasResource(String path) {
}
private static String getOSName() {
-return System.getProperty("os.name");
+return System.getProperty("os.name").toLowerCase();
+}
+
+private static String getOSArch() {
+return System.getProperty("os.arch").toLowerCase();
+}
+
+private static String getOSArchString() {
+String os = getOSName();
+if(os.startsWith("linux")) {
+os = "linux";
+} else if(os.startsWith("mac")) {
+os = "darwin";
+} else if(os.startsWith("windows")) {
+os = "windows";
+}
+
+String arch = getOSArch();
+if(arch.equals("amd64") || arch.equals("x86_64")) {
+arch = "x64";
+} else if(arch.endsWith("86")) {
+arch = "x86";
+} else if(arch.indexOf("arm64") != -1) {
+arch = "arm64";
+} else if(arch.indexOf("armv6") != -1) {
+arch = "armv6";
+} else if(arch.indexOf("armv7") != -1) {
+arch = "armv7";
+} else if(arch.indexOf("ppc") != -1) {
+arch = "ppc64le";
+}
+
+return os + "-" + arch;
--- End diff --
that's so, I'll mod it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user amaya382 commented on a diff in the pull request: https://github.com/apache/incubator-hivemall/pull/80#discussion_r117665209 --- Diff: bin/build_xgboost.sh --- @@ -1,87 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -# xgboost requires g++-4.6 or higher (https://github.com/dmlc/xgboost/blob/master/doc/build.md), -# so we need to first check if the requirement is satisfied. -COMPILER_REQUIRED_VERSION="4.6" -COMPILER_VERSION=`g++ --version 2> /dev/null` - -# Check if GNU g++ installed -if [ $? = 127 ]; then --- End diff -- @maropu No, building xgboost depends on gcc. because dockcross includes gcc, skips to check compiler. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/80#discussion_r117657316
--- Diff: xgboost/src/main/java/hivemall/xgboost/NativeLibLoader.java ---
@@ -54,15 +55,47 @@ private static boolean hasResource(String path) {
}
private static String getOSName() {
-return System.getProperty("os.name");
+return System.getProperty("os.name").toLowerCase();
+}
+
+private static String getOSArch() {
+return System.getProperty("os.arch").toLowerCase();
+}
+
+private static String getOSArchString() {
+String os = getOSName();
+if(os.startsWith("linux")) {
+os = "linux";
+} else if(os.startsWith("mac")) {
+os = "darwin";
+} else if(os.startsWith("windows")) {
+os = "windows";
+}
+
+String arch = getOSArch();
+if(arch.equals("amd64") || arch.equals("x86_64")) {
+arch = "x64";
+} else if(arch.endsWith("86")) {
+arch = "x86";
+} else if(arch.indexOf("arm64") != -1) {
+arch = "arm64";
+} else if(arch.indexOf("armv6") != -1) {
+arch = "armv6";
+} else if(arch.indexOf("armv7") != -1) {
+arch = "armv7";
+} else if(arch.indexOf("ppc") != -1) {
+arch = "ppc64le";
+}
+
+return os + "-" + arch;
--- End diff --
I think you could refer [the
code](https://github.com/xerial/snappy-java/blob/master/src/main/java/org/xerial/snappy/OSInfo.java)
in `snappy-java` to handle almost all the case for detecting arch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
Github user maropu commented on a diff in the pull request: https://github.com/apache/incubator-hivemall/pull/80#discussion_r117657090 --- Diff: bin/build_xgboost.sh --- @@ -1,87 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. -# - -# xgboost requires g++-4.6 or higher (https://github.com/dmlc/xgboost/blob/master/doc/build.md), -# so we need to first check if the requirement is satisfied. -COMPILER_REQUIRED_VERSION="4.6" -COMPILER_VERSION=`g++ --version 2> /dev/null` - -# Check if GNU g++ installed -if [ $? = 127 ]; then --- End diff -- @amaya382 Does the new script works for both gcc and clang? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hivemall pull request #80: [WIP][HIVEMALL-99] Cross-compilation of...
GitHub user amaya382 opened a pull request:
https://github.com/apache/incubator-hivemall/pull/80
[WIP][HIVEMALL-99] Cross-compilation of XGBoost using Docker
## What changes were proposed in this pull request?
Cross compiling xgboost using dockcross
## What type of PR is it?
Improvement
## What is the Jira issue?
[[HIVEMALL-99]](https://issues.apache.org/jira/browse/HIVEMALL-99)
## How was this patch tested?
(Need to decide)
## How to use this feature?
```sh
# make xgboost-${arch}
make xgboost-linux-x64
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/amaya382/incubator-hivemall cross-compiling
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hivemall/pull/80.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #80
commit b7d162de9f0f1bf97e4d17d0266f939e0d96134f
Author: amaya
Date: 2017-05-17T02:38:55Z
WIP
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
