[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568693#comment-17568693 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran closed pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1 URL: https://github.com/apache/parquet-mr/pull/970 > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568692#comment-17568692 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1189454282 resolved by #973 > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17563878#comment-17563878 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1177929495 oh, upgrading is better! > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17561764#comment-17561764 ] ASF GitHub Bot commented on PARQUET-2150: - shangxinli commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1172920665 @steveloughran Thanks for the explanation! Do you have concerns if we use [PR-973](https://github.com/apache/parquet-mr/pull/973)? It seems we can rely on proto-buf itself to solve the issue. @sunchao please add @steveloughran as co-author for your PR. > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556450#comment-17556450 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1160638077 this patch is based on Dongjoon;s one for hadoop, tells maven to use the x86 artifact on macbook m1 builds. the sunchao one switches to a version of protobuf with a genuine mac m1 artifacts, a version which should also include some CVE fixes. > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556125#comment-17556125 ] ASF GitHub Bot commented on PARQUET-2150: - sunchao commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1159806290 @shangxinli my approach is different from @steveloughran 's one. Since newer version of protobuf already provides M1 artifacts, upgrade will solve the issue > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17555823#comment-17555823 ] ASF GitHub Bot commented on PARQUET-2150: - shangxinli commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1159351267 @sunchao I see your change is to upgrade the protobuf version. Is that required to solve this problem, which I don't see in this PR. > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552481#comment-17552481 ] ASF GitHub Bot commented on PARQUET-2150: - sunchao commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1151738855 Oops @steveloughran just found out this PR. I opened #973 for the same purpose > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545826#comment-17545826 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1145835507 tested lasted patch on * macbook m1 * macbook x86 * ec2 x86 i would set up my raspberry pi for a build too -I just fear having to build the thrift compiler > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543966#comment-17543966 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1141216939 ok, somehow I've broken the default classifier logic, which is of course critical. downgrading this to a draft until it's ready to be reviewed ``` Error: Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (generate-sources) on project parquet-protobuf: Error resolving artifact: com.google.protobuf:protoc:3.16.1:exe:${os.detected.classifier}: Could not find artifact com.google.protobuf:protoc:exe:${os.detected.classifier}:3.16.1 in jitpack.io (https://jitpack.io/) Error: Error: Try downloading the file manually from the project website. Error: Error: Then, install it using the command: Error: mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=${os.detected.classifier} -Dpackaging=exe -Dfile=/path/to/file Error: Error: Alternatively, if you host your own repository you can deploy the file there: Error: mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=${os.detected.classifier} -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] ``` > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541655#comment-17541655 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1136296983 its the maven protoc compiler plugin; wants to pull in something native. i have no idea why, but do know, if it is not there, you get to see a stack trace. ``` [ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (generate-sources) on project parquet-protobuf: Error resolving artifact: com.google.protobuf:protoc:3.16.1: Could not find artifact com.google.protobuf:protoc:exe:osx-aarch_64:3.16.1 in jitpack.io (https://jitpack.io) [ERROR] [ERROR] Try downloading the file manually from the project website. [ERROR] [ERROR] Then, install it using the command: [ERROR] mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file [ERROR] [ERROR] Alternatively, if you host your own repository you can deploy the file there: [ERROR] mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] [ERROR] [ERROR] [ERROR] com.google.protobuf:protoc:exe:3.16.1 [ERROR] [ERROR] from the specified remote repositories: [ERROR] jitpack.io (https://jitpack.io, releases=true, snapshots=true), [ERROR] apache.snapshots (https://repository.apache.org/snapshots, releases=false, snapshots=true), [ERROR] maven-central (https://repo1.maven.org/maven2, releases=true, snapshots=false) [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :parquet-protobuf ``` > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541585#comment-17541585 ] ASF GitHub Bot commented on PARQUET-2150: - theosib-amazon commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1136102990 I looked up parquet-protobuf and only found Java code. Is there some use of JNI here that I'm not finding? > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541089#comment-17541089 ] ASF GitHub Bot commented on PARQUET-2150: - dongjoon-hyun commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1135005341 Thank you for pinging me, @steveloughran . > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541087#comment-17541087 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran opened a new pull request, #970: URL: https://github.com/apache/parquet-mr/pull/970 Force use of x86 protocol executable in builds onl macOS systems, including those with ARM64 architectures, for which native binaries do not exist. ### Jira - [X] My PR addresses the following [Parquet Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR" - https://issues.apache.org/jira/browse/PARQUET-XXX - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Tests - [X] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: It can only be validated on an M1 mac. the normal CI test runs will verify the changes do not cause regressions elsewhere. Note: this patch was based on hadoop's HADOOP-17939 patch https://github.com/apache/hadoop/pull/3486 -this was needed to fix the hadoop build ### Commits - [X] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [X] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain Javadoc that explain what it does > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541088#comment-17541088 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1135004012 fyi @dongjoon-hyun > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541068#comment-17541068 ] Steve Loughran commented on PARQUET-2150: - same issue and solution as HADOOP-17939 > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)