[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568693#comment-17568693 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran closed pull request #970: PARQUET-2150: parquet-protobuf to compile on Mac M1 URL: https://github.com/apache/parquet-mr/pull/970 > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568692#comment-17568692 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1189454282 resolved by #973 > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17563878#comment-17563878 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1177929495 oh, upgrading is better! > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17561764#comment-17561764 ] ASF GitHub Bot commented on PARQUET-2150: - shangxinli commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1172920665 @steveloughran Thanks for the explanation! Do you have concerns if we use [PR-973](https://github.com/apache/parquet-mr/pull/973)? It seems we can rely on proto-buf itself to solve the issue. @sunchao please add @steveloughran as co-author for your PR. > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556450#comment-17556450 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1160638077 this patch is based on Dongjoon;s one for hadoop, tells maven to use the x86 artifact on macbook m1 builds. the sunchao one switches to a version of protobuf with a genuine mac m1 artifacts, a version which should also include some CVE fixes. > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556125#comment-17556125 ] ASF GitHub Bot commented on PARQUET-2150: - sunchao commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1159806290 @shangxinli my approach is different from @steveloughran 's one. Since newer version of protobuf already provides M1 artifacts, upgrade will solve the issue > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17555823#comment-17555823 ] ASF GitHub Bot commented on PARQUET-2150: - shangxinli commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1159351267 @sunchao I see your change is to upgrade the protobuf version. Is that required to solve this problem, which I don't see in this PR. > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17552481#comment-17552481 ] ASF GitHub Bot commented on PARQUET-2150: - sunchao commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1151738855 Oops @steveloughran just found out this PR. I opened #973 for the same purpose > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545826#comment-17545826 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1145835507 tested lasted patch on * macbook m1 * macbook x86 * ec2 x86 i would set up my raspberry pi for a build too -I just fear having to build the thrift compiler > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[
https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543966#comment-17543966
]
ASF GitHub Bot commented on PARQUET-2150:
-
steveloughran commented on PR #970:
URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1141216939
ok, somehow I've broken the default classifier logic, which is of course
critical.
downgrading this to a draft until it's ready to be reviewed
```
Error: Failed to execute goal
com.github.os72:protoc-jar-maven-plugin:3.11.4:run (generate-sources) on
project parquet-protobuf: Error resolving artifact:
com.google.protobuf:protoc:3.16.1:exe:${os.detected.classifier}: Could not find
artifact com.google.protobuf:protoc:exe:${os.detected.classifier}:3.16.1 in
jitpack.io (https://jitpack.io/)
Error:
Error: Try downloading the file manually from the project website.
Error:
Error: Then, install it using the command:
Error: mvn install:install-file -DgroupId=com.google.protobuf
-DartifactId=protoc -Dversion=3.16.1 -Dclassifier=${os.detected.classifier}
-Dpackaging=exe -Dfile=/path/to/file
Error:
Error: Alternatively, if you host your own repository you can deploy the
file there:
Error: mvn deploy:deploy-file -DgroupId=com.google.protobuf
-DartifactId=protoc -Dversion=3.16.1 -Dclassifier=${os.detected.classifier}
-Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
```
> parquet-protobuf to compile on mac M1
> -
>
> Key: PARQUET-2150
> URL: https://issues.apache.org/jira/browse/PARQUET-2150
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-protobuf
>Affects Versions: 1.13.0
>Reporter: Steve Loughran
>Priority: Major
>
> parquet-protobuf module fails to compile on Mac M1 because the maven protoc
> plugin cannot find the native osx-aarch_64:3.16.1 binary.
> the build needs to be tweaked to pick up the x86 binaries
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541655#comment-17541655 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1136296983 its the maven protoc compiler plugin; wants to pull in something native. i have no idea why, but do know, if it is not there, you get to see a stack trace. ``` [ERROR] Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.11.4:run (generate-sources) on project parquet-protobuf: Error resolving artifact: com.google.protobuf:protoc:3.16.1: Could not find artifact com.google.protobuf:protoc:exe:osx-aarch_64:3.16.1 in jitpack.io (https://jitpack.io) [ERROR] [ERROR] Try downloading the file manually from the project website. [ERROR] [ERROR] Then, install it using the command: [ERROR] mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file [ERROR] [ERROR] Alternatively, if you host your own repository you can deploy the file there: [ERROR] mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.16.1 -Dclassifier=osx-aarch_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] [ERROR] [ERROR] [ERROR] com.google.protobuf:protoc:exe:3.16.1 [ERROR] [ERROR] from the specified remote repositories: [ERROR] jitpack.io (https://jitpack.io, releases=true, snapshots=true), [ERROR] apache.snapshots (https://repository.apache.org/snapshots, releases=false, snapshots=true), [ERROR] maven-central (https://repo1.maven.org/maven2, releases=true, snapshots=false) [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :parquet-protobuf ``` > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541585#comment-17541585 ] ASF GitHub Bot commented on PARQUET-2150: - theosib-amazon commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1136102990 I looked up parquet-protobuf and only found Java code. Is there some use of JNI here that I'm not finding? > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541089#comment-17541089 ] ASF GitHub Bot commented on PARQUET-2150: - dongjoon-hyun commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1135005341 Thank you for pinging me, @steveloughran . > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[
https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541087#comment-17541087
]
ASF GitHub Bot commented on PARQUET-2150:
-
steveloughran opened a new pull request, #970:
URL: https://github.com/apache/parquet-mr/pull/970
Force use of x86 protocol executable in builds onl macOS systems,
including those with ARM64 architectures, for which native binaries
do not exist.
### Jira
- [X] My PR addresses the following [Parquet
Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references
them in the PR title. For example, "PARQUET-1234: My Parquet PR"
- https://issues.apache.org/jira/browse/PARQUET-XXX
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
It can only be validated on an M1 mac. the normal CI test runs will verify
the changes do not cause regressions elsewhere.
Note: this patch was based on hadoop's HADOOP-17939 patch
https://github.com/apache/hadoop/pull/3486 -this was needed to fix the hadoop
build
### Commits
- [X] My commits all reference Jira issues in their subject lines. In
addition, my commits follow the guidelines from "[How to write a good git
commit message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain Javadoc that
explain what it does
> parquet-protobuf to compile on mac M1
> -
>
> Key: PARQUET-2150
> URL: https://issues.apache.org/jira/browse/PARQUET-2150
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-protobuf
>Affects Versions: 1.13.0
>Reporter: Steve Loughran
>Priority: Major
>
> parquet-protobuf module fails to compile on Mac M1 because the maven protoc
> plugin cannot find the native osx-aarch_64:3.16.1 binary.
> the build needs to be tweaked to pick up the x86 binaries
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541088#comment-17541088 ] ASF GitHub Bot commented on PARQUET-2150: - steveloughran commented on PR #970: URL: https://github.com/apache/parquet-mr/pull/970#issuecomment-1135004012 fyi @dongjoon-hyun > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2150) parquet-protobuf to compile on mac M1
[ https://issues.apache.org/jira/browse/PARQUET-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541068#comment-17541068 ] Steve Loughran commented on PARQUET-2150: - same issue and solution as HADOOP-17939 > parquet-protobuf to compile on mac M1 > - > > Key: PARQUET-2150 > URL: https://issues.apache.org/jira/browse/PARQUET-2150 > Project: Parquet > Issue Type: Improvement > Components: parquet-protobuf >Affects Versions: 1.13.0 >Reporter: Steve Loughran >Priority: Major > > parquet-protobuf module fails to compile on Mac M1 because the maven protoc > plugin cannot find the native osx-aarch_64:3.16.1 binary. > the build needs to be tweaked to pick up the x86 binaries -- This message was sent by Atlassian Jira (v8.20.7#820007)
