LouisGariepy commented on PR #6466: URL: https://github.com/apache/arrow-datafusion/pull/6466#issuecomment-1565595402
**TLDR:** I definitely have `gcc` installed, I stumbled upon this problem while trying to build DataFusion on my main computer (which I use for daily Rust dev). > You shouldn't need protobuf to build DataFusion This seems to be incorrect (evidence below). As far as I can tell `cargo build` errors out saying that Protobuf is required, and in the case of Fedora Linux, the required dependencies are indeed acquired from `protobuf-devel`. I provided the whole reproducer process below! --- @tustvold Here's a reproducer for a more convincing argument. Sorry, it's a bit lengthy, but I wanted to make everything crystal clear: # Reproducer ### Starting up a fresh Fedora container ```bash podman run -it fedora bash ``` ### Installing basic development libraries ```bash # Linker dnf install -y gcc # VCS dnf install -y git ``` ### Installing Rust ```bash curl https://sh.rustup.rs -sSf | sh -s -- -y source "$HOME/.cargo/env" ``` ### Checking that Rust is installed correctly with all the required dependencies ```bash cargo new hello-world cd hello-world cargo run cd .. rm -rf hello-world ``` You should be greeted with a familiar sight: ``` Created binary (application) `hello-world` package Compiling hello-world v0.1.0 (/hello-world) Finished dev [unoptimized + debuginfo] target(s) in 0.22s Running `target/debug/hello-world` Hello, world! ``` This clearly shows that we are not missing any generic build dependencies. Rust is good to go! ### Clone and build DataFusion ```bash # Let's pull in submodules for the sake of showing that we're not missing anything git clone --recurse-submodules https://github.com/apache/arrow-datafusion.git cd arrow-datafusion cargo build ``` You will see this: ``` thread 'main' panicked at 'Could not find `protoc` installation and this build crate cannot proceed without this knowledge. If `protoc` is installed and this crate had trouble finding it, you can set the `PROTOC` environment variable with the specific path to your installed `protoc` binary.If you're on debian, try `apt-get install protobuf-compiler` or download it from https://github.com/protocolbuffers/protobuf/releases ``` **Oh no**. ### Fixing the error by reading the docs So, apparently Protobufs **are** required to build DataFusion. The savvy user will go straight to the contributor guide and see a reassuring line of code: > `dnf install -y protobuf-compiler` This should work, right? Let's try again: ```bash cargo build ``` **Oh my!** We get a crazy cursed error message! Open the `details` collapsible section below to get a good scare. <details> Error: Custom { kind: Other, error: "protoc failed: google/protobuf/any.proto: File not found.\nsubstrait/extensions/extensions.proto:6:1: Import \"google/protobuf/any.proto\" was not found or had errors.\nsubstrait/extensions/extensions.proto:77:3: \"google.protobuf.Any\" is not defined.\nsubstrait/extensions/extensions.proto:80:3: \"google.protobuf.Any\" is not defined.\ngoogle/protobuf/empty.proto: File not found.\nsubstrait/type.proto:6:1: Import \"google/protobuf/empty.proto\" was not found or had errors.\nsubstrait/type.proto:192:7: \"google.protobuf.Empty\" is not defined.\nsubstrait/algebra.proto:6:1: Import \"google/protobuf/any.proto\" was not found or had errors.\nsubstrait/algebra.proto:7:1: Import \"substrait/extensions/extensions.proto\" was not found or had errors.\nsubstrait/algebra.proto:8:1: Import \"substrait/type.proto\" was not found or had errors.\nsubstrait/algebra.proto:46:7: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.prot o:52:7: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:40:5: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:25:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:78:5: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:89:5: \"google.protobuf.Any\" is not defined.\nsubstrait/algebra.proto:140:9: \"google.protobuf.Any\" is not defined.\nsubstrait/algebra.proto:95:5: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:60:3: \"NamedStruct\" is not defined.\nsubstrait/algebra.proto:64:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:152:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:178:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:187:3: \"substrait.extensions.AdvancedExtension\" is not defined.\n substrait/algebra.proto:198:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:214:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:236:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:244:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:254:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:271:3: \"google.protobuf.Any\" is not defined.\nsubstrait/algebra.proto:277:3: \"google.protobuf.Any\" is not defined.\nsubstrait/algebra.proto:284:3: \"google.protobuf.Any\" is not defined.\nsubstrait/algebra.proto:338:7: \"google.protobuf.Any\" is not defined.\nsubstrait/algebra.proto:303:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:380:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:386:3: \"google.protobuf.Any\" is not defined.\n substrait/algebra.proto:397:3: \"NamedStruct\" is not defined.\nsubstrait/algebra.proto:447:3: \"NamedStruct\" is not defined.\nsubstrait/algebra.proto:510:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:537:3: \"substrait.extensions.AdvancedExtension\" is not defined.\nsubstrait/algebra.proto:544:5: \"Type\" is not defined.\nsubstrait/algebra.proto:696:16: \"Type.Parameter\" is not defined.\nsubstrait/algebra.proto:700:7: \"google.protobuf.Any\" is not defined.\nsubstrait/algebra.proto:627:7: \"Type\" is not defined.\nsubstrait/algebra.proto:629:7: \"Type.List\" is not defined.\nsubstrait/algebra.proto:630:7: \"Type.Map\" is not defined.\nsubstrait/algebra.proto:772:5: \"Type\" is not defined.\nsubstrait/algebra.proto:806:5: \"Type\" is not defined.\nsubstrait/algebra.proto:905:5: \"Type\" is not defined.\nsubstrait/algebra.proto:943:5: \"Type\" is not defined.\nsubstrait/algebra.proto:1258:3: \"Type\" is not defined.\n" } </details> ### Here's where my suggestion comes in Looking at the error message, we can see lines like > `protoc failed: google/protobuf/any.proto: File not found` Google maintains protobufs, and the error says we're missing some protobuf files, so my reasoning was that installing the development version of protobuf would solve this issue. Let's remove the protobuf dependency we installed. This is not strictly required, but I want to showcase that my solution works without anything else being needed: ```bash dnf remove -y protobuf-compiler ``` Now let's install the protobuf development version ```bash dnf install -y protobuf-devel ``` And we try once again: ```bash cargo build ``` Tadaa! :) Everything builds this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
