visitor-4137 opened a new issue #10553:
URL: https://github.com/apache/arrow/issues/10553
I had written a cpp code to run on a rhel 8 machine which reads a csv file
through a record batch reader and then writes these record batches to a feather
file.. while compiling the program, i get the following error -
> /lib/../lib64/libarrow.so: undefined reference to
`LZ4F_resetDecompressionContext'
The code i have is -
```
#include <arrow/api.h>
#include <arrow/filesystem/filesystem.h>
#include <arrow/filesystem/localfs.h>
#include <arrow/ipc/feather.h>
#include <arrow/ipc/writer.h>
#include <chrono>
#include <iostream>
#include <string>
#include <arrow/csv/api.h>
arrow::fs::LocalFileSystem file_system;
void make_table_from_csv(std::string file_name,std::string output_file) {
arrow::io::IOContext io_context = arrow::io::default_io_context();
// open the input file
std::shared_ptr<arrow::io::InputStream> input =
file_system.OpenInputStream(file_name).ValueOrDie();
std::shared_ptr<arrow::io::OutputStream> output =
file_system.OpenOutputStream(output_file).ValueOrDie();
auto read_options = arrow::csv::ReadOptions::Defaults();
read_options.block_size = (1 << 28); // decides what size of chunk we will
read in a single record batch, increasing block_size increases size of each
chunk.
auto parse_options = arrow::csv::ParseOptions::Defaults();
auto convert_options = arrow::csv::ConvertOptions::Defaults();
auto stream_reader =
arrow::csv::StreamingReader::Make(io_context,input,read_options,parse_options,convert_options);
if (!stream_reader.ok()) {
std::cerr << "Error while reading csv file." << std::endl;
}
std::shared_ptr<arrow::csv::StreamingReader> reader = *stream_reader;
std::shared_ptr <arrow::RecordBatch> chunk;
reader -> ReadNext(&chunk);
auto schema = chunk -> schema();
auto rec_writer =
arrow::ipc::MakeStreamWriter(output.get(),schema).ValueOrDie();
rec_writer -> WriteRecordBatch(*chunk);
std::unique_ptr <arrow::RecordBatchBuilder> rec_builder;
arrow::RecordBatchBuilder::Make(schema,arrow::default_memory_pool(),&rec_builder);
while(true){
rec_builder -> Flush(&chunk);
reader -> ReadNext(&chunk);
if(chunk == NULL){
break;
}
else{
rec_writer -> WriteRecordBatch(*chunk);
}
}
output -> Close();
return;
}
int main(int argc, char *argv[]) {
make_table_from_csv(std::string(argv[1]),"data.feather");
return 0;
}
```
Built the program with -
> g++ -O3 a.cpp -larrow
The LZ4 version is -
> *** LZ4 command line interface 64-bits v1.7.5, by Yann Collet ***
What is the error and why is it not working ?
I know updating the LZ4 could be a simple solution to this but as far as i
know there is no such restriction of using a certain LZ4 version, correct me in
case if i am wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
