vera-rubin opened a new issue, #2719:
URL: https://github.com/apache/orc/issues/2719
The default C++ writer stores FLOAT statistics from the double input before
narrowing it to binary32. A one-row file containing `0.1` returns no rows for
an equality search argument whose literal exactly equals the stored FLOAT value.
## Version and build
Apache ORC `3.0.0-SNAPSHOT`, commit
`2069271e06c92197c66def9d4a0214753ac9a3b8`. Tested with GCC 13.3.0, CMake
3.28.3 and Python 3.12.3 on Ubuntu, Linux 6.17.0-1020-oracle, x86_64. C++
Release build, default numeric vectors and ZSTD compression. CMake Java and C++
tests were off. As checked on 2026-10-12, upstream `main` matches this commit.
## Reproduction
Save this as `repro.cc`, then run the Bash commands below from the same
directory. Reproducer: CC0-1.0.
```cpp
#include <orc/OrcFile.hh>
#include <orc/Statistics.hh>
#include <orc/Vector.hh>
#include <orc/sargs/SearchArgument.hh>
#include <iostream>
#include <iomanip>
int main(int argc,char**argv){
if(argc!=2)return 2;
auto schema=orc::Type::buildTypeFromString("struct<x:float>");
{
auto out=orc::writeLocalFile(argv[1]);
auto writer=orc::createWriter(*schema,out.get(),orc::WriterOptions());
auto batch=writer->createRowBatch(1);
auto& root=dynamic_cast<orc::StructVectorBatch&>(*batch);
auto& x=dynamic_cast<orc::DoubleVectorBatch&>(*root.fields[0]);
root.numElements=x.numElements=1;x.data[0]=0.1;
writer->add(*batch);writer->close();
}
auto
reader=orc::createReader(orc::readLocalFile(argv[1]),orc::ReaderOptions());
auto stats=reader->getColumnStatistics(1);
auto& f=dynamic_cast<orc::DoubleColumnStatistics&>(*stats);
std::cout<<std::setprecision(17)<<"minimum "<<f.getMinimum()<<"\nmaximum
"<<f.getMaximum()<<"\nsum "<<f.getSum()<<"\n";
for(bool filtered:{false,true}){
orc::RowReaderOptions opts;
if(filtered)
opts.searchArgument(orc::SearchArgumentFactory::newBuilder()->startAnd()
.equals("x",orc::PredicateDataType::FLOAT,orc::Literal(0.100000001490116119384765625)).end().build());
auto rows=reader->createRowReader(opts);auto
batch=rows->createRowBatch(1);uint64_t count=0;
while(rows->next(*batch)){
auto& root=dynamic_cast<orc::StructVectorBatch&>(*batch);
auto& x=dynamic_cast<orc::DoubleVectorBatch&>(*root.fields[0]);
for(uint64_t i=0;i<root.numElements;i++)
std::cout<<(filtered?"filtered_value ":"plain_value ")<<x.data[i]<<"\n";
count+=root.numElements;
}
std::cout<<(filtered?"filtered_rows ":"plain_rows ")<<count<<"\n";
}
}
```
```sh
git clone https://github.com/apache/orc.git orc
git -C orc checkout --detach 2069271e06c92197c66def9d4a0214753ac9a3b8
cmake -S orc -B orc-build \
-DBUILD_JAVA=OFF -DBUILD_CPP_TESTS=OFF -DBUILD_TOOLS=ON \
-DCMAKE_BUILD_TYPE=Release -DSTOP_BUILD_ON_WARNING=OFF
cmake --build orc-build --parallel 2
g++ -std=c++17 -O2 -Iorc/c++/include -Iorc-build/c++/include \
repro.cc orc-build/c++/src/liborc.a \
orc-build/_deps/protobuf-build/liborc_vendored_protobuf.a \
orc-build/_deps/zlib-build/liborc_vendored_zlib.a \
orc-build/_deps/snappy-build/liborc_vendored_snappy.a \
orc-build/_deps/lz4-build/liborc_vendored_lz4.a \
orc-build/_deps/zstd-build/lib/liborc_vendored_zstd.a \
-pthread -o repro
./repro input.orc
```
## Expected and observed
The [column statistics
specification](https://github.com/apache/orc/blob/2069271e06c92197c66def9d4a0214753ac9a3b8/site/specification/ORCv1.md#L260-L265)
defines statistics over column values. [FLOAT
encoding](https://github.com/apache/orc/blob/2069271e06c92197c66def9d4a0214753ac9a3b8/site/specification/ORCv1.md#L1051-L1055)
uses IEEE 754 binary32. The [C++ vector
documentation](https://github.com/apache/orc/blob/2069271e06c92197c66def9d4a0214753ac9a3b8/site/_docs/core-cpp.md#L40-L52)
represents FLOAT input with `DoubleVectorBatch`.
The stored value has bits `0x3dcccccd`. Its exact value is
`13421773/134217728`, or `0.100000001490116119384765625`. With one non-null
row, minimum, maximum and sum should all equal that value. The equality search
argument should retain the row.
Observed:
```text
minimum 0.10000000000000001
maximum 0.10000000000000001
sum 0.10000000000000001
plain_value 0.10000000149011612
plain_rows 1
filtered_rows 0
```
An independent byte decoder confirms the DATA value and the mismatching
statistics at file, stripe and row-group levels. The literal equals the decoded
FLOAT exactly.
## Source and scope
In
[FloatingColumnWriter::add](https://github.com/apache/orc/blob/2069271e06c92197c66def9d4a0214753ac9a3b8/c%2B%2B/src/ColumnWriter.cc#L869-L881),
DATA uses `static_cast<float>(doubleData[i])`, while statistics use the
original double. Statistics should use the value serialized as FLOAT.
Independent C++ controls pass with pre-rounded input, tight float vectors,
exact `0.5`, and DOUBLE. Negative `-0.1` also reproduces. No read-type
conversion is used. The independent harness initializes row counts and null
flags and closes the writer before reading. Predicate pushdown may retain extra
rows, but it must retain this matching row.
Java's FLOAT writer was compared with source at the same commit only. No
Java writer runtime comparison or other compiler or architecture test is
claimed.
Found by ChipLab's AI verification agents and independently reproduced.
ChipLab ref: CLAB-2026-863
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
