Hello,
I am trying to use TypedColumnReader<DType>::ReadBatchSpaced to read
ByteArrays.
In my use case, I am only reading flat data but it is data with nulls,
which is why I am using ReadBatchSpaced, because its the only way I have
found to be able to read data and also know which values are null. My code
looks something like this:
int64_t total_values_read = 0;
int64_t valid_bits_offset = 0;
int64_t levels_read = 0;
int64_t values_read = 0;
int64_t null_count = -1;
std::vector<parquet::ByteArray> values(numRecords);
std::vector<int16_t> dresult(numRecords, -1);
std::vector<int16_t> rresult(numRecords, -1);
std::vector<uint8_t> valid_bits(numRecords, 255);
while (total_values_read < numRecords){
int64_t rows_read = parquetTypeReader->ReadBatchSpaced(numRecords,
dresult.begin() + total_values_read, rresult.data() + total_values_read,
values.begin() + total_values_read, valid_bits.data() + total_values_read,
valid_bits_offset, &levels_read, &values_read, &null_count);
total_values_read += rows_read;
}
When I follow this pattern, and I need to do multiple calls to
ReadBatchSpaced, I can get garbage results in my vector of ByteArrays after
the first call in the loop. If I were reading a more primitive data type, I
do not have this issue.
So far the only way I have been able to get this to work is by using an
intermediary buffer to hold the ByteArray data, which would look more
something like this:
std::vector<parquet::ByteArray> values(numRecords);
std::vector<parquet::ByteArray> intermediateBuffer(numRecords);
std::vector<int16_t> dresult(numRecords, -1);
std::vector<int16_t> rresult(numRecords, -1);
std::vector<uint8_t> valid_bits(numRecords, 255);
while (total_values_read < numRecords){
int64_t rows_read = parquetTypeReader->ReadBatchSpaced(numRecords,
dresult.begin() + total_values_read, rresult.data() + total_values_read,
&(intermediateBuffer[0]), valid_bits.data() + total_values_read,
valid_bits_offset, &levels_read, &values_read, &null_count);
std::copy(intermediateBuffer.begin(), intermediateBuffer.begin() +
rows_read, values.begin() + total_values_read);
total_values_read += rows_read;
}
Any ideas as to what I am doing incorrectly in my first example? Do I
always need to use an intermediate buffer?
Thanks!
William
[image: BlazingDB] <https://htmlsig.com/t/000001C10NAQ>
William Malpica / VP of Engineering
[email protected] / 859.619.0708
BlazingDB
www.blazingdb.com
[image: Twitter] <https://htmlsig.com/t/000001C0FQGA> [image: Facebook]
<https://htmlsig.com/t/000001C3NKBJ> [image: LinkedIn]
<https://htmlsig.com/t/000001BZZ9SC> [image: Vimeo]
<https://htmlsig.com/t/000001BYP7S2>
