schaefer01 opened a new issue, #35477:
URL: https://github.com/apache/arrow/issues/35477
### Describe the bug, including details regarding any error messages,
version, and platform.
I found the index by searching for the initial data as a pattern in the data
block
But this isn't the right way to do this
my code:
pa_result = pa.array(result, type=pa.struct([('r', pa.int16()), ('i',
pa.int16())]))
print("len(pa_result)=",len(result))
8192
xfer_block = pa_to_bytes(pa_result)
len(xfer_block)=",len(xfer_block)
33224
8K bytes = (8*1024) * 4 bytes per complex value
33224 - (1024*8*4) = 456 -> this is overhead, 448 in front and 8
(checksum?) at the end
I use NATS on a mac to xmit the bytes to a NATS server on ubuntu which
sends the data back
to a NATS receiver on the mac (xmit and receive are in the same python
program using asynio),
and then use the arrow stream buffer reader to read the block. The
purpose is to test py_arrow and
NATS, then use py_arrow and MQTT for benchmarking which protocol, NATS or
MQTT is better.
The rational for py_arrow is a potential customer recommended arrow.
buf = memoryview(data)
with pa.input_stream(buf) as stream:
size = stream.size()
metadata = stream.metadata()
print("size=",size)
size= 33224
print("metadata=",metadata)
metadata= {}
stream.seek(448)
recovery = stream.read(16)
The first 16 bytes match, I'm just looking for an explanation and perhaps a
less "hacky" way to
determine the size and placement of overhead.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]