marchostau opened a new issue, #36101:
URL: https://github.com/apache/arrow/issues/36101
### Describe the enhancement requested
I'm trying to put dataframes of different types in multiprocessing shared
memory. When I'm using dataframes that contain python types I've no problem,
but when I use pyarrow types I've problems closing the Shared Memories. I've a
function to put a dataframe to the Shared Memory and another to receive this
dataframe. I have two problems when I'm using Arrow types like string[pyarrow]
inside a dataframe:
When I call the close on the sm_put and after receiving the dataframe I call
the close on the sm_get I get instantly an error, that causes the code to stop
executing: sm_get.close() -> self.close() -> self._buf.release() ->
BufferError: memoryview has 1 exported buffer.
When I call the close on the sm_put, and I don't call the close on the
sm_get, I've no problem instantly, but I get a warning after a while (that
surprises me because I don't call any function self.close()): self.close() ->
self._buf.release() -> BufferError: memoryview has 1 exported buffer.
My functions to put the dataframe to the shared memory and to receive the
dataframe from shared memory are these:
> def put_df(data, logTime, logSize, usingArrow=False):
> if(usingArrow):
> table = (pyarrow.Table.from_pandas(data)).combine_chunks()
> record_batch = table.to_batches(max_chunksize = sys.maxsize)[0]
> else:
> record_batch = pa.RecordBatch.from_pandas(data)
> # Determine the size of buffer to request from the shared memory
> mock_sink = pa.MockOutputStream()
> stream_writer = pa.RecordBatchStreamWriter(mock_sink,
record_batch.schema)
> stream_writer.write_batch(record_batch)
> stream_writer.close()
> data_size = mock_sink.size()
>
> sm_put = SharedMemory(create=True, size=data_size)
> buffer = pa.py_buffer(sm_put.buf)
>
> # Write the PyArrow RecordBatch to SharedMemory
> stream = pa.FixedSizeBufferWriter(buffer)
> stream_writer = pa.RecordBatchStreamWriter(stream, record_batch.schema)
> stream_writer.write_batch(record_batch)
> stream_writer.close()
>
> del stream_writer
> del stream
> del buffer
> if(usingArrow):
> del table
> del record_batch
>
> sm_put.close()
> return sm_put.name
>
> def get_df(sm_get_name, logTime, logSize, usingArrow=False):
> sm_get = SharedMemory(name = sm_get_name, create = False)
> buffer = pa.BufferReader(sm_get.buf)
>
> # Convert object back into an Arrow RecordBatch
> reader = pa.RecordBatchStreamReader(buffer)
> record_batch = reader.read_next_batch()
>
> # Convert back into Pandas
> if(usingArrow):
> data = record_batch.to_pandas(types_mapper = pd.ArrowDtype)
> else:
> data = record_batch.to_pandas()
>
> del buffer
> del reader
> del record_batch
>
> sm_get.close()
> sm_get.unlink()
>
> return data
>
> if __name__ == '__main__':
> names = [ "0", "1"]
> types = { "0": "string[pyarrow]", "1": "string[pyarrow]" }
> df = pd.read_csv(open(file, "r"), index_col=False, header=None,
names=names, dtype=types, engine="c")
> sm_get_name = put_df(df)
> data = get_df(sm_get_name=sm_get_name)
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]