barbuz opened a new issue, #34898:
URL: https://github.com/apache/arrow/issues/34898

   ### Describe the enhancement requested
   
   ### Motivation
   I need to store a schema in a permanent way, as I'm building a process where 
several CSV files with the same structure will be converted to Parquet over 
time, but inferring the schema of the data from each file does not guarantee 
consistency. I could use something like pickle to dump a binary version of the 
schema but this is not very portable, and a human-readable representation would 
allow to do simple modifications by hand if the need arises (e.g. adding or 
removing columns). For this reason I'm trying to save the schema in a json file 
and then to read it and re-build a pyarrow Schema. 
   
   The first step is easy thanks to each type having a nice string 
representation, but going backwards is harder as I could not find any way of 
building a type or a Field from a string that did not break on more complex 
types such as `timestep[ms, tz=utc]`. I ended up having to implement my own 
function to parse string representations of types and build the appropriate 
pyarrow objects.
   
   ### Idea
   The basic idea would be implementing a function that takes a string 
representation of a type and returns the corresponding pyarrow type object. 
   Other things that could make this process easier would be a `to_dict()` 
function for Schema that basically builds the dictionary obtained combining the 
`names` and `types` lists of a Schema, or a `from_string(s)` function that can 
reverse the `to_string()` function.
   
   I can contribute my code to convert strings to type objects, but as I've 
never contributed to this project before I would like some advice on whether 
this is something desired and what would be the best way to integrate this 
functionality with the rest of the code base.
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to