zeroshade commented on issue #133: URL: https://github.com/apache/arrow-go/issues/133#issuecomment-2366859423
> More philosophically, is this behavior actually in the arrow spec? Or is the spec just about the in-memory format, and implementations are free to define their compute/casting behavior as desired? It seems funny/non-ideal to me that it is possible for the go implementation of this casting to drift away from the c++ implementation. I would think it would be better if there was some test that verified they both had the same behavior? This is the case, the spec *just* defines the in-memory format. Anything implementing compute and casting behavior is outside the scope of the spec itself as different compute engines are free to have different behaviors. For a spec attempting to create standardized behavior on compute functionality, there is https://substrait.io That said, I agree the ideal would be for the go implementation to follow the C++ implementation where possible for these things. > I went looking for an equivalent nullable to non-nullable cast test for non-struct arrays, but couldn't find any. Is this possibly not tested? Or do I misunderstand the arrow spec, and all datatypes EXCEPT struct fields are nullable, so there is no eg non-nullable uint16? By the Arrow spec, "nullability" is a feature of "fields" as it is simply a piece of metadata to inform consumers of data. So you can only mark something as non-nullable by way of a schema (i.e. top-level columns in a record batch can be marked non-nullable via their fields in the schema) or by virtue of it being a nested field of some other type (i.e. struct, list, map, etc. type) which contains fields to describe its children. So you can have a non-nullable uint16, if the schema/parent struct/parent list type defines the column field as non-nullable. Since casting doesn't accept record batches, only Array or ChunkedArray, the only cases where you might run into this nullability issue are in the child fields of these nested types. Since we do provide factories for constructing lists with non-nullable element fields, we probably should add tests to verify we handle nullability/non-nullability for lists/maps/etc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
