Hello, It is an proposition to add new logical types for the Apache Arrow data format.
As Melik-Adamyan said, it is quite easy to convert 5-bytes FixedSizeBinary to PostgreSQL's inet data type by the Arrow_Fdw module (an extension of PostgreSQL responsible to data conversion), however, it is not obvious for readers whether it is network-address or just a bunch of small binary. https://www.postgresql.org/docs/11/sql-importforeignschema.html PostgreSQL has IMPORT FOREIGN SCHEMA command; that allows to define a foreign table according to schema information of the external data source. In case of Arrow_Fdw, we can define a foreign table without manual listing of columns with data types as follows: IMPORT FOREIGN SCHEMA foo FROM arrow_fdw INTO public OPTIONS (file '/opt/nvme/foo.arrow'); In this case, Schema definition in the 'foo.arrow' can tell PostgreSQL how many columns are defined and its name, data types and so on. However, PostgreSQL may be confusing to convert the FixedSizeBinary (width=5) without any metadata support. It may be 'inet4' data type, and it also may be 'char(5)'. One idea is utilization of custom_metadata field in the Field-node. We may be able to mark it is a network address, not a blob. However, I didn't find out specification of the custom_metadata. I expect network address is widely used for log-data processing area, and not small number of application will support it. If so, it is not too niche requirement for a new logical data type definition in the Apache Arrow data format. Best regards, 2019年4月30日(火) 15:13 Micah Kornfield <emkornfi...@gmail.com>: > > Hi KaiGai Kohei, > Can you clarify if you are looking for advice on modelling these types or > proposing to add new logical types to the Arrow specification? > > Thanks, > Micah > > On Monday, April 29, 2019, Kohei KaiGai <kai...@heterodb.com> wrote: > > > Hello folks, > > > > How about your opinions about network address types support in Apache > > Arrow data format? > > Network address always appears at network logs massively generated by > > any network facilities, > > and it is a significant information when people analyze their backward > > logs. > > > > I'm working on Apache Arrow format mapping on PostgreSQL. > > http://heterodb.github.io/pg-strom/arrow_fdw/ > > > > This extension allows to read Arrow files as if PostgreSQL's table > > using foreign table. > > Data types of Arrow shall be mapped to relevant PostgreSQL's data type > > according to the above > > documentation. > > > > https://www.postgresql.org/docs/current/datatype-net-types.html > > PostgreSQL supports some network address types and operators. > > For example, we can put a qualifier like: WHERE addr <<= inet > > '192.168.1.0/24' , to find out all > > the records in the subnet of '192.168.1.0/24'. > > > > Probably, these three data types are now sufficient for most network > > logs: inet4, inet6 and macaddr. > > * inet4 is 32bit + optional 8bit (for netmask) fixed length array > > * inet6 is 128bit + optional 8bit (for netmask) fixed length array > > * macaddr is 48bit fixed length array. > > > > I don't favor to map the inetX types on flexible length Binary data > > type, because it takes 32bit offset > > to indicate 32 or 40bit value, inefficient so much, even though > > PostgreSQL allows to mix inet4/inet6 > > data types in a same column. > > > > Thanks, > > -- > > HeteroDB, Inc / The PG-Strom Project > > KaiGai Kohei <kai...@heterodb.com> > > -- HeteroDB, Inc / The PG-Strom Project KaiGai Kohei <kai...@heterodb.com>
