Matt, Good overview.
>Avro is similar, but the schema must always be provided with the data. In >the case of NiFi DataFlows, it's likely more efficient to send the schema >once as an initialization packet (I can't remember the real term in NiFi), >then the rows can be streamed individually, in batches of user-defined size, >sampled, etc. Do you mean "Initial Information Packet" or "IIP" ? Mr. Morrison classical FBP includes such functionality, used often for configuration. As far I know NiFi don't have such concept. But NiFi ExecuteSQL uses Avro with schema for query result. Result is one big FlowFile which includes both schema and all rows. Processor just creates schema from JDBC metadata, writes it to Avro container and next all rows are written to Avro container. Writing and reading such file is done using streaming, so result can very big. Thanks Toivo -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Common-data-exchange-formats-and-tabular-data-tp3508p4271.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
