I was watching a video presentation by Jay Kreps where we was talking about some data challenges he was dealing with that he solved with Avro. The thing is, he glosses over the details.
I am in a situation where I am dealing with having to ingest CSVs. The files are picked up by SSIS and imported into a data warehouse. My problem is the files are created by a system that apparently isn’t stable. The developers of the system like to add columns without warning. What is particularly annoying is they can’t seem to decide how to represent negative numbers. Sometimes they have a negative sign which is fine. Sometimes numbers come in accounting notation with parentheses to denote negative numbers. That is not fine as SQL Server doesn’t understand that as a negative. Can I somehow use Avro to ENSURE that the file from the third party system comes as something expected? Adaryl "Bob" Wakefield, MBA Principal Mass Street Analytics, LLC 913.938.6685 www.massstreet.net www.linkedin.com/in/bobwakefieldmba Twitter: @BobLovesData
