Attendees:
* Alex
* Dan
* Zhenxiao
* Mickael
* Blue
* Ippo
* Julien
* Jason
Performance improvements
* not very performant when compared to record-oriented storage (even for
flat, but wide tables)
* Both write side and read side
* Check encodings
- Dictionary defaults may be bad
- What are reasonable numbers?
* Refactoring the read path
- Decode columns at one time
- Decode column for an entire record at a time
- Layers of abstraction can be reduced (depends on API definition)
* Unions are problematic - can we special case?
Implemented predicate push-down using dictionaries, stats in presto
* Mind-boggling numbers for certain queries
* Sorted data based on metrics, getting speed-ups of 170x
Vectorized read path
* Going to address Julien's comments to try to get it in
* ~2x out of it
* Let's make sure packages are marked public/private
ByteBuffer update:
* Defer writing nulls into empty maps may be causing problems
* More work to make semver happy
Compat testing is happening
Things that need to be reviewed:
* Dictionary encoding to the thrift impl (Blue will review)
* PR for getting dictionaries (Reviewed)
* Page size check implmentation (Blue will review)
* Mem manager flaky tests PR
Hadoop world in NY is next week. See you there!
rb
--
Ryan Blue
