Thanks Sidd for the context! On Mon, Jun 1, 2020 at 9:38 AM siddharth teotia <siddharthteo...@gmail.com> wrote:
> Thanks, Haibo > > I can help change the release notes accordingly. Also, the issue is not > about backward incompatibility. The format change is backward compatible. > Let me explain > > I had done two format changes for raw index: > > https://github.com/apache/incubator-pinot/pull/5256 > > To handle very large STRING/BYTES column values, derive number of docs per > chunk using metadata. Before this change, the docs per chunk was hardcoded > to 1000. > > https://github.com/apache/incubator-pinot/pull/5285 > > Follow up to previous one -- if the raw index is greater than 2GB, the > offset of each chunk has to be 8-byte (LONG) in the file header instead of > 4-byte (INT) > > Both the above changes were backward compatible. So the new code can read > old segments (both with 1000 docs per chunk and 4-byte chunk offsets). The > first change caused performance issues for high QPS use cases since based > on the access pattern, you are going to uncompress a bigger chunk to fish > out the rows from forward index. For example, if we now pack 5000 rows per > chunk, and the docIds to select are 0, 5, 1000, 1500, 2000. In the old > code, we will uncompress 2 chunks of 1000 rows each. In the new code, we > will uncompress a single chunk of 5000 rows. Due to this, the first change > is made configurable. So the release notes have to changed. > > Thanks, > Sidd > > On Mon, Jun 1, 2020 at 9:02 AM H <hbwan...@gmail.com> wrote: > > > Please ignore this version. > > > > A backward incompatible issue was reported. I will cherry-pick 5470 > > <https://github.com/apache/incubator-pinot/pull/5470> and 5472 > > <https://github.com/apache/incubator-pinot/pull/5472> and send a new > > release candidate. Please let me know if there are other changes that > need > > to be included. > > > > Thanks > > > > On Mon, Jun 1, 2020 at 8:39 AM H <hbwan...@gmail.com> wrote: > > > > > Hi Pinot Community, > > > > > > This is a call for vote to the release Apache Pinot (incubating) > version > > > 0.4.0. > > > > > > The release candidate: > > > > > > > > > https://dist.apache.org/repos/dist/dev/incubator/pinot/apache-pinot-incubating-0.4.0-rc1 > > > > > > Git tag for this release: > > > https://github.com/apache/incubator-pinot/tree/release-0.4.0-rc1 > > > > > > Git hash for this release: > > > fcfd016ae9c4f922c31b46bd8dc3fd624e1a297a > > > > > > The artifacts have been signed with key: 6CC169A6FC19C470, which can be > > > found in the following KEYS file. > > > https://dist.apache.org/repos/dist/release/incubator/pinot/KEYS > > > > > > Release notes: > > > GitHub: > > > > https://github.com/apache/incubator-pinot/releases/tag/release-0.4.0-rc1 > > > > > > Staging repository: > > > https://repository.apache.org/content/repositories/orgapachepinot-1012 > > > > > > Documentation on verifying a release candidate: > > > > > > > > > https://cwiki.apache.org/confluence/display/PINOT/Validating+a+release+candidate > > > > > > > > > The vote will be open for at least 72 hours or until necessary number > of > > > votes are reached. > > > > > > Please vote accordingly, > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove with the reason > > > > > > Thanks, > > > Apache Pinot (incubating) team > > > > > > > > -- > *Best Regards,* > *SIDDHARTH TEOTIA* > *2008C6PS540G* > *BITS PILANI- GOA CAMPUS* > > *+91 87911 75932* >
