Hi Fokko, I've looked at FLINK-11911 (Make KafkaTopicPartition a POJO), and left a comment in the PR.
Main gist is that this change would break state compatibility for previous versions of the FlinkKafkaConsumers, since changing that behind the scenes Flink would use different serializers for the type. Specifically speaking, we would be changing from the `KryoSerializer` to a `PojoSerializer` for `KafkaTopicPartition`'s serialization. While with the current API it is possible to migrate state serialization schema for this case (we were actually already thinking about enabling it), this is a rather delicate change and I would not suggest rushing it into the 1.8.0 release. For the time being, I've opened FLINK-11917 [1] to track this improvement. Cheers, Gordon [1] https://issues.apache.org/jira/browse/FLINK-11917 On Thu, Mar 14, 2019 at 6:41 AM Driesprong, Fokko <fo...@driesprong.frl> wrote: > Thanks for all the effort of releasing Flink 1.8. From experiences, I know > that releasing a new version is very time-consuming. > > Aljoscha, It is unfortunate to hear that the Parquet compression isn't > getting in Flink 1.8. Can we attach a version to the ticket > <https://issues.apache.org/jira/browse/FLINK-11401>, so I know when we can > expect this? > > Today I've spent some time to update our Flink job to Flink 1.8, including > the new KafkaDeserializationSchema, which allows us to extract some Kafka > headers. I've noticed that there where some serialization related log lines > that the KafkaTopicPartition class cannot be encoded as a POJO. We also > write Kafka's topic, partition and offset to Parquet, so we can use this > later for debugging. Potentially the KafkaTopicPartition is being pulled in > when serializing, I think it would be nice to be able to encode this class > as a POJO as well for the sake of performance. I've opened up a PR which > fixes it for me: https://github.com/apache/flink/pull/7979 > > Let me know if this can still make it into Flink 1.8. > > Cheers, Fokko > > Op wo 13 mrt. 2019 om 14:59 schreef Aljoscha Krettek <aljos...@apache.org > >: > > > @Robert I was planning to do that, yes! :-) > > > > > On 13. Mar 2019, at 13:04, Robert Metzger <rmetz...@apache.org> wrote: > > > > > > Aljoscha, can you CC the user@ list on the next RC, so that a wider > > > audience of people sees it and can help testing with their prod jobs? > > > > > > On Wed, Mar 13, 2019 at 12:47 PM jincheng sun < > sunjincheng...@gmail.com> > > > wrote: > > > > > >> Thanks for prepare the new RC. Aljoscha! > > >> > > >> I also merged the FLINK-11892 > > >> <https://issues.apache.org/jira/browse/FLINK-11892> (end-to-end test > > doc > > >> improve) into release-1.8 branch, and thanks for the quick fix @Yu. > > >> Looking forward to the new RC. > > >> > > >> Best, > > >> Jincheng > > >> > > >> Aljoscha Krettek <aljos...@apache.org> 于2019年3月13日周三 下午7:25写道： > > >> > > >>> Hi, > > >>> > > >>> I’m hereby canceling the vote for RC1 of Flink 1.8.0 because of the > > >>> aforementioned issues. I’ll create a new RC as soon as those issues > are > > >>> resolved. > > >>> > > >>> Best, > > >>> Aljoscha > > >>> > > >>>> On 13. Mar 2019, at 12:24, Aljoscha Krettek <aljos...@apache.org> > > >> wrote: > > >>>> > > >>>> Hi, > > >>>> > > >>>> Thanks for the verification performed so far. It seems I have to > > cancel > > >>> this vote and create a new RC. I’ll wait for - > > >>> https://issues.apache.org/jira/browse/FLINK-11901 < > > >>> https://issues.apache.org/jira/browse/FLINK-11901> to be resolved. > > >>>> > > >>>> There have also been two blockers that are now resolved: > > >>>> - https://issues.apache.org/jira/browse/FLINK-11420 < > > >>> https://issues.apache.org/jira/browse/FLINK-11420> > > >>>> - https://issues.apache.org/jira/browse/FLINK-11865 < > > >>> https://issues.apache.org/jira/browse/FLINK-11865> > > >>>> > > >>>> Regarding https://github.com/apache/flink/pull/7547 < > > >>> https://github.com/apache/flink/pull/7547> and > > >>> https://github.com/apache/flink/pull/7508 < > > >>> https://github.com/apache/flink/pull/7508>. The latter can be merged > > >> onto > > >>> release-1.8 at any time since it only updates documentation and that > is > > >>> always built from the corresponding release branch. For the former, > I’m > > >>> afraid we shouldn’t do that anymore now because the feature freeze > for > > >>> Flink 1.8.0 happened a few weeks ago and we shouldn’t introduce new > > >>> functionality now. > > >>>> > > >>>> Best, > > >>>> Aljoscha > > >>>> > > >>>> > > >>>>> On 13. Mar 2019, at 12:02, Timo Walther <twal...@apache.org> > wrote: > > >>>>> > > >>>>> I just found https://issues.apache.org/jira/browse/FLINK-11901 > > >>>>> > > >>>>> According to Chesnay, this is a release blocker. > > >>>>> > > >>>>> Regards, > > >>>>> Timo > > >>>>> > > >>>>> Am 13.03.19 um 09:48 schrieb Driesprong, Fokko: > > >>>>>> -1 (non-binding) > > >>>>>> > > >>>>>> I'd like to see both get into Flink 1.8: > > >>>>>> https://github.com/apache/flink/pull/7547 > > >>>>>> https://github.com/apache/flink/pull/7508 > > >>>>>> > > >>>>>> Cheers, Fokko > > >>>>>> > > >>>>>> > > >>>>>> Op di 12 mrt. 2019 om 15:59 schreef jincheng sun < > > >>> sunjincheng...@gmail.com>: > > >>>>>> > > >>>>>>> I agree with @Robert Metzger <rmetz...@apache.org> 's > suggestion! > > >>>>>>> The 1.8 version is a big release, we have a lot of enhancements, > > and > > >>>>>>> letting users participate in the test is very helpful for the > > >> quality > > >>> of > > >>>>>>> the release. > > >>>>>>> > > >>>>>>> BTW: > > >>>>>>> I am also doing RC verification, but I haven't fully verified it > > >> yet, > > >>> And I > > >>>>>>> have already found the end-to-end test error (not sure if it's > the > > >>> blocker, > > >>>>>>> I am trying to find out the reason...). > > >>>>>>> I will share the check result ASAP. > > >>>>>>> > > >>>>>>> Best, > > >>>>>>> Jincheng > > >>>>>>> > > >>>>>>> Robert Metzger <rmetz...@apache.org> 于2019年3月12日周二 下午6:34写道： > > >>>>>>> > > >>>>>>>> I'm wondering whether we should tweet about this release > > candidate, > > >>> and > > >>>>>>>> also post on the user@ mailing list to ask for some help with > > >>> testing > > >>>>>>> the > > >>>>>>>> RC in production environments? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Tue, Mar 12, 2019 at 11:23 AM Kurt Young <ykt...@gmail.com> > > >>> wrote: > > >>>>>>>> > > >>>>>>>>> +1 (non-binding) > > >>>>>>>>> > > >>>>>>>>> Checked items: > > >>>>>>>>> - checked checksums and GPG files > > >>>>>>>>> - verified that the source archives do not contains any > binaries > > >>>>>>>>> - checked that all POM files point to the same version > > >>>>>>>>> - build from source > > >>>>>>>>> > > >>>>>>>>> Best, > > >>>>>>>>> Kurt > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Tue, Mar 12, 2019 at 9:20 AM Congxian Qiu < > > >>> qcx978132...@gmail.com> > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> +1 (non-binding) > > >>>>>>>>>> > > >>>>>>>>>> Checked items: > > >>>>>>>>>> - run mvn clean install locally with success > > >>>>>>>>>> - run program on yarn, success > > >>>>>>>>>> - test state schema evolution(POJO state type) with the > > following > > >>>>>>> paths > > >>>>>>>>>> success(heap&rocks) > > >>>>>>>>>> - 1.6 -> 1.8 -> 1.8 > > >>>>>>>>>> - 1.7 -> 1.8 -> 1.8 > > >>>>>>>>>> - 1.8 -> 1.8 > > >>>>>>>>>> > > >>>>>>>>>> Best, Congxian > > >>>>>>>>>> On Mar 11, 2019, 21:07 +0800, Aljoscha Krettek < > > >>> aljos...@apache.org > > >>>>>>>> , > > >>>>>>>>>> wrote: > > >>>>>>>>>>> Hi everyone, > > >>>>>>>>>>> Please review and vote on the release candidate 1 for Flink > > >> 1.8.0, > > >>>>>>> as > > >>>>>>>>>> follows: > > >>>>>>>>>>> [ ] +1, Approve the release > > >>>>>>>>>>> [ ] -1, Do not approve the release (please provide specific > > >>>>>>> comments) > > >>>>>>>>>>> > > >>>>>>>>>>> The complete staging area is available for your review, which > > >>>>>>>> includes: > > >>>>>>>>>>> * JIRA release notes [1], > > >>>>>>>>>>> * the official Apache source release and binary convenience > > >>>>>>> releases > > >>>>>>>> to > > >>>>>>>>>> be deployed to dist.apache.org [2], which are signed with the > > >> key > > >>>>>>> with > > >>>>>>>>>> fingerprint F2A67A8047499BBB3908D17AA8F4FD97121D7293 [3], > > >>>>>>>>>>> * all artifacts to be deployed to the Maven Central > Repository > > >>> [4], > > >>>>>>>>>>> * source code tag "release-1.2.3-rc3" [5], > > >>>>>>>>>>> * website pull request listing the new release [6] > > >>>>>>>>>>> * website pull request adding announcement blog post [7]. > > >>>>>>>>>>> > > >>>>>>>>>>> The vote will be open for at least 72 hours. It is adopted by > > >>>>>>>> majority > > >>>>>>>>>> approval, with at least 3 PMC affirmative votes. > > >>>>>>>>>>> Thanks, > > >>>>>>>>>>> Aljoscha > > >>>>>>>>>>> > > >>>>>>>>>>> [1] > > >>>>>>> > > >>> > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12344274 > > >>>>>>>>>>> [2] > > >> https://dist.apache.org/repos/dist/dev/flink/flink-1.8.0-rc1/ > > >>>>>>>>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > >>>>>>>>>>> [4] > > >>>>>>>> > > >>> > > https://repository.apache.org/content/repositories/orgapacheflink-1210/ > > >>>>>>>>>>> [5] > > >>>>>>> > > >>> > > >> > > > https://gitbox.apache.org/repos/asf?p=flink.git;a=tag;h=2d00b1c26d7b4554707063ab0d1d6cc236cfe8a5 > > >>>>>>>>>>> [6] https://github.com/apache/flink-web/pull/180 < > > >>>>>>>>>> https://github.com/apache/flink-web/pull/180> > > >>>>>>>>>>> [7] https://github.com/apache/flink-web/pull/179 > > >>>>> > > >>>>> > > >>>> > > >>> > > >>> > > >> > > > > >
