Re: Add EventTime to pulsar messages

2017-08-23 Thread Sahaya Andrews
Ok, got it. I know you had specific reason for this proposal, but wasn't clear initially. Thanks for clarifying. Andrews. On Wed, Aug 23, 2017 at 3:56 PM, Sijie Guo wrote: > On Wed, Aug 23, 2017 at 3:19 PM, Sahaya Andrews wrote: > >> If this is not going

Re: Add EventTime to pulsar messages

2017-08-23 Thread Rajan Dhabalia
> Didn't get this, can you clarify? I think I misinterpreted previous reply where I thought we want to add "eventTIme" field to avoid deserialization of Message-Metadata so, I asked question how to retrieve "eventTime" field without deserializing. > It's certain that an application can use a

Re: Add EventTime to pulsar messages

2017-08-23 Thread Sijie Guo
On Wed, Aug 23, 2017 at 4:23 PM, Rajan Dhabalia wrote: > > because this field might be accessed when processing every message. the > overhead in key/value property is much higher than an int64 field. > > I have the same concern as Andrews mentioned. > But a very quick

Re: Add EventTime to pulsar messages

2017-08-23 Thread Matteo Merli
> So, I believe in your usecase, we want to skip metadata-deserialization and > idea is to read only "EventTime" field (field-12) without deserializing > message-metada using ByteBuf-offset (eg: read last int64 using end of > metadata-index).? Don't you think it will prevent us to add additional >

Re: Add EventTime to pulsar messages

2017-08-23 Thread Rajan Dhabalia
> because this field might be accessed when processing every message. the overhead in key/value property is much higher than an int64 field. I have the same concern as Andrews mentioned. But a very quick implementation question: properties

Re: Add EventTime to pulsar messages

2017-08-23 Thread Sijie Guo
On Wed, Aug 23, 2017 at 3:19 PM, Sahaya Andrews wrote: > If this is not going to be used by the broker for anything, why can't > this be set in message property key/value? > Currently it is not used by broker. However, we might leverage field later on to provide advanced

Re: Add EventTime to pulsar messages

2017-08-23 Thread Sahaya Andrews
If this is not going to be used by the broker for anything, why can't this be set in message property key/value? Andrews On Wed, Aug 23, 2017 at 3:03 PM, Matteo Merli wrote: > Proposal looks good to me. > > Added to wiki: >

Re: Add EventTime to pulsar messages

2017-08-23 Thread Matteo Merli
Proposal looks good to me. Added to wiki: https://github.com/apache/incubator-pulsar/wiki/PIP-5:-Event-time On Wed, Aug 23, 2017 at 2:52 PM Sijie Guo wrote: > Hi pulsar folks, > > Currently Pulsar messages has fields for `publish_time`. However in some > use cases for

Add EventTime to pulsar messages

2017-08-23 Thread Sijie Guo
Hi pulsar folks, Currently Pulsar messages has fields for `publish_time`. However in some use cases for example stream computing, `event_time` is required. I'd like to start a proposal to add `event_time` support to pulsar messages. This proposal only covers adding `event_time` to pulsar