Re: Avro schema getting changed dynamically
On Mon, Aug 29, 2016 at 11:17 PM, Manjeet Singh wrote: > Hi All, > > I have use case to put data in avro format in Hbase , I have frequent read > write operations but its not a problem. > > Problem is what if my avro schema get changed how shuld I deal with it? > This should in mind what about older data which already inserted in Hbase > and now we have new schema. > > can anyone suggest me solution for the same > > Do a bit of research on Avro. Usually schema is stored apart from data. To read old format you must volunteer the old schema. But you can set it up otherwise to embed schema IIRC. Also, I'm no expert, but adding fields is probably fine. Have you tried reading the old format w/ new schema? For more, poke around here, https://github.com/kijiproject, a mature project that used avro serializing data with built-in schema-migration, etc. St.Ack > Thanks > Manjeet > > -- > luv all >
Re: Avro schema getting changed dynamically
Moving dev@ to bcc. Please don't email the developer mailing list with
questions about how to set up HBase for your use case.
On Monday, August 29, 2016, Manjeet Singh
wrote:
> I want ot add few more points
>
> I am using Java native Api for Hbase get/put
>
> and below is the example
>
> assume i have below schema and I am inserting data by using this schema
> into Hbase but later I have new new values coming and i need to fit this
> into my schema for this I need to create new schema as showing in below
> example
> in example 2 I have new field time stamp
>
> example 1
> { "type" : "record", "name" : "twitter_schema", "namespace" :
> "com.miguno.avro", "fields" : [ { "name" : "username", "type" : "string",
> "doc" : "Name of the user account on Twitter.com" }, { "name" : "tweet",
> "type" : "string", "doc" : "The content of the user's Twitter message" } ]
> }
>
>
> example 2
>
>
> { "type" : "record", "name" : "twitter_schema", "namespace" :
> "com.miguno.avro", "fields" : [ { "name" : "username", "type" : "string",
> "doc" : "Name of the user account on Twitter.com" }, { "name" : "tweet",
> "type" : "string", "doc" : "The content of the user's Twitter message" }, {
> "name" : "timestamp", "type" : "long", "doc" : "Unix epoch time in seconds"
> } ], }
>
>
>
>
> Thanks
> Manjeet
>
>
>
>
>
> On Tue, Aug 30, 2016 at 11:47 AM, Manjeet Singh <
> [email protected] >
> wrote:
>
> > Hi All,
> >
> > I have use case to put data in avro format in Hbase , I have frequent
> read
> > write operations but its not a problem.
> >
> > Problem is what if my avro schema get changed how shuld I deal with it?
> > This should in mind what about older data which already inserted in Hbase
> > and now we have new schema.
> >
> > can anyone suggest me solution for the same
> >
> > Thanks
> > Manjeet
> >
> > --
> > luv all
> >
>
>
>
> --
> luv all
>
--
-Dima
Re: Avro schema getting changed dynamically
I want ot add few more points
I am using Java native Api for Hbase get/put
and below is the example
assume i have below schema and I am inserting data by using this schema
into Hbase but later I have new new values coming and i need to fit this
into my schema for this I need to create new schema as showing in below
example
in example 2 I have new field time stamp
example 1
{ "type" : "record", "name" : "twitter_schema", "namespace" :
"com.miguno.avro", "fields" : [ { "name" : "username", "type" : "string",
"doc" : "Name of the user account on Twitter.com" }, { "name" : "tweet",
"type" : "string", "doc" : "The content of the user's Twitter message" } ] }
example 2
{ "type" : "record", "name" : "twitter_schema", "namespace" :
"com.miguno.avro", "fields" : [ { "name" : "username", "type" : "string",
"doc" : "Name of the user account on Twitter.com" }, { "name" : "tweet",
"type" : "string", "doc" : "The content of the user's Twitter message" }, {
"name" : "timestamp", "type" : "long", "doc" : "Unix epoch time in seconds"
} ], }
Thanks
Manjeet
On Tue, Aug 30, 2016 at 11:47 AM, Manjeet Singh
wrote:
> Hi All,
>
> I have use case to put data in avro format in Hbase , I have frequent read
> write operations but its not a problem.
>
> Problem is what if my avro schema get changed how shuld I deal with it?
> This should in mind what about older data which already inserted in Hbase
> and now we have new schema.
>
> can anyone suggest me solution for the same
>
> Thanks
> Manjeet
>
> --
> luv all
>
--
luv all
