Re: Branch for "Per Table Write ID" implementation

[Sankar Hariappan]

Thanks Thejas, Eugene and Gopal for the feedback!
Will go ahead and create the branch!

Best regards
Sankar








On 11/01/18, 11:03 AM, "Eugene Koifman"  wrote:

>+1
>
>
>On 1/10/18, 8:18 PM, "Thejas Nair"  wrote:
>
>+1
>Makes sense to split the changes into multiple smaller patches that are
>easier to review, and creating this branch would help with that.
>
>
>
>On Tue, Jan 9, 2018 at 10:55 PM, Sankar Hariappan <
>shariap...@hortonworks.com> wrote:
>
>> Hi all,
>>
>> "Hive Replication” feature is advancing to support ACID tables 
> (HIVE-18320<
>> https://issues.apache.org/jira/browse/HIVE-18320>).
>> “Per Table Write ID” is an important requirement to support replication
>> for ACID tables especially for the use case of “Analytics workload
>> off-loading for scalability”. Details are available in the design 
> document
>> attached in the JIRA.
>>
>> Per table Write ID implementation have several changes.
>>
>>   1.  Add metadata tables to allocate and manage write ID. Also, map it
>> against global transaction.
>>   2.  Handle snapshot isolation for ACID/MM table reads by using
>> ValidWriteIDList instead of ValidTxnList.
>>   3.  Modify ORC/Hive row readers to use ValidWriteIDList instead of
>> ValidTxnList to read valid delta/base directories.
>>   4.  Update ValidCompactorTxnList to use table Write Ids.
>>   5.  Upgrade from existing Hive versions by migrating the ACID/MM tables
>> to use Write ID instead of global transaction ID.
>>   6.  Correct the UT test scripts to use ValidWriteIDList instead of
>> ValidTxnList for snapshot isolation tests.
>>   7.  Rename the method/variable names of several classes to use WriteId
>> instead of TxnId.
>>
>> As part of HIVE-18192,
>> I have implemented first 3 changes in the list which makes ACID 
> read/write
>> to work with Write ID change. But, this feature will be incomplete 
> without
>> rest of the changes.
>>
>> Hence, I would like to create a branch (branch-per-table-writeid) from
>> master to commit this feature with multiple patches. This branch is
>> expected to be short-lived for 2 to 3 weeks.
>>
>> Request feedback from the community.
>>
>> Best regards
>> Sankar
>>
>>
>
>

Re: Branch for "Per Table Write ID" implementation

[Gopal Vijayaraghavan]

+1

Cheers,
Gopal


On 1/9/18, 10:55 PM, "Sankar Hariappan"  wrote:

Hi all,

"Hive Replication” feature is advancing to support ACID tables 
(HIVE-18320).
“Per Table Write ID” is an important requirement to support replication for 
ACID tables especially for the use case of “Analytics workload off-loading for 
scalability”. Details are available in the design document attached in the JIRA.

Per table Write ID implementation have several changes.

  1.  Add metadata tables to allocate and manage write ID. Also, map it 
against global transaction.
  2.  Handle snapshot isolation for ACID/MM table reads by using 
ValidWriteIDList instead of ValidTxnList.
  3.  Modify ORC/Hive row readers to use ValidWriteIDList instead of 
ValidTxnList to read valid delta/base directories.
  4.  Update ValidCompactorTxnList to use table Write Ids.
  5.  Upgrade from existing Hive versions by migrating the ACID/MM tables 
to use Write ID instead of global transaction ID.
  6.  Correct the UT test scripts to use ValidWriteIDList instead of 
ValidTxnList for snapshot isolation tests.
  7.  Rename the method/variable names of several classes to use WriteId 
instead of TxnId.

As part of HIVE-18192, I 
have implemented first 3 changes in the list which makes ACID read/write to 
work with Write ID change. But, this feature will be incomplete without rest of 
the changes.

Hence, I would like to create a branch (branch-per-table-writeid) from 
master to commit this feature with multiple patches. This branch is expected to 
be short-lived for 2 to 3 weeks.

Request feedback from the community.

Best regards
Sankar

Re: Branch for "Per Table Write ID" implementation

[Eugene Koifman]

+1


On 1/10/18, 8:18 PM, "Thejas Nair"  wrote:

+1
Makes sense to split the changes into multiple smaller patches that are
easier to review, and creating this branch would help with that.



On Tue, Jan 9, 2018 at 10:55 PM, Sankar Hariappan <
shariap...@hortonworks.com> wrote:

> Hi all,
>
> "Hive Replication” feature is advancing to support ACID tables 
(HIVE-18320<
> https://issues.apache.org/jira/browse/HIVE-18320>).
> “Per Table Write ID” is an important requirement to support replication
> for ACID tables especially for the use case of “Analytics workload
> off-loading for scalability”. Details are available in the design document
> attached in the JIRA.
>
> Per table Write ID implementation have several changes.
>
>   1.  Add metadata tables to allocate and manage write ID. Also, map it
> against global transaction.
>   2.  Handle snapshot isolation for ACID/MM table reads by using
> ValidWriteIDList instead of ValidTxnList.
>   3.  Modify ORC/Hive row readers to use ValidWriteIDList instead of
> ValidTxnList to read valid delta/base directories.
>   4.  Update ValidCompactorTxnList to use table Write Ids.
>   5.  Upgrade from existing Hive versions by migrating the ACID/MM tables
> to use Write ID instead of global transaction ID.
>   6.  Correct the UT test scripts to use ValidWriteIDList instead of
> ValidTxnList for snapshot isolation tests.
>   7.  Rename the method/variable names of several classes to use WriteId
> instead of TxnId.
>
> As part of HIVE-18192,
> I have implemented first 3 changes in the list which makes ACID read/write
> to work with Write ID change. But, this feature will be incomplete without
> rest of the changes.
>
> Hence, I would like to create a branch (branch-per-table-writeid) from
> master to commit this feature with multiple patches. This branch is
> expected to be short-lived for 2 to 3 weeks.
>
> Request feedback from the community.
>
> Best regards
> Sankar
>
>

Re: Branch for "Per Table Write ID" implementation

[Thejas Nair]

+1
Makes sense to split the changes into multiple smaller patches that are
easier to review, and creating this branch would help with that.



On Tue, Jan 9, 2018 at 10:55 PM, Sankar Hariappan <
shariap...@hortonworks.com> wrote:

> Hi all,
>
> "Hive Replication” feature is advancing to support ACID tables (HIVE-18320<
> https://issues.apache.org/jira/browse/HIVE-18320>).
> “Per Table Write ID” is an important requirement to support replication
> for ACID tables especially for the use case of “Analytics workload
> off-loading for scalability”. Details are available in the design document
> attached in the JIRA.
>
> Per table Write ID implementation have several changes.
>
>   1.  Add metadata tables to allocate and manage write ID. Also, map it
> against global transaction.
>   2.  Handle snapshot isolation for ACID/MM table reads by using
> ValidWriteIDList instead of ValidTxnList.
>   3.  Modify ORC/Hive row readers to use ValidWriteIDList instead of
> ValidTxnList to read valid delta/base directories.
>   4.  Update ValidCompactorTxnList to use table Write Ids.
>   5.  Upgrade from existing Hive versions by migrating the ACID/MM tables
> to use Write ID instead of global transaction ID.
>   6.  Correct the UT test scripts to use ValidWriteIDList instead of
> ValidTxnList for snapshot isolation tests.
>   7.  Rename the method/variable names of several classes to use WriteId
> instead of TxnId.
>
> As part of HIVE-18192,
> I have implemented first 3 changes in the list which makes ACID read/write
> to work with Write ID change. But, this feature will be incomplete without
> rest of the changes.
>
> Hence, I would like to create a branch (branch-per-table-writeid) from
> master to commit this feature with multiple patches. This branch is
> expected to be short-lived for 2 to 3 weeks.
>
> Request feedback from the community.
>
> Best regards
> Sankar
>
>

Branch for "Per Table Write ID" implementation

[Sankar Hariappan]

Hi all,

"Hive Replication” feature is advancing to support ACID tables 
(HIVE-18320).
“Per Table Write ID” is an important requirement to support replication for 
ACID tables especially for the use case of “Analytics workload off-loading for 
scalability”. Details are available in the design document attached in the JIRA.

Per table Write ID implementation have several changes.

  1.  Add metadata tables to allocate and manage write ID. Also, map it against 
global transaction.
  2.  Handle snapshot isolation for ACID/MM table reads by using 
ValidWriteIDList instead of ValidTxnList.
  3.  Modify ORC/Hive row readers to use ValidWriteIDList instead of 
ValidTxnList to read valid delta/base directories.
  4.  Update ValidCompactorTxnList to use table Write Ids.
  5.  Upgrade from existing Hive versions by migrating the ACID/MM tables to 
use Write ID instead of global transaction ID.
  6.  Correct the UT test scripts to use ValidWriteIDList instead of 
ValidTxnList for snapshot isolation tests.
  7.  Rename the method/variable names of several classes to use WriteId 
instead of TxnId.

As part of HIVE-18192, I have 
implemented first 3 changes in the list which makes ACID read/write to work 
with Write ID change. But, this feature will be incomplete without rest of the 
changes.

Hence, I would like to create a branch (branch-per-table-writeid) from master 
to commit this feature with multiple patches. This branch is expected to be 
short-lived for 2 to 3 weeks.

Request feedback from the community.

Best regards
Sankar