[jira] [Comment Edited] (OMID-262) Move from commons-configuration:commons-configuration to org.apache.commons:commons-configuration2

[Istvan Toth (Jira)]

[ 
https://issues.apache.org/jira/browse/OMID-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810240#comment-17810240
 ] 

Istvan Toth edited comment on OMID-262 at 1/24/24 7:31 AM:
---

I couldn't find any reference to commons-configuration in the code.
Looks like we could simply remove it.



was (Author: stoty):
I couldn't find any reference to commons-configuration in the code.
Looks like we could simply remove it.

commons-configuration2 is a 

> Move from commons-configuration:commons-configuration to 
> org.apache.commons:commons-configuration2
> --
>
> Key: OMID-262
> URL: https://issues.apache.org/jira/browse/OMID-262
> Project: Phoenix Omid
>  Issue Type: Improvement
>Reporter: Nihal Jain
>Priority: Major
>
> Move from 
> [commons-configuration:commons-configuration|https://mvnrepository.com/artifact/commons-configuration/commons-configuration]
>  to 
> [org.apache.commons:commons-configuration2.|https://mvnrepository.com/artifact/org.apache.commons/commons-configuration2]
> Current version used by omid is 1.10, which was released in Oct 24, 2013 and 
> is very very old. We should move to latest which is moved to 
> org.apache.commons:commons-configuration2



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OMID-262) Move from commons-configuration:commons-configuration to org.apache.commons:commons-configuration2

[Istvan Toth (Jira)]

[ 
https://issues.apache.org/jira/browse/OMID-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810240#comment-17810240
 ] 

Istvan Toth commented on OMID-262:
--

I couldn't find any reference to commons-configuration in the code.
Looks like we could simply remove it.

commons-configuration2 is a 

> Move from commons-configuration:commons-configuration to 
> org.apache.commons:commons-configuration2
> --
>
> Key: OMID-262
> URL: https://issues.apache.org/jira/browse/OMID-262
> Project: Phoenix Omid
>  Issue Type: Improvement
>Reporter: Nihal Jain
>Priority: Major
>
> Move from 
> [commons-configuration:commons-configuration|https://mvnrepository.com/artifact/commons-configuration/commons-configuration]
>  to 
> [org.apache.commons:commons-configuration2.|https://mvnrepository.com/artifact/org.apache.commons/commons-configuration2]
> Current version used by omid is 1.10, which was released in Oct 24, 2013 and 
> is very very old. We should move to latest which is moved to 
> org.apache.commons:commons-configuration2



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (PHOENIX-7189) Update Omid to 1.1.0

[Istvan Toth (Jira)]

Istvan Toth created PHOENIX-7189:


 Summary: Update Omid to 1.1.0
 Key: PHOENIX-7189
 URL: https://issues.apache.org/jira/browse/PHOENIX-7189
 Project: Phoenix
  Issue Type: Task
  Components: core
Affects Versions: 5.2.0, 5.1.4
Reporter: Istvan Toth






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (PHOENIX-7189) Update Omid to 1.1.1

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth updated PHOENIX-7189:
-
Summary: Update Omid to 1.1.1  (was: Update Omid to 1.1.0)

> Update Omid to 1.1.1
> 
>
> Key: PHOENIX-7189
> URL: https://issues.apache.org/jira/browse/PHOENIX-7189
> Project: Phoenix
>  Issue Type: Task
>  Components: core
>Affects Versions: 5.2.0, 5.1.4
>Reporter: Istvan Toth
>Priority: Blocker
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [CANCEL][VOTE] Release of Apache Phoenix Omid 1.1.1 RC0

[rajeshb...@apache.org]

Thanks Istvan, Richard, Cong Luo for fixing the issues quickly.


Will create another RC today.

Thanks,
Rajeshbabu.


On Wed, Jan 24, 2024, 12:34 PM IstvaToth  wrote:

> Every problem found in  RC0 should be fixed now.
> (Plus I have also merged OMID-275)
>
> Istvan
>
> On Tue, Jan 23, 2024 at 8:34 AM Istvan Toth  wrote:
>
> > It would also be nice if OMID-275
> >  could make it into
> 1.1.1.
> >
> > It's a small change, but it would open the possibility of building
> > 5.1.4 with future Omid releases which will have to remove
> > TTable.getHgetTableDescriptor() for HBase 3 compatibility.
> >
> > best regards
> > Istvan
> >
> >
> > On Fri, Jan 19, 2024 at 2:58 PM Istvan Toth  wrote:
> >
> >> We have worked on this with Richard.
> >>
> >> I have made some changes to force the test to use IPv6 on my Linux
> >> machine, and it still works fine.
> >> It is some funky mac networking issue:
> >>
> >>
> https://medium.com/@quelgar/java-sockets-broken-for-ipv6-on-mac-5aae72f06b21
> >>
> >> Please review https://issues.apache.org/jira/browse/OMID-249, which
> >> fixes the issue (at least on Richard's particular mac)
> >>
> >> best regards
> >> Istvan
> >>
> >> On Fri, Jan 19, 2024 at 8:07 AM Istvan Toth  wrote:
> >>
> >>> Someone who has access to a Mac please try to track down which change
> >>> broke the tests.
> >>>
> >>>
> >>> On Fri, Jan 19, 2024 at 8:05 AM Istvan Toth 
> wrote:
> >>>
>  PR for JDK17 support is up:
>  https://issues.apache.org/jira/browse/OMID-272
> 
>  On Thu, Jan 18, 2024 at 6:39 PM rajeshb...@apache.org <
>  chrajeshbab...@gmail.com> wrote:
> 
> > The vote has been canceled to address issues with JDK and Ipv6
> > interface in
> > MacOS
> >
> > Thanks,
> > Rajeshbabu.
> >
> > On Thu, Jan 18, 2024 at 11:05 PM rajeshb...@apache.org <
> > chrajeshbab...@gmail.com> wrote:
> >
> > > Sure, let me cancel the RC and create another RC after fixes with
> the
> > > network interface and JDK 17.
> > >
> > > Thanks, Richard and Istvan for finding issues, and Cong Luo for
> your
> > > support.
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Jan 18, 2024 at 9:52 PM Istvan Toth
> > 
> > > wrote:
> > >
> > >> I have found another problem, Omid doesn't work with JDK 17.
> > >> Not necessarily a blocker, but if we do another RC, it would be
> > nice to
> > >> include it.
> > >>
> > >> I'm also working on adding HBase 3 support, which requires a small
> > API
> > >> change (that also affects Phoenix)
> > >> Again, not a blocker, but we'll have to do another release for 5.2
> > sooner
> > >> than later for HBase 3 support.
> > >>
> > >>
> > >> On Thu, Jan 18, 2024 at 9:11 AM Cong Luo  wrote:
> > >>
> > >> > Hi Istvan,
> > >> > Using ipv6 on linux operating systems is available, so the
> reason
> > may be
> > >> > limited to users using Mac OS. And then, I using the ping the
> ipv6
> > >> address
> > >> > on Mac OS, got the 'Unknown host', obviously macos uses ipv6 as
> > the host
> > >> > name. based on this default behavior of the operating system, I
> > >> recommend
> > >> > using macos with ipv4 and others with ipv4/ipv6.
> > >> >
> > >> > On 2024/01/18 06:54:14 Istvan Toth wrote:
> > >> > > I think we should look into this a bit deeper.
> > >> > > Why does this fail with IPv6 ?
> > >> > >
> > >> > > Are we explicitly binding TSO to the IPv4 address only ?
> > >> > > Do we only do that in the tests, or does that happen in
> > production ?
> > >> > > Is it even possible to use Omid with IPv6 ?
> > >> > >
> > >> > > I suspect that fixing
> > https://issues.apache.org/jira/browse/OMID-249
> > >> > would
> > >> > > improve on this.
> > >> > >
> > >> > > Generally, I think that having a single IP per TSO can be a
> > problem
> > >> in a
> > >> > > mixed IPv4 / IPv6 network, and with some network
> configurations.
> > >> > > Maybe we should store the name of the active server instead of
> > >> addresses
> > >> > in
> > >> > > ZK, and kick this problem down to the network/DNS
> administrator.
> > >> > >
> > >> > > If Omid is known to not work with IPV6 in 1.1.0 , then I'm OK
> > with
> > >> Cong's
> > >> > > proposed solution, but if it breaks pre-existing Ipv6 support,
> > then we
> > >> > > should
> > >> > > fix this properly for 1.1.1.
> > >> > >
> > >> > > We should create a follow-up ticket to make sure the Omid is
> > IPv6
> > >> clean
> > >> > for
> > >> > > the next release at the very least.
> > >> > >
> > >> > > Istvan
> > >> > >
> > >> > >
> > >> > > On Thu, Jan 18, 2024 at 6:56 AM Cong Luo 
> > wrote:
> > >> > >
> > >> > > > Hi Richard,
> > >> > > >   Actually, I 

[jira] [Created] (PHOENIX-7188) Remove Omid TTable.getTableDescriptor() calls

[Istvan Toth (Jira)]

Istvan Toth created PHOENIX-7188:


 Summary: Remove Omid TTable.getTableDescriptor() calls
 Key: PHOENIX-7188
 URL: https://issues.apache.org/jira/browse/PHOENIX-7188
 Project: Phoenix
  Issue Type: Improvement
  Components: core
Affects Versions: 5.2.0, 5.1.4
Reporter: Istvan Toth


Once we have upgraded to Omid 1.1.1, replace TTable#getTableDescriptor() calls 
with
TTable#getHBaseTable()#getTableDescriptor().

This is to allow for potentially building with future Omid versions, which will 
remove TTable#getTableDescriptor().



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [CANCEL][VOTE] Release of Apache Phoenix Omid 1.1.1 RC0

[Istvan Toth]

Every problem found in  RC0 should be fixed now.
(Plus I have also merged OMID-275)

Istvan

On Tue, Jan 23, 2024 at 8:34 AM Istvan Toth  wrote:

> It would also be nice if OMID-275
>  could make it into 1.1.1.
>
> It's a small change, but it would open the possibility of building
> 5.1.4 with future Omid releases which will have to remove
> TTable.getHgetTableDescriptor() for HBase 3 compatibility.
>
> best regards
> Istvan
>
>
> On Fri, Jan 19, 2024 at 2:58 PM Istvan Toth  wrote:
>
>> We have worked on this with Richard.
>>
>> I have made some changes to force the test to use IPv6 on my Linux
>> machine, and it still works fine.
>> It is some funky mac networking issue:
>>
>> https://medium.com/@quelgar/java-sockets-broken-for-ipv6-on-mac-5aae72f06b21
>>
>> Please review https://issues.apache.org/jira/browse/OMID-249, which
>> fixes the issue (at least on Richard's particular mac)
>>
>> best regards
>> Istvan
>>
>> On Fri, Jan 19, 2024 at 8:07 AM Istvan Toth  wrote:
>>
>>> Someone who has access to a Mac please try to track down which change
>>> broke the tests.
>>>
>>>
>>> On Fri, Jan 19, 2024 at 8:05 AM Istvan Toth  wrote:
>>>
 PR for JDK17 support is up:
 https://issues.apache.org/jira/browse/OMID-272

 On Thu, Jan 18, 2024 at 6:39 PM rajeshb...@apache.org <
 chrajeshbab...@gmail.com> wrote:

> The vote has been canceled to address issues with JDK and Ipv6
> interface in
> MacOS
>
> Thanks,
> Rajeshbabu.
>
> On Thu, Jan 18, 2024 at 11:05 PM rajeshb...@apache.org <
> chrajeshbab...@gmail.com> wrote:
>
> > Sure, let me cancel the RC and create another RC after fixes with the
> > network interface and JDK 17.
> >
> > Thanks, Richard and Istvan for finding issues, and Cong Luo for your
> > support.
> >
> >
> >
> >
> >
> > On Thu, Jan 18, 2024 at 9:52 PM Istvan Toth
> 
> > wrote:
> >
> >> I have found another problem, Omid doesn't work with JDK 17.
> >> Not necessarily a blocker, but if we do another RC, it would be
> nice to
> >> include it.
> >>
> >> I'm also working on adding HBase 3 support, which requires a small
> API
> >> change (that also affects Phoenix)
> >> Again, not a blocker, but we'll have to do another release for 5.2
> sooner
> >> than later for HBase 3 support.
> >>
> >>
> >> On Thu, Jan 18, 2024 at 9:11 AM Cong Luo  wrote:
> >>
> >> > Hi Istvan,
> >> > Using ipv6 on linux operating systems is available, so the reason
> may be
> >> > limited to users using Mac OS. And then, I using the ping the ipv6
> >> address
> >> > on Mac OS, got the 'Unknown host', obviously macos uses ipv6 as
> the host
> >> > name. based on this default behavior of the operating system, I
> >> recommend
> >> > using macos with ipv4 and others with ipv4/ipv6.
> >> >
> >> > On 2024/01/18 06:54:14 Istvan Toth wrote:
> >> > > I think we should look into this a bit deeper.
> >> > > Why does this fail with IPv6 ?
> >> > >
> >> > > Are we explicitly binding TSO to the IPv4 address only ?
> >> > > Do we only do that in the tests, or does that happen in
> production ?
> >> > > Is it even possible to use Omid with IPv6 ?
> >> > >
> >> > > I suspect that fixing
> https://issues.apache.org/jira/browse/OMID-249
> >> > would
> >> > > improve on this.
> >> > >
> >> > > Generally, I think that having a single IP per TSO can be a
> problem
> >> in a
> >> > > mixed IPv4 / IPv6 network, and with some network configurations.
> >> > > Maybe we should store the name of the active server instead of
> >> addresses
> >> > in
> >> > > ZK, and kick this problem down to the network/DNS administrator.
> >> > >
> >> > > If Omid is known to not work with IPV6 in 1.1.0 , then I'm OK
> with
> >> Cong's
> >> > > proposed solution, but if it breaks pre-existing Ipv6 support,
> then we
> >> > > should
> >> > > fix this properly for 1.1.1.
> >> > >
> >> > > We should create a follow-up ticket to make sure the Omid is
> IPv6
> >> clean
> >> > for
> >> > > the next release at the very least.
> >> > >
> >> > > Istvan
> >> > >
> >> > >
> >> > > On Thu, Jan 18, 2024 at 6:56 AM Cong Luo 
> wrote:
> >> > >
> >> > > > Hi Richard,
> >> > > >   Actually, I had the same issue before, but it was resolved
> >> locally:
> >> > if
> >> > > > the NIC was assigned both ipv4 and ipv6 addresses, use ipv4
> first.
> >> If
> >> > we
> >> > > > all think this is acceptable, then I will create a PR
> submission.
> >> > > >
> >> > > > On 2024/01/17 14:30:50 Richárd Antal wrote:
> >> > > > > -1
> >> > > > >
> >> > > > > Signature: OK
> >> > > > > Checksum: OK
> >> > > > > 

[jira] [Resolved] (OMID-275) Expose backing HBase Table from TTable

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/OMID-275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth resolved OMID-275.
--
Fix Version/s: 1.1.1
   Resolution: Fixed

Committed.
Thanks for the review [~gjacoby]

> Expose backing HBase Table from TTable
> --
>
> Key: OMID-275
> URL: https://issues.apache.org/jira/browse/OMID-275
> Project: Phoenix Omid
>  Issue Type: Improvement
>Affects Versions: 1.1.1
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Fix For: 1.1.1
>
>
> HBase 3.0 removes the HTableDescriptor class.
> However, OmidTransactionTable in Phoenix needs to implement 
> getTableDescriptor() when build for HBase 2.
> The way to break this to expose the backing HBase Table object from TTable.
> That way, we can still get getTableDescriptor from that for HBase 2.
> If we implement this for 1.1.1, then we can use this new API for 5.1.4 (and 
> possibly 5.2.0), which would hoepfully let them be built with future Omid 
> releases.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OMID-275) Expose backing HBase Table from TTable

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/OMID-275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810228#comment-17810228
 ] 

ASF GitHub Bot commented on OMID-275:
-

stoty commented on PR #155:
URL: https://github.com/apache/phoenix-omid/pull/155#issuecomment-1907483237

   merged manually.




> Expose backing HBase Table from TTable
> --
>
> Key: OMID-275
> URL: https://issues.apache.org/jira/browse/OMID-275
> Project: Phoenix Omid
>  Issue Type: Improvement
>Affects Versions: 1.1.1
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
>
> HBase 3.0 removes the HTableDescriptor class.
> However, OmidTransactionTable in Phoenix needs to implement 
> getTableDescriptor() when build for HBase 2.
> The way to break this to expose the backing HBase Table object from TTable.
> That way, we can still get getTableDescriptor from that for HBase 2.
> If we implement this for 1.1.1, then we can use this new API for 5.1.4 (and 
> possibly 5.2.0), which would hoepfully let them be built with future Omid 
> releases.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OMID-275) Expose backing HBase Table from TTable

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/OMID-275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810227#comment-17810227
 ] 

ASF GitHub Bot commented on OMID-275:
-

stoty closed pull request #155: OMID-275 Expose backing HBase Table from TTable
URL: https://github.com/apache/phoenix-omid/pull/155




> Expose backing HBase Table from TTable
> --
>
> Key: OMID-275
> URL: https://issues.apache.org/jira/browse/OMID-275
> Project: Phoenix Omid
>  Issue Type: Improvement
>Affects Versions: 1.1.1
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
>
> HBase 3.0 removes the HTableDescriptor class.
> However, OmidTransactionTable in Phoenix needs to implement 
> getTableDescriptor() when build for HBase 2.
> The way to break this to expose the backing HBase Table object from TTable.
> That way, we can still get getTableDescriptor from that for HBase 2.
> If we implement this for 1.1.1, then we can use this new API for 5.1.4 (and 
> possibly 5.2.0), which would hoepfully let them be built with future Omid 
> releases.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (PHOENIX-7001) Change Data Capture leveraging Max Lookback and Uncovered Indexes

[Kadir Ozdemir (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kadir Ozdemir updated PHOENIX-7001:
---
Description: 
The use cases for a Change Data Capture (CDC) feature are centered around 
capturing changes to a given table (or updatable view) as these changes happen 
in near real-time. A CDC application can retrieve changes in real-time or with 
some delay, or even retrieves the same set of changes multiple times. This 
means the CDC use case can be generalized as time range queries where the time 
range is typically short such as last x minutes or hours or expressed as a 
specific time range in the last n days where n is typically less than 7.

A change is an update in a row. That is, a change is either updating one or 
more columns of a table for a given row or deleting a row. It is desirable to 
provide these changes in the order of their arrival. One can visualize the 
delivery of these changes through a stream from a Phoenix table to the 
application that is initiated by the application similar to the delivery of any 
other Phoenix query results. The difference is that a regular query result 
includes at most one result row for each row satisfying the query and the 
deleted rows are not visible to the query result while the CDC stream/result 
can include multiple result rows for each row and the result includes deleted 
rows. Some use cases need to also get the pre and/or post image of the row 
along with a change on the row. 

The design proposed here leverages Phoenix Max Lookback and Uncovered (Global 
or Local) Indexes. The max lookback feature retains recent changes to a table, 
that is, the changes that have been done in the last x days typically. This 
means that the max lookback feature already captures the changes to a given 
table. Currently, the max lookback age is configurable at the cluster level. We 
need to extend this capability to be able to configure the max lookback age at 
the table level so that each table can have a different max lookback age based 
on its CDC application requirements.

To deliver the changes in the order of their arrival, we need a time based 
index. This index should be uncovered as the changes are already retained in 
the table by the max lookback feature. The arrival time will be defined as the 
mutation timestamp generated by the server. An uncovered index would allow us 
to efficiently and orderly access to the changes. Changes to an index table are 
also preserved by the max lookback feature.

A CDC feature can be composed of the following components:
 * {*}CDCUncoveredIndexRegionScanner{*}: This is a server side scanner on an 
uncovered index used for CDC. This can inherit UncoveredIndexRegionScanner. It 
goes through index table rows using a raw scan to identify data table rows and 
retrieves these rows using a raw scan. Using the time range, it forms a JSON 
blob to represent changes to the row including pre and/or post row images.
 * {*}CDC Query Compiler{*}: This is a client side component. It prepares the 
scan object based on the given CDC query statement. 
 * {*}CDC DDL Compiler{*}: This is a client side component. It creates the time 
based uncovered (global/local) index based on the given CDC DDL statement and a 
virtual table of CDC type. CDC will be a new table type. 

A CDC DDL syntax to create CDC on a (data) table can be as follows: 

Create CDC  on  INCLUDE (pre | post | latest | 
all) INDEX =  SALT_BUCKETS=

The above CDC DDL creates a virtual CDC table and an uncovered index. The CDC 
table PK columns start with the timestamp and continue with the data table PK 
columns. The CDC table includes one non-PK column which is a JSON column. The 
change is expressed in this JSON column in multiple ways based on the CDC DDL 
or query statement. The change can be expressed as just the mutation for the 
change, the latest image of the row, the pre image of the row (the image before 
the change), the post image, or any combination of these. The CDC table is not 
a physical table on disk. It is just a virtual table to be used in a CDC query. 
Phoenix stores just the metadata for this virtual table. 

A CDC query can be as follow:

Select * from  where PHOENIX_ROW_TIMESTAMP() >= TO_DATE( …) AND 
PHOENIX_ROW_TIMESTAMP() < TO_DATE( …)

This query would return the rows of the CDC table which is constructed on the 
server side by CDCUncoveredIndexRegionScanner by joining the uncovered index 
row versions with the corresponding data table row version (using raw scans). 
The above select query can be hinted at by using a new CDC hint to return just 
the actual change, pre, pos, or latest image of the row, or a combination of 
them to overwrite the default JSON column format defined by the CDC DDL 
statement. 

The CDC application will run the above query in a loop. When the difference 
between the current time of the application and the upper limit 

[jira] [Updated] (PHOENIX-7001) Change Data Capture leveraging Max Lookback and Uncovered Indexes

[Kadir Ozdemir (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kadir Ozdemir updated PHOENIX-7001:
---
Description: 
The use cases for a Change Data Capture (CDC) feature are centered around 
capturing changes to a given table (or updatable view) as these changes happen 
in near real-time. A CDC application can retrieve changes in real-time or with 
some delay, or even retrieves the same set of changes multiple times. This 
means the CDC use case can be generalized as time range queries where the time 
range is typically short such as last x minutes or hours or expressed as a 
specific time range in the last n days where n is typically less than 7.

A change is an update in a row. That is, a change is either updating one or 
more columns of a table for a given row or deleting a row. It is desirable to 
provide these changes in the order of their arrival. One can visualize the 
delivery of these changes through a stream from a Phoenix table to the 
application that is initiated by the application similar to the delivery of any 
other Phoenix query results. The difference is that a regular query result 
includes at most one result row for each row satisfying the query and the 
deleted rows are not visible to the query result while the CDC stream/result 
can include multiple result rows for each row and the result includes deleted 
rows. Some use cases need to also get the pre and/or post image of the row 
along with a change on the row. 

The design proposed here leverages Phoenix Max Lookback and Uncovered (Global 
or Local) Indexes. The max lookback feature retains recent changes to a table, 
that is, the changes that have been done in the last x days typically. This 
means that the max lookback feature already captures the changes to a given 
table. Currently, the max lookback age is configurable at the cluster level. We 
need to extend this capability to be able to configure the max lookback age at 
the table level so that each table can have a different max lookback age based 
on its CDC application requirements.

To deliver the changes in the order of their arrival, we need a time based 
index. This index should be uncovered as the changes are already retained in 
the table by the max lookback feature. The arrival time will be defined as the 
mutation timestamp generated by the server. An uncovered index would allow us 
to efficiently and orderly access to the changes. Changes to an index table are 
also preserved by the max lookback feature.

A CDC feature can be composed of the following components:
 * {*}CDCUncoveredIndexRegionScanner{*}: This is a server side scanner on an 
uncovered index used for CDC. This can inherit UncoveredIndexRegionScanner. It 
goes through index table rows using a raw scan to identify data table rows and 
retrieves these rows using a raw scan. Using the time range, it forms a JSON 
blob to represent changes to the row including pre and/or post row images.
 * {*}CDC Query Compiler{*}: This is a client side component. It prepares the 
scan object based on the given CDC query statement. 
 * {*}CDC DDL Compiler{*}: This is a client side component. It creates the time 
based uncovered (global/local) index based on the given CDC DDL statement and a 
virtual table of CDC type. CDC will be a new table type. 

A CDC DDL syntax to create CDC on a (data) table can be as follows: 

Create CDC  on  INCLUDE (pre | post | latest | 
all) MAX_LOOKBACK_AGE =  INDEX =  
SALT_BUCKETS=

The above CDC DDL creates a virtual CDC table and an uncovered index. The CDC 
table PK columns start with the timestamp and continue with the data table PK 
columns. The CDC table includes one non-PK column which is a JSON column. The 
change is expressed in this JSON column in multiple ways based on the CDC DDL 
or query statement. The change can be expressed as just the mutation for the 
change, the latest image of the row, the pre image of the row (the image before 
the change), the post image, or any combination of these. The CDC table is not 
a physical table on disk. It is just a virtual table to be used in a CDC query. 
Phoenix stores just the metadata for this virtual table. 

A CDC query can be as follow:

Select * from  where PHOENIX_ROW_TIMESTAMP() >= TO_DATE( …) AND 
PHOENIX_ROW_TIMESTAMP() < TO_DATE( …)

This query would return the rows of the CDC table which is constructed on the 
server side by CDCUncoveredIndexRegionScanner by joining the uncovered index 
row versions with the corresponding data table row version (using raw scans). 
The above select query can be hinted at by using a new CDC hint to return just 
the actual change, pre, pos, or latest image of the row, or a combination of 
them to overwrite the default JSON column format defined by the CDC DDL 
statement. 

The CDC application will run the above query in a loop. When the difference 
between the current time of the application 

[jira] [Assigned] (PHOENIX-7187) Improvement of Integration test case with Explain Plan for Partial Index

[Nikita Pande (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nikita Pande reassigned PHOENIX-7187:
-

Assignee: Nikita Pande

> Improvement of Integration test case with Explain Plan for Partial Index
> 
>
> Key: PHOENIX-7187
> URL: https://issues.apache.org/jira/browse/PHOENIX-7187
> Project: Phoenix
>  Issue Type: Improvement
>Reporter: Nikita Pande
>Assignee: Nikita Pande
>Priority: Major
>
> Currently in partialIndexIT.java, currently the schema name and table name of 
> the index table in the PhoenixResultSet context match the expected schema 
> name and table name is validated.
> There is no test case available for the Explain Plan.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (PHOENIX-7187) Improvement of Integration test case with Explain Plan for Partial Index

[Nikita Pande (Jira)]

Nikita Pande created PHOENIX-7187:
-

 Summary: Improvement of Integration test case with Explain Plan 
for Partial Index
 Key: PHOENIX-7187
 URL: https://issues.apache.org/jira/browse/PHOENIX-7187
 Project: Phoenix
  Issue Type: Improvement
Reporter: Nikita Pande


Currently in partialIndexIT.java, currently the schema name and table name of 
the index table in the PhoenixResultSet context match the expected schema name 
and table name is validated.
There is no test case available for the Explain Plan.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] Using reflection to support HBase 2 and 3 in Omid

[Istvan Toth]

Hi!

I'm working on Hbase 3 support, starting with Omid.
While the API changes are not major, HBase 3 does remove HTableDescriptor
and some methods in Filter.

We COULD solve this with extra modules and profiles, but since the
difference is rather small, I'd like to avoid that and use reflection to
handle the missing methods.

The Filter code IS performance sensitive, but my micro-benchmarks show that
reflection doesn't cause a performance hit in this case.

I'm looking for your feedback for this solution.
Did I make a mistake when benchmarking ?
Is there some other reason not to use this solution ?

Ticket:
https://issues.apache.org/jira/browse/OMID-271
Draft PR:
https://github.com/apache/phoenix-omid/pull/152
Class using reflection:
https://github.com/apache/phoenix-omid/pull/152/files#diff-7cfca9b7c88aeeb17c7a662a0215e904acf0854d8eeaeadacbdb45b4ebeec2e3
Reflection performance microbenchmark:
https://github.com/apache/phoenix-omid/pull/152/files#diff-e9f0ca82e9c730fbe803d28531ddef05b6486e903ef46de3c09c340cb861082b

best regards
Istvan

[jira] [Updated] (PHOENIX-7081) Replace /tmp with PHOENIX-7175

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth updated PHOENIX-7081:
-
Summary: Replace /tmp with PHOENIX-7175  (was: Replace /tmp with )

> Replace /tmp with PHOENIX-7175
> --
>
> Key: PHOENIX-7081
> URL: https://issues.apache.org/jira/browse/PHOENIX-7081
> Project: Phoenix
>  Issue Type: Bug
>  Components: core
>Reporter: Istvan Toth
>Assignee: Divneet Kaur
>Priority: Minor
>  Labels: beginner, test
>
> I was running two test suites in parallel on a large VM, and it seems that 
> OrphanViewToolIT cannot handle that.
> {noformat}
> java.io.FileNotFoundException: /tmp/OrphanView.txt (No such file or 
> directory){noformat}
> Use some directory here that is under the source code's root.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (PHOENIX-7081) Replace /tmp with {java.io.tmpdir} in tests

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth updated PHOENIX-7081:
-
Summary: Replace /tmp with {java.io.tmpdir} in tests  (was: Replace /tmp 
with PHOENIX-7175)

> Replace /tmp with {java.io.tmpdir} in tests
> ---
>
> Key: PHOENIX-7081
> URL: https://issues.apache.org/jira/browse/PHOENIX-7081
> Project: Phoenix
>  Issue Type: Bug
>  Components: core
>Reporter: Istvan Toth
>Assignee: Divneet Kaur
>Priority: Minor
>  Labels: beginner, test
>
> I was running two test suites in parallel on a large VM, and it seems that 
> OrphanViewToolIT cannot handle that.
> {noformat}
> java.io.FileNotFoundException: /tmp/OrphanView.txt (No such file or 
> directory){noformat}
> Use some directory here that is under the source code's root.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (PHOENIX-7081) Replace /tmp with

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth updated PHOENIX-7081:
-
Summary: Replace /tmp with   (was: OrphanViewToolIT uses global /tmp)

> Replace /tmp with 
> --
>
> Key: PHOENIX-7081
> URL: https://issues.apache.org/jira/browse/PHOENIX-7081
> Project: Phoenix
>  Issue Type: Bug
>  Components: core
>Reporter: Istvan Toth
>Assignee: Divneet Kaur
>Priority: Minor
>  Labels: beginner, test
>
> I was running two test suites in parallel on a large VM, and it seems that 
> OrphanViewToolIT cannot handle that.
> {noformat}
> java.io.FileNotFoundException: /tmp/OrphanView.txt (No such file or 
> directory){noformat}
> Use some directory here that is under the source code's root.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (PHOENIX-7175) Set java.io.tmpdir to the maven build directory for tests

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth resolved PHOENIX-7175.
--
Fix Version/s: 5.2.0
   5.1.4
   Resolution: Fixed

Merged to master and 5.1.
Thank you [~divneet18].

> Set java.io.tmpdir to the maven build directory for tests
> -
>
> Key: PHOENIX-7175
> URL: https://issues.apache.org/jira/browse/PHOENIX-7175
> Project: Phoenix
>  Issue Type: Bug
>  Components: connectors, core, queryserver
>Reporter: Istvan Toth
>Assignee: Divneet Kaur
>Priority: Minor
> Fix For: 5.2.0, 5.1.4
>
>
> Our tests are currently using a global tmpdir.
> This cuses conflicts when running multiple test runs on the same machine.
> Set java.io.tmpdir to the build directory.
> We can copy this from HBase:
> https://github.com/apache/hbase/blob/a09305d5854fc98300426271fad3b53a69d2ae71/pom.xml#L1879



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OMID-270) Set java.io.tmpdir to the maven build directory for tests

[Istvan Toth (Jira)]

[ 
https://issues.apache.org/jira/browse/OMID-270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809824#comment-17809824
 ] 

Istvan Toth commented on OMID-270:
--

Do you want to pick this one up [~divneet18] ?

> Set java.io.tmpdir to the maven build directory for tests
> -
>
> Key: OMID-270
> URL: https://issues.apache.org/jira/browse/OMID-270
> Project: Phoenix Omid
>  Issue Type: Bug
>Reporter: Istvan Toth
>Priority: Major
>
> Same as PHOENIX-7175



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (PHOENIX-7175) Set java.io.tmpdir to the maven build directory for tests

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth reopened PHOENIX-7175:
--

> Set java.io.tmpdir to the maven build directory for tests
> -
>
> Key: PHOENIX-7175
> URL: https://issues.apache.org/jira/browse/PHOENIX-7175
> Project: Phoenix
>  Issue Type: Bug
>  Components: connectors, core, queryserver
>Reporter: Istvan Toth
>Assignee: Divneet Kaur
>Priority: Minor
>  Labels: test
> Fix For: 5.2.0, 5.1.4
>
>
> Our tests are currently using a global tmpdir.
> This cuses conflicts when running multiple test runs on the same machine.
> Set java.io.tmpdir to the build directory.
> We can copy this from HBase:
> https://github.com/apache/hbase/blob/a09305d5854fc98300426271fad3b53a69d2ae71/pom.xml#L1879



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (PHOENIX-7175) Set java.io.tmpdir to the maven build directory for tests

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/PHOENIX-7175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth updated PHOENIX-7175:
-
Labels: test  (was: )

> Set java.io.tmpdir to the maven build directory for tests
> -
>
> Key: PHOENIX-7175
> URL: https://issues.apache.org/jira/browse/PHOENIX-7175
> Project: Phoenix
>  Issue Type: Bug
>  Components: connectors, core, queryserver
>Reporter: Istvan Toth
>Assignee: Divneet Kaur
>Priority: Minor
>  Labels: test
> Fix For: 5.2.0, 5.1.4
>
>
> Our tests are currently using a global tmpdir.
> This cuses conflicts when running multiple test runs on the same machine.
> Set java.io.tmpdir to the build directory.
> We can copy this from HBase:
> https://github.com/apache/hbase/blob/a09305d5854fc98300426271fad3b53a69d2ae71/pom.xml#L1879



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (OMID-273) TestTSOClientConnectionToTSO fails on Mac due to IPv6 connection failure

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/OMID-273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth resolved OMID-273.
--
Fix Version/s: 1.1.1
   Resolution: Duplicate

> TestTSOClientConnectionToTSO fails on Mac due to IPv6 connection failure
> 
>
> Key: OMID-273
> URL: https://issues.apache.org/jira/browse/OMID-273
> Project: Phoenix Omid
>  Issue Type: Bug
>Reporter: Richárd Antal
>Assignee: Istvan Toth
>Priority: Major
> Fix For: 1.1.1
>
>
> During the OMID 1.1.1 RC0 release we found that TestTSOClientConnectionToTSO 
> had failing tests:
> testSuccessfulConnectionToTSOThroughZK and
> testSuccessOfTSOClientReconnectionsToARestartedTSOWithZKPublishing
> It might be because it uses IPv6
> 2024-01-17T15:17:45,858 INFO 
> [TestNGInvoker-testSuccessfulConnectionToTSOThroughZK()] client.TSOClient: * 
> Current TSO host:port found in ZK: [fe80:0:0:0:aede:48ff:fe00:1122%en5]:52934 
> Epoch 0
> 2024-01-17T15:17:45,859 INFO  [tsofsm-0] client.TSOClient: Trying to
> connect to TSO [/fe80:0:0:0:aede:48ff:fe00:1122%en5:52934]
> 2024-01-17T15:17:45,863 ERROR [tsoclient-worker-0] client.TSOClient: Failed 
> connection attempt to TSO [/fe80:0:0:0:aede:48ff:fe00:1122%en5:52934] failed. 
> Channel [id: 0x9d8c0b44] 
> I've checked this en5 interface and it has only IPv6 and no IPv4
> I ran a git bisect but it wasn't very useful. Before OMID-248 
> TestTSOClientConnectionToTSO used IPv4 and after it, it is using IPv6 and 
> failing on my Mac



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (OMID-249) Improve default network address logic

[Istvan Toth (Jira)]

 [ 
https://issues.apache.org/jira/browse/OMID-249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Istvan Toth resolved OMID-249.
--
Fix Version/s: 1.1.1
   Resolution: Fixed

Committed.
Thanks for the review [~RichardAntal].

> Improve default network address logic
> -
>
> Key: OMID-249
> URL: https://issues.apache.org/jira/browse/OMID-249
> Project: Phoenix Omid
>  Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
> Fix For: 1.1.1
>
>
> NetworkUtils.getNetworkInterface() seems to be only used for determining the 
> host and port when registering TSO to ZK for HA.
> We should get the TSO public IP from the ZK TCP connection on demand, and not 
> worry about the default network interface at all.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OMID-249) Improve default network address logic

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/OMID-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809811#comment-17809811
 ] 

ASF GitHub Bot commented on OMID-249:
-

stoty closed pull request #154: OMID-249 Improve default network address logic
URL: https://github.com/apache/phoenix-omid/pull/154




> Improve default network address logic
> -
>
> Key: OMID-249
> URL: https://issues.apache.org/jira/browse/OMID-249
> Project: Phoenix Omid
>  Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
>
> NetworkUtils.getNetworkInterface() seems to be only used for determining the 
> host and port when registering TSO to ZK for HA.
> We should get the TSO public IP from the ZK TCP connection on demand, and not 
> worry about the default network interface at all.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (OMID-249) Improve default network address logic

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/OMID-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17809812#comment-17809812
 ] 

ASF GitHub Bot commented on OMID-249:
-

stoty commented on PR #154:
URL: https://github.com/apache/phoenix-omid/pull/154#issuecomment-1905593916

   merged manually




> Improve default network address logic
> -
>
> Key: OMID-249
> URL: https://issues.apache.org/jira/browse/OMID-249
> Project: Phoenix Omid
>  Issue Type: Improvement
>Affects Versions: 1.1.0
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Major
>
> NetworkUtils.getNetworkInterface() seems to be only used for determining the 
> host and port when registering TSO to ZK for HA.
> We should get the TSO public IP from the ZK TCP connection on demand, and not 
> worry about the default network interface at all.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)