Re: [ANNOUNCE] Lokesh Khurana as Phoenix Committer
Congratulations Lokesh! On Thu, Jan 18, 2024 at 10:58 PM Rushabh Shah wrote: > Congratulations Lokesh !! > > > > > On Wed, Jan 17, 2024 at 9:26 AM Viraj Jasani wrote: > > > On behalf of the Apache Phoenix PMC, I'm pleased to announce that Lokesh > > Khurana has accepted the PMC's invitation to become a committer on Apache > > Phoenix. > > > > We appreciate all of the great contributions Lokesh has made to the > > community thus far and we look forward to his continued involvement. > > > > Congratulations and Welcome, Lokesh! > > >
Re: [ANNOUNCE] New Phoenix PMC member Jacob Isaac
Congratulations Jacob Isaac! Well deserved! On Mon, Nov 6, 2023 at 6:12 PM rajeshb...@apache.org < chrajeshbab...@gmail.com> wrote: > Congratulations Jacob Isaac!! > > Thanks, > Rajeshbabu. > > On Tue, Nov 7, 2023, 7:37 AM Viraj Jasani wrote: > > > On behalf of the Apache Phoenix PMC, I'm pleased to announce that Jacob > > Isaac has accepted the invitation to join the PMC. > > > > We appreciate all of the great contributions Jacob has made to the > > community thus far and we look forward to his continued involvement. > > > > Please join me in congratulating Jacob Isaac! > > >
Re: [ANNOUNCE] Rushabh Shah as Phoenix Committer
Congratulations Rushabh! On Tue, Aug 15, 2023 at 10:46 AM Viraj Jasani wrote: > On behalf of the Apache Phoenix PMC, I'm pleased to announce that Rushabh > Shah has accepted the PMC's invitation to become a committer on Apache > Phoenix. > > We appreciate all of the great contributions Rushabh has made to the > community thus far and we look forward to his continued involvement. > > Congratulations and Welcome, Rushabh! >
Re: [ANNOUNCE] New Phoenix PMC member Tanuj Khurana
Congratulations Tanuj! Well deserved. On Tue, Jan 3, 2023 at 9:17 AM Xinyi Yan wrote: > Congratulations! > > Sent from my iPhone > > > On Jan 3, 2023, at 8:50 AM, Rushabh Shah > > > wrote: > > > > Congratulations Tanuj ! Well deserved. > > > > > >> On Mon, Jan 2, 2023 at 8:17 PM Viraj Jasani wrote: > >> > >> On behalf of the Apache Phoenix PMC, I'm pleased to announce that Tanuj > >> Khurana has accepted the invitation to join the PMC. > >> > >> We appreciate all of the great contributions Tanuj has made to the > >> community thus far and we look forward to his continued involvement. > >> > >> Please join me in congratulating Tanuj Khurana! > >> >
Re: [ANNOUNCE] Richard Antal joins Phoenix PMC
Congratulations, Richard! On Fri, Sep 30, 2022 at 8:38 AM Geoffrey Jacoby wrote: > Congratulations, Richard! > > On Thu, Sep 29, 2022 at 10:12 PM rajeshb...@apache.org < > chrajeshbab...@gmail.com> wrote: > >> On behalf of the Apache Phoenix PMC, I'm pleased to announce that >> Richard Antal >> has accepted our invitation to join the PMC. >> >> We appreciate all of the great contributions Richard has made to the >> community thus far and we look forward to his continued involvement. >> >> Please join me in congratulating Richard Antal! >> >> Thanks, >> Rajeshbabu. >> >
Re: Invitation: Apache Phoenix Tech Talk @ Thu Apr 7, 2022 9am - 10:30am (PDT) (d...@phoenix.apache.org)
The slides and recording have been uploaded at https://phoenix.apache.org/tech_talks.html. Thank you for your interest and patience. Kadir On Thu, Apr 7, 2022 at 8:52 PM Cong Luo wrote: > Hello, > > When will we upload the meeting record files? I can't wait any longer. > > On 2022/03/31 19:01:21 ka...@gsuite.cloud.apache.org wrote: > > You have been invited to the following event. > > > > Title: Apache Phoenix Tech Talk > > Title: JSON Support for Apache PhoenixAbstract: JSON is a popular data > > format that is used for exchanging data in many applications. Many > RESTful > > services return or accept data formatted as JSON. Also several > databases > > support JSON as their primary data format. This talk proposes changes > which > > will introduce JSON support in Phoenix to better support our > > customers.Presenter : Gokcen IskenderMeeting > > Link : > https://urldefense.com/v3/__https://salesforce.zoom.us/j/84383596313?pwd=RlZ0OWhlZk1DQ1NhZDg0cDFFeUdvdz09This__;!!DCbAVzZNrAf4!XWopl5rZfh9HRJJz8wrkaD0P5078mHUc7kFCA-UtT-6diaSWUYy8GcUg4c88Mfkimg$ > > > meeting will be recorded. > > When: Thu Apr 7, 2022 9am – 10:30am Pacific Time - Los Angeles > > > > Joining info: Join with Google Meet > > > https://urldefense.com/v3/__https://meet.google.com/vtd-zebv-vst?hs=224__;!!DCbAVzZNrAf4!XWopl5rZfh9HRJJz8wrkaD0P5078mHUc7kFCA-UtT-6diaSWUYy8GcUg4c9mACLE-Q$ > > > > Join by phone > > (US) +1 470-655-0545 (PIN: 865431500) > > > > More phone numbers: > https://urldefense.com/v3/__https://tel.meet/vtd-zebv-vst?pin=2105560975173=0__;!!DCbAVzZNrAf4!XWopl5rZfh9HRJJz8wrkaD0P5078mHUc7kFCA-UtT-6diaSWUYy8GcUg4c_qrnPz1A$ > > > > Calendar: d...@phoenix.apache.org > > Who: > > * ka...@gsuite.cloud.apache.org - organizer > > * d...@phoenix.apache.org > > * user@phoenix.apache.org > > > > Event details: > > > https://urldefense.com/v3/__https://calendar.google.com/calendar/event?action=VIEW=MTBudG1taGNqMHRyb3JuN3BoNjgzcDJsdXUgZGV2QHBob2VuaXguYXBhY2hlLm9yZw=Mjkja2FkaXJAZ3N1aXRlLmNsb3VkLmFwYWNoZS5vcmc0ZTZhMGQxMzlkMzY0ZWE1Nzg4MjE5ZTZlYzQ2Nzg2NThkMDNmMjdj=America*2FLos_Angeles=en=0__;JQ!!DCbAVzZNrAf4!XWopl5rZfh9HRJJz8wrkaD0P5078mHUc7kFCA-UtT-6diaSWUYy8GcUg4c-yea_lHA$ > > > > Invitation from Google Calendar: > https://urldefense.com/v3/__https://calendar.google.com/calendar/__;!!DCbAVzZNrAf4!XWopl5rZfh9HRJJz8wrkaD0P5078mHUc7kFCA-UtT-6diaSWUYy8GcUg4c963xMDrg$ > > > > You are receiving this courtesy email at the account > d...@phoenix.apache.org > > because you are an attendee of this event. > > > > To stop receiving future updates for this event, decline this event. > > Alternatively you can sign up for a Google account at > > > https://urldefense.com/v3/__https://calendar.google.com/calendar/__;!!DCbAVzZNrAf4!XWopl5rZfh9HRJJz8wrkaD0P5078mHUc7kFCA-UtT-6diaSWUYy8GcUg4c963xMDrg$ > and control your notification > > settings for your entire calendar. > > > > Forwarding this invitation could allow any recipient to send a response > to > > the organizer and be added to the guest list, or invite others > regardless > > of their own invitation status, or to modify your RSVP. Learn more at > > > https://urldefense.com/v3/__https://support.google.com/calendar/answer/37135*forwarding__;Iw!!DCbAVzZNrAf4!XWopl5rZfh9HRJJz8wrkaD0P5078mHUc7kFCA-UtT-6diaSWUYy8GcUg4c9KLgwP5A$ > > >
Tech Talk - JSON Support for Apache Phoenix
Hi All, We will have another very interesting tech talk session tomorrow. Gokcen Iskender will talk about upcoming JSON support in Phoenix at 9 am (PDT). You can join the meeting using this link <https://salesforce.zoom.us/j/84383596313?pwd=RlZ0OWhlZk1DQ1NhZDg0cDFFeUdvdz09> . For more information on Phoenix tech talks, please see the web page <https://phoenix.apache.org/tech_talks.html>. Thanks, Kadir
Invitation: Apache Phoenix Tech Talk @ Thu Apr 7, 2022 9am - 10:30am (PDT) (user@phoenix.apache.org)
BEGIN:VCALENDAR PRODID:-//Google Inc//Google Calendar 70.9054//EN VERSION:2.0 CALSCALE:GREGORIAN METHOD:REQUEST BEGIN:VEVENT DTSTART:20220407T16Z DTEND:20220407T173000Z DTSTAMP:20220331T190121Z ORGANIZER;CN=ka...@gsuite.cloud.apache.org:mailto:kadir@gsuite.cloud.apache .org UID:10ntmmhcj0trorn7ph683p2...@google.com ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE ;CN=ka...@gsuite.cloud.apache.org;X-NUM-GUESTS=0:mailto:kadir@gsuite.cloud. apache.org ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=d...@phoenix.apache.org;X-NUM-GUESTS=0:mailto:d...@phoenix.apache.org ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=user@phoenix.apache.org;X-NUM-GUESTS=0:mailto:user@phoenix.apache.o rg X-GOOGLE-CONFERENCE:https://meet.google.com/vtd-zebv-vst X-MICROSOFT-CDO-OWNERAPPTID:356854106 CREATED:20220331T190120Z DESCRIPTION:Title: JSON Support for Apache PhoenixAbstract: JSON is a popular data format that is used for exchanging data in many application s. Many RESTful services return or accept data formatted as JSON. Also seve ral databases support JSON as their primary data format. This talk proposes changes which will introduce JSON support in Phoenix to better support our customers.Presenter : Gokcen IskenderMeeting Link :\; \;https://salesforce.zoom.us/j/84383596313?pwd=RlZ0OWhlZk 1DQ1NhZDg0cDFFeUdvdz09">https://salesforce.zoom.us/j/84383596313?pwd=RlZ0OW hlZk1DQ1NhZDg0cDFFeUdvdz09This meeting will be recorded.\n\n-:: ~:~::~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~: ~::~:~::-\nDo not edit this section of the description.\n\nThis event has a video call.\nJoin: https://meet.google.com/vtd-zebv-vst\n(US) +1 470-655-0 545 PIN: 865431500#\nView more phone numbers: https://tel.meet/vtd-zebv-vst ?pin=2105560975173=7\n\nView your event at https://calendar.google.com/c alendar/event?action=VIEW=MTBudG1taGNqMHRyb3JuN3BoNjgzcDJsdXUgdXNlckBwa G9lbml4LmFwYWNoZS5vcmc=Mjkja2FkaXJAZ3N1aXRlLmNsb3VkLmFwYWNoZS5vcmcxMzg3 MGJmNTkwOTM5YWU4NDQzY2NmM2FlZGJjZGMyOTRjYmZmOGU5=America%2FLos_Angeles& hl=en=1.\n-::~:~::~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~: ~:~:~:~:~:~:~:~:~::~:~::- LAST-MODIFIED:20220331T190120Z LOCATION: SEQUENCE:0 STATUS:CONFIRMED SUMMARY:Apache Phoenix Tech Talk TRANSP:OPAQUE END:VEVENT END:VCALENDAR invite.ics Description: application/ics
Re: [ANNOUNCE] Tanuj Khurana as Phoenix Committer
Tanuj, Thank you for your contributions! Congratulations! On Fri, Dec 17, 2021 at 1:57 AM Viraj Jasani wrote: > Congratulations Tanuj! Well deserved! > > > On Fri, 17 Dec 2021 at 12:47 AM, Geoffrey Jacoby > wrote: > > > On behalf of the Apache Phoenix PMC, I'm pleased to announce that Tanuj > > Khurana has accepted the PMC's invitation to become a committer on Apache > > Phoenix. > > > > We appreciate all of the great contributions Tanuj has made to the > > community > > thus far and we look forward to his continued involvement. > > > > Welcome, Tanuj! > > > > Geoffrey Jacoby > > >
Re: Next Tech Talk on Online Data Format Change in Phoenix
We had the tech talk on Online Data Format Change in Phoenix today. The slides and recording for this tech talk are posted at https://phoenix.apache.org/tech_talks.html. Thanks, Kadir On Wed, Sep 29, 2021 at 11:53 AM Kadir Ozdemir wrote: > Hi All, > > Did you ever need to change the underlying data format of your table which > is already in production and serving your customers. For example, you > wanted to change the primary key of your table, or its column encoding or > even its storage format. And you wanted to do this without interrupting the > services depending on this table. If you did, then you would want to > attend the tech talk by Gokcen Iskender on Oct 07 (next Thursday at 9:00 > PST) and see how this will be possible in Phoenix. Hope to see you there. > Please see details at https://phoenix.apache.org/tech_talks.html. > > Thanks, > Kadir >
Next Tech Talk on Online Data Format Change in Phoenix
Hi All, Did you ever need to change the underlying data format of your table which is already in production and serving your customers. For example, you wanted to change the primary key of your table, or its column encoding or even its storage format. And you wanted to do this without interrupting the services depending on this table. If you did, then you would want to attend the tech talk by Gokcen Iskender on Oct 07 (next Thursday at 9:00 PST) and see how this will be possible in Phoenix. Hope to see you there. Please see details at https://phoenix.apache.org/tech_talks.html. Thanks, Kadir
Invitation: Apache Phoenix Tech Talk @ Thu Oct 7, 2021 9am - 10:30am (PDT) (user@phoenix.apache.org)
BEGIN:VCALENDAR PRODID:-//Google Inc//Google Calendar 70.9054//EN VERSION:2.0 CALSCALE:GREGORIAN METHOD:REQUEST BEGIN:VEVENT DTSTART:20211007T16Z DTEND:20211007T173000Z DTSTAMP:20210929T182931Z ORGANIZER;CN=ka...@gsuite.cloud.apache.org:mailto:kadir@gsuite.cloud.apache .org UID:2839cfd3o2o7grqo05pa55e...@google.com ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=user@phoenix.apache.org;X-NUM-GUESTS=0:mailto:user@phoenix.apache.o rg ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE ;CN=ka...@gsuite.cloud.apache.org;X-NUM-GUESTS=0:mailto:kadir@gsuite.cloud. apache.org ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=d...@phoenix.apache.org;X-NUM-GUESTS=0:mailto:d...@phoenix.apache.org X-MICROSOFT-CDO-OWNERAPPTID:-2090667828 CREATED:20210929T182930Z DESCRIPTION:Title: Online Data Format Change in PhoenixAbstract: Ph oenix lets the users to make certain changes to the table/index schemas via ALTER TABLE and ALTER INDEX. Some schema changes such as changing the row key (primary keys)\, the type of a column\, the table storage format\, the column encoding\, etc. requires table/index rows to be re-written. This tal k introduces Online Data Format change feature which will let Phoenix users to make those changes with no service interruption. The focus will be on c hanging column encoding and table storage format.Presenter : Gokcen IskenderMeeting Link : https://salesforce.zoom.us/j/8624511?pw d=MXFNTkYrUzFoRFkxYnlybnlwNnc4UT09This meeting will be recorded.Please see https://phoenix.apache.org/tech_talks.html;>https: //phoenix.apache.org/tech_talks.html for the info on upcomi ng meetings\, and the recordings of the previous meetings.\n\n-::~:~::~:~:~ :~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~::~:~::-\ nDo not edit this section of the description.\n\nView your event at https:/ /calendar.google.com/calendar/event?action=VIEW=MjgzOWNmZDNvMm83Z3JxbzA 1cGE1NWVuZ3IgdXNlckBwaG9lbml4LmFwYWNoZS5vcmc=Mjkja2FkaXJAZ3N1aXRlLmNsb3 VkLmFwYWNoZS5vcmc2Y2ZiYjY0OTU1YzMzYmZjNTBlZjVjNzJkMzEyNzljODdjYjAwNTBj= America%2FLos_Angeles=en=1.\n-::~:~::~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~: ~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~::~:~::- LAST-MODIFIED:20210929T182930Z LOCATION: SEQUENCE:0 STATUS:CONFIRMED SUMMARY:Apache Phoenix Tech Talk TRANSP:OPAQUE END:VEVENT END:VCALENDAR invite.ics Description: application/ics
Re: Urgent problem with Phoenix function LAST_VALUE and joins
Geoffrey,
Thank you for pointing out this. I mixed up release numbers. Yes, the
paging feature cannot be the reason here. My comment for disabling the
paging feature should be ignored.
Kadir
On Tue, Sep 28, 2021 at 7:44 AM Geoffrey Jacoby
wrote:
> Kadir,
>
> The server-side paging changes are in Phoenix 5.1, which is at feature
> parity with 4.16 for HBase 1.x, not 5.0 (which was released in 2018 and is
> roughly equivalent to Phoenix 4.14).
>
> So if Simon's environment is Phoenix 5.0, it can't be affected by the
> server-side changes. (Though I see that he's running HBase 2.2, which I
> thought wasn't supported until Phoenix 5.1?)
>
> Geoffrey
>
> On Tue, Sep 28, 2021 at 4:07 AM Simon Mottram <
> simon.mott...@cucumber.co.nz> wrote:
>
>> Hi thanks for the quick response
>>
>> I'm going to appear thick now, brace yourself. By client do you mean
>> the queryserver running in EMR which does have an hbase-site.xml or my Java
>> API as a property, we don't have an hbase-site.xml (unless it's embedded in
>> the thin client jar.
>>
>> Cheers
>>
>> S
>> --
>> *From:* Kadir Ozdemir
>> *Sent:* 28 September 2021 7:45 PM
>> *To:* user
>> *Subject:* Re: Urgent problem with Phoenix function LAST_VALUE and joins
>>
>> In Phoenix 5.0 we introduced a new server side paging feature to
>> eliminate timeouts due long running server side computations such
>> aggregation and joins. I wonder if this new feature caused your issue. If
>> so, the work around is to disable it by
>> setting phoenix.server.paging.enabled to false in hbase-site.xml. This is a
>> client side config param so you just need to restart your client. Hope this
>> will fix your issue.
>>
>> On Mon, Sep 27, 2021 at 9:23 PM Simon Mottram <
>> simon.mott...@cucumber.co.nz> wrote:
>>
>> Hi
>>
>> Got my fingers crossed that there's a work around for this as this really
>> is a big problem for us
>>
>> We are using:
>>
>> Amazon EMR
>>
>> Release label:emr-6.1.0
>> Hadoop distribution:Amazon
>> Applications:Hbase 2.2.5, Hive 3.1.2, Phoenix 5.0.0, Pig 0.17.0
>>
>> Thin Client version:
>> phoenix-5.0.0-HBase-2.0-thin-client.jar
>>
>> We get the following error when doing an LAST_VALUE aggregation where
>>
>> 1. A JOIN is empty
>> 2. The column is INTEGER or DATETIME
>>
>> Remote driver error: IllegalArgumentException: offset (25) + length (4)
>> exceed the capacity of the array: 25
>>
>> The query that breaks is:
>>
>> SELECT
>> "BIOMATERIAL_NAME",
>> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
>> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
>> FROM
>> VARIABLE_VALUES_QA.OBSERVATION
>> JOIN VARIABLE_VALUES_QA.OBSERVATION_TAG_INDEX
>> ON DOCID = OBSERVATION_VALUE_ID
>> AND TAGNAME = 'TRIAL_ID'
>> AND TAGVALUE = 'f62dd8e0-d2ea-4d9a-9ab6-2049601bb9fe'
>> GROUP BY
>> "BIOMATERIAL_NAME"
>> LIMIT 10
>> OFFSET 0;
>>
>> I can refactor this using EXIST but get same error, presumably the driver
>> knows to treat them the same:
>>
>> SELECT
>> "BIOMATERIAL_NAME",
>> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
>> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
>> FROM
>> VARIABLE_VALUES_QA.OBSERVATION
>> WHERE
>> EXISTS (
>> SELECT
>> DOCID
>> FROM
>> VARIABLE_VALUES_QA.OBSERVATION_TAG_INDEX
>> WHERE
>> DOCID = OBSERVATION_VALUE_ID
>> AND TAGNAME = 'TRIAL_ID'
>> AND TAGVALUE = 'f62dd8e0-d2ea-4d9a-9ab6-2049601bb9fe')
>> GROUP BY
>> "BIOMATERIAL_NAME"
>> LIMIT 10
>> OFFSET 0;
>>
>> If we remove the external reference we get no error, regardless of
>> whether there are any hits or not
>>
>> -- these all work
>> There are no hits for this query
>>
>> SELECT
>> "BIOMATERIAL_NAME",
>> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
>> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
>> FROM
>> VARIABLE_VALUES_QA.OBSERVATION
>> WHERE
>> BIOMATERIAL_TYPE = 'aardvark'
>> GROUP BY
>> "BIOMATERIAL_NAME"
>> LIMIT 10
>> OFFSET 0;
>>
>> Lots of hits for this query:
>>
>> SELECT
>> "BIOMATERIAL_NAME",
>> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
>> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
>>
Re: Urgent problem with Phoenix function LAST_VALUE and joins
In Phoenix 5.0 we introduced a new server side paging feature to eliminate
timeouts due long running server side computations such aggregation and
joins. I wonder if this new feature caused your issue. If so, the work
around is to disable it by setting phoenix.server.paging.enabled to false
in hbase-site.xml. This is a client side config param so you just need to
restart your client. Hope this will fix your issue.
On Mon, Sep 27, 2021 at 9:23 PM Simon Mottram
wrote:
> Hi
>
> Got my fingers crossed that there's a work around for this as this really
> is a big problem for us
>
> We are using:
>
> Amazon EMR
>
> Release label:emr-6.1.0
> Hadoop distribution:Amazon
> Applications:Hbase 2.2.5, Hive 3.1.2, Phoenix 5.0.0, Pig 0.17.0
>
> Thin Client version:
> phoenix-5.0.0-HBase-2.0-thin-client.jar
>
> We get the following error when doing an LAST_VALUE aggregation where
>
> 1. A JOIN is empty
> 2. The column is INTEGER or DATETIME
>
> Remote driver error: IllegalArgumentException: offset (25) + length (4)
> exceed the capacity of the array: 25
>
> The query that breaks is:
>
> SELECT
> "BIOMATERIAL_NAME",
> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
> FROM
> VARIABLE_VALUES_QA.OBSERVATION
> JOIN VARIABLE_VALUES_QA.OBSERVATION_TAG_INDEX
> ON DOCID = OBSERVATION_VALUE_ID
> AND TAGNAME = 'TRIAL_ID'
> AND TAGVALUE = 'f62dd8e0-d2ea-4d9a-9ab6-2049601bb9fe'
> GROUP BY
> "BIOMATERIAL_NAME"
> LIMIT 10
> OFFSET 0;
>
> I can refactor this using EXIST but get same error, presumably the driver
> knows to treat them the same:
>
> SELECT
> "BIOMATERIAL_NAME",
> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
> FROM
> VARIABLE_VALUES_QA.OBSERVATION
> WHERE
> EXISTS (
> SELECT
> DOCID
> FROM
> VARIABLE_VALUES_QA.OBSERVATION_TAG_INDEX
> WHERE
> DOCID = OBSERVATION_VALUE_ID
> AND TAGNAME = 'TRIAL_ID'
> AND TAGVALUE = 'f62dd8e0-d2ea-4d9a-9ab6-2049601bb9fe')
> GROUP BY
> "BIOMATERIAL_NAME"
> LIMIT 10
> OFFSET 0;
>
> If we remove the external reference we get no error, regardless of whether
> there are any hits or not
>
> -- these all work
> There are no hits for this query
>
> SELECT
> "BIOMATERIAL_NAME",
> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
> FROM
> VARIABLE_VALUES_QA.OBSERVATION
> WHERE
> BIOMATERIAL_TYPE = 'aardvark'
> GROUP BY
> "BIOMATERIAL_NAME"
> LIMIT 10
> OFFSET 0;
>
> Lots of hits for this query:
>
> SELECT
> "BIOMATERIAL_NAME",
> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
> FROM
> VARIABLE_VALUES_QA.OBSERVATION
> GROUP BY
> "BIOMATERIAL_NAME"
> LIMIT 10 OFFSET 0;
>
> I've tried weird things like:
>
> Comparing exists to TRUE to try and force it into a normal BOOLEAN value,
> same IllegalArgumentException.
>
> SELECT
> "BIOMATERIAL_NAME",
> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
> FROM
> VARIABLE_VALUES_QA.OBSERVATION
> WHERE
> (EXISTS (
> SELECT
> DOCID
> FROM
> VARIABLE_VALUES_QA.OBSERVATION_TAG_INDEX
> WHERE
> DOCID = OBSERVATION_VALUE_ID
> AND TAGNAME = 'TRIAL_ID'
> AND TAGVALUE = 'f62dd8e0-d2ea-4d9a-9ab6-2049601bb9fe')) = TRUE
> GROUP BY
> "BIOMATERIAL_NAME"
> LIMIT 10
> OFFSET 0;
>
> And are you prepared for this one, which throws exact same error:
>
> SELECT
> "BIOMATERIAL_NAME",
> FIRST_VALUE("PLANT_FRUIT_COUNT") WITHIN GROUP (
> ORDER BY OBSERVATION_DATE DESC) AS "Plant Fruit Count"
> FROM
> VARIABLE_VALUES_QA.OBSERVATION
> WHERE
> TRUE
> GROUP BY
> "BIOMATERIAL_NAME"
> LIMIT 10
> OFFSET 0;
>
> Change FIRST_VALUE to AVG and it works fine.
>
Next Tech Talk Meeting in September 02
Hi All, The next tech talk meeting will be on September 02. We have not received any proposal for the August meeting yet and so we will skip it. I will update https://phoenix.apache.org/tech_talks.html accordingly. If you are thinking about being a host for the next meeting, please let us know soon. Thanks, Kadir
Tech Talk on Thursday : Phoenix High Availability
Hi All, I am hoping that you will be able to join the tech talk meeting on Thursday. Mingliang Liu, Daniel Wong, and Abhishek Singh Chouhan will introduce a new feature, Phoenix High Availability (HA) which allows Phoenix users to interact with multiple Phoenix/HBase clusters in order to achieve additional availability. Please see https://phoenix.apache.org/ tech_talks.html <https://www.google.com/url?q=https://phoenix.apache.org/tech_talks.html=D=calendar=1624940767308000=AOvVaw1Q7PV9bDzTi7_MDpvK_bVT> for details. Thanks, Kadir
Invitation: Invitation: Apache Phoenix Tech Talks @ Thu May 6, 2021 9... @ Thu Jul 1, 2021 9am - 10:30am (PDT) (user@phoenix.apache.org)
BEGIN:VCALENDAR PRODID:-//Google Inc//Google Calendar 70.9054//EN VERSION:2.0 CALSCALE:GREGORIAN METHOD:REQUEST BEGIN:VEVENT DTSTART:20210701T16Z DTEND:20210701T173000Z DTSTAMP:20210624T042309Z ORGANIZER;CN=ka...@gsuite.cloud.apache.org:mailto:kadir@gsuite.cloud.apache .org UID:37csinvqaddg6u6rffe9rtv...@google.com ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=user@phoenix.apache.org;X-NUM-GUESTS=0:mailto:user@phoenix.apache.o rg ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE ;CN=ka...@gsuite.cloud.apache.org;X-NUM-GUESTS=0:mailto:kadir@gsuite.cloud. apache.org ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=d...@phoenix.apache.org;X-NUM-GUESTS=0:mailto:d...@phoenix.apache.org X-MICROSOFT-CDO-OWNERAPPTID:1628347771 CREATED:20210624T042308Z DESCRIPTION:Title: Phoenix High AvailabilityAbstract: This talk int roduces Phoenix High Availability (HA) feature which allows Phoenix users t o interact with multiple Phoenix/HBase clusters in order to achieve additio nal availability compared to a single cluster. In particular we target the common deployment configuration of two HBase clusters with master/master as ynchronous replication enabled between the queried tables\, but with consid eration to future extensions in use cases\, replication\, and number of clu sters.Presenters : Mingliang Liu\, Daniel Wong\, and Abhishek Singh ChouhanMeeting Link : https://salesforce.zoom.us/j/89205670836?pwd =NWdwTG9DQTBqRlBtUWRFUFBXNWYxQT09This meeting will be recorded. Please see https://phoenix.apache.org/tech_talks.html;>https:/ /phoenix.apache.org/tech_talks.html for the info on upcoming meeti ngs\, and the recordings of the previous meetings.\n\n-::~:~::~:~:~:~:~:~:~ :~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~::~:~::-\nDo not edit this section of the description.\n\nThis event has a video call.\nJoin : https://meet.google.com/gom-yhzk-cty\n(US) +1 347-871-9216 PIN: 864842583 #\nView more phone numbers: https://tel.meet/gom-yhzk-cty?pin=3037665245088 =7\n\nView your event at https://calendar.google.com/calendar/event?acti on=VIEW=Mzdjc2ludnFhZGRnNnU2cmZmZTlydHZocXMgdXNlckBwaG9lbml4LmFwYWNoZS5 vcmc=Mjkja2FkaXJAZ3N1aXRlLmNsb3VkLmFwYWNoZS5vcmdjMGUxNjI1ZDMyOWY1NGVhYj dlNWE4N2QxMjVhMWU1N2VlNzBkZThh=America%2FLos_Angeles=en=1.\n-::~: ~::~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~: :~:~::- LAST-MODIFIED:20210624T042308Z LOCATION: SEQUENCE:0 STATUS:CONFIRMED SUMMARY:Invitation: Apache Phoenix Tech Talks @ Thu May 6\, 2021 9am - 10:3 0am (PDT) (user@phoenix.apache.org) TRANSP:OPAQUE END:VEVENT END:VCALENDAR invite.ics Description: application/ics
Re: [ANNOUNCE] New Committer Jacob Isaac
Congratulations Jacob! On Wed, Jun 2, 2021 at 8:17 PM Xinyi Yan wrote: > On behalf of the Apache Phoenix PMC, I'm pleased to announce that Jacob > Isaac has accepted the PMC's invitation to become a committer on Apache > Phoenix. > > We appreciate all of the great contributions Jacob has made to the > community thus far and we look forward to his continued involvement. > > Welcome, Jacob! > > Xinyi >
Re: June Tech Talk Meeting
We will skip the June meeting as the meeting time did not work out. However, the Salesforce team working on the Phoenix HA project is planning to present it for the July meeting. The Phoenix HA project is currently an internal project at Salesforce and uses two Phoenix/HBase clusters to improve availability for immutable use cases and/or provide disaster recovery. I look forward to seeing the abstract and agenda posted here in the next couple of weeks. Thanks, Kadir On Wed, May 19, 2021 at 11:49 AM Kadir Ozdemir wrote: > Hi All, > > Our next tech talk meeting is on June 03 and we have not received a topic > suggestion yet. Please let us know by May 27 if you would like to be a > presenter (see https://phoenix.apache.org/tech_talks.html). > > Thanks, > Kadir > >
June Tech Talk Meeting
Hi All, Our next tech talk meeting is on June 03 and we have not received a topic suggestion yet. Please let us know by May 27 if you would like to be a presenter (see https://phoenix.apache.org/tech_talks.html). Thanks, Kadir
Missed Tech Talks meeting?
Have you missed our monthly tech talks? No worries, you can still access the meeting slides and recordings at https://phoenix.apache.org/tech_talks.html. Our next meeting is on June 03. If you would like to present in the next meeting, please let us know your topic suggestion. Thanks, Kadir
Invitation: Apache Phoenix Tech Talks @ Thu May 6, 2021 9am - 10:30am (PDT) (user@phoenix.apache.org)
BEGIN:VCALENDAR PRODID:-//Google Inc//Google Calendar 70.9054//EN VERSION:2.0 CALSCALE:GREGORIAN METHOD:REQUEST BEGIN:VEVENT DTSTART:20210506T16Z DTEND:20210506T173000Z DTSTAMP:20210429T221853Z ORGANIZER;CN=ka...@gsuite.cloud.apache.org:mailto:kadir@gsuite.cloud.apache .org UID:5pt3n9aia930o2hdrtbig5a...@google.com ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=user@phoenix.apache.org;X-NUM-GUESTS=0:mailto:user@phoenix.apache.o rg ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE ;CN=ka...@gsuite.cloud.apache.org;X-NUM-GUESTS=0:mailto:kadir@gsuite.cloud. apache.org ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=d...@phoenix.apache.org;X-NUM-GUESTS=0:mailto:d...@phoenix.apache.org X-MICROSOFT-CDO-OWNERAPPTID:1688338849 CREATED:20210429T221853Z DESCRIPTION:Abstract:Atomic upserts are supported for tables without gl obal indexes. The current design of atomic upserts cannot be expanded to su pport tables with global indexes since doing index table updates results in cluster wide deadlocks. This talk covers how atomic upserts can be used\, details around how it is implemented today\, its limitations and how we can leverage the new global secondary indexing design to support atomic upsert s on tables with global indexes.Presenter: Tanuj KhuranaMee ting Link :\;https://salesforce.zoom.us/j/87348086990?pwd=Q0J HMXZUZGpOUlJzcEVsV2d2alVSQT09">https://salesforce.zoom.us/j/87348086990?pwd =Q0JHMXZUZGpOUlJzcEVsV2d2alVSQT09This meeting will be recorded. Please see https://phoenix.apache.org/tech_talks.html;>htt ps://phoenix.apache.org/tech_talks.html for the info on upcoming meetin gs\, and the recordings of the previous meetings.\n\n-::~:~::~:~:~:~:~:~:~: ~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~::~:~::-\nPlease d o not edit this section of the description.\n\nThis event has a video call. \nJoin: https://meet.google.com/tbi-vbdp-nxs\n(US) +1 518-732-5992 PIN: 684 572676#\nView more phone numbers: https://tel.meet/tbi-vbdp-nxs?pin=2371462 143464=7\n\nView your event at https://calendar.google.com/calendar/even t?action=VIEW=NXB0M245YWlhOTMwbzJoZHJ0YmlnNWFmbnQgdXNlckBwaG9lbml4LmFwY WNoZS5vcmc=Mjkja2FkaXJAZ3N1aXRlLmNsb3VkLmFwYWNoZS5vcmc4ZjU5NGJjYmI1ODYx YjFiOGNhYWRjMzAwYjk2OTA5ZjRmNWNiNWUw=America%2FLos_Angeles=en=1.\ n-::~:~::~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~: ~:~:~::~:~::- LAST-MODIFIED:20210429T221853Z LOCATION: SEQUENCE:0 STATUS:CONFIRMED SUMMARY:Apache Phoenix Tech Talks TRANSP:OPAQUE END:VEVENT END:VCALENDAR invite.ics Description: application/ics
Re: Topic suggestions for May Tech Talk
Hi Tanuj, This sounds great! Thank you for the topic suggestion. You will be the host for the next meeting. PHOENIX-6387 <https://issues.apache.org/jira/browse/PHOENIX-6387> redesigns atomic upserts and I am sure many of us would like to know more about it. I look forward to another engaging tech talk session. Kadir On Thu, Apr 22, 2021 at 10:46 AM Tanuj Khurana wrote: > Hi Kadir, > > I would like to present on Atomic Upserts in Phoenix. My talk will cover > how atomic upserts can be used in Phoenix, details around how its > implemented today, its limitations and how we can leverage the new > secondary indexing design to support atomic upserts on tables with indexes > (PHOENIX-6387). > > Thanks, > Tanuj > > On Thu, 15 Apr 2021 at 13:08, Kadir Ozdemir > wrote: > >> Hi All, >> >> Our next tech talk meeting will be held at 9AM PST on May 06. If you >> like to present a topic for the next meeting, please let us know by April >> 22. For more information about our tech talks, and the slides and >> recordings for the previous presentations, please visit >> https://phoenix.apache.org/tech_talks.html. >> >> Thanks, >> Kadir >> >
Topic suggestions for May Tech Talk
Hi All, Our next tech talk meeting will be held at 9AM PST on May 06. If you like to present a topic for the next meeting, please let us know by April 22. For more information about our tech talks, and the slides and recordings for the previous presentations, please visit https://phoenix.apache.org/tech_talks.html. Thanks, Kadir
Re: Speedup initial index creation
Also, there are config params to reduce the impact of the IndexTool on your
cluster, such as reducing the number of mappers and the amount of work to
be done in each RPC call initiated by a mapper to the server so that you
can run IndexTool outside the maintenance window.
On Thu, Apr 1, 2021 at 9:08 PM Kadir Ozdemir wrote:
> If you have only one version for the rows, then you should not see much
> difference between old and new. Can you provide some info on your
> performance testing? For example, number of rows in your data table, number
> of indexes, number of data table regions, number of MapReduce mappers,
> IndexTool MapReduce job duration, number of data nodes in your cluster, any
> IndexTool config params setting, etc.
>
> On Thu, Apr 1, 2021 at 6:12 PM Alexander Batyrshin <0x62...@gmail.com>
> wrote:
>
>>
>>
>> 2 апр. 2021 г., в 03:55, Kadir Ozdemir написал(а):
>>
>>
>> 1) I was thinking about the bulk load tool (
>> https://phoenix.apache.org/bulk_dataload.html). However, in this case,
>> you are not interested in bulk loading into the data table and its index
>> but just the index table. Now, I see that it would not work for you. You
>> are supposed to build a strongly consistent index once when you create the
>> index. I am curious why you are so concerned about its performance.
>>
>>
>> I need minimum maintenance time window on our cluster.
>>
>> 2) I thought you wanted to disable WAL only during index rebuild for the
>> index table, not all the time. You should be able to still use the ALTER
>> TABLE command with the new index design. Please note that in this case you
>> would disable WAL for the main table too. Is that what you are looking
>> for? If you are willing to disable WAL, then there is no point in using
>> strongly consistent indexes because you would lose recently written data if
>> region servers crash. By the way, you can use IndexUpgradeTool to downgrade
>> your tables to the old design (to replace IndexRegionObserver with
>> Indexer), see https://phoenix.apache.org/secondary_indexing.html
>>
>>
>> I know about possibility of data loosing. But it’s not a problem if main
>> table do not receive mutation during index creation (maintenance window).
>>
>> Old indexes goes inconsistent too often, so it not the way.
>>
>>
>> 3) Delete markers will be added each time you run the index create
>> command whenever the data table rows have multiple versions and the
>> versions of a row have different values for indexed columns.
>>
>>
>> My table has 1 version per row after major-compaction. Also main table
>> has no mutation during index creation
>>
>> On Thu, Apr 1, 2021 at 3:28 PM Alexander Batyrshin <0x62...@gmail.com>
>> wrote:
>>
>>> 1) How to create index old way via intermediate HFiles?
>>>
>>> I see “direct” option for IndexTool but description says its disabled:
>>>
>>> private static final Option DIRECT_API_OPTION = new Option("direct",
>>> "direct", false,
>>> "This parameter is deprecated. Direct mode will be used whether it
>>> is set or not. Keeping it for backwards compatibility.”);
>>>
>>>
>>> 2) On phoenix-4.14.2 (old indexes) WAL disabling for index table was
>>> possible by “ALTER TABLE main_table SET DISABLE_WAL=true”
>>> Maybe we can add this feature to 4.16+ ?
>>>
>>>
>>> 3) My main table has VERSIONS=>1. Anyway I decided to major-compacted
>>> before next run and still got Delete mutations
>>>
>>> From table metrics ~ 10% of mutations is Delete
>>>
>>>
>>> I checked my main table, it has loaded IndexRegionObserver:
>>>
>>> coprocessor$1 =>
>>> '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
>>> coprocessor$2 =>
>>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
>>> coprocessor$3 =>
>>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
>>> coprocessor$4 =>
>>> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
>>> coprocessor$5 =>
>>> '|org.apache.phoenix.hbase.index.IndexRegionObserver|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'
>>>
>>>
>>> By the way I split index table for more regions, increased
>>> hbase.hregion.memstore.flush.size, hbase.hstore.blockingStoreFiles and get
>
Re: Speedup initial index creation
If you have only one version for the rows, then you should not see much
difference between old and new. Can you provide some info on your
performance testing? For example, number of rows in your data table, number
of indexes, number of data table regions, number of MapReduce mappers,
IndexTool MapReduce job duration, number of data nodes in your cluster, any
IndexTool config params setting, etc.
On Thu, Apr 1, 2021 at 6:12 PM Alexander Batyrshin <0x62...@gmail.com>
wrote:
>
>
> 2 апр. 2021 г., в 03:55, Kadir Ozdemir написал(а):
>
>
> 1) I was thinking about the bulk load tool (
> https://phoenix.apache.org/bulk_dataload.html). However, in this case,
> you are not interested in bulk loading into the data table and its index
> but just the index table. Now, I see that it would not work for you. You
> are supposed to build a strongly consistent index once when you create the
> index. I am curious why you are so concerned about its performance.
>
>
> I need minimum maintenance time window on our cluster.
>
> 2) I thought you wanted to disable WAL only during index rebuild for the
> index table, not all the time. You should be able to still use the ALTER
> TABLE command with the new index design. Please note that in this case you
> would disable WAL for the main table too. Is that what you are looking
> for? If you are willing to disable WAL, then there is no point in using
> strongly consistent indexes because you would lose recently written data if
> region servers crash. By the way, you can use IndexUpgradeTool to downgrade
> your tables to the old design (to replace IndexRegionObserver with
> Indexer), see https://phoenix.apache.org/secondary_indexing.html
>
>
> I know about possibility of data loosing. But it’s not a problem if main
> table do not receive mutation during index creation (maintenance window).
>
> Old indexes goes inconsistent too often, so it not the way.
>
>
> 3) Delete markers will be added each time you run the index create command
> whenever the data table rows have multiple versions and the versions of a
> row have different values for indexed columns.
>
>
> My table has 1 version per row after major-compaction. Also main table has
> no mutation during index creation
>
> On Thu, Apr 1, 2021 at 3:28 PM Alexander Batyrshin <0x62...@gmail.com>
> wrote:
>
>> 1) How to create index old way via intermediate HFiles?
>>
>> I see “direct” option for IndexTool but description says its disabled:
>>
>> private static final Option DIRECT_API_OPTION = new Option("direct",
>> "direct", false,
>> "This parameter is deprecated. Direct mode will be used whether it is
>> set or not. Keeping it for backwards compatibility.”);
>>
>>
>> 2) On phoenix-4.14.2 (old indexes) WAL disabling for index table was
>> possible by “ALTER TABLE main_table SET DISABLE_WAL=true”
>> Maybe we can add this feature to 4.16+ ?
>>
>>
>> 3) My main table has VERSIONS=>1. Anyway I decided to major-compacted
>> before next run and still got Delete mutations
>>
>> From table metrics ~ 10% of mutations is Delete
>>
>>
>> I checked my main table, it has loaded IndexRegionObserver:
>>
>> coprocessor$1 =>
>> '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
>> coprocessor$2 =>
>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
>> coprocessor$3 =>
>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
>> coprocessor$4 =>
>> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
>> coprocessor$5 =>
>> '|org.apache.phoenix.hbase.index.IndexRegionObserver|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'
>>
>>
>> By the way I split index table for more regions, increased
>> hbase.hregion.memstore.flush.size, hbase.hstore.blockingStoreFiles and get
>> ~ 30% speedup.
>> This is still very slow compared to old index creation.
>>
>> On 31 Mar 2021, at 02:55, Kadir Ozdemir
>> wrote:
>>
>> I assume that your base table has several versions for a given row. If
>> so, creating a consistent index on this base table can be slower
>> than creating an old design index. This is because the new design creates
>> an index row for every data table row version. It simply replays the
>> mutations on a row without updating the data table but makes necessary
>> mutations on the index table. It does this to make sure that if you use SCN
>> connections to do point-in-time queries,
Re: Speedup initial index creation
1) I was thinking about the bulk load tool (
https://phoenix.apache.org/bulk_dataload.html). However, in this case, you
are not interested in bulk loading into the data table and its index but
just the index table. Now, I see that it would not work for you. You are
supposed to build a strongly consistent index once when you create the
index. I am curious why you are so concerned about its performance.
2) I thought you wanted to disable WAL only during index rebuild for the
index table, not all the time. You should be able to still use the ALTER
TABLE command with the new index design. Please note that in this case you
would disable WAL for the main table too. Is that what you are looking
for? If you are willing to disable WAL, then there is no point in using
strongly consistent indexes because you would lose recently written data if
region servers crash. By the way, you can use IndexUpgradeTool to downgrade
your tables to the old design (to replace IndexRegionObserver with
Indexer), see https://phoenix.apache.org/secondary_indexing.html.
3) Delete markers will be added each time you run the index create command
whenever the data table rows have multiple versions and the versions of a
row have different values for indexed columns.
On Thu, Apr 1, 2021 at 3:28 PM Alexander Batyrshin <0x62...@gmail.com>
wrote:
> 1) How to create index old way via intermediate HFiles?
>
> I see “direct” option for IndexTool but description says its disabled:
>
> private static final Option DIRECT_API_OPTION = new Option("direct",
> "direct", false,
> "This parameter is deprecated. Direct mode will be used whether it is
> set or not. Keeping it for backwards compatibility.”);
>
>
> 2) On phoenix-4.14.2 (old indexes) WAL disabling for index table was
> possible by “ALTER TABLE main_table SET DISABLE_WAL=true”
> Maybe we can add this feature to 4.16+ ?
>
>
> 3) My main table has VERSIONS=>1. Anyway I decided to major-compacted
> before next run and still got Delete mutations
>
> From table metrics ~ 10% of mutations is Delete
>
> I checked my main table, it has loaded IndexRegionObserver:
>
> coprocessor$1 =>
> '|org.apache.phoenix.coprocessor.ScanRegionObserver|805306366|',
> coprocessor$2 =>
> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|805306366|',
> coprocessor$3 =>
> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|805306366|',
> coprocessor$4 =>
> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|805306366|',
> coprocessor$5 =>
> '|org.apache.phoenix.hbase.index.IndexRegionObserver|805306366|org.apache.hadoop.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'
>
>
> By the way I split index table for more regions, increased
> hbase.hregion.memstore.flush.size, hbase.hstore.blockingStoreFiles and get
> ~ 30% speedup.
> This is still very slow compared to old index creation.
>
> On 31 Mar 2021, at 02:55, Kadir Ozdemir
> wrote:
>
> I assume that your base table has several versions for a given row. If so,
> creating a consistent index on this base table can be slower than creating
> an old design index. This is because the new design creates an index row
> for every data table row version. It simply replays the mutations on a row
> without updating the data table but makes necessary mutations on the index
> table. It does this to make sure that if you use SCN connections to do
> point-in-time queries, the index will return correct results. During these
> replays, index rows will be deleted if index columns are modified. This is
> the reason I think you see delete mutations on the index table.
>
> 1) Yes
> 2) No
> 3) No
>
> It will be a good improvement to have an option to support (3) by just
> creating indexes using the last data row versions. Please feel free to
> create an improvement Jira for this.
>
> Did you create your base table using 4.16? If not, have you upgraded it to
> the new index design using IndexUpgradeTool? I am asking this to make sure
> that your index actually uses the new index design. You can verify this
> using the HBase shell by describing the data table and checking if the
> IndexRegionObserver coproc is loaded on your base table.
>
>
> On Tue, Mar 30, 2021 at 3:10 PM Alexander Batyrshin <0x62...@gmail.com>
> wrote:
>
>> I tried on phoenix-4.16.0
>>
>> > On 31 Mar 2021, at 00:54, Alexander Batyrshin <0x62...@gmail.com>
>> wrote:
>> >
>> > Hello,
>> > I tried to create new consistent index on mutable table and found out
>> that IndexTool MapReduce works 3-5 times slower compared to old indexes on
>> 4.14.2
>> > So I have some question;
>> >
>> > 1) Is it possible to create index old way via intermediate HFiles and
>> bulk-loading?
>> > 2) Is it possible to disable WAL on HBase index table for creation time?
>> > 3) My main table has no updates, but I observe Delete mutations on
>> index table. Is it possible to disable this for initial index creation time?
>> >
>>
>>
>
Re: Speedup initial index creation
I assume that your base table has several versions for a given row. If so, creating a consistent index on this base table can be slower than creating an old design index. This is because the new design creates an index row for every data table row version. It simply replays the mutations on a row without updating the data table but makes necessary mutations on the index table. It does this to make sure that if you use SCN connections to do point-in-time queries, the index will return correct results. During these replays, index rows will be deleted if index columns are modified. This is the reason I think you see delete mutations on the index table. 1) Yes 2) No 3) No It will be a good improvement to have an option to support (3) by just creating indexes using the last data row versions. Please feel free to create an improvement Jira for this. Did you create your base table using 4.16? If not, have you upgraded it to the new index design using IndexUpgradeTool? I am asking this to make sure that your index actually uses the new index design. You can verify this using the HBase shell by describing the data table and checking if the IndexRegionObserver coproc is loaded on your base table. On Tue, Mar 30, 2021 at 3:10 PM Alexander Batyrshin <0x62...@gmail.com> wrote: > I tried on phoenix-4.16.0 > > > On 31 Mar 2021, at 00:54, Alexander Batyrshin <0x62...@gmail.com> wrote: > > > > Hello, > > I tried to create new consistent index on mutable table and found out > that IndexTool MapReduce works 3-5 times slower compared to old indexes on > 4.14.2 > > So I have some question; > > > > 1) Is it possible to create index old way via intermediate HFiles and > bulk-loading? > > 2) Is it possible to disable WAL on HBase index table for creation time? > > 3) My main table has no updates, but I observe Delete mutations on index > table. Is it possible to disable this for initial index creation time? > > > >
Invitation: Apache Phoenix Tech Talks @ Thu Apr 1, 2021 9am - 10:30am (PDT) (user@phoenix.apache.org)
BEGIN:VCALENDAR PRODID:-//Google Inc//Google Calendar 70.9054//EN VERSION:2.0 CALSCALE:GREGORIAN METHOD:REQUEST BEGIN:VEVENT DTSTART:20210401T16Z DTEND:20210401T173000Z DTSTAMP:20210329T051335Z ORGANIZER;CN=ka...@gsuite.cloud.apache.org:mailto:kadir@gsuite.cloud.apache .org UID:37oinhhlrk8f7es5a4hjuqm...@google.com ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=user@phoenix.apache.org;X-NUM-GUESTS=0:mailto:user@phoenix.apache.o rg ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=TRUE ;CN=ka...@gsuite.cloud.apache.org;X-NUM-GUESTS=0:mailto:kadir@gsuite.cloud. apache.org ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=NEEDS-ACTION;RSVP= TRUE;CN=d...@phoenix.apache.org;X-NUM-GUESTS=0:mailto:d...@phoenix.apache.org X-MICROSOFT-CDO-OWNERAPPTID:-126994816 CREATED:20210329T051334Z DESCRIPTION:PhoenixDB is a new project which is a Python library for intera cting with Phoenix with Phoenix Query Server (PQS). Hue is an open source S QL assistant for databases and data warehouses. In this talk\, we give an o verview of PQS\, and recently completed and planned improvements for it\, a nd introduce the PhoenixDB project. Then\, we present Hue\, touch on the in tegration of PhoenixDB into Hue. Finally\, we present a live demo of using PhoenixDB within Hue.Agenda :Introductions: Josh Elser< /li>Phoenix Query Server and PhoenixDB overview: Istvan TothHu e and PhoenixDB integration: Romain Rigaux or Ayush Kumal GoyalQ mp\;AMeeting Link : https://salesforce.zoom.us/j/84174813152? pwd=eXN0U2pzZm1iZTl2SHZoQWVXVWpvdz09This meeting will be recorded.< br>Please see https://phoenix.apache.org/tech_talks.html for the info o n upcoming meetings\, and the recordings of the previous meetings.\n\n-::~: ~::~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~: :~:~::-\nPlease do not edit this section of the description.\n\nView your e vent at https://calendar.google.com/calendar/event?action=VIEW=MzdvaW5o aGxyazhmN2VzNWE0aGp1cW01cTkgdXNlckBwaG9lbml4LmFwYWNoZS5vcmc=Mjkja2FkaXJ AZ3N1aXRlLmNsb3VkLmFwYWNoZS5vcmc4YjYzMzQ1YmFlNjEzNzM1ZTQ0Zjk5NzU5YjY4ZDZmNT NhMzVjNTE3=America%2FLos_Angeles=en=1.\n-::~:~::~:~:~:~:~:~:~:~:~ :~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~:~::~:~::- LAST-MODIFIED:20210329T051334Z LOCATION: SEQUENCE:0 STATUS:CONFIRMED SUMMARY:Apache Phoenix Tech Talks TRANSP:OPAQUE END:VEVENT END:VCALENDAR invite.ics Description: application/ics
Re: Topic suggestions for April Tech Talk
Lars, Thank you for suggestions and it will be great to see you presenting in an upcoming meeting, hopefully very soon. Josh, Thank you for volunteering for the April meeting. You are the host for it. It will be great if you can write a brief abstract/description about the meeting topic that I can send out with the meeting invitation. I do not know exactly how you want to structure the meeting but I can think of that you give a brief overview of Hue, PQS, and the Python library, explain what specific problems they address, and what challenges they have and/or what we would like to do for them in near future to initiate discussions. Thank you again! Kadir On Fri, Mar 19, 2021 at 10:39 AM la...@apache.org wrote: > An interesting topic would BigData ecosystem integration, which is clearly > one of the strengths of Phoenix. > > I.e. with Trino (formerly Presto), Spark, and classical MapReduce. Perhaps > this could cover also HBase snapshot support. > > I'm happy to cover this. But I can't to that for April 1st - juggling too > many things right now. > > -- Lars > > On Tuesday, March 9, 2021, 10:54:16 AM PST, Kadir Ozdemir < > ka...@gsuite.cloud.apache.org> wrote: > > > > > > Hi All, > > This is a friendly reminder that our next tech talk meeting will be held at > 9AM PST on April 01. If you like to present a topic for the next meeting, > please let us know by March 18. For more information about our tech talks, > and the slides and recording for the previous presentation, please visit > https://phoenix.apache.org/tech_talks.html. > > I look forward to your suggestions for the next meeting. > > Thanks, > Kadir >
Topic suggestions for April Tech Talk
Hi All, This is a friendly reminder that our next tech talk meeting will be held at 9AM PST on April 01. If you like to present a topic for the next meeting, please let us know by March 18. For more information about our tech talks, and the slides and recording for the previous presentation, please visit https://phoenix.apache.org/tech_talks.html. I look forward to your suggestions for the next meeting. Thanks, Kadir
Tech Talks Meeting Tomorrow
The invitation for the tech talks meeting did not get posted here. So, I wanted to send out this note to remind you that we are starting our tech talks series and the first meeting is at 9AM PST tomorrow. You can join the meeting using the link at https://phoenix.apache.org/tech_talks.html. Hope you can make it. Thanks, Kadir
Re: [Discuss] Phoenix Tech Talks
Hi All, The page for Phoenix Tech Talks has been added to the Apache Phoenix web site (see https://phoenix.apache.org/tech_talks.html) where all the information for upcoming and previous tech talks will be provided. The abstract for the talk at 9 AM PST on March 4 has been posted there. We are still working on finalizing the technical details of the video conferencing meeting, and I will update the meeting link on the page later. Thanks, Kadir On Wed, Feb 10, 2021 at 9:58 AM Kadir Ozdemir wrote: > I wanted to give an update on this. We are planning to have our first > Phoenix tech talk meeting on March 4th at 9am, and working on the details. > I will add a page on the Phoenix website for the tech talks and provide the > details there by the end of next week. Stay tuned! > > On Mon, Feb 8, 2021 at 2:00 PM Kadir Ozdemir > wrote: > >> Ankit, I was planning to include the user mailing list and it is good >> that you already did it. I like the idea of expanding talks to include >> use-cases. Thank you for the topic suggestions. >> >> >> On Mon, Feb 8, 2021 at 12:42 PM Ankit Singhal >> wrote: >> >>> This is excellent, thanks Kadir for initiating it, I am keen to listen to >>> your presentation on "strongly consistent global indexes". >>> Logistics looks great, however, if required can be adjusted as per the >>> feedback after the first meetup. >>> >>> Some of the current topics comes to my mind (in case someone wants to >>> share >>> their experiences with them) >>> * Improved python driver support ( Could be completely technical about >>> implementation or on the use-case >>> that how easy it is for any python developer to get started with >>> Phoenix/HBase) >>> * Key features in 4.16 and 5.1 release and upgrade path >>> * Adopted projects Apache Omid and Apache Tephra key differences. >>> >>> >>> +user to expand the scope of talks if someone >>> wants to present their use-cases. >>> >>> On Mon, Feb 8, 2021 at 9:06 AM Viraj Jasani wrote: >>> >>> > +1 to the idea of monthly meet-ups, and I will also do my best to join >>> in. >>> > Logistics part (when, durations etc) looks good to me. >>> > >>> > >>> > On Fri, 5 Feb 2021 at 1:36 AM, Josh Elser wrote: >>> > >>> > > Love it! I'll do my best to join in and listen (and participate later >>> > > on, too ;)) >>> > > >>> > > I joined one from Calcite a week or two ago. They did a signup via >>> > > Meetup.com and hosted it through Zoom. It felt very professional. >>> > > >>> > > On 2/4/21 12:10 PM, Kadir Ozdemir wrote: >>> > > > We are very excited to propose an idea that brings the Phoenix >>> > community >>> > > > together to have technical discussions on a recurring basis. The >>> goal >>> > is >>> > > to >>> > > > have a forum where we share technical knowledge we have acquired by >>> > > working >>> > > > on various aspects of Phoenix and to continue to bring innovation >>> and >>> > > > improvements as a community into Phoenix. We’d love to get >>> feedback on >>> > > this >>> > > > idea and determine the logistics for these meetings. >>> > > > >>> > > > Here is what we were thinking: >>> > > > >>> > > > - Come together as a community by hosting *Phoenix tech talks* >>> > once a >>> > > > month >>> > > > - The topics for these meetings can be any technical subject >>> > related >>> > > to >>> > > > Phoenix, including the architecture, internals, features and >>> > > interfaces of >>> > > > Phoenix, its operational aspects in the first party data >>> centers >>> > and >>> > > cloud, >>> > > > the technologies that it leverages (e.g., HBase and >>> Zookeeper), and >>> > > > technologies it can possibly leverage, adapt or follow >>> > > > >>> > > > *Logistics*: >>> > > > >>> > > > - *When*: First Thursday of each month at 9AM PST >>> > > > - *Duration*: 90 minutes (to allow the audience to participate >>> and >>> > > ask >>> > > > questions) >>> > > > - We will conduct these meetings over a video conference and >>> make >>> > the >>> > > > recordings available (we are sorting out the specifics) >>> > > > - The meeting agenda and past recordings will be available on >>> the >>> > > Apache >>> > > > Phoenix site >>> > > > >>> > > > We need a coordinator for these meetings to set the agenda and >>> manage >>> > its >>> > > > logistics. I will volunteer to organize these meetings and curate >>> the >>> > > > topics for the tech talks, at least initially. To get the ball >>> > rolling, I >>> > > > will present the strongly consistent global indexes in the first >>> > meeting. >>> > > > What do you think about this proposal? >>> > > > >>> > > > Thanks, >>> > > > Kadir >>> > > > >>> > > >>> > >>> >>
Re: [Discuss] Phoenix Tech Talks
I wanted to give an update on this. We are planning to have our first Phoenix tech talk meeting on March 4th at 9am, and working on the details. I will add a page on the Phoenix website for the tech talks and provide the details there by the end of next week. Stay tuned! On Mon, Feb 8, 2021 at 2:00 PM Kadir Ozdemir wrote: > Ankit, I was planning to include the user mailing list and it is good that > you already did it. I like the idea of expanding talks to include > use-cases. Thank you for the topic suggestions. > > > On Mon, Feb 8, 2021 at 12:42 PM Ankit Singhal > wrote: > >> This is excellent, thanks Kadir for initiating it, I am keen to listen to >> your presentation on "strongly consistent global indexes". >> Logistics looks great, however, if required can be adjusted as per the >> feedback after the first meetup. >> >> Some of the current topics comes to my mind (in case someone wants to >> share >> their experiences with them) >> * Improved python driver support ( Could be completely technical about >> implementation or on the use-case >> that how easy it is for any python developer to get started with >> Phoenix/HBase) >> * Key features in 4.16 and 5.1 release and upgrade path >> * Adopted projects Apache Omid and Apache Tephra key differences. >> >> >> +user to expand the scope of talks if someone >> wants to present their use-cases. >> >> On Mon, Feb 8, 2021 at 9:06 AM Viraj Jasani wrote: >> >> > +1 to the idea of monthly meet-ups, and I will also do my best to join >> in. >> > Logistics part (when, durations etc) looks good to me. >> > >> > >> > On Fri, 5 Feb 2021 at 1:36 AM, Josh Elser wrote: >> > >> > > Love it! I'll do my best to join in and listen (and participate later >> > > on, too ;)) >> > > >> > > I joined one from Calcite a week or two ago. They did a signup via >> > > Meetup.com and hosted it through Zoom. It felt very professional. >> > > >> > > On 2/4/21 12:10 PM, Kadir Ozdemir wrote: >> > > > We are very excited to propose an idea that brings the Phoenix >> > community >> > > > together to have technical discussions on a recurring basis. The >> goal >> > is >> > > to >> > > > have a forum where we share technical knowledge we have acquired by >> > > working >> > > > on various aspects of Phoenix and to continue to bring innovation >> and >> > > > improvements as a community into Phoenix. We’d love to get feedback >> on >> > > this >> > > > idea and determine the logistics for these meetings. >> > > > >> > > > Here is what we were thinking: >> > > > >> > > > - Come together as a community by hosting *Phoenix tech talks* >> > once a >> > > > month >> > > > - The topics for these meetings can be any technical subject >> > related >> > > to >> > > > Phoenix, including the architecture, internals, features and >> > > interfaces of >> > > > Phoenix, its operational aspects in the first party data centers >> > and >> > > cloud, >> > > > the technologies that it leverages (e.g., HBase and Zookeeper), >> and >> > > > technologies it can possibly leverage, adapt or follow >> > > > >> > > > *Logistics*: >> > > > >> > > > - *When*: First Thursday of each month at 9AM PST >> > > > - *Duration*: 90 minutes (to allow the audience to participate >> and >> > > ask >> > > > questions) >> > > > - We will conduct these meetings over a video conference and >> make >> > the >> > > > recordings available (we are sorting out the specifics) >> > > > - The meeting agenda and past recordings will be available on >> the >> > > Apache >> > > > Phoenix site >> > > > >> > > > We need a coordinator for these meetings to set the agenda and >> manage >> > its >> > > > logistics. I will volunteer to organize these meetings and curate >> the >> > > > topics for the tech talks, at least initially. To get the ball >> > rolling, I >> > > > will present the strongly consistent global indexes in the first >> > meeting. >> > > > What do you think about this proposal? >> > > > >> > > > Thanks, >> > > > Kadir >> > > > >> > > >> > >> >
Re: [ANNOUNCE] New Phoenix committer Viraj Jasani
Congratulations, Viraj! On Tue, Feb 9, 2021 at 3:25 PM Ankit Singhal wrote: > Congratulations, Viraj! > > On Tue, Feb 9, 2021 at 2:18 PM Xinyi Yan wrote: > >> Congratulations and welcome, Viraj! >> >> On Tue, Feb 9, 2021 at 2:07 PM Geoffrey Jacoby >> wrote: >> >>> On behalf of the Apache Phoenix PMC, I'm pleased to announce that Viraj >>> Jasani has accepted the PMC's invitation to become a committer on Apache >>> Phoenix. >>> >>> We appreciate all of the great contributions Viraj has made to the >>> community thus far and we look forward to his continued involvement. >>> >>> Congratulations and welcome, Viraj! >>> >>
Re: [Discuss] Phoenix Tech Talks
Ankit, I was planning to include the user mailing list and it is good that you already did it. I like the idea of expanding talks to include use-cases. Thank you for the topic suggestions. On Mon, Feb 8, 2021 at 12:42 PM Ankit Singhal wrote: > This is excellent, thanks Kadir for initiating it, I am keen to listen to > your presentation on "strongly consistent global indexes". > Logistics looks great, however, if required can be adjusted as per the > feedback after the first meetup. > > Some of the current topics comes to my mind (in case someone wants to share > their experiences with them) > * Improved python driver support ( Could be completely technical about > implementation or on the use-case > that how easy it is for any python developer to get started with > Phoenix/HBase) > * Key features in 4.16 and 5.1 release and upgrade path > * Adopted projects Apache Omid and Apache Tephra key differences. > > > +user to expand the scope of talks if someone > wants to present their use-cases. > > On Mon, Feb 8, 2021 at 9:06 AM Viraj Jasani wrote: > > > +1 to the idea of monthly meet-ups, and I will also do my best to join > in. > > Logistics part (when, durations etc) looks good to me. > > > > > > On Fri, 5 Feb 2021 at 1:36 AM, Josh Elser wrote: > > > > > Love it! I'll do my best to join in and listen (and participate later > > > on, too ;)) > > > > > > I joined one from Calcite a week or two ago. They did a signup via > > > Meetup.com and hosted it through Zoom. It felt very professional. > > > > > > On 2/4/21 12:10 PM, Kadir Ozdemir wrote: > > > > We are very excited to propose an idea that brings the Phoenix > > community > > > > together to have technical discussions on a recurring basis. The goal > > is > > > to > > > > have a forum where we share technical knowledge we have acquired by > > > working > > > > on various aspects of Phoenix and to continue to bring innovation and > > > > improvements as a community into Phoenix. We’d love to get feedback > on > > > this > > > > idea and determine the logistics for these meetings. > > > > > > > > Here is what we were thinking: > > > > > > > > - Come together as a community by hosting *Phoenix tech talks* > > once a > > > > month > > > > - The topics for these meetings can be any technical subject > > related > > > to > > > > Phoenix, including the architecture, internals, features and > > > interfaces of > > > > Phoenix, its operational aspects in the first party data centers > > and > > > cloud, > > > > the technologies that it leverages (e.g., HBase and Zookeeper), > and > > > > technologies it can possibly leverage, adapt or follow > > > > > > > > *Logistics*: > > > > > > > > - *When*: First Thursday of each month at 9AM PST > > > > - *Duration*: 90 minutes (to allow the audience to participate > and > > > ask > > > > questions) > > > > - We will conduct these meetings over a video conference and make > > the > > > > recordings available (we are sorting out the specifics) > > > > - The meeting agenda and past recordings will be available on the > > > Apache > > > > Phoenix site > > > > > > > > We need a coordinator for these meetings to set the agenda and manage > > its > > > > logistics. I will volunteer to organize these meetings and curate the > > > > topics for the tech talks, at least initially. To get the ball > > rolling, I > > > > will present the strongly consistent global indexes in the first > > meeting. > > > > What do you think about this proposal? > > > > > > > > Thanks, > > > > Kadir > > > > > > > > > >
