[jira] [Assigned] (PHOENIX-5939) Publish PhoenixDB to PyPI
[ https://issues.apache.org/jira/browse/PHOENIX-5939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth reassigned PHOENIX-5939: Assignee: Istvan Toth > Publish PhoenixDB to PyPI > - > > Key: PHOENIX-5939 > URL: https://issues.apache.org/jira/browse/PHOENIX-5939 > Project: Phoenix > Issue Type: Task > Components: queryserver >Affects Versions: queryserver-1.0.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > The original PhoenixDB driver was published to PyPI. > The improved version in phoenix-queryserver is only available from there. > We should start publishing the driver again. > Some questions to answer: > * Can we take over the old PyPI project ? > * Do we want to ? > * What should be the project/artifact name (if not the old one) > * Version numbering ? > * Do we want to publish development versions ? > * What is the process / who should do the publishing ? > * Any blockers before we start publishing ? -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [Discuss] Twill removal and Guava update plan
Thanks, Istvan for putting up a detailed plan +1 for removing twill(as I can see it is recently moved to Attic) and adding whatever convenience it's bringing in Omid/tephra project itself. Instead of creating a shaded artifact of Guava and then refer it in Omid and Tephra, can't we just short-circuit your step 2 with 3 and use relocation of the maven-shaded plugin to produce the shaded artifacts for Tephra/Omid as we are anyways shading everything including public API? On Tue, Jun 2, 2020 at 8:25 AM Josh Elser wrote: > Sounds like a well-thought-out plan to me. > > If we're going through and changing Guava, it may also be worthwhile to > try to eliminate the use of Guava in our "public API". While the shaded > guava eliminates classpath compatibility issues, Guava could (at any > point) drop a class that we're using in our API and still break us. That > could be a "later" thing. > > The only thing I think differently is that 4.x could (at some point) > pick up the shaded guava artifact you describe and make the change. > However, that's just for the future -- the call can be made if/when > someone wants to do that :) > > On 6/2/20 10:01 AM, Istvan Toth wrote: > > Hi! > > > > There are two related dependency issues that I believe should be solved > in > > Phoenix to keep it healthy and supportable. > > > > The Twill project has been officially terminated. Both Tephra and Omid > > depend on it, and so transitively Phoenix does as well. > > > > Hadoop 3.3 has updated its Guava to 29, while Phoenix (master) is still > on > > 13. > > None of Twill, Omid, Tephra, or Phoenix will run or compile against > recent > > Guava versions, which are pulled in by Hadoop 3.3. > > > > If we want to work with Hadoop 3.3, we either have to update all > > dependencies to a recent Guava version, or we have to build our artifacts > > with shaded Guava. > > Since Guava 13 has known vulnerabilities, including in the classpath > causes > > a barrier to adaptation. Some potential Phoenix users consider including > > dependencies with > > known vulnerabilities a show-stopper, they do not care if the > vulnerability > > affects Phoenix or not. > > > > I propose that we take following steps to ensure compatibility with > > upcoming Hadoop versions: > > > > *1. Remove the Twill dependency from Omid and Tephra* > > It is generally not healthy to depend on abandoned projects, but the fact > > Twill also depends (heavily) on Guava 13, makes removal the best > solution. > > As far as I can see, Omid and Tephra mostly use the ZK client from Twill, > > as well as the (transitively included) Guava service model. > > Refactoring to use the native ZK client, and to use the Guava service > > classes directly shouldn't be too difficult. > > > > *2. Create a shaded guava artifact for Omid and Tephra* > > Since Omid and Tephra needs to work with Hadoop2 and Hadoop3 (including > the > > upcoming Hadoop 3.3), which already pull in Guava, we need to use > different > > Guava internally. > > (similar to the HBase-thirdparty solution, but we need a separate one). > > This artifact could live under the Phoenix groupId, but we'll need to be > > careful with the circular dependencies. > > > > *3. Update the Omid and Tephra to use the shaded Guava artifact* > > Apart from handling the mostly trivial, "let's break API compatibility > for > > the heck of it" Guava changes, the Guava Service API that both Omid and > > Tephra build on has changed significantly. > > This will mean changes in the public (Phoenix facing) APIs. All Guava > > references will have to be replaced with the shaded guava classes from > step > > 2. > > > > *3. Define self-contained public APIs for Omid and Tephra* > > To break the public API's dependency on Guava, redefine the public APIs > in > > such a way that they do not have Guava classes as ancestors. > > This doesn't mean that we decouple the internal implementation from > Guava, > > simply defining a set of java Interfaces that matches the existing > (updated > > to recent Guava Service API) > > interface's signature, but is self-contained under the Tephra/Omid > > namespace should do the trick. > > > > *4. Update Phoenix to use new Omid/Tephra API* > > i.e. use the new Interface that we defined in step 3. > > > > *5. Update Phoenix to work with Guava 13-29.* > > We need to somehow get Phoenix work with both old and new Guava. > > Probably the least disruptive way to do this is reduce the Guava use to > the > > common subset of 13.0 and 29.0, and replace/reimplement the parts that > > cannot be resolved. > > Alternatively, we could rebase to the same shaded guava-thirdparty > library > > that we use for Omid and Tephra. > > > > For *4.x*, since we cannot get rid of Guava 13 ever, *Step 5 *is not > > necessary > > > > I am very interested in your opinion on the above plan. > > Does anyone have any objections ? > > Does anyone have a better solution ? > > Is there some hidden pitfall that I hadn't considered (there certainly >
[jira] [Updated] (PHOENIX-5932) View Index rebuild results in surplus rows from other view indexes
[ https://issues.apache.org/jira/browse/PHOENIX-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated PHOENIX-5932: Fix Version/s: 5.1.0 > View Index rebuild results in surplus rows from other view indexes > -- > > Key: PHOENIX-5932 > URL: https://issues.apache.org/jira/browse/PHOENIX-5932 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.16.0 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-5932-4.x.001.patch, PHOENIX-5932-4.x.001.patch, > PHOENIX-5932-4.x.patch, PHOENIX-5932-master.patch, > PHOENIX-5932-wip-4.x.001.patch, PHOENIX-5932-wip-4.x.patch, > PHOENIX-5932-wip-master.001.patch, PHOENIX-5932-wip-master.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > > Steps to reproduce > # Create a table with PK as COL1, COL2 > # Create 2 views with different view constants on COL2. Also add other > columns > # Upsert rows into the views > # Create indexes on the views. Rebuild will result in each view index having > rows from other view also > This is because we set the filter on scan to null during rebuild > [here|[https://github.com/apache/phoenix/blob/4.x/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java#L1084]] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (PHOENIX-5946) Implement SchemaExtractionTool utility to get effective DDL from cluster
[ https://issues.apache.org/jira/browse/PHOENIX-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swaroopa Kadam reassigned PHOENIX-5946: --- Assignee: (was: Swaroopa Kadam) > Implement SchemaExtractionTool utility to get effective DDL from cluster > > > Key: PHOENIX-5946 > URL: https://issues.apache.org/jira/browse/PHOENIX-5946 > Project: Phoenix > Issue Type: Improvement >Reporter: Swaroopa Kadam >Priority: Major > > The utility will take table/view/index and schema as a parameter to generate > effective DDL for those entities in a cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (PHOENIX-5946) Implement SchemaExtractionTool utility to get effective DDL from cluster
[ https://issues.apache.org/jira/browse/PHOENIX-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swaroopa Kadam reassigned PHOENIX-5946: --- Assignee: Swaroopa Kadam > Implement SchemaExtractionTool utility to get effective DDL from cluster > > > Key: PHOENIX-5946 > URL: https://issues.apache.org/jira/browse/PHOENIX-5946 > Project: Phoenix > Issue Type: Improvement >Reporter: Swaroopa Kadam >Assignee: Swaroopa Kadam >Priority: Major > > The utility will take table/view/index and schema as a parameter to generate > effective DDL for those entities in a cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5949) Add support for "all" keyword in the utility
Swaroopa Kadam created PHOENIX-5949: --- Summary: Add support for "all" keyword in the utility Key: PHOENIX-5949 URL: https://issues.apache.org/jira/browse/PHOENIX-5949 Project: Phoenix Issue Type: Sub-task Reporter: Swaroopa Kadam when passed an -all keyword to utility, it will query the sys cat and emit the effective DDL for all views, indexes, and tables on the cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5948) Add initial support to read table and schema from CommandLine
Swaroopa Kadam created PHOENIX-5948: --- Summary: Add initial support to read table and schema from CommandLine Key: PHOENIX-5948 URL: https://issues.apache.org/jira/browse/PHOENIX-5948 Project: Phoenix Issue Type: Sub-task Reporter: Swaroopa Kadam -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5947) Add Unit tests and Integration tests for utility
Swaroopa Kadam created PHOENIX-5947: --- Summary: Add Unit tests and Integration tests for utility Key: PHOENIX-5947 URL: https://issues.apache.org/jira/browse/PHOENIX-5947 Project: Phoenix Issue Type: Sub-task Reporter: Swaroopa Kadam -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-5946) Implement SchemaExtractionTool utility to get effective DDL from cluster
Swaroopa Kadam created PHOENIX-5946: --- Summary: Implement SchemaExtractionTool utility to get effective DDL from cluster Key: PHOENIX-5946 URL: https://issues.apache.org/jira/browse/PHOENIX-5946 Project: Phoenix Issue Type: Improvement Reporter: Swaroopa Kadam The utility will take table/view/index and schema as a parameter to generate effective DDL for those entities in a cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-5932) View Index rebuild results in surplus rows from other view indexes
[ https://issues.apache.org/jira/browse/PHOENIX-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Singh Chouhan updated PHOENIX-5932: Attachment: PHOENIX-5932-4.x.001.patch > View Index rebuild results in surplus rows from other view indexes > -- > > Key: PHOENIX-5932 > URL: https://issues.apache.org/jira/browse/PHOENIX-5932 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.16.0 >Reporter: Abhishek Singh Chouhan >Assignee: Abhishek Singh Chouhan >Priority: Major > Fix For: 4.16.0 > > Attachments: PHOENIX-5932-4.x.001.patch, PHOENIX-5932-4.x.001.patch, > PHOENIX-5932-4.x.patch, PHOENIX-5932-master.patch, > PHOENIX-5932-wip-4.x.001.patch, PHOENIX-5932-wip-4.x.patch, > PHOENIX-5932-wip-master.001.patch, PHOENIX-5932-wip-master.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Steps to reproduce > # Create a table with PK as COL1, COL2 > # Create 2 views with different view constants on COL2. Also add other > columns > # Upsert rows into the views > # Create indexes on the views. Rebuild will result in each view index having > rows from other view also > This is because we set the filter on scan to null during rebuild > [here|[https://github.com/apache/phoenix/blob/4.x/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java#L1084]] -- This message was sent by Atlassian Jira (v8.3.4#803005)
