https://issues.apache.org/jira/browse/HAWQ-1445
Jon Roberts On Tue, Apr 25, 2017 at 6:31 AM, Jon Roberts <[email protected]> wrote: > No I did not. My post was intended to start a discussion about the plugin > and I thought the original contributor would chime in. But I can make the > tickets now if you think that is the best way to move forward. > > Jon Roberts > Principal Engineer | [email protected] | 615-426-8661 <(615)%20426-8661> > > On Tue, Apr 25, 2017 at 2:58 AM, Michael Pearce <[email protected]> > wrote: > >> Hi Jon, >> >> Did you make JIRA issues for the two improvements you suggested to be >> made? >> >> Cheers >> Mike >> >> On 04/04/2017, 04:15, "Michael Pearce" <[email protected]> wrote: >> >> Hi Jon >> >> I think on issue 1 and 2 are valid improvements that can be made, add >> some Jira's for these. Looking at issue 1 it seems quite trivial for >> someone to contribute the solution for. >> >> On the question front: >> >> 1) from my understanding and using this plugin so far this is what >> the partion by interval allows you to control the number of partitions that >> would be generated, e.g. This is why the sample in document has 1 year so >> it only creates 2 fragments. You can also partition by an enum field here >> the number of partitions is the number of enums. >> >> 2) I haven't checked specifically, saying that I haven't run into an >> issue when trialling this myself. Obviously this is contributed by Devin he >> will be the definitive source here. Though I guess can always specially >> test these scenarios. >> >> Sent using OWA for iPhone >> ________________________________________ >> From: Jon Roberts <[email protected]> >> Sent: Monday, April 3, 2017 11:57:59 PM >> To: [email protected] >> Subject: Re: PXF JDBC plugin >> >> https://github.com/apache/incubator-hawq/tree/master/pxf/pxf-jdbc >> >> Issue 1: Security >> The example has the username and password in the connection string. >> >> LOCATION ('pxf://localhost:51200/demodb.myclass' >> '?PROFILE=JDBC' >> '&JDBC_DRIVER=com.mysql.jdbc.Driver' >> '&DB_URL=jdbc:mysql:// >> 192.168.200.6:3306/demodb&USER=root&PASS=root' >> ) >> >> Any chance we can get this changed to a connection profile that >> points to a >> file outside of the database? For Greenplum database and S3, the >> LOCATION >> syntax includes "config=/path/to/config_file". The config_file >> contains >> the S3 credentials. This seems like a good pattern to follow. >> >> As it is right now, anyone that can connect to the database will be >> able to >> see the username and password of the JDBC connection. >> >> Issue 2: Extra Properties >> >> Some JDBC drivers will need many additional properties beyond the URL >> and >> this requires setting it with a put to a Properties variable. An >> example >> of this is Oracle's defaultRowPrefetch property that needs to be >> updated >> from the default of 10 which is designed for OLTP to something larger >> like >> 2000 which is more ideal for data extracts. >> >> Additionally, you will need the ability to set the isolation level >> which is >> done with setTransactionIsolation on the Connection. I don't believe >> you >> can set this on the connection URL either. Many SQL Server and DB2 >> database still don't use snapshot isolation and use dirty reads >> instead. >> Without a dirty read here, your query will block modifications to the >> table >> with a "blocking lock". >> >> So for the configuration file will need both an extra properties >> variable >> that is delimited so you can multiple and a isolation level indicator. >> >> Questions. >> 1. How do you manage the maximum number of pxf instances? For >> example, if >> you partition like this: >> "PARTITION_BY=cdate:date&RANGE=2008-01-01:2010-01-01&INTERVAL=1:day", >> will >> you create 730 pxf instances and thus 730 concurrent queries to the >> source >> database? >> 2. Have you tested special characters like null, escape, carriage >> return, >> newline, and your delimiter? Maybe that is handled automatically by >> pxfwritable_import so this isn't an issue. >> >> >> >> Jon Roberts >> Principal Engineer | [email protected] | 615-426-8661 >> >> On Mon, Apr 3, 2017 at 3:18 PM, Michael Pearce <[email protected] >> > >> wrote: >> >> > I mean the readme file :) at that location. >> > ________________________________________ >> > From: Michael Pearce >> > Sent: Monday, April 3, 2017 9:15:48 PM >> > To: [email protected] >> > Subject: Re: PXF JDBC plugin >> > >> > On master there is the document that was added. >> > >> > https://github.com/apache/incubator-hawq/tree/master/pxf/pxf-jdbc >> > >> > Reading this I think answers some of the questions. >> > >> > >> > >> > >> > >> > Sent using OWA for iPhone >> > ________________________________________ >> > From: Vineet Goel <[email protected]> >> > Sent: Monday, April 3, 2017 6:33:07 PM >> > To: [email protected] >> > Subject: Re: PXF JDBC plugin >> > >> > Devin and others, >> > >> > It would be great if you can expand a bit more on the >> implementation or >> > point to a document? >> > >> > Thanks! >> > >> > >> > >> > On Mon, Apr 3, 2017 at 7:53 AM Jon Roberts <[email protected]> >> wrote: >> > >> > > JDBC PXF is pretty exciting. Are there details on how this works? >> > > >> > > Does PXF randomly pick a segment to connect to the JDBC source or >> do you >> > > specify a particular node to execute the query? I'm assuming it >> isn't a >> > > parallel query so you don't flood the JDBC source with x number of >> > "select >> > > *" queries but I could be wrong. >> > > >> > > How do you register the JDBC connection string with PXF? >> > > >> > > How do you manage credentials? >> > > >> > > How do you register the JDBC jar files? >> > > >> > > >> > > Jon Roberts >> > > >> > > On Sat, Apr 1, 2017 at 5:05 PM, Ed Espino <[email protected]> >> wrote: >> > > >> > > > Thank you Michael, this is a well-deserved report highlight. I >> have >> > added >> > > > the following to the report: >> > > > >> > > > 3. Community contribution highlight(s): >> > > > >> > > > * Leveraging the extensible PXF design, a JDBC PXF plugin >> > > > was contributed by Devin Jia (github id: jiadexin). This >> > > > contribution came from the community and not from the >> > > > company which originally donated HAWQ to the ASF. >> > > > >> > > > Warm regrds, >> > > > -=e >> > > > >> > > > On Sat, Apr 1, 2017 at 1:14 PM, Michael André Pearce < >> > > > [email protected]> wrote: >> > > > >> > > > > >> > > > > I would add note about the new JDBC PXF plugin being >> contributed by >> > > Devin >> > > > > Jia. (Kudos to him) >> > > > > >> > > > > It's a good show of the community contributing and growing >> outside of >> > > > just >> > > > > Pivotal staff (I think for incubator review this is >> important) and >> > also >> > > > the >> > > > > benefit of the extensible PXF design the project has done. >> > > >> > The information contained in this email is strictly confidential >> and for >> > the use of the addressee only, unless otherwise indicated. If you >> are not >> > the intended recipient, please do not read, copy, use or disclose >> to others >> > this message or any attachment. Please also notify the sender by >> replying >> > to this email or by telephone (+44(020 7896 0011) and then delete >> the email >> > and any copies of it. Opinions, conclusion (etc) that do not relate >> to the >> > official business of this company shall be understood as neither >> given nor >> > endorsed by it. IG is a trading name of IG Markets Limited (a >> company >> > registered in England and Wales, company number 04008957) and IG >> Index >> > Limited (a company registered in England and Wales, company number >> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate >> Hill, >> > London EC4R 2YA. Both IG Markets Limited (register number 195355) >> and IG >> > Index Limited (register number 114059) are authorised and regulated >> by the >> > Financial Conduct Authority. >> > >> >> >> The information contained in this email is strictly confidential and for >> the use of the addressee only, unless otherwise indicated. If you are not >> the intended recipient, please do not read, copy, use or disclose to others >> this message or any attachment. Please also notify the sender by replying >> to this email or by telephone (+44(020 7896 0011) and then delete the email >> and any copies of it. Opinions, conclusion (etc) that do not relate to the >> official business of this company shall be understood as neither given nor >> endorsed by it. IG is a trading name of IG Markets Limited (a company >> registered in England and Wales, company number 04008957) and IG Index >> Limited (a company registered in England and Wales, company number >> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill, >> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG >> Index Limited (register number 114059) are authorised and regulated by the >> Financial Conduct Authority. >> > >
