Hi Niranda, Are we going to solve those limitations before the GA? Specially limitation no.2. Over time we can have stat table with thousands of records, so are we going to remove all the records and reinsert every time that spark script runs?
Regards, Gihan On Tue, Aug 11, 2015 at 7:13 AM, Niranda Perera <[email protected]> wrote: > Hi all, > > we have implemented a custom Spark JDBC connector to be used in the Carbon > environment. > > this enables the following > > 1. Now, temporary tables can be created in the Spark environment by > specifying an analytics datasource (configured by the > analytics-datasources.xml) and a table > 2. Spark uses "SELECT 1 FROM $table LIMIT 1" query to check the > existence of a table and the LIMIT query is not provided by all dbs. With > the new connector, this query can be provided with as a config. (this > config is still WIP) > 3. Adding new spark dialects related for various dbs (WIP) > > the idea is to test this for the following dbs > > - mysql > - h2 > - mssql > - oracle > - postgres > - db2 > > I have loosely tested the connector with MySQL, and I would like the APIM > team to use it with the API usage stats use-case, and provide us some > feedback. > > this connector can be accessed as follows. (docs are still not updated. I > will do that ASAP) > > create temporary table <temp_table> using CarbonJDBC options (dataSource > "<datasource name>", tableName "<table name>"); > > select * from <temp_table> > > insert into/overwrite table <temp_table> <some select statement> > > known limitations > > 1. when creating a temp table, it should already be created in the > underlying datasource > 2. "insert overwrite table" deletes the existing table and creates it > again > > > would be very grateful if you could use this connector in your current > JDBC use cases and provide us with feedback. > > best > -- > *Niranda Perera* > Software Engineer, WSO2 Inc. > Mobile: +94-71-554-8430 > Twitter: @n1r44 <https://twitter.com/N1R44> > https://pythagoreanscript.wordpress.com/ > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- W.G. Gihan Anuruddha Senior Software Engineer | WSO2, Inc. M: +94772272595
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
