On Tue, Sep 20, 2016 at 1:18 PM, Benjamin Kim <bbuil...@gmail.com> wrote:
> Now that Kudu 1.0.0 is officially out and ready for production use, where > do we find the spark connector jar for this release? > > It's available in the official ASF maven repository: https://repository.apache.org/#nexus-search;quick~kudu-spark <dependency> <groupId>org.apache.kudu</groupId> <artifactId>kudu-spark_2.10</artifactId> <version>1.0.0</version> </dependency> -Todd > On Jun 17, 2016, at 11:08 AM, Dan Burkert <d...@cloudera.com> wrote: > > Hi Ben, > > To your first question about `CREATE TABLE` syntax with Kudu/Spark SQL, I > do not think we support that at this point. I haven't looked deeply into > it, but we may hit issues specifying Kudu-specific options (partitioning, > column encoding, etc.). Probably issues that can be worked through > eventually, though. If you are interested in contributing to Kudu, this is > an area that could obviously use improvement! Most or all of our Spark > features have been completely community driven to date. > > >> I am assuming that more Spark support along with semantic changes below >> will be incorporated into Kudu 0.9.1. >> > > As a rule we do not release new features in patch releases, but the good > news is that we are releasing regularly, and our next scheduled release is > for the August timeframe (see JD's roadmap > <https://lists.apache.org/thread.html/1a3b949e715a74d7f26bd9c102247441a06d16d077324ba39a662e2a@1455234076@%3Cdev.kudu.apache.org%3E> > email > about what we are aiming to include). Also, Cloudera does publish snapshot > versions of the Spark connector here > <https://repository.cloudera.com/cloudera/cloudera-repos/org/kududb/>, so > the jars are available if you don't mind using snapshots. > > >> Anyone know of a better way to make unique primary keys other than using >> UUID to make every row unique if there is no unique column (or combination >> thereof) to use. >> > > Not that I know of. In general it's pretty rare to have a dataset without > a natural primary key (even if it's just all of the columns), but in those > cases UUID is a good solution. > > >> This is what I am using. I know auto incrementing is coming down the line >> (don’t know when), but is there a way to simulate this in Kudu using Spark >> out of curiosity? >> > > To my knowledge there is no plan to have auto increment in Kudu. > Distributed, consistent, auto incrementing counters is a difficult problem, > and I don't think there are any known solutions that would be fast enough > for Kudu (happy to be proven wrong, though!). > > - Dan > > >> >> Thanks, >> Ben >> >> On Jun 14, 2016, at 6:08 PM, Dan Burkert <d...@cloudera.com> wrote: >> >> I'm not sure exactly what the semantics will be, but at least one of them >> will be upsert. These modes come from spark, and they were really designed >> for file-backed storage and not table storage. We may want to do append = >> upsert, and overwrite = truncate + insert. I think that may match the >> normal spark semantics more closely. >> >> - Dan >> >> On Tue, Jun 14, 2016 at 6:00 PM, Benjamin Kim <bbuil...@gmail.com> wrote: >> >>> Dan, >>> >>> Thanks for the information. That would mean both “append” and >>> “overwrite” modes would be combined or not needed in the future. >>> >>> Cheers, >>> Ben >>> >>> On Jun 14, 2016, at 5:57 PM, Dan Burkert <d...@cloudera.com> wrote: >>> >>> Right now append uses an update Kudu operation, which requires the row >>> already be present in the table. Overwrite maps to insert. Kudu very >>> recently got upsert support baked in, but it hasn't yet been integrated >>> into the Spark connector. So pretty soon these sharp edges will get a lot >>> better, since upsert is the way to go for most spark workloads. >>> >>> - Dan >>> >>> On Tue, Jun 14, 2016 at 5:41 PM, Benjamin Kim <bbuil...@gmail.com> >>> wrote: >>> >>>> I tried to use the “append” mode, and it worked. Over 3.8 million rows >>>> in 64s. I would assume that now I can use the “overwrite” mode on existing >>>> data. Now, I have to find answers to these questions. What would happen if >>>> I “append” to the data in the Kudu table if the data already exists? What >>>> would happen if I “overwrite” existing data when the DataFrame has data in >>>> it that does not exist in the Kudu table? I need to evaluate the best way >>>> to simulate the UPSERT behavior in HBase because this is what our use case >>>> is. >>>> >>>> Thanks, >>>> Ben >>>> >>>> >>>> >>>> On Jun 14, 2016, at 5:05 PM, Benjamin Kim <bbuil...@gmail.com> wrote: >>>> >>>> Hi, >>>> >>>> Now, I’m getting this error when trying to write to the table. >>>> >>>> import scala.collection.JavaConverters._ >>>> val key_seq = Seq(“my_id") >>>> val key_list = List(“my_id”).asJava >>>> kuduContext.createTable(tableName, df.schema, key_seq, new >>>> CreateTableOptions().setNumReplicas(1).addHashPartitions(key_list, >>>> 100)) >>>> >>>> df.write >>>> .options(Map("kudu.master" -> kuduMaster,"kudu.table" -> tableName)) >>>> .mode("overwrite") >>>> .kudu >>>> >>>> java.lang.RuntimeException: failed to write 1000 rows from DataFrame to >>>> Kudu; sample errors: Not found: key not found (error 0)Not found: key not >>>> found (error 0)Not found: key not found (error 0)Not found: key not found >>>> (error 0)Not found: key not found (error 0) >>>> >>>> Does the key field need to be first in the DataFrame? >>>> >>>> Thanks, >>>> Ben >>>> >>>> On Jun 14, 2016, at 4:28 PM, Dan Burkert <d...@cloudera.com> wrote: >>>> >>>> >>>> >>>> On Tue, Jun 14, 2016 at 4:20 PM, Benjamin Kim <bbuil...@gmail.com> >>>> wrote: >>>> >>>>> Dan, >>>>> >>>>> Thanks! It got further. Now, how do I set the Primary Key to be a >>>>> column(s) in the DataFrame and set the partitioning? Is it like this? >>>>> >>>>> kuduContext.createTable(tableName, df.schema, Seq(“my_id"), new >>>>> CreateTableOptions().setNumReplicas(1).addHashPartitions(“my_id")) >>>>> >>>>> java.lang.IllegalArgumentException: Table partitioning must be >>>>> specified using setRangePartitionColumns or addHashPartitions >>>>> >>>> >>>> Yep. The `Seq("my_id")` part of that call is specifying the set of >>>> primary key columns, so in this case you have specified the single PK >>>> column "my_id". The `addHashPartitions` call adds hash partitioning to the >>>> table, in this case over the column "my_id" (which is good, it must be over >>>> one or more PK columns, so in this case "my_id" is the one and only valid >>>> combination). However, the call to `addHashPartition` also takes the >>>> number of buckets as the second param. You shouldn't get the >>>> IllegalArgumentException as long as you are specifying either >>>> `addHashPartitions` or `setRangePartitionColumns`. >>>> >>>> - Dan >>>> >>>> >>>>> >>>>> Thanks, >>>>> Ben >>>>> >>>>> >>>>> On Jun 14, 2016, at 4:07 PM, Dan Burkert <d...@cloudera.com> wrote: >>>>> >>>>> Looks like we're missing an import statement in that example. Could >>>>> you try: >>>>> >>>>> import org.kududb.client._ >>>>> >>>>> and try again? >>>>> >>>>> - Dan >>>>> >>>>> On Tue, Jun 14, 2016 at 4:01 PM, Benjamin Kim <bbuil...@gmail.com> >>>>> wrote: >>>>> >>>>>> I encountered an error trying to create a table based on the >>>>>> documentation from a DataFrame. >>>>>> >>>>>> <console>:49: error: not found: type CreateTableOptions >>>>>> kuduContext.createTable(tableName, df.schema, >>>>>> Seq("key"), new CreateTableOptions().setNumReplicas(1)) >>>>>> >>>>>> Is there something I’m missing? >>>>>> >>>>>> Thanks, >>>>>> Ben >>>>>> >>>>>> On Jun 14, 2016, at 3:00 PM, Jean-Daniel Cryans <jdcry...@apache.org> >>>>>> wrote: >>>>>> >>>>>> It's only in Cloudera's maven repo: https://repository. >>>>>> cloudera.com/cloudera/cloudera-repos/org/kududb/ >>>>>> kudu-spark_2.10/0.9.0/ >>>>>> >>>>>> J-D >>>>>> >>>>>> On Tue, Jun 14, 2016 at 2:59 PM, Benjamin Kim <bbuil...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi J-D, >>>>>>> >>>>>>> I installed Kudu 0.9.0 using CM, but I can’t find the kudu-spark jar >>>>>>> for spark-shell to use. Can you show me where to find it? >>>>>>> >>>>>>> Thanks, >>>>>>> Ben >>>>>>> >>>>>>> >>>>>>> On Jun 8, 2016, at 1:19 PM, Jean-Daniel Cryans <jdcry...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>> What's in this doc is what's gonna get released: https://github.com/ >>>>>>> cloudera/kudu/blob/master/docs/developing.adoc#kudu- >>>>>>> integration-with-spark >>>>>>> >>>>>>> J-D >>>>>>> >>>>>>> On Tue, Jun 7, 2016 at 8:52 PM, Benjamin Kim <bbuil...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Will this be documented with examples once 0.9.0 comes out? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Ben >>>>>>>> >>>>>>>> >>>>>>>> On May 28, 2016, at 3:22 PM, Jean-Daniel Cryans < >>>>>>>> jdcry...@apache.org> wrote: >>>>>>>> >>>>>>>> It will be in 0.9.0. >>>>>>>> >>>>>>>> J-D >>>>>>>> >>>>>>>> On Sat, May 28, 2016 at 8:31 AM, Benjamin Kim <bbuil...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Chris, >>>>>>>>> >>>>>>>>> Will all this effort be rolled into 0.9.0 and be ready for use? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ben >>>>>>>>> >>>>>>>>> >>>>>>>>> On May 18, 2016, at 9:01 AM, Chris George < >>>>>>>>> christopher.geo...@rms.com> wrote: >>>>>>>>> >>>>>>>>> There is some code in review that needs some more refinement. >>>>>>>>> It will allow upsert/insert from a dataframe using the datasource >>>>>>>>> api. It will also allow the creation and deletion of tables from a >>>>>>>>> dataframe >>>>>>>>> http://gerrit.cloudera.org:8080/#/c/2992/ >>>>>>>>> >>>>>>>>> Example usages will look something like: >>>>>>>>> http://gerrit.cloudera.org:8080/#/c/2992/5/docs/developing.adoc >>>>>>>>> >>>>>>>>> -Chris George >>>>>>>>> >>>>>>>>> >>>>>>>>> On 5/18/16, 9:45 AM, "Benjamin Kim" <bbuil...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Can someone tell me what the state is of this Spark work? >>>>>>>>> >>>>>>>>> Also, does anyone have any sample code on how to update/insert >>>>>>>>> data in Kudu using DataFrames? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ben >>>>>>>>> >>>>>>>>> >>>>>>>>> On Apr 13, 2016, at 8:22 AM, Chris George < >>>>>>>>> christopher.geo...@rms.com> wrote: >>>>>>>>> >>>>>>>>> SparkSQL cannot support these type of statements but we may be >>>>>>>>> able to implement similar functionality through the api. >>>>>>>>> -Chris >>>>>>>>> >>>>>>>>> On 4/12/16, 5:19 PM, "Benjamin Kim" <bbuil...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> It would be nice to adhere to the SQL:2003 standard for an >>>>>>>>> “upsert” if it were to be implemented. >>>>>>>>> >>>>>>>>> MERGE INTO table_name USING table_reference ON (condition) >>>>>>>>> WHEN MATCHED THEN >>>>>>>>> UPDATE SET column1 = value1 [, column2 = value2 ...] >>>>>>>>> WHEN NOT MATCHED THEN >>>>>>>>> INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 …]) >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Ben >>>>>>>>> >>>>>>>>> On Apr 11, 2016, at 12:21 PM, Chris George < >>>>>>>>> christopher.geo...@rms.com> wrote: >>>>>>>>> >>>>>>>>> I have a wip kuduRDD that I made a few months ago. I pushed it >>>>>>>>> into gerrit if you want to take a look. http://gerrit.cloudera. >>>>>>>>> org:8080/#/c/2754/ >>>>>>>>> It does pushdown predicates which the existing input formatter >>>>>>>>> based rdd does not. >>>>>>>>> >>>>>>>>> Within the next two weeks I’m planning to implement a datasource >>>>>>>>> for spark that will have pushdown predicates and insertion/update >>>>>>>>> functionality (need to look more at cassandra and the hbase >>>>>>>>> datasource for >>>>>>>>> best way to do this) I agree that server side upsert would be helpful. >>>>>>>>> Having a datasource would give us useful data frames and also make >>>>>>>>> spark sql usable for kudu. >>>>>>>>> >>>>>>>>> My reasoning for having a spark datasource and not using Impala >>>>>>>>> is: 1. We have had trouble getting impala to run fast with high >>>>>>>>> concurrency >>>>>>>>> when compared to spark 2. We interact with datasources which do not >>>>>>>>> integrate with impala. 3. We have custom sql query planners for >>>>>>>>> extended >>>>>>>>> sql functionality. >>>>>>>>> >>>>>>>>> -Chris George >>>>>>>>> >>>>>>>>> >>>>>>>>> On 4/11/16, 12:22 PM, "Jean-Daniel Cryans" <jdcry...@apache.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> You guys make a convincing point, although on the upsert side >>>>>>>>> we'll need more support from the servers. Right now all you can do is >>>>>>>>> an >>>>>>>>> INSERT then, if you get a dup key, do an UPDATE. I guess we could at >>>>>>>>> least >>>>>>>>> add an API on the client side that would manage it, but it wouldn't be >>>>>>>>> atomic. >>>>>>>>> >>>>>>>>> J-D >>>>>>>>> >>>>>>>>> On Mon, Apr 11, 2016 at 9:34 AM, Mark Hamstra < >>>>>>>>> m...@clearstorydata.com> wrote: >>>>>>>>> >>>>>>>>>> It's pretty simple, actually. I need to support versioned >>>>>>>>>> datasets in a Spark SQL environment. Instead of a hack on top of a >>>>>>>>>> Parquet >>>>>>>>>> data store, I'm hoping (among other reasons) to be able to use >>>>>>>>>> Kudu's write >>>>>>>>>> and timestamp-based read operations to support not only appending >>>>>>>>>> data, but >>>>>>>>>> also updating existing data, and even some schema migration. The >>>>>>>>>> most >>>>>>>>>> typical use case is a dataset that is updated periodically (e.g., >>>>>>>>>> weekly or >>>>>>>>>> monthly) in which the the preliminary data in the previous window >>>>>>>>>> (week or >>>>>>>>>> month) is updated with values that are expected to remain unchanged >>>>>>>>>> from >>>>>>>>>> then on, and a new set of preliminary values for the current window >>>>>>>>>> need to >>>>>>>>>> be added/appended. >>>>>>>>>> >>>>>>>>>> Using Kudu's Java API and developing additional functionality on >>>>>>>>>> top of what Kudu has to offer isn't too much to ask, but the ease of >>>>>>>>>> integration with Spark SQL will gate how quickly we would move to >>>>>>>>>> using >>>>>>>>>> Kudu and how seriously we'd look at alternatives before making that >>>>>>>>>> decision. >>>>>>>>>> >>>>>>>>>> On Mon, Apr 11, 2016 at 8:14 AM, Jean-Daniel Cryans < >>>>>>>>>> jdcry...@apache.org> wrote: >>>>>>>>>> >>>>>>>>>>> Mark, >>>>>>>>>>> >>>>>>>>>>> Thanks for taking some time to reply in this thread, glad it >>>>>>>>>>> caught the attention of other folks! >>>>>>>>>>> >>>>>>>>>>> On Sun, Apr 10, 2016 at 12:33 PM, Mark Hamstra < >>>>>>>>>>> m...@clearstorydata.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Do they care being able to insert into Kudu with SparkSQL >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I care about insert into Kudu with Spark SQL. I'm currently >>>>>>>>>>>> delaying a refactoring of some Spark SQL-oriented insert >>>>>>>>>>>> functionality >>>>>>>>>>>> while trying to evaluate what to expect from Kudu. Whether Kudu >>>>>>>>>>>> does a >>>>>>>>>>>> good job supporting inserts with Spark SQL will be a key >>>>>>>>>>>> consideration as >>>>>>>>>>>> to whether we adopt Kudu. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I'd like to know more about why SparkSQL inserts in necessary >>>>>>>>>>> for you. Is it just that you currently do it that way into some >>>>>>>>>>> database or >>>>>>>>>>> parquet so with minimal refactoring you'd be able to use Kudu? Would >>>>>>>>>>> re-writing those SQL lines into Scala and directly use the Java >>>>>>>>>>> API's >>>>>>>>>>> KuduSession be too much work? >>>>>>>>>>> >>>>>>>>>>> Additionally, what do you expect to gain from using Kudu VS your >>>>>>>>>>> current solution? If it's not completely clear, I'd love to help >>>>>>>>>>> you think >>>>>>>>>>> through it. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Apr 10, 2016 at 12:23 PM, Jean-Daniel Cryans < >>>>>>>>>>>> jdcry...@apache.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yup, starting to get a good idea. >>>>>>>>>>>>> >>>>>>>>>>>>> What are your DS folks looking for in terms of functionality >>>>>>>>>>>>> related to Spark? A SparkSQL integration that's as fully featured >>>>>>>>>>>>> as >>>>>>>>>>>>> Impala's? Do they care being able to insert into Kudu with >>>>>>>>>>>>> SparkSQL or just >>>>>>>>>>>>> being able to query real fast? Anything more specific to Spark >>>>>>>>>>>>> that I'm >>>>>>>>>>>>> missing? >>>>>>>>>>>>> >>>>>>>>>>>>> FWIW the plan is to get to 1.0 in late Summer/early Fall. At >>>>>>>>>>>>> Cloudera all our resources are committed to making things happen >>>>>>>>>>>>> in time, >>>>>>>>>>>>> and a more fully featured Spark integration isn't in our plans >>>>>>>>>>>>> during that >>>>>>>>>>>>> period. I'm really hoping someone in the community will help with >>>>>>>>>>>>> Spark, >>>>>>>>>>>>> the same way we got a big contribution for the Flume sink. >>>>>>>>>>>>> >>>>>>>>>>>>> J-D >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Apr 10, 2016 at 11:29 AM, Benjamin Kim < >>>>>>>>>>>>> bbuil...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, we took Kudu for a test run using 0.6 and 0.7 versions. >>>>>>>>>>>>>> But, since it’s not “production-ready”, upper management doesn’t >>>>>>>>>>>>>> want to >>>>>>>>>>>>>> fully deploy it yet. They just want to keep an eye on it though. >>>>>>>>>>>>>> Kudu was >>>>>>>>>>>>>> so much simpler and easier to use in every aspect compared to >>>>>>>>>>>>>> HBase. Impala >>>>>>>>>>>>>> was great for the report writers and analysts to experiment with >>>>>>>>>>>>>> for the >>>>>>>>>>>>>> short time it was up. But, once again, the only blocker was the >>>>>>>>>>>>>> lack of >>>>>>>>>>>>>> Spark support for our Data Developers/Scientists. So, >>>>>>>>>>>>>> production-level data >>>>>>>>>>>>>> population won’t happen until then. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I hope this helps you get an idea where I am coming from… >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Ben >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 10, 2016, at 11:08 AM, Jean-Daniel Cryans < >>>>>>>>>>>>>> jdcry...@apache.org> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Apr 10, 2016 at 12:30 AM, Benjamin Kim < >>>>>>>>>>>>>> bbuil...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> J-D, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The main thing I hear that Cassandra is being used as an >>>>>>>>>>>>>>> updatable hot data store to ensure that duplicates are taken >>>>>>>>>>>>>>> care of and >>>>>>>>>>>>>>> idempotency is maintained. Whether data was directly retrieved >>>>>>>>>>>>>>> from >>>>>>>>>>>>>>> Cassandra for analytics, reports, or searches, it was not clear >>>>>>>>>>>>>>> as to what >>>>>>>>>>>>>>> was its main use. Some also just used it for a staging area to >>>>>>>>>>>>>>> populate >>>>>>>>>>>>>>> downstream tables in parquet format. The last thing I heard was >>>>>>>>>>>>>>> that CQL >>>>>>>>>>>>>>> was terrible, so that rules out much use of direct queries >>>>>>>>>>>>>>> against it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm no C* expert, but I don't think CQL is meant for real >>>>>>>>>>>>>> analytics, just ease of use instead of plainly using the APIs. >>>>>>>>>>>>>> Even then, >>>>>>>>>>>>>> Kudu should beat it easily on big scans. Same for HBase. We've >>>>>>>>>>>>>> done >>>>>>>>>>>>>> benchmarks against the latter, not the former. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As for our company, we have been looking for an updatable >>>>>>>>>>>>>>> data store for a long time that can be quickly queried directly >>>>>>>>>>>>>>> either >>>>>>>>>>>>>>> using Spark SQL or Impala or some other SQL engine and still >>>>>>>>>>>>>>> handle TB or >>>>>>>>>>>>>>> PB of data without performance degradation and many >>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>> headaches. For now, we are using HBase to take on this role >>>>>>>>>>>>>>> with Phoenix as >>>>>>>>>>>>>>> a fast way to directly query the data. I can see Kudu as the >>>>>>>>>>>>>>> best way to >>>>>>>>>>>>>>> fill this gap easily, especially being the closest thing to >>>>>>>>>>>>>>> other >>>>>>>>>>>>>>> relational databases out there in familiarity for the many SQL >>>>>>>>>>>>>>> analytics >>>>>>>>>>>>>>> people in our company. The other alternative would be to go >>>>>>>>>>>>>>> with AWS >>>>>>>>>>>>>>> Redshift for the same reasons, but it would come at a cost, of >>>>>>>>>>>>>>> course. If >>>>>>>>>>>>>>> we went with either solutions, Kudu or Redshift, it would get >>>>>>>>>>>>>>> rid of the >>>>>>>>>>>>>>> need to extract from HBase to parquet tables or export to >>>>>>>>>>>>>>> PostgreSQL to >>>>>>>>>>>>>>> support more of the SQL language using by analysts or the >>>>>>>>>>>>>>> reporting >>>>>>>>>>>>>>> software we use.. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, the usual then *smile*. Looks like we're not too far off >>>>>>>>>>>>>> with Kudu. Have you folks tried Kudu with Impala yet with those >>>>>>>>>>>>>> use cases? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I hope this helps. >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> It does, thanks for nice reply. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Ben >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 9, 2016, at 2:00 PM, Jean-Daniel Cryans < >>>>>>>>>>>>>>> jdcry...@apache.org> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ha first time I'm hearing about SMACK. Inside Cloudera we >>>>>>>>>>>>>>> like to refer to "Impala + Kudu" as Kimpala, but yeah it's not >>>>>>>>>>>>>>> as sexy. My >>>>>>>>>>>>>>> colleagues who were also there did say that the hype around >>>>>>>>>>>>>>> Spark isn't >>>>>>>>>>>>>>> dying down. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There's definitely an overlap in the use cases that >>>>>>>>>>>>>>> Cassandra, HBase, and Kudu cater to. I wouldn't go as far as >>>>>>>>>>>>>>> saying that C* >>>>>>>>>>>>>>> is just an interim solution for the use case you describe. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Nothing significant happened in Kudu over the past month, >>>>>>>>>>>>>>> it's a storage engine so things move slowly *smile*. I'd love >>>>>>>>>>>>>>> to see more >>>>>>>>>>>>>>> contributions on the Spark front. I know there's code out there >>>>>>>>>>>>>>> that could >>>>>>>>>>>>>>> be integrated in kudu-spark, it just needs to land in gerrit. >>>>>>>>>>>>>>> I'm sure >>>>>>>>>>>>>>> folks will happily review it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you have relevant experiences you can share? I'd love to >>>>>>>>>>>>>>> learn more about the use cases for which you envision using >>>>>>>>>>>>>>> Kudu as a C* >>>>>>>>>>>>>>> replacement. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> J-D >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Apr 8, 2016 at 12:45 PM, Benjamin Kim < >>>>>>>>>>>>>>> bbuil...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi J-D, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> My colleagues recently came back from Strata in San Jose. >>>>>>>>>>>>>>>> They told me that everything was about Spark and there is a >>>>>>>>>>>>>>>> big buzz about >>>>>>>>>>>>>>>> the SMACK stack (Spark, Mesos, Akka, Cassandra, Kafka). I >>>>>>>>>>>>>>>> still think that >>>>>>>>>>>>>>>> Cassandra is just an interim solution as a low-latency, easily >>>>>>>>>>>>>>>> queried data >>>>>>>>>>>>>>>> store. I was wondering if anything significant happened in >>>>>>>>>>>>>>>> regards to Kudu, >>>>>>>>>>>>>>>> especially on the Spark front. Plus, can you come up with your >>>>>>>>>>>>>>>> own proposed >>>>>>>>>>>>>>>> stack acronym to promote? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>> Ben >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 1, 2016, at 12:20 PM, Jean-Daniel Cryans < >>>>>>>>>>>>>>>> jdcry...@apache.org> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Ben, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> AFAIK no one in the dev community committed to any >>>>>>>>>>>>>>>> timeline. I know of one person on the Kudu Slack who's working >>>>>>>>>>>>>>>> on a better >>>>>>>>>>>>>>>> RDD, but that's about it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> J-D >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Mar 1, 2016 at 11:00 AM, Benjamin Kim < >>>>>>>>>>>>>>>> b...@amobee.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi J-D, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Quick question… Is there an ETA for KUDU-1214? I want to >>>>>>>>>>>>>>>>> target a version of Kudu to begin real testing of Spark >>>>>>>>>>>>>>>>> against it for our >>>>>>>>>>>>>>>>> devs. At least, I can tell them what timeframe to anticipate. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Just curious, >>>>>>>>>>>>>>>>> *Benjamin Kim* >>>>>>>>>>>>>>>>> *Data Solutions Architect* >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [a•mo•bee] *(n.)* the company defining digital marketing. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> *Mobile: +1 818 635 2900 <%2B1%20818%20635%202900>* >>>>>>>>>>>>>>>>> 3250 Ocean Park Blvd, Suite 200 | Santa Monica, CA >>>>>>>>>>>>>>>>> 90405 | www.amobee.com >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Feb 24, 2016, at 3:51 PM, Jean-Daniel Cryans < >>>>>>>>>>>>>>>>> jdcry...@apache.org> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The DStream stuff isn't there at all. I'm not sure if it's >>>>>>>>>>>>>>>>> needed either. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The kuduRDD is just leveraging the MR input format, >>>>>>>>>>>>>>>>> ideally we'd use scans directly. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The SparkSQL stuff is there but it doesn't do any sort of >>>>>>>>>>>>>>>>> pushdown. It's really basic. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The goal was to provide something for others to contribute >>>>>>>>>>>>>>>>> to. We have some basic unit tests that others can easily >>>>>>>>>>>>>>>>> extend. None of us >>>>>>>>>>>>>>>>> on the team are Spark experts, but we'd be really happy to >>>>>>>>>>>>>>>>> assist one >>>>>>>>>>>>>>>>> improve the kudu-spark code. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> J-D >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Feb 24, 2016 at 3:41 PM, Benjamin Kim < >>>>>>>>>>>>>>>>> bbuil...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> J-D, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> It looks like it fulfills most of the basic requirements >>>>>>>>>>>>>>>>>> (kudu RDD, kudu DStream) in KUDU-1214. Am I right? Besides >>>>>>>>>>>>>>>>>> shoring up more >>>>>>>>>>>>>>>>>> Spark SQL functionality (Dataframes) and doing the >>>>>>>>>>>>>>>>>> documentation, what more >>>>>>>>>>>>>>>>>> needs to be done? Optimizations? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I believe that it’s a good place to start using Spark >>>>>>>>>>>>>>>>>> with Kudu and compare it to HBase with Spark (not clean). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> Ben >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Feb 24, 2016, at 3:10 PM, Jean-Daniel Cryans < >>>>>>>>>>>>>>>>>> jdcry...@apache.org> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> AFAIK no one is working on it, but we did manage to get >>>>>>>>>>>>>>>>>> this in for 0.7.0: https://issues. >>>>>>>>>>>>>>>>>> cloudera.org/browse/KUDU-1321 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> It's a really simple wrapper, and yes you can use >>>>>>>>>>>>>>>>>> SparkSQL on Kudu, but it will require a lot more work to >>>>>>>>>>>>>>>>>> make it >>>>>>>>>>>>>>>>>> fast/useful. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hope this helps, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> J-D >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wed, Feb 24, 2016 at 3:08 PM, Benjamin Kim < >>>>>>>>>>>>>>>>>> bbuil...@gmail.com> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I see this KUDU-1214 >>>>>>>>>>>>>>>>>>> <https://issues.cloudera.org/browse/KUDU-1214> targeted >>>>>>>>>>>>>>>>>>> for 0.8.0, but I see no progress on it. When this is >>>>>>>>>>>>>>>>>>> complete, will this >>>>>>>>>>>>>>>>>>> mean that Spark will be able to work with Kudu both >>>>>>>>>>>>>>>>>>> programmatically and as >>>>>>>>>>>>>>>>>>> a client via Spark SQL? Or is there more work that needs to >>>>>>>>>>>>>>>>>>> be done on the >>>>>>>>>>>>>>>>>>> Spark side for it to work? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Just curious. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>> Ben >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >> >> > > -- Todd Lipcon Software Engineer, Cloudera
