Thanks!
> On Sep 20, 2016, at 3:02 PM, Jordan Birdsell <jordantbirds...@gmail.com>
> wrote:
>
> http://kudu.apache.org/docs/developing.html#_kudu_integration_with_spark
> <http://kudu.apache.org/docs/developing.html#_kudu_integration_with_spark>
>
> On Tue, Sep 20, 2016 at 5:00 PM Benjamin Kim <bbuil...@gmail.com
> <mailto:bbuil...@gmail.com>> wrote:
> I see that the API has changed a bit so my old code doesn’t work anymore. Can
> someone direct me to some code samples?
>
> Thanks,
> Ben
>
>
>> On Sep 20, 2016, at 1:44 PM, Todd Lipcon <t...@cloudera.com
>> <mailto:t...@cloudera.com>> wrote:
>>
>> On Tue, Sep 20, 2016 at 1:18 PM, Benjamin Kim <bbuil...@gmail.com
>> <mailto:bbuil...@gmail.com>> wrote:
>> Now that Kudu 1.0.0 is officially out and ready for production use, where do
>> we find the spark connector jar for this release?
>>
>>
>> It's available in the official ASF maven repository:
>> https://repository.apache.org/#nexus-search;quick~kudu-spark
>> <https://repository.apache.org/#nexus-search;quick~kudu-spark>
>>
>> <dependency>
>> <groupId>org.apache.kudu</groupId>
>> <artifactId>kudu-spark_2.10</artifactId>
>> <version>1.0.0</version>
>> </dependency>
>>
>>
>> -Todd
>>
>>
>>
>>> On Jun 17, 2016, at 11:08 AM, Dan Burkert <d...@cloudera.com
>>> <mailto:d...@cloudera.com>> wrote:
>>>
>>> Hi Ben,
>>>
>>> To your first question about `CREATE TABLE` syntax with Kudu/Spark SQL, I
>>> do not think we support that at this point. I haven't looked deeply into
>>> it, but we may hit issues specifying Kudu-specific options (partitioning,
>>> column encoding, etc.). Probably issues that can be worked through
>>> eventually, though. If you are interested in contributing to Kudu, this is
>>> an area that could obviously use improvement! Most or all of our Spark
>>> features have been completely community driven to date.
>>>
>>> I am assuming that more Spark support along with semantic changes below
>>> will be incorporated into Kudu 0.9.1.
>>>
>>> As a rule we do not release new features in patch releases, but the good
>>> news is that we are releasing regularly, and our next scheduled release is
>>> for the August timeframe (see JD's roadmap
>>> <https://lists.apache.org/thread.html/1a3b949e715a74d7f26bd9c102247441a06d16d077324ba39a662e2a@1455234076@%3Cdev.kudu.apache.org%3E>
>>> email about what we are aiming to include). Also, Cloudera does publish
>>> snapshot versions of the Spark connector here
>>> <https://repository.cloudera.com/cloudera/cloudera-repos/org/kududb/>, so
>>> the jars are available if you don't mind using snapshots.
>>>
>>> Anyone know of a better way to make unique primary keys other than using
>>> UUID to make every row unique if there is no unique column (or combination
>>> thereof) to use.
>>>
>>> Not that I know of. In general it's pretty rare to have a dataset without
>>> a natural primary key (even if it's just all of the columns), but in those
>>> cases UUID is a good solution.
>>>
>>> This is what I am using. I know auto incrementing is coming down the line
>>> (don’t know when), but is there a way to simulate this in Kudu using Spark
>>> out of curiosity?
>>>
>>> To my knowledge there is no plan to have auto increment in Kudu.
>>> Distributed, consistent, auto incrementing counters is a difficult problem,
>>> and I don't think there are any known solutions that would be fast enough
>>> for Kudu (happy to be proven wrong, though!).
>>>
>>> - Dan
>>>
>>>
>>> Thanks,
>>> Ben
>>>
>>>> On Jun 14, 2016, at 6:08 PM, Dan Burkert <d...@cloudera.com
>>>> <mailto:d...@cloudera.com>> wrote:
>>>>
>>>> I'm not sure exactly what the semantics will be, but at least one of them
>>>> will be upsert. These modes come from spark, and they were really
>>>> designed for file-backed storage and not table storage. We may want to do
>>>> append = upsert, and overwrite = truncate + insert. I think that may
>>>> match the normal spark semantics more closely.
>>>>
>>>> - Dan
>>>>
>>>> On Tue, Jun 14, 2016 at 6:00 PM, Benjamin Kim <bbuil...@gmail.com
>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>> Dan,
>>>>
>>>> Thanks for the information. That would mean both “append” and “overwrite”
>>>> modes would be combined or not needed in the future.
>>>>
>>>> Cheers,
>>>> Ben
>>>>
>>>>> On Jun 14, 2016, at 5:57 PM, Dan Burkert <d...@cloudera.com
>>>>> <mailto:d...@cloudera.com>> wrote:
>>>>>
>>>>> Right now append uses an update Kudu operation, which requires the row
>>>>> already be present in the table. Overwrite maps to insert. Kudu very
>>>>> recently got upsert support baked in, but it hasn't yet been integrated
>>>>> into the Spark connector. So pretty soon these sharp edges will get a
>>>>> lot better, since upsert is the way to go for most spark workloads.
>>>>>
>>>>> - Dan
>>>>>
>>>>> On Tue, Jun 14, 2016 at 5:41 PM, Benjamin Kim <bbuil...@gmail.com
>>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>>> I tried to use the “append” mode, and it worked. Over 3.8 million rows in
>>>>> 64s. I would assume that now I can use the “overwrite” mode on existing
>>>>> data. Now, I have to find answers to these questions. What would happen
>>>>> if I “append” to the data in the Kudu table if the data already exists?
>>>>> What would happen if I “overwrite” existing data when the DataFrame has
>>>>> data in it that does not exist in the Kudu table? I need to evaluate the
>>>>> best way to simulate the UPSERT behavior in HBase because this is what
>>>>> our use case is.
>>>>>
>>>>> Thanks,
>>>>> Ben
>>>>>
>>>>>
>>>>>
>>>>>> On Jun 14, 2016, at 5:05 PM, Benjamin Kim <bbuil...@gmail.com
>>>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Now, I’m getting this error when trying to write to the table.
>>>>>>
>>>>>> import scala.collection.JavaConverters._
>>>>>> val key_seq = Seq(“my_id")
>>>>>> val key_list = List(“my_id”).asJava
>>>>>> kuduContext.createTable(tableName, df.schema, key_seq, new
>>>>>> CreateTableOptions().setNumReplicas(1).addHashPartitions(key_list, 100))
>>>>>>
>>>>>> df.write
>>>>>> .options(Map("kudu.master" -> kuduMaster,"kudu.table" -> tableName))
>>>>>> .mode("overwrite")
>>>>>> .kudu
>>>>>>
>>>>>> java.lang.RuntimeException: failed to write 1000 rows from DataFrame to
>>>>>> Kudu; sample errors: Not found: key not found (error 0)Not found: key
>>>>>> not found (error 0)Not found: key not found (error 0)Not found: key not
>>>>>> found (error 0)Not found: key not found (error 0)
>>>>>>
>>>>>> Does the key field need to be first in the DataFrame?
>>>>>>
>>>>>> Thanks,
>>>>>> Ben
>>>>>>
>>>>>>> On Jun 14, 2016, at 4:28 PM, Dan Burkert <d...@cloudera.com
>>>>>>> <mailto:d...@cloudera.com>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jun 14, 2016 at 4:20 PM, Benjamin Kim <bbuil...@gmail.com
>>>>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>>>>> Dan,
>>>>>>>
>>>>>>> Thanks! It got further. Now, how do I set the Primary Key to be a
>>>>>>> column(s) in the DataFrame and set the partitioning? Is it like this?
>>>>>>>
>>>>>>> kuduContext.createTable(tableName, df.schema, Seq(“my_id"), new
>>>>>>> CreateTableOptions().setNumReplicas(1).addHashPartitions(“my_id"))
>>>>>>>
>>>>>>> java.lang.IllegalArgumentException: Table partitioning must be
>>>>>>> specified using setRangePartitionColumns or addHashPartitions
>>>>>>>
>>>>>>> Yep. The `Seq("my_id")` part of that call is specifying the set of
>>>>>>> primary key columns, so in this case you have specified the single PK
>>>>>>> column "my_id". The `addHashPartitions` call adds hash partitioning to
>>>>>>> the table, in this case over the column "my_id" (which is good, it must
>>>>>>> be over one or more PK columns, so in this case "my_id" is the one and
>>>>>>> only valid combination). However, the call to `addHashPartition` also
>>>>>>> takes the number of buckets as the second param. You shouldn't get the
>>>>>>> IllegalArgumentException as long as you are specifying either
>>>>>>> `addHashPartitions` or `setRangePartitionColumns`.
>>>>>>>
>>>>>>> - Dan
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ben
>>>>>>>
>>>>>>>
>>>>>>>> On Jun 14, 2016, at 4:07 PM, Dan Burkert <d...@cloudera.com
>>>>>>>> <mailto:d...@cloudera.com>> wrote:
>>>>>>>>
>>>>>>>> Looks like we're missing an import statement in that example. Could
>>>>>>>> you try:
>>>>>>>>
>>>>>>>> import org.kududb.client._
>>>>>>>> and try again?
>>>>>>>>
>>>>>>>> - Dan
>>>>>>>>
>>>>>>>> On Tue, Jun 14, 2016 at 4:01 PM, Benjamin Kim <bbuil...@gmail.com
>>>>>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>> I encountered an error trying to create a table based on the
>>>>>>>> documentation from a DataFrame.
>>>>>>>>
>>>>>>>> <console>:49: error: not found: type CreateTableOptions
>>>>>>>> kuduContext.createTable(tableName, df.schema,
>>>>>>>> Seq("key"), new CreateTableOptions().setNumReplicas(1))
>>>>>>>>
>>>>>>>> Is there something I’m missing?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ben
>>>>>>>>
>>>>>>>>> On Jun 14, 2016, at 3:00 PM, Jean-Daniel Cryans <jdcry...@apache.org
>>>>>>>>> <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>
>>>>>>>>> It's only in Cloudera's maven repo:
>>>>>>>>> https://repository.cloudera.com/cloudera/cloudera-repos/org/kududb/kudu-spark_2.10/0.9.0/
>>>>>>>>>
>>>>>>>>> <https://repository.cloudera.com/cloudera/cloudera-repos/org/kududb/kudu-spark_2.10/0.9.0/>
>>>>>>>>>
>>>>>>>>> J-D
>>>>>>>>>
>>>>>>>>> On Tue, Jun 14, 2016 at 2:59 PM, Benjamin Kim <bbuil...@gmail.com
>>>>>>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>>> Hi J-D,
>>>>>>>>>
>>>>>>>>> I installed Kudu 0.9.0 using CM, but I can’t find the kudu-spark jar
>>>>>>>>> for spark-shell to use. Can you show me where to find it?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Ben
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Jun 8, 2016, at 1:19 PM, Jean-Daniel Cryans <jdcry...@apache.org
>>>>>>>>>> <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>>
>>>>>>>>>> What's in this doc is what's gonna get released:
>>>>>>>>>> https://github.com/cloudera/kudu/blob/master/docs/developing.adoc#kudu-integration-with-spark
>>>>>>>>>>
>>>>>>>>>> <https://github.com/cloudera/kudu/blob/master/docs/developing.adoc#kudu-integration-with-spark>
>>>>>>>>>>
>>>>>>>>>> J-D
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 7, 2016 at 8:52 PM, Benjamin Kim <bbuil...@gmail.com
>>>>>>>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>>>> Will this be documented with examples once 0.9.0 comes out?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Ben
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On May 28, 2016, at 3:22 PM, Jean-Daniel Cryans
>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> It will be in 0.9.0.
>>>>>>>>>>>
>>>>>>>>>>> J-D
>>>>>>>>>>>
>>>>>>>>>>> On Sat, May 28, 2016 at 8:31 AM, Benjamin Kim <bbuil...@gmail.com
>>>>>>>>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>>>>> Hi Chris,
>>>>>>>>>>>
>>>>>>>>>>> Will all this effort be rolled into 0.9.0 and be ready for use?
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Ben
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On May 18, 2016, at 9:01 AM, Chris George
>>>>>>>>>>>> <christopher.geo...@rms.com <mailto:christopher.geo...@rms.com>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> There is some code in review that needs some more refinement.
>>>>>>>>>>>> It will allow upsert/insert from a dataframe using the datasource
>>>>>>>>>>>> api. It will also allow the creation and deletion of tables from a
>>>>>>>>>>>> dataframe
>>>>>>>>>>>> http://gerrit.cloudera.org:8080/#/c/2992/
>>>>>>>>>>>> <http://gerrit.cloudera.org:8080/#/c/2992/>
>>>>>>>>>>>>
>>>>>>>>>>>> Example usages will look something like:
>>>>>>>>>>>> http://gerrit.cloudera.org:8080/#/c/2992/5/docs/developing.adoc
>>>>>>>>>>>> <http://gerrit.cloudera.org:8080/#/c/2992/5/docs/developing.adoc>
>>>>>>>>>>>>
>>>>>>>>>>>> -Chris George
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 5/18/16, 9:45 AM, "Benjamin Kim" <bbuil...@gmail.com
>>>>>>>>>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Can someone tell me what the state is of this Spark work?
>>>>>>>>>>>>
>>>>>>>>>>>> Also, does anyone have any sample code on how to update/insert
>>>>>>>>>>>> data in Kudu using DataFrames?
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Ben
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On Apr 13, 2016, at 8:22 AM, Chris George
>>>>>>>>>>>>> <christopher.geo...@rms.com <mailto:christopher.geo...@rms.com>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> SparkSQL cannot support these type of statements but we may be
>>>>>>>>>>>>> able to implement similar functionality through the api.
>>>>>>>>>>>>> -Chris
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 4/12/16, 5:19 PM, "Benjamin Kim" <bbuil...@gmail.com
>>>>>>>>>>>>> <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> It would be nice to adhere to the SQL:2003 standard for an
>>>>>>>>>>>>> “upsert” if it were to be implemented.
>>>>>>>>>>>>>
>>>>>>>>>>>>> MERGE INTO table_name USING table_reference ON (condition)
>>>>>>>>>>>>> WHEN MATCHED THEN
>>>>>>>>>>>>> UPDATE SET column1 = value1 [, column2 = value2 ...]
>>>>>>>>>>>>> WHEN NOT MATCHED THEN
>>>>>>>>>>>>> INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 …])
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Ben
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Apr 11, 2016, at 12:21 PM, Chris George
>>>>>>>>>>>>>> <christopher.geo...@rms.com <mailto:christopher.geo...@rms.com>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have a wip kuduRDD that I made a few months ago. I pushed it
>>>>>>>>>>>>>> into gerrit if you want to take a look.
>>>>>>>>>>>>>> http://gerrit.cloudera.org:8080/#/c/2754/
>>>>>>>>>>>>>> <http://gerrit.cloudera.org:8080/#/c/2754/>
>>>>>>>>>>>>>> It does pushdown predicates which the existing input formatter
>>>>>>>>>>>>>> based rdd does not.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Within the next two weeks I’m planning to implement a datasource
>>>>>>>>>>>>>> for spark that will have pushdown predicates and
>>>>>>>>>>>>>> insertion/update functionality (need to look more at cassandra
>>>>>>>>>>>>>> and the hbase datasource for best way to do this) I agree that
>>>>>>>>>>>>>> server side upsert would be helpful.
>>>>>>>>>>>>>> Having a datasource would give us useful data frames and also
>>>>>>>>>>>>>> make spark sql usable for kudu.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My reasoning for having a spark datasource and not using Impala
>>>>>>>>>>>>>> is: 1. We have had trouble getting impala to run fast with high
>>>>>>>>>>>>>> concurrency when compared to spark 2. We interact with
>>>>>>>>>>>>>> datasources which do not integrate with impala. 3. We have
>>>>>>>>>>>>>> custom sql query planners for extended sql functionality.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -Chris George
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 4/11/16, 12:22 PM, "Jean-Daniel Cryans" <jdcry...@apache.org
>>>>>>>>>>>>>> <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You guys make a convincing point, although on the upsert side
>>>>>>>>>>>>>> we'll need more support from the servers. Right now all you can
>>>>>>>>>>>>>> do is an INSERT then, if you get a dup key, do an UPDATE. I
>>>>>>>>>>>>>> guess we could at least add an API on the client side that would
>>>>>>>>>>>>>> manage it, but it wouldn't be atomic.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> J-D
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 11, 2016 at 9:34 AM, Mark Hamstra
>>>>>>>>>>>>>> <m...@clearstorydata.com <mailto:m...@clearstorydata.com>>wrote:
>>>>>>>>>>>>>> It's pretty simple, actually. I need to support versioned
>>>>>>>>>>>>>> datasets in a Spark SQL environment. Instead of a hack on top
>>>>>>>>>>>>>> of a Parquet data store, I'm hoping (among other reasons) to be
>>>>>>>>>>>>>> able to use Kudu's write and timestamp-based read operations to
>>>>>>>>>>>>>> support not only appending data, but also updating existing
>>>>>>>>>>>>>> data, and even some schema migration. The most typical use case
>>>>>>>>>>>>>> is a dataset that is updated periodically (e.g., weekly or
>>>>>>>>>>>>>> monthly) in which the the preliminary data in the previous
>>>>>>>>>>>>>> window (week or month) is updated with values that are expected
>>>>>>>>>>>>>> to remain unchanged from then on, and a new set of preliminary
>>>>>>>>>>>>>> values for the current window need to be added/appended.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Using Kudu's Java API and developing additional functionality on
>>>>>>>>>>>>>> top of what Kudu has to offer isn't too much to ask, but the
>>>>>>>>>>>>>> ease of integration with Spark SQL will gate how quickly we
>>>>>>>>>>>>>> would move to using Kudu and how seriously we'd look at
>>>>>>>>>>>>>> alternatives before making that decision.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Apr 11, 2016 at 8:14 AM, Jean-Daniel Cryans
>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>>wrote:
>>>>>>>>>>>>>> Mark,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for taking some time to reply in this thread, glad it
>>>>>>>>>>>>>> caught the attention of other folks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Apr 10, 2016 at 12:33 PM, Mark
>>>>>>>>>>>>>> Hamstra<m...@clearstorydata.com
>>>>>>>>>>>>>> <mailto:m...@clearstorydata.com>> wrote:
>>>>>>>>>>>>>> Do they care being able to insert into Kudu with SparkSQL
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I care about insert into Kudu with Spark SQL. I'm currently
>>>>>>>>>>>>>> delaying a refactoring of some Spark SQL-oriented insert
>>>>>>>>>>>>>> functionality while trying to evaluate what to expect from Kudu.
>>>>>>>>>>>>>> Whether Kudu does a good job supporting inserts with Spark SQL
>>>>>>>>>>>>>> will be a key consideration as to whether we adopt Kudu.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd like to know more about why SparkSQL inserts in necessary
>>>>>>>>>>>>>> for you. Is it just that you currently do it that way into some
>>>>>>>>>>>>>> database or parquet so with minimal refactoring you'd be able to
>>>>>>>>>>>>>> use Kudu? Would re-writing those SQL lines into Scala and
>>>>>>>>>>>>>> directly use the Java API's KuduSession be too much work?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Additionally, what do you expect to gain from using Kudu VS your
>>>>>>>>>>>>>> current solution? If it's not completely clear, I'd love to help
>>>>>>>>>>>>>> you think through it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Apr 10, 2016 at 12:23 PM, Jean-Daniel Cryans
>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>>>>>> Yup, starting to get a good idea.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What are your DS folks looking for in terms of functionality
>>>>>>>>>>>>>> related to Spark? A SparkSQL integration that's as fully
>>>>>>>>>>>>>> featured as Impala's? Do they care being able to insert into
>>>>>>>>>>>>>> Kudu with SparkSQL or just being able to query real fast?
>>>>>>>>>>>>>> Anything more specific to Spark that I'm missing?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> FWIW the plan is to get to 1.0 in late Summer/early Fall. At
>>>>>>>>>>>>>> Cloudera all our resources are committed to making things happen
>>>>>>>>>>>>>> in time, and a more fully featured Spark integration isn't in
>>>>>>>>>>>>>> our plans during that period. I'm really hoping someone in the
>>>>>>>>>>>>>> community will help with Spark, the same way we got a big
>>>>>>>>>>>>>> contribution for the Flume sink.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> J-D
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sun, Apr 10, 2016 at 11:29 AM, Benjamin Kim
>>>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>>wrote:
>>>>>>>>>>>>>> Yes, we took Kudu for a test run using 0.6 and 0.7 versions.
>>>>>>>>>>>>>> But, since it’s not “production-ready”, upper management doesn’t
>>>>>>>>>>>>>> want to fully deploy it yet. They just want to keep an eye on it
>>>>>>>>>>>>>> though. Kudu was so much simpler and easier to use in every
>>>>>>>>>>>>>> aspect compared to HBase. Impala was great for the report
>>>>>>>>>>>>>> writers and analysts to experiment with for the short time it
>>>>>>>>>>>>>> was up. But, once again, the only blocker was the lack of Spark
>>>>>>>>>>>>>> support for our Data Developers/Scientists. So, production-level
>>>>>>>>>>>>>> data population won’t happen until then.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I hope this helps you get an idea where I am coming from…
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Ben
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Apr 10, 2016, at 11:08 AM, Jean-Daniel Cryans
>>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Sun, Apr 10, 2016 at 12:30 AM, Benjamin Kim
>>>>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>>>>>>>>> J-D,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The main thing I hear that Cassandra is being used as an
>>>>>>>>>>>>>>> updatable hot data store to ensure that duplicates are taken
>>>>>>>>>>>>>>> care of and idempotency is maintained. Whether data was
>>>>>>>>>>>>>>> directly retrieved from Cassandra for analytics, reports, or
>>>>>>>>>>>>>>> searches, it was not clear as to what was its main use. Some
>>>>>>>>>>>>>>> also just used it for a staging area to populate downstream
>>>>>>>>>>>>>>> tables in parquet format. The last thing I heard was that CQL
>>>>>>>>>>>>>>> was terrible, so that rules out much use of direct queries
>>>>>>>>>>>>>>> against it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm no C* expert, but I don't think CQL is meant for real
>>>>>>>>>>>>>>> analytics, just ease of use instead of plainly using the APIs.
>>>>>>>>>>>>>>> Even then, Kudu should beat it easily on big scans. Same for
>>>>>>>>>>>>>>> HBase. We've done benchmarks against the latter, not the former.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As for our company, we have been looking for an updatable data
>>>>>>>>>>>>>>> store for a long time that can be quickly queried directly
>>>>>>>>>>>>>>> either using Spark SQL or Impala or some other SQL engine and
>>>>>>>>>>>>>>> still handle TB or PB of data without performance degradation
>>>>>>>>>>>>>>> and many configuration headaches. For now, we are using HBase
>>>>>>>>>>>>>>> to take on this role with Phoenix as a fast way to directly
>>>>>>>>>>>>>>> query the data. I can see Kudu as the best way to fill this gap
>>>>>>>>>>>>>>> easily, especially being the closest thing to other relational
>>>>>>>>>>>>>>> databases out there in familiarity for the many SQL analytics
>>>>>>>>>>>>>>> people in our company. The other alternative would be to go
>>>>>>>>>>>>>>> with AWS Redshift for the same reasons, but it would come at a
>>>>>>>>>>>>>>> cost, of course. If we went with either solutions, Kudu or
>>>>>>>>>>>>>>> Redshift, it would get rid of the need to extract from HBase to
>>>>>>>>>>>>>>> parquet tables or export to PostgreSQL to support more of the
>>>>>>>>>>>>>>> SQL language using by analysts or the reporting software we
>>>>>>>>>>>>>>> use..
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ok, the usual then *smile*. Looks like we're not too far off
>>>>>>>>>>>>>>> with Kudu. Have you folks tried Kudu with Impala yet with those
>>>>>>>>>>>>>>> use cases?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I hope this helps.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It does, thanks for nice reply.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Ben
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Apr 9, 2016, at 2:00 PM, Jean-Daniel Cryans
>>>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ha first time I'm hearing about SMACK. Inside Cloudera we like
>>>>>>>>>>>>>>>> to refer to "Impala + Kudu" as Kimpala, but yeah it's not as
>>>>>>>>>>>>>>>> sexy. My colleagues who were also there did say that the hype
>>>>>>>>>>>>>>>> around Spark isn't dying down.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There's definitely an overlap in the use cases that Cassandra,
>>>>>>>>>>>>>>>> HBase, and Kudu cater to. I wouldn't go as far as saying that
>>>>>>>>>>>>>>>> C* is just an interim solution for the use case you describe.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nothing significant happened in Kudu over the past month, it's
>>>>>>>>>>>>>>>> a storage engine so things move slowly *smile*. I'd love to
>>>>>>>>>>>>>>>> see more contributions on the Spark front. I know there's code
>>>>>>>>>>>>>>>> out there that could be integrated in kudu-spark, it just
>>>>>>>>>>>>>>>> needs to land in gerrit. I'm sure folks will happily review it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Do you have relevant experiences you can share? I'd love to
>>>>>>>>>>>>>>>> learn more about the use cases for which you envision using
>>>>>>>>>>>>>>>> Kudu as a C* replacement.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> J-D
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Apr 8, 2016 at 12:45 PM, Benjamin Kim
>>>>>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>>>>>>>>>> Hi J-D,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My colleagues recently came back from Strata in San Jose. They
>>>>>>>>>>>>>>>> told me that everything was about Spark and there is a big
>>>>>>>>>>>>>>>> buzz about the SMACK stack (Spark, Mesos, Akka, Cassandra,
>>>>>>>>>>>>>>>> Kafka). I still think that Cassandra is just an interim
>>>>>>>>>>>>>>>> solution as a low-latency, easily queried data store. I was
>>>>>>>>>>>>>>>> wondering if anything significant happened in regards to Kudu,
>>>>>>>>>>>>>>>> especially on the Spark front. Plus, can you come up with your
>>>>>>>>>>>>>>>> own proposed stack acronym to promote?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Ben
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mar 1, 2016, at 12:20 PM, Jean-Daniel Cryans
>>>>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Ben,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> AFAIK no one in the dev community committed to any timeline.
>>>>>>>>>>>>>>>>> I know of one person on the Kudu Slack who's working on a
>>>>>>>>>>>>>>>>> better RDD, but that's about it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> J-D
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Mar 1, 2016 at 11:00 AM, Benjamin Kim
>>>>>>>>>>>>>>>>> <b...@amobee.com <mailto:b...@amobee.com>> wrote:
>>>>>>>>>>>>>>>>> Hi J-D,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Quick question… Is there an ETA for KUDU-1214? I want to
>>>>>>>>>>>>>>>>> target a version of Kudu to begin real testing of Spark
>>>>>>>>>>>>>>>>> against it for our devs. At least, I can tell them what
>>>>>>>>>>>>>>>>> timeframe to anticipate.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Just curious,
>>>>>>>>>>>>>>>>> Benjamin Kim
>>>>>>>>>>>>>>>>> Data Solutions Architect
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [a•mo•bee] (n.) the company defining digital marketing.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Mobile: +1 818 635 2900 <tel:%2B1%20818%20635%202900>
>>>>>>>>>>>>>>>>> 3250 Ocean Park Blvd, Suite 200 | Santa Monica, CA 90405 |
>>>>>>>>>>>>>>>>> www.amobee.com <http://www.amobee.com/>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Feb 24, 2016, at 3:51 PM, Jean-Daniel Cryans
>>>>>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The DStream stuff isn't there at all. I'm not sure if it's
>>>>>>>>>>>>>>>>>> needed either.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The kuduRDD is just leveraging the MR input format, ideally
>>>>>>>>>>>>>>>>>> we'd use scans directly.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The SparkSQL stuff is there but it doesn't do any sort of
>>>>>>>>>>>>>>>>>> pushdown. It's really basic.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The goal was to provide something for others to contribute
>>>>>>>>>>>>>>>>>> to. We have some basic unit tests that others can easily
>>>>>>>>>>>>>>>>>> extend. None of us on the team are Spark experts, but we'd
>>>>>>>>>>>>>>>>>> be really happy to assist one improve the kudu-spark code.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> J-D
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Feb 24, 2016 at 3:41 PM, Benjamin Kim
>>>>>>>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>> J-D,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It looks like it fulfills most of the basic requirements
>>>>>>>>>>>>>>>>>> (kudu RDD, kudu DStream) in KUDU-1214. Am I right? Besides
>>>>>>>>>>>>>>>>>> shoring up more Spark SQL functionality (Dataframes) and
>>>>>>>>>>>>>>>>>> doing the documentation, what more needs to be done?
>>>>>>>>>>>>>>>>>> Optimizations?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I believe that it’s a good place to start using Spark with
>>>>>>>>>>>>>>>>>> Kudu and compare it to HBase with Spark (not clean).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Ben
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Feb 24, 2016, at 3:10 PM, Jean-Daniel Cryans
>>>>>>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> AFAIK no one is working on it, but we did manage to get
>>>>>>>>>>>>>>>>>>> this in for 0.7.0:
>>>>>>>>>>>>>>>>>>> https://issues.cloudera.org/browse/KUDU-1321
>>>>>>>>>>>>>>>>>>> <https://issues.cloudera.org/browse/KUDU-1321>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It's a really simple wrapper, and yes you can use SparkSQL
>>>>>>>>>>>>>>>>>>> on Kudu, but it will require a lot more work to make it
>>>>>>>>>>>>>>>>>>> fast/useful.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hope this helps,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> J-D
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Feb 24, 2016 at 3:08 PM, Benjamin Kim
>>>>>>>>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote:
>>>>>>>>>>>>>>>>>>> I see this KUDU-1214
>>>>>>>>>>>>>>>>>>> <https://issues.cloudera.org/browse/KUDU-1214> targeted for
>>>>>>>>>>>>>>>>>>> 0.8.0, but I see no progress on it. When this is complete,
>>>>>>>>>>>>>>>>>>> will this mean that Spark will be able to work with Kudu
>>>>>>>>>>>>>>>>>>> both programmatically and as a client via Spark SQL? Or is
>>>>>>>>>>>>>>>>>>> there more work that needs to be done on the Spark side for
>>>>>>>>>>>>>>>>>>> it to work?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Just curious.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>> Ben
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>
