Re: Spark on Kudu

[Benjamin Kim]

Thanks!

> On Sep 20, 2016, at 3:02 PM, Jordan Birdsell  
> wrote:
> 
> http://kudu.apache.org/docs/developing.html#_kudu_integration_with_spark 
> 
> 
> On Tue, Sep 20, 2016 at 5:00 PM Benjamin Kim  > wrote:
> I see that the API has changed a bit so my old code doesn’t work anymore. Can 
> someone direct me to some code samples?
> 
> Thanks,
> Ben
> 
> 
>> On Sep 20, 2016, at 1:44 PM, Todd Lipcon > > wrote:
>> 
>> On Tue, Sep 20, 2016 at 1:18 PM, Benjamin Kim > > wrote:
>> Now that Kudu 1.0.0 is officially out and ready for production use, where do 
>> we find the spark connector jar for this release?
>> 
>> 
>> It's available in the official ASF maven repository:  
>> https://repository.apache.org/#nexus-search;quick~kudu-spark 
>> 
>> 
>> 
>>   org.apache.kudu
>>   kudu-spark_2.10
>>   1.0.0
>> 
>> 
>> 
>> -Todd
>>  
>> 
>> 
>>> On Jun 17, 2016, at 11:08 AM, Dan Burkert >> > wrote:
>>> 
>>> Hi Ben,
>>> 
>>> To your first question about `CREATE TABLE` syntax with Kudu/Spark SQL, I 
>>> do not think we support that at this point.  I haven't looked deeply into 
>>> it, but we may hit issues specifying Kudu-specific options (partitioning, 
>>> column encoding, etc.).  Probably issues that can be worked through 
>>> eventually, though.  If you are interested in contributing to Kudu, this is 
>>> an area that could obviously use improvement!  Most or all of our Spark 
>>> features have been completely community driven to date.
>>>  
>>> I am assuming that more Spark support along with semantic changes below 
>>> will be incorporated into Kudu 0.9.1.
>>> 
>>> As a rule we do not release new features in patch releases, but the good 
>>> news is that we are releasing regularly, and our next scheduled release is 
>>> for the August timeframe (see JD's roadmap 
>>> 
>>>  email about what we are aiming to include).  Also, Cloudera does publish 
>>> snapshot versions of the Spark connector here 
>>> , so 
>>> the jars are available if you don't mind using snapshots.
>>>  
>>> Anyone know of a better way to make unique primary keys other than using 
>>> UUID to make every row unique if there is no unique column (or combination 
>>> thereof) to use.
>>> 
>>> Not that I know of.  In general it's pretty rare to have a dataset without 
>>> a natural primary key (even if it's just all of the columns), but in those 
>>> cases UUID is a good solution.
>>>  
>>> This is what I am using. I know auto incrementing is coming down the line 
>>> (don’t know when), but is there a way to simulate this in Kudu using Spark 
>>> out of curiosity?
>>> 
>>> To my knowledge there is no plan to have auto increment in Kudu.  
>>> Distributed, consistent, auto incrementing counters is a difficult problem, 
>>> and I don't think there are any known solutions that would be fast enough 
>>> for Kudu (happy to be proven wrong, though!).
>>> 
>>> - Dan
>>>  
>>> 
>>> Thanks,
>>> Ben
>>> 
 On Jun 14, 2016, at 6:08 PM, Dan Burkert > wrote:
 
 I'm not sure exactly what the semantics will be, but at least one of them 
 will be upsert.  These modes come from spark, and they were really 
 designed for file-backed storage and not table storage.  We may want to do 
 append = upsert, and overwrite = truncate + insert.  I think that may 
 match the normal spark semantics more closely.
 
 - Dan
 
 On Tue, Jun 14, 2016 at 6:00 PM, Benjamin Kim > wrote:
 Dan,
 
 Thanks for the information. That would mean both “append” and “overwrite” 
 modes would be combined or not needed in the future.
 
 Cheers,
 Ben
 
> On Jun 14, 2016, at 5:57 PM, Dan Burkert  > wrote:
> 
> Right now append uses an update Kudu operation, which requires the row 
> already be present in the table. Overwrite maps to insert.  Kudu very 
> recently got upsert support baked in, but it hasn't yet been integrated 
> into the Spark connector.  So pretty soon these sharp edges will get a 
> lot better, since upsert is the way to go for most spark workloads.
> 
> - Dan
> 
> On Tue, Jun 14, 2016 at 5:41 PM, Benjamin Kim  > wrote:
> I tried to use the “append” mode, and it worked. Over 3.8 million rows in 
> 64s. I would 

Re: [ANNOUNCE] Apache Kudu 1.0.0 release

[Benjamin Kim]

Todd,

Thanks. I’ll look into those.

Cheers,
Ben


> On Sep 20, 2016, at 12:11 AM, Todd Lipcon  wrote:
> 
> The Apache Kudu team is happy to announce the release of Kudu 1.0.0!
> 
> Kudu is an open source storage engine for structured data which supports 
> low-latency random access together with efficient analytical access patterns. 
> It is designed within the context of the Apache Hadoop ecosystem and supports 
> many integrations with other data analytics projects both inside and outside 
> of the Apache Software Foundation.
> 
> This latest version adds several new features, including:
> 
> - Removal of multiversion concurrency control (MVCC) history is now 
> supported. This allows Kudu to reclaim disk space, where previously Kudu 
> would keep a full history of all changes made to a given table since the 
> beginning of time.
> 
> - Most of Kudu’s command line tools have been consolidated under a new 
> top-level "kudu" tool. This reduces the number of large binaries distributed 
> with Kudu and also includes much-improved help output.
> 
> - Administrative tools including "kudu cluster ksck" now support running 
> against multi-master Kudu clusters.
> 
> - The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND mode. 
> This can provide higher throughput for ingest workloads.
> 
> This release also includes many bug fixes, optimizations, and other 
> improvements, detailed in the release notes available at:
> http://kudu.apache.org/releases/1.0.0/docs/release_notes.html 
> 
> 
> Download the source release here:
> http://kudu.apache.org/releases/1.0.0/ 
> 
> 
> Convenience binary artifacts for the Java client and various Java 
> integrations (eg Spark, Flume) are also now available via the ASF Maven 
> repository.
> 
> Enjoy the new release!
> 
> - The Apache Kudu team

Re: [ANNOUNCE] Apache Kudu 1.0.0 release

[Todd Lipcon]

-announce


On Tue, Sep 20, 2016 at 11:34 AM, Benjamin Kim  wrote:

> This is awesome!!! Great!!!
>
> Do you know if any improvements were also made to the Spark plugin jar?
>

Looks like a few changes based on the git log:
https://gist.github.com/4fa3ccc3b9be787227fed89c1bd42837

as well as a number of changes to the Java client (which gets pulled into
the Spark jar):
https://gist.github.com/e2a8ca78e51773fabb70aae34207199f


In particular, I think the partition pruning work in the Java client should
reduce the number of Spark partitions if you have predicates on your data
frames. (though I haven't personally verified it)

-Todd



> On Sep 20, 2016, at 12:11 AM, Todd Lipcon  wrote:
>
> The Apache Kudu team is happy to announce the release of Kudu 1.0.0!
>
> Kudu is an open source storage engine for structured data which supports
> low-latency random access together with efficient analytical access
> patterns. It is designed within the context of the Apache Hadoop ecosystem
> and supports many integrations with other data analytics projects both
> inside and outside of the Apache Software Foundation.
>
> This latest version adds several new features, including:
>
> - Removal of multiversion concurrency control (MVCC) history is now
> supported. This allows Kudu to reclaim disk space, where previously Kudu
> would keep a full history of all changes made to a given table since the
> beginning of time.
>
> - Most of Kudu’s command line tools have been consolidated under a new
> top-level "kudu" tool. This reduces the number of large binaries
> distributed with Kudu and also includes much-improved help output.
>
> - Administrative tools including "kudu cluster ksck" now support running
> against multi-master Kudu clusters.
>
> - The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND
> mode. This can provide higher throughput for ingest workloads.
>
> This release also includes many bug fixes, optimizations, and other
> improvements, detailed in the release notes available at:
> http://kudu.apache.org/releases/1.0.0/docs/release_notes.html
>
> Download the source release here:
> http://kudu.apache.org/releases/1.0.0/
>
> Convenience binary artifacts for the Java client and various Java
> integrations (eg Spark, Flume) are also now available via the ASF Maven
> repository.
>
> Enjoy the new release!
>
> - The Apache Kudu team
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Re: [ANNOUNCE] Apache Kudu 1.0.0 release

[Benjamin Kim]

This is awesome!!! Great!!!

Do you know if any improvements were also made to the Spark plugin jar?

Thanks,
Ben

> On Sep 20, 2016, at 12:11 AM, Todd Lipcon  wrote:
> 
> The Apache Kudu team is happy to announce the release of Kudu 1.0.0!
> 
> Kudu is an open source storage engine for structured data which supports 
> low-latency random access together with efficient analytical access patterns. 
> It is designed within the context of the Apache Hadoop ecosystem and supports 
> many integrations with other data analytics projects both inside and outside 
> of the Apache Software Foundation.
> 
> This latest version adds several new features, including:
> 
> - Removal of multiversion concurrency control (MVCC) history is now 
> supported. This allows Kudu to reclaim disk space, where previously Kudu 
> would keep a full history of all changes made to a given table since the 
> beginning of time.
> 
> - Most of Kudu’s command line tools have been consolidated under a new 
> top-level "kudu" tool. This reduces the number of large binaries distributed 
> with Kudu and also includes much-improved help output.
> 
> - Administrative tools including "kudu cluster ksck" now support running 
> against multi-master Kudu clusters.
> 
> - The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND mode. 
> This can provide higher throughput for ingest workloads.
> 
> This release also includes many bug fixes, optimizations, and other 
> improvements, detailed in the release notes available at:
> http://kudu.apache.org/releases/1.0.0/docs/release_notes.html 
> 
> 
> Download the source release here:
> http://kudu.apache.org/releases/1.0.0/ 
> 
> 
> Convenience binary artifacts for the Java client and various Java 
> integrations (eg Spark, Flume) are also now available via the ASF Maven 
> repository.
> 
> Enjoy the new release!
> 
> - The Apache Kudu team

Re: Casual meetup/happy hour at Strata?

[Todd Lipcon]

Sounds like there's enough interest (I got a few other people ping me via
Slack).

Any locals have a good suggestion of a spot to meet? Probably somewhere
near the Javits would be easiest for everyone, though I'd be happy to go
elsewhere as well if anyone knows a nice spot that wouldn't be too crowded
during the conference.

-Todd

On Sat, Sep 17, 2016 at 7:12 PM, Clifford Resnick 
wrote:

> +1. We're just starting with Kudu, but it would be nice to meet other
> users, and a casual Q & A would be great if you're up for it!
>
> On Sep 17, 2016 9:05 PM, Todd Lipcon  wrote:
>
> Hey Kudu users,
>
> I'll be in NYC for the last week in September for Strata/Hadoop World. I
> imagine some others might be as well, and wanted to gauge interest in doing
> a casual meetup or happy hour on Wednesday night of that week. No
> presentations or anything, just pick a time and place and whoever's around
> can drop by and put some faces to names.
>
> Let me know if you're interested - if not enough people are around, I'll
> can the idea, but if it seems there are at least a few people in town it
> might be fun.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Re: [ANNOUNCE] Apache Kudu 1.0.0 release

[Aminul Islam]

Congrats
On Sep 20, 2016 9:35 PM, "Jacques Nadeau"  wrote:

> Congrats to everyone. This is a great accomplishment!
>
> On Tue, Sep 20, 2016 at 12:11 AM, Todd Lipcon  wrote:
>
>> The Apache Kudu team is happy to announce the release of Kudu 1.0.0!
>>
>> Kudu is an open source storage engine for structured data which supports
>> low-latency random access together with efficient analytical access
>> patterns. It is designed within the context of the Apache Hadoop ecosystem
>> and supports many integrations with other data analytics projects both
>> inside and outside of the Apache Software Foundation.
>>
>> This latest version adds several new features, including:
>>
>> - Removal of multiversion concurrency control (MVCC) history is now
>> supported. This allows Kudu to reclaim disk space, where previously Kudu
>> would keep a full history of all changes made to a given table since the
>> beginning of time.
>>
>> - Most of Kudu’s command line tools have been consolidated under a new
>> top-level "kudu" tool. This reduces the number of large binaries
>> distributed with Kudu and also includes much-improved help output.
>>
>> - Administrative tools including "kudu cluster ksck" now support running
>> against multi-master Kudu clusters.
>>
>> - The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND
>> mode. This can provide higher throughput for ingest workloads.
>>
>> This release also includes many bug fixes, optimizations, and other
>> improvements, detailed in the release notes available at:
>> http://kudu.apache.org/releases/1.0.0/docs/release_notes.html
>>
>> Download the source release here:
>> http://kudu.apache.org/releases/1.0.0/
>>
>> Convenience binary artifacts for the Java client and various Java
>> integrations (eg Spark, Flume) are also now available via the ASF Maven
>> repository.
>>
>> Enjoy the new release!
>>
>> - The Apache Kudu team
>>
>
>

Re: [ANNOUNCE] Apache Kudu 1.0.0 release

[Jacques Nadeau]

Congrats to everyone. This is a great accomplishment!

On Tue, Sep 20, 2016 at 12:11 AM, Todd Lipcon  wrote:

> The Apache Kudu team is happy to announce the release of Kudu 1.0.0!
>
> Kudu is an open source storage engine for structured data which supports
> low-latency random access together with efficient analytical access
> patterns. It is designed within the context of the Apache Hadoop ecosystem
> and supports many integrations with other data analytics projects both
> inside and outside of the Apache Software Foundation.
>
> This latest version adds several new features, including:
>
> - Removal of multiversion concurrency control (MVCC) history is now
> supported. This allows Kudu to reclaim disk space, where previously Kudu
> would keep a full history of all changes made to a given table since the
> beginning of time.
>
> - Most of Kudu’s command line tools have been consolidated under a new
> top-level "kudu" tool. This reduces the number of large binaries
> distributed with Kudu and also includes much-improved help output.
>
> - Administrative tools including "kudu cluster ksck" now support running
> against multi-master Kudu clusters.
>
> - The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND
> mode. This can provide higher throughput for ingest workloads.
>
> This release also includes many bug fixes, optimizations, and other
> improvements, detailed in the release notes available at:
> http://kudu.apache.org/releases/1.0.0/docs/release_notes.html
>
> Download the source release here:
> http://kudu.apache.org/releases/1.0.0/
>
> Convenience binary artifacts for the Java client and various Java
> integrations (eg Spark, Flume) are also now available via the ASF Maven
> repository.
>
> Enjoy the new release!
>
> - The Apache Kudu team
>

[ANNOUNCE] Apache Kudu 1.0.0 release

[Todd Lipcon]

The Apache Kudu team is happy to announce the release of Kudu 1.0.0!

Kudu is an open source storage engine for structured data which supports
low-latency random access together with efficient analytical access
patterns. It is designed within the context of the Apache Hadoop ecosystem
and supports many integrations with other data analytics projects both
inside and outside of the Apache Software Foundation.

This latest version adds several new features, including:

- Removal of multiversion concurrency control (MVCC) history is now
supported. This allows Kudu to reclaim disk space, where previously Kudu
would keep a full history of all changes made to a given table since the
beginning of time.

- Most of Kudu’s command line tools have been consolidated under a new
top-level "kudu" tool. This reduces the number of large binaries
distributed with Kudu and also includes much-improved help output.

- Administrative tools including "kudu cluster ksck" now support running
against multi-master Kudu clusters.

- The C++ client API now supports writing data in AUTO_FLUSH_BACKGROUND
mode. This can provide higher throughput for ingest workloads.

This release also includes many bug fixes, optimizations, and other
improvements, detailed in the release notes available at:
http://kudu.apache.org/releases/1.0.0/docs/release_notes.html

Download the source release here:
http://kudu.apache.org/releases/1.0.0/

Convenience binary artifacts for the Java client and various Java
integrations (eg Spark, Flume) are also now available via the ASF Maven
repository.

Enjoy the new release!

- The Apache Kudu team