Re: Re: KUDU Space is not Released

[Matthew Jacobs]

Can you elaborate how you're observing that "the local disk space is
not released"? Are you saying that the behavior is different when
using Impala vs. other Kudu clients?

On Tue, Aug 15, 2017 at 6:16 PM, sky <x_h...@163.com> wrote:
> No,it is a internal table, comand is "create table .. stored as kudu"
>
>
>
>
>
>
>
> At 2017-08-15 23:33:55, "Matthew Jacobs" <m...@cloudera.com> wrote:
>>Was it an external table?
>>
>>On Tue, Aug 15, 2017 at 1:29 AM sky <x_h...@163.com> wrote:
>>
>>> Hi all,
>>> When deleted the kudu table via impala-shell, I find that the local
>>> disk space is not released. How do I free up the space?
>>
>>--
>>Sent from my iPhone
>
>
>
>

Re: KUDU Space is not Released

[Matthew Jacobs]

Was it an external table?

On Tue, Aug 15, 2017 at 1:29 AM sky  wrote:

> Hi all,
> When deleted the kudu table via impala-shell, I find that the local
> disk space is not released. How do I free up the space?

-- 
Sent from my iPhone

Re: Kudu Scanner not Found

[Matthew Jacobs]

Please share the query profile and Impala daemon coordinator log.

On Sat, Aug 12, 2017 at 9:42 AM, sky  wrote:
>
> Hi all,
> Using command “create table  . stored as kudu select  from table” 
> to create kudu table , I get the errors:
> Query progress can be monitored at: 
> http://node.sky.org:25000/query_plan?query_id=b243c1b8987db9dd
> :b520b3be
> WARNINGS:
> Unable to advance iterator: Timed out: Scan RPC to 192.168.1.27:7065 timed 
> out after 0.000s (SENT)
> : Not found: Scanner not found
> Unable to advance iterator: Timed out: Scan RPC to 192.168.1.27:7065 timed 
> out after 0.000s (SENT)
> : Not found: Scanner not found
>
>
> How to do it ?
>

Re: kudu field type is not supported in Impala

[Matthew Jacobs]

In the version of Impala that you're using, unixtime_micros will not
be supported by Impala, though I don't know why the error is
complaining about 'bool'.

I can't reproduce this issue on Impala master (which now supports
UNIXTIME_MICROS):

import kudu
from kudu.client import Partitioning
builder = kudu.schema_builder()
builder.add_column('key').type(kudu.int64).nullable(False).primary_key()
builder.add_column('id1').type(kudu.int8).nullable(False)
builder.add_column('id2').type(kudu.int32).nullable(False)
builder.add_column('id3').type(kudu.int16).nullable(False)
builder.add_column('id4').type(kudu.float).nullable(False)
builder.add_column('id5').type(kudu.double).nullable(False)
builder.add_column('id6').type(kudu.bool).nullable(False)
builder.add_column('str', type_=kudu.string, nullable=False, compression='lz4')
builder.add_column('time', type_=kudu.unixtime_micros, nullable=False,
compression='lz4')
schema = builder.build()
partitioning = Partitioning().add_hash_partitions(column_names=['key'],
num_buckets=3)
c.create_table('foo', schema, partitioning)


In Impala:
[localhost:21000] > create external table foo stored as kudu TBLPROPERTIES (
  'kudu.table_name' = 'foo' );
Query: create external table foo stored as kudu TBLPROPERTIES (
  'kudu.table_name' = 'foo' )

Fetched 0 row(s) in 0.05s
[localhost:21000] > describe foo;
Query: describe foo
+--+---+-+-+--+---+---+-++
| name | type  | comment | primary_key | nullable | default_value
| encoding  | compression | block_size |
+--+---+-+-+--+---+---+-++
| key  | bigint| | true| false|
| AUTO_ENCODING | DEFAULT_COMPRESSION | 0  |
| id1  | tinyint   | | false   | false|
| AUTO_ENCODING | DEFAULT_COMPRESSION | 0  |
| id2  | int   | | false   | false|
| AUTO_ENCODING | DEFAULT_COMPRESSION | 0  |
| id3  | smallint  | | false   | false|
| AUTO_ENCODING | DEFAULT_COMPRESSION | 0  |
| id4  | float | | false   | false|
| AUTO_ENCODING | DEFAULT_COMPRESSION | 0  |
| id5  | double| | false   | false|
| AUTO_ENCODING | DEFAULT_COMPRESSION | 0  |
| id6  | boolean   | | false   | false|
| AUTO_ENCODING | DEFAULT_COMPRESSION | 0  |
| str  | string| | false   | false|
| AUTO_ENCODING | LZ4 | 0  |
| time | timestamp | | false   | false|
| AUTO_ENCODING | LZ4 | 0  |
+--+---+-+-+--+---+---+-++
Fetched 9 row(s) in 4.75s


On Thu, Jun 1, 2017 at 10:34 PM, x_hsky  wrote:
> impala versoin: cdh5.10.0
> kudu verson: 1.3.0-SNAPSHOT
>
>
> Create the table by  kudu api, it is ok:
> builder = kudu.schema_builder()
> builder.add_column('key').type(kudu.int64).nullable(False).primary_key()
> builder.add_column('id1').type(kudu.int8).nullable(False)
> builder.add_column('id2').type(kudu.int32).nullable(False)
> builder.add_column('id3').type(kudu.int16).nullable(False)
> builder.add_column('id4').type(kudu.float).nullable(False)
> builder.add_column('id5').type(kudu.double).nullable(False)
> builder.add_column('id6').type(kudu.bool).nullable(False)
> builder.add_column('str', type_=kudu.string, nullable=False, 
> compression='lz4')
> builder.add_column('time', type_=kudu.unixtime_micros, nullable=False, 
> compression='lz4')
> schema = builder.build()
>
>
> However, when I create a kudu table from impala-shell, I will report errors：
> [db.sky.org:21000] > create table test(id int primary key,id2 binary) 
> partition by hash partitions 8 stored as kudu;
> Query: create table test(id int primary key,id2 binary) partition by hash 
> partitions 8 stored as kudu
> ERROR: AnalysisException: Unsupported data type: BINARY
> [db.sky.org:21000] > create external table test stored as kudu 
> TBLPROPERTIES('kudu.table_name'='test_type');
> Query: create external table test stored as kudu 
> TBLPROPERTIES('kudu.table_name'='test_type')
> ERROR:
> ImpalaRuntimeException: Error loading schema of table 'test_type'
> CAUSED BY: ImpalaRuntimeException: Kudu type 'bool' is not supported in Impala
> [db.sky.org:21000] >
> I found that, in addition to integer and string type, the other kudu field 
> types can not be supported in the impala-shell.why ?
>

Re: kudu field type is not supported in impala

[Matthew Jacobs]

0) What versions of Impala and Kudu are you using?

1) BINARY is not supported by Impala. Use STRING instead.

2) Can you show how you created the table test_type in Kudu? I
wouldn't expect to have issues with boolean types.

On Thu, Jun 1, 2017 at 6:40 PM, x_hsky  wrote:
> Hi all:
> I can create the kudu table using the field type by kudu api,but it is 
> not supported by impala shell.why?
> Here are two examples：
>
>
> [db.sky.org:21000] >
> [db.sky.org:21000] > create table test(id int primary key,id2 binary) 
> partition by hash partitions 8 stored as kudu;
> Query: create table test(id int primary key,id2 binary) partition by hash 
> partitions 8 stored as kudu
> ERROR: AnalysisException: Unsupported data type: BINARY
>
>
> [db.sky.org:21000] >
> [db.sky.org:21000] > create external table test stored as kudu 
> TBLPROPERTIES('kudu.table_name'='test_type');
> Query: create external table test stored as kudu 
> TBLPROPERTIES('kudu.table_name'='test_type')
> ERROR:
> ImpalaRuntimeException: Error loading schema of table 'test_type'
> CAUSED BY: ImpalaRuntimeException: Kudu type 'bool' is not supported in Impala
>
>
> [db.sky.org:21000] >
>
>
>
>
>
>
>

Re: Proposal for 1.3 branch and release

[Matthew Jacobs]

+1

On Tue, Feb 21, 2017 at 11:22 AM, Dan Burkert  wrote:
> +1 SGTM, thanks!
>
> - Dan
>
> On Tue, Feb 21, 2017 at 11:19 AM, Hao Hao  wrote:
>
>> +1 Thanks a lot Todd for putting this together and volunteered to be the
>> RM.
>>
>> Best,
>> Hao
>>
>> On Tue, Feb 21, 2017 at 10:40 AM, Dave Wang  wrote:
>>
>> > Strong +1 on your proposal Todd.
>> >
>> > - Dave
>> >
>> > On Tue, Feb 21, 2017 at 10:39 AM, Jean-Daniel Cryans <
>> jdcry...@apache.org>
>> > wrote:
>> >
>> > > +1 on the plan and +1 on you being the RM since you're so close to all
>> > the
>> > > security-related patches.
>> > >
>> > > J-D
>> > >
>> > > On Tue, Feb 21, 2017 at 10:32 AM, Todd Lipcon  wrote:
>> > >
>> > > > Hi folks,
>> > > >
>> > > > I don't think the topic of Kudu 1.3 has been brought up on the dev
>> list
>> > > yet
>> > > > (though I just hinted at it on a user@ thread). I wanted to open
>> this
>> > > > topic
>> > > > to discussion by proposing a release plan.
>> > > >
>> > > > Back in December, the discussion around the 1.2 release was to hold
>> off
>> > > on
>> > > > the security features, and push them to 1.3. So, obviously security
>> has
>> > > > been a big focus of development over the last month or so. I've been
>> > > pretty
>> > > > involved with this development and it seems from my perspective that
>> > the
>> > > > initial phase of development (authentication without authorization)
>> is
>> > > > wrapping up, and would be good to get out to users.
>> > > >
>> > > > Aside from security, there are a bunch of good bug fixes and
>> > > improvements.
>> > > > From a quick scan of the git log, a few things of note are:
>> > > >
>> > > > - fixes to various cases where more disk space was consumed than
>> > actually
>> > > > used for data
>> > > > - support for using Breakpad for crash analysis
>> > > > - new default encodings/compression to save space and increase
>> > > performance
>> > > > - compression of the WAL and deltafiles (performance and space)
>> > > > - support for IS NULL and IS NOT NULL predicates
>> > > > - improved hash-partition pruning for IN-list predicates
>> > > >
>> > > > ...plus a few nice bug fixes and stability improvements.
>> > > >
>> > > > All of the above seems like it should make for a compelling 1.3
>> > release.
>> > > >
>> > > > Following the pattern of 1.2, I think we should plan to branch this
>> > week,
>> > > > and then spend a week or two on stabilization and testing before
>> making
>> > > the
>> > > > release candidate.
>> > > >
>> > > > I'll volunteer to be the RM again unles anyone else feels like
>> taking a
>> > > > turn.
>> > > >
>> > > > Any comments or concerns?
>> > > >
>> > > > -Todd
>> > > >
>> > > > --
>> > > > Todd Lipcon
>> > > > Software Engineer, Cloudera
>> > > >
>> > >
>> >
>>