http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/docs/xdocs/language_manual/data-manipulation-statements.xml ---------------------------------------------------------------------- diff --git a/docs/xdocs/language_manual/data-manipulation-statements.xml b/docs/xdocs/language_manual/data-manipulation-statements.xml deleted file mode 100644 index 214a0dc..0000000 --- a/docs/xdocs/language_manual/data-manipulation-statements.xml +++ /dev/null @@ -1,234 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. ---> - -<document> - - <properties> - <title>Hadoop Hive- Data Manipulation Statements</title> - <author email="[email protected]">Hadoop Hive Documentation Team</author> - </properties> - - <body> - - <section name="Create Table Syntax" href="create_table_syntax"> - - <source><![CDATA[ -CREATE [EXTERNAL] TABLE [IF NOT EXISTS] table_name - [(col_name data_type [COMMENT col_comment], ...)] - [COMMENT table_comment] - [PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)] - [CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS] - [ROW FORMAT row_format] - [STORED AS file_format] - [LOCATION hdfs_path] - [TBLPROPERTIES (property_name=property_value, ...)] - [AS select_statement] - -CREATE [EXTERNAL] TABLE [IF NOT EXISTS] table_name - LIKE existing_table_name - [LOCATION hdfs_path] - -data_type - : primitive_type - | array_type - | map_type - | struct_type - -primitive_type - : TINYINT - | SMALLINT - | INT - | BIGINT - | BOOLEAN - | FLOAT - | DOUBLE - | STRING - -array_type - : ARRAY < data_type > - -map_type - : MAP < primitive_type, data_type > - -struct_type - : STRUCT < col_name : data_type [COMMENT col_comment], ...> - -row_format - : DELIMITED [FIELDS TERMINATED BY char] [COLLECTION ITEMS TERMINATED BY char] - [MAP KEYS TERMINATED BY char] [LINES TERMINATED BY char] - | SERDE serde_name [WITH SERDEPROPERTIES (property_name=property_value, property_name=property_value, ...)] - -file_format: - : SEQUENCEFILE - | TEXTFILE - | INPUTFORMAT input_format_classname OUTPUTFORMAT output_format_classname -]]></source> - -<p> -CREATE TABLE creates a table with the given name. An error is thrown if a table or view with the same name already exists. You can use IF NOT EXISTS to skip the error. -</p> - -<p> -The EXTERNAL keyword lets you create a table and provide a LOCATION so that Hive does not use a default location for this table. This comes in handy if you already have data generated. When dropping an EXTERNAL table, data in the table is NOT deleted from the file system. -</p> -The LIKE form of CREATE TABLE allows you to copy an existing table definition exactly (without copying its data). - -<p> -You can create tables with custom SerDe or using native SerDe. A native SerDe is used if ROW FORMAT is not specified or ROW FORMAT DELIMITED is specified. You can use the DELIMITED clause to read delimited files. Use the SERDE clause to create a table with custom SerDe. Refer to SerDe section of the User Guide for more information on SerDe. -</p> - -<p> -You must specify a list of a columns for tables that use a native SerDe. Refer to the Types part of the User Guide for the allowable column types. A list of columns for tables that use a custom SerDe may be specified but Hive will query the SerDe to determine the actual list of columns for this table. -</p> - -<p> -Use STORED AS TEXTFILE if the data needs to be stored as plain text files. Use STORED AS SEQUENCEFILE if the data needs to be compressed. Please read more about Hive/CompressedStorage if you are planning to keep data compressed in your Hive tables. Use INPUTFORMAT and OUTPUTFORMAT to specify the name of a corresponding InputFormat and OutputFormat class as a string literal, e.g. 'org.apache.hadoop.hive.contrib.fileformat.base64.Base64TextInputFormat'. -</p> - -<p> -Partitioned tables can be created using the PARTITIONED BY clause. A table can have one or more partition columns and a separate data directory is created for each distinct value combination in the partition columns. Further, tables or partitions can be bucketed using CLUSTERED BY columns, and data can be sorted within that bucket via SORT BY columns. This can improve performance on certain kinds of queries. -</p> - -<p> -Table names and column names are case insensitive but SerDe and property names are case sensitive. Table and column comments are string literals (single-quoted). The TBLPROPERTIES clause allows you to tag the table definition with your own metadata key/value pairs. -</p> - -<p>A create table example:</p> - <source><![CDATA[CREATE TABLE page_view(viewTime INT, userid BIGINT, - page_url STRING, referrer_url STRING, - ip STRING COMMENT 'IP Address of the User') - COMMENT 'This is the page view table' - PARTITIONED BY(dt STRING, country STRING) - STORED AS SEQUENCEFILE;]]></source> - - <p>The statement above creates the page_view table with viewTime, userid, page_url, referrer_url, and ip columns (including comments). The table is also partitioned and data is stored in sequence files. The data format in the files is assumed to be field-delimited by ctrl-A and row-delimited by newline. - </p> - -</section> - -<section name="Create Table as Select (CTAS)" href="ctas?"> - - <p> - Tables can also be created and populated by the results of a query in one create-table-as-select (CTAS) statement. The table created by CTAS is atomic, meaning that the table is not seen by other users until all the query results are populated. So other users will either see the table with the complete results of the query or will not see the table at all. - </p> - - <p> - There are two parts in CTAS, the SELECT part can be any SELECT statement supported by HiveQL. The CREATE part of the CTAS takes the resulting schema from the SELECT part and creates the target table with other table properties such as the SerDe and storage format. The only restrictions in CTAS is that the target table cannot be a partitioned table (nor can it be an external table). - </p> - - <source><![CDATA[CREATE TABLE page_view(viewTime INT, userid BIGINT, - page_url STRING, referrer_url STRING, - ip STRING COMMENT 'IP Address of the User') - COMMENT 'This is the page view table' - PARTITIONED BY(dt STRING, country STRING) - STORED AS SEQUENCEFILE; -]]></source> - -</section> - -<section name="Using SerDes" href="SerDes"> - -<p> -This example CTAS statement creates the target table new_key_value_store with the -schema (new_key DOUBLE, key_value_pair STRING) derived from the results of the -SELECT statement. If the SELECT statement does not specify column aliases, the -column names will be automatically assigned to _col0, _col1, and _col2 etc. -In addition, the new target table is created using a specific SerDe and a storage -format independent of the source tables in the SELECT statement. -</p> - -<source><![CDATA[CREATE TABLE new_key_value_store - ROW FORMAT SERDE "org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe" - STORED AS RCFile AS -SELECT (key % 1024) new_key, concat(key, value) key_value_pair -FROM key_value_store -SORT BY new_key, key_value_pair; -]]></source> - -<p> -<b>Being able to select data from one table to another is one of the most -powerful features of Hive. Hive handles the conversion of the data from the source -format to the destination format as the query is being executed!</b> -</p> - -</section> - -<section name="Bucketed Sorted Table" href="bucketed_sorted_table"> - -<source><![CDATA[CREATE TABLE page_view(viewTime INT, userid BIGINT, - page_url STRING, referrer_url STRING, - ip STRING COMMENT 'IP Address of the User') - COMMENT 'This is the page view table' - PARTITIONED BY(dt STRING, country STRING) - CLUSTERED BY(userid) SORTED BY(viewTime) INTO 32 BUCKETS - ROW FORMAT DELIMITED - FIELDS TERMINATED BY '\001' - COLLECTION ITEMS TERMINATED BY '\002' - MAP KEYS TERMINATED BY '\003' - STORED AS SEQUENCEFILE; -]]></source> - -<p>In the example above, the page_view table is bucketed (clustered by) userid and within each bucket the data is sorted in increasing order of viewTime. Such an organization allows the user to do efficient sampling on the clustered column - in this case userid. The sorting property allows internal operators to take advantage of the better-known data structure while evaluating queries, also increasing efficiency. MAP KEYS and COLLECTION ITEMS keywords can be used if any of the columns are lists or maps. -</p> - -<p> -The CLUSTERED BY and SORTED BY creation commands do not affect how data is inserted into a table -- only how it is read. This means that users must be careful to insert data correctly by specifying the number of reducers to be equal to the number of buckets, and using CLUSTER BY and SORT BY commands in their query. See -<a href="working_with_bucketed_tables.html">Working with Bucketed tables</a> to see how these -are used. -</p> - -</section> - -<section name="External Tables" href="external_table?"> - -<p> -Unless a table is specified as EXTERNAL it will be stored inside a folder specified by the -configuration property hive.metastore.warehouse.dir. -EXTERNAL tables points to any hdfs location for its storage. You still have to make sure that the data is format is specified to match the data. - -</p> -<source><![CDATA[CREATE EXTERNAL TABLE page_view(viewTime INT, userid BIGINT, - page_url STRING, referrer_url STRING, - ip STRING COMMENT 'IP Address of the User', - country STRING COMMENT 'country of origination') - COMMENT 'This is the staging page view table' - ROW FORMAT DELIMITED FIELDS TERMINATED BY '\054' - STORED AS TEXTFILE - LOCATION '<hdfs_location>'; - ]]></source> - -</section> - -<section name="Create Table ... Like" href="create_table_like?"> - -<p>The statement above creates a new empty_key_value_store table whose definition exactly matches the existing key_value_store in all particulars other than table name. The new table contains no rows. -</p> - -<source><![CDATA[CREATE TABLE empty_key_value_store -LIKE key_value_store; -]]></source> - -</section> - -<section name="drop" href="drop"> -<p>Drop it like it is hot</p> -</section> - </body> -</document>
http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/docs/xdocs/language_manual/joins.xml ---------------------------------------------------------------------- diff --git a/docs/xdocs/language_manual/joins.xml b/docs/xdocs/language_manual/joins.xml deleted file mode 100644 index 190ecd4..0000000 --- a/docs/xdocs/language_manual/joins.xml +++ /dev/null @@ -1,212 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. ---> - -<document> - - <properties> - <title>Hadoop Hive- Joins</title> - <author email="[email protected]">Hadoop Hive Documentation Team</author> - </properties> - - <body> - - <section name="Join Syntax" href="join_syntax"> - - <source><![CDATA[join_table: - table_reference [INNER] JOIN table_factor [join_condition] - | table_reference {LEFT|RIGHT|FULL} [OUTER] JOIN table_reference join_condition - | table_reference LEFT SEMI JOIN table_reference join_condition - -table_reference: - table_factor - | join_table - -table_factor: - tbl_name [alias] - | table_subquery alias - | ( table_references ) - -join_condition: - ON equality_expression ( AND equality_expression )* - -equality_expression: - expression = expression -]]></source> - -<p> -Only equality joins, outer joins, and left semi joins are supported in Hive. Hive does not support join conditions that are not equality conditions as it is very difficult to express such conditions as a map/reduce job. Also, more than two tables can be joined in Hive. -</p> - -<b>Allowed Equality Joins</b> - -<source><![CDATA[SELECT a.* FROM a JOIN b ON (a.id = b.id) -]]></source> - -<source><![CDATA[SELECT a.* FROM a JOIN b ON (a.id = b.id AND a.department = b.department) -]]></source> - -<b>Disallowed Joins</b> - -<source><![CDATA[SELECT a.* FROM a JOIN b ON (a.id <> b.id) -]]></source> - -<p>Multiple Tables can be joined in the same query</p> - -<source><![CDATA[SELECT a.val, b.val, c.val FROM a JOIN b ON (a.key = b.key1) JOIN c ON (c.key = b.key2) -]]></source> - - - -<source><![CDATA[ -]]></source> - -</section> - -<section name="Join implementation with Map Reduce" href="join_map_reduce"> - -<p>Hive converts joins over multiple tables into a single map/reduce job if for every table the same column is used in the join clauses. The query below is -converted into a single map/reduce job as only key1 column for b is involved in the join.</p> - -<source><![CDATA[SELECT a.val, b.val, c.val FROM a JOIN b ON (a.key = b.key1) JOIN c ON (c.key = b.key1)]]></source> -<i>It is very interesting to note that any number of tables can be joined in single map/reduce process as long as they fit the above criteria.</i> - -<p>However if the join colums are not the same for all tables the is converted into multiple map/reduce jobs</p> - -<source><![CDATA[SELECT a.val, b.val, c.val FROM a JOIN b ON (a.key = b.key1) JOIN c ON (c.key = b.key2) -]]></source> - -<p>In this case the first map/reduce job joins a with b and the results are then joined with c in the second map/reduce job. </p> -</section> - -<section name="Largest Table LAST" href="lagest_table_last"> - -<p>In every map/reduce stage of the join, the last table in the sequence is streamed through the reducers where as the others are buffered. Therefore, it helps to reduce the memory needed in the reducer for buffering the rows for a particular value of the join key by organizing the tables such that the largest tables appear last in the sequence. e.g. in</p> - -<source><![CDATA[SELECT a.val, b.val, c.val FROM a JOIN b ON (a.key = b.key1) JOIN c ON (c.key = b.key1)]]></source> - -<p>all the three tables are joined in a single map/reduce job and the values for a particular value of the key for tables a and b are buffered in the memory in the reducers. Then for each row retrieved from c, the join is computed with the buffered rows.</p> - -<p>For the query:</p> - -<source><![CDATA[SELECT a.val, b.val, c.val FROM a JOIN b ON (a.key = b.key1) JOIN c ON (c.key = b.key2)]]></source> - -<p> * there are two map/reduce jobs involved in computing the join. The first of these joins a with b and buffers the values of a while streaming the values of b in the reducers. The second of one of these jobs buffers the results of the first join while streaming the values of c through the reducers. </p> - -</section> - -<section name="Streamtable hint" href="stream_table_hint"> - -<p>In every map/reduce stage of the join, the table to be streamed can be specified via a hint:</p> - -<source><![CDATA[SELECT /*+ STREAMTABLE(a) */ a.val, b.val, c.val -FROM a JOIN b ON (a.key = b.key1) JOIN c ON (c.key = b.key1)]]></source> - -<p>All the three tables are joined in a single map/reduce job and the values for a particular value of the key for tables b and c are buffered in the memory in the reducers. Then for each row retrieved from a, the join is computed with the buffered rows. -</p> - -</section> - -<section name="Outer Joins" href="outer_joins"> - -<p>LEFT, RIGHT, and FULL OUTER joins exist in order to provide more control over ON clauses for which there is no match. For example:</p> - -<source><![CDATA[SELECT a.val, b.val FROM a LEFT OUTER JOIN b ON (a.key=b.key) -]]></source> - -<p>The above query will return a row for every row in a. This output row will be a.val,b.val when there is a b.key that equals a.key, and the output row will be a.val,NULL when there is no corresponding b.key. Rows from b which have no corresponding a.key will be dropped. The syntax "FROM a LEFT OUTER JOIN b" must be written on one line in order to understand how it works--a is to the LEFT of b in this query, and so all rows from a are kept; a RIGHT OUTER JOIN will keep all rows from b, and a FULL OUTER JOIN will keep all rows from a and all rows from b. OUTER JOIN semantics should conform to standard SQL specs. -</p> - -<p>Joins occur BEFORE WHERE CLAUSES. So, if you want to restrict the OUTPUT of a join, a requirement should be in the WHERE clause, otherwise it should be in the JOIN clause. A big point of confusion for this issue is partitioned tables</p> - -<source><![CDATA[SELECT a.val, b.val FROM a LEFT OUTER JOIN b ON (a.key=b.key) - WHERE a.ds='2009-07-07' AND b.ds='2009-07-07']]></source> - -<p>will join a on b, producing a list of a.val and b.val. The WHERE clause, however, can also reference other columns of a and b that are in the output of the join, and then filter them out. However, whenever a row from the JOIN has found a key for a and no key for b, all of the columns of b will be NULL, including the ds column. This is to say, you will filter out all rows of join output for which there was no valid b.key, and thus you have outsmarted your LEFT OUTER requirement. In other words, the LEFT OUTER part of the join is irrelevant if you reference any column of b in the WHERE clause. Instead, when OUTER JOINing, use this syntax:</p> - -<source><![CDATA[SELECT a.val, b.val FROM a LEFT OUTER JOIN b - ON (a.key=b.key AND b.ds='2009-07-07' AND a.ds='2009-07-07')]]></source> - -<p>Joins are NOT commutative! Joins are left-associative regardless of whether they are LEFT or RIGHT joins. </p> - -<source><![CDATA[SELECT a.val1, a.val2, b.val, c.val -FROM a -JOIN b ON (a.key = b.key) -LEFT OUTER JOIN c ON (a.key = c.key)]]></source> - -<p>The above query first joins a on b, throwing away everything in a or b that does not have a corresponding key in the other table. The reduced table is then joined on c. This provides unintuitive results if there is a key that exists in both a and c, but not b: The whole row (including a.val1, a.val2, and a.key) is dropped in the "a JOIN b" step, so when the result of that is joined with c, any row with a c.key that had a corresponding a.key or b.key (but not both) will show up as NULL, NULL, NULL, c.val.</p> -</section> - -<section name="Left Semi Join" href="left_semi_join"> - -<p>LEFT SEMI JOIN implements the correlated IN/EXISTS subquery semantics in an efficient way. Since Hive currently does not support IN/EXISTS subqueries, you can rewrite your queries using LEFT SEMI JOIN. The restrictions of using LEFT SEMI JOIN is that the right-hand-side table should only be referenced in the join condition (ON-clause), but not in WHERE- or SELECT-clauses etc.</p> - -<p>This type of query</p> -<source><![CDATA[SELECT a.key, a.value -FROM a -WHERE a.key in -(SELECT b.key -FROM B);]]></source> - -<p>Can be written as:</p> - -<source><![CDATA[SELECT a.key, a.val -FROM a LEFT SEMI JOIN b on (a.key = b.key)]]></source> - -</section> - -<section name="Map Side Join" href="map_side_join"> - -<p>If all but one of the tables being joined are small, the join can be performed as a map only job. The query -does not need a reducer. For every mapper a,b is read completely. A restriction is that a <b>FULL/RIGHT OUTER JOIN b</b> cannot be performed. </p> - -<source><![CDATA[SELECT /*+ MAPJOIN(b) */ a.key, a.value -FROM a join b on a.key = b.key]]></source> - -</section> - -<section name="Bucketed Map Join" href="bucket_map_join"> - -<p>If the tables being joined are bucketized, and the buckets are a multiple of each other, the buckets can be joined with each other. If table A has 8 buckets are table B has 4 buckets, the following join:</p> - -<source><![CDATA[SELECT /*+ MAPJOIN(b) */ a.key, a.value -FROM a join b on a.key = b.key]]></source> - -<p>can be done on the mapper only. Instead of fetching B completely for each mapper of A, only the required buckets are fetched. For the query above, the mapper processing bucket 1 for A will only fetch bucket 1 of B. It is not the default behavior, and is governed by the following parameter </p> - -<i>set hive.optimize.bucketmapjoin = true</i> - -<p>If the tables being joined are sorted and bucketized, and the number of buckets are same, a sort-merge join can be performed. The corresponding buckets are joined with each other at the mapper. If both A and B have 4 buckets</p> - -<source><![CDATA[ SELECT /*+ MAPJOIN(b) */ a.key, a.value -FROM A a join B b on a.key = b.key]]></source> - -<p>can be done on the mapper only. The mapper for the bucket for A will traverse the corresponding bucket for B. This is not the default behavior, and the following parameters need to be set:</p> - -<source><![CDATA[set hive.input.format=org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat; -set hive.optimize.bucketmapjoin = true; -set hive.optimize.bucketmapjoin.sortedmerge = true;]]></source> - -</section> - -</body> - - - -</document> http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/docs/xdocs/language_manual/var_substitution.xml ---------------------------------------------------------------------- diff --git a/docs/xdocs/language_manual/var_substitution.xml b/docs/xdocs/language_manual/var_substitution.xml deleted file mode 100644 index 3f3b04b..0000000 --- a/docs/xdocs/language_manual/var_substitution.xml +++ /dev/null @@ -1,130 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. ---> - -<document> - - <properties> - <title>Hadoop Hive- Variable Substitution</title> - <author email="[email protected]">Hadoop Hive Documentation Team</author> - </properties> - - <body> -<h3>Hive Variable Substitution</h3> -<section name="Introduction" href="Introduction"> - -<p>Hive is used for both interactive queries as well as part. The hive variable substitution mechanism was -designed to avoid some of the code that was getting baked into the scripting language ontop of hive. For example:</p> - -<source><![CDATA[$ a=b -$ hive -e " describe $a " -]]></source> - -<p> -are becoming common place. This is frustrating as hive becomes closely coupled with scripting languages. The hive -startup time of a couple seconds is non-trivial when doing thousands of manipulations multiple hive -e invocations.</p> - -<p> -Hive Variables combine the set capability you know and love with some limited yet powerful (evil laugh) substitution -ability. For example:</p> - -<source><![CDATA[$ bin/hive -hiveconf a=b -e 'set a; set hiveconf:a; \ -create table if not exists b (col int); describe ${hiveconf:a}' -]]></source> - -<p>Results in:</p> -<source><![CDATA[Hive history file=/tmp/edward/hive_job_log_edward_201011240906_1463048967.txt -a=b -hiveconf:a=b -OK -Time taken: 5.913 seconds -OK -col int -Time taken: 0.754 seconds -]]></source> - -</section> - -<section name="Using variables" href="using_variables"> - -<p>There are three namespaces for variables hiveconf,system, and env. hiveconf variables are set as normal:</p> - -<source><![CDATA[set x=myvalue -]]></source> - -<p>However they are retrieved using</p> - -<source><![CDATA[${hiveconf:x} -]]></source> - -<p>Annotated examples of usage from the test case ql/src/test/queries/clientpositive/set_processor_namespaces.q</p> - -<source><![CDATA[set zzz=5; --- sets zzz=5 -set zzz; - -set system:xxx=5; -set system:xxx; --- sets a system property xxx to 5 - -set system:yyy=${system:xxx}; -set system:yyy; --- sets yyy with value of xxx - -set go=${hiveconf:zzz}; -set go; --- sets go base on value on zzz - -set hive.variable.substitute=false; -set raw=${hiveconf:zzz}; -set raw; --- disable substitution set a value to the literal - -set hive.variable.substitute=true; - -EXPLAIN SELECT * FROM src where key=${hiveconf:zzz}; -SELECT * FROM src where key=${hiveconf:zzz}; ---use a variable in a query - -set a=1; -set b=a; -set c=${hiveconf:${hiveconf:b}}; -set c; ---uses nested variables. - - -set jar=../lib/derby.jar; - -add file ${hiveconf:jar}; -list file; -delete file ${hiveconf:jar}; -list file; -]]></source> -</section> - -<section name="Disabling" href="disable"> - <p>Variable substitution is on by default. If this causes an issue with an already existing script disable it.</p> - -<source><![CDATA[set hive.variable.substitute=false; -]]></source> - -</section> - -</body> -</document> http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/docs/xdocs/language_manual/working_with_bucketed_tables.xml ---------------------------------------------------------------------- diff --git a/docs/xdocs/language_manual/working_with_bucketed_tables.xml b/docs/xdocs/language_manual/working_with_bucketed_tables.xml deleted file mode 100644 index de4b599..0000000 --- a/docs/xdocs/language_manual/working_with_bucketed_tables.xml +++ /dev/null @@ -1,87 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. ---> - -<document> - - <properties> - <title>Hadoop Hive- Working with Bucketed Tables</title> - <author email="[email protected]">Hadoop Hive Documentation Team</author> - </properties> - - <body> - - <section name="Defining Bucketed Tables" href="defining_bucketed_tables?"> - -<p> -This is a brief example on creating a populating bucketed tables. Bucketed tables -are fantastic in that they allow much more efficient sampling than do non-bucketed -tables, and they may later allow for time saving operations such as mapside joins. -However, the bucketing specified at table creation is not enforced when the table -is written to, and so it is possible for the table's metadata to advertise -properties which are not upheld by the table's actual layout. This should obviously -be avoided. Here's how to do it right. -</p> -<p>First thereâs table creation:</p> - - <source><![CDATA[CREATE TABLE user_info_bucketed(user_id BIGINT, firstname STRING, lastname STRING) -COMMENT 'A bucketed copy of user_info' -PARTITIONED BY(ds STRING) -CLUSTERED BY(user_id) INTO 256 BUCKETS; - ]]></source> - -<p>notice that we define user_id as the bucket column</p> -</section> - -<section name="Populating Bucketed Tables" href="populating_bucketed_tables?"> - - <source><![CDATA[set hive.enforce.bucketing = true; -FROM user_id -INSERT OVERWRITE TABLE user_info_bucketed -PARTITION (ds='2009-02-25') -SELECT userid, firstname, lastname WHERE ds='2009-02-25'; - ]]></source> - -<p>The command <strong>set hive.enforce.bucketing = true;</strong> allows the -correct number of reducers and the cluster by column to be automatically selected -based on the table. Otherwise, you would need to set the number of reducers to be -the same as the number of buckets with -<strong>set mapred.reduce.tasks = 256;</strong> and have a -<strong>CLUSTER BY ...</strong> clause in the select.</p> - -</section> - -<section name="Bucketing Explained" href="bucketing_explained?"> -<p> -How does Hive distribute the rows across the buckets? In general, the bucket number is determined by the expression hash_function(bucketing_column) mod num_buckets. (There's a '0x7FFFFFFF in there too, but that's not that important). The hash_function depends on the type of the bucketing column. For an int, it's easy, hash_int(i) == i. For example, if user_id were an int, and there were 10 buckets, we would expect all user_id's that end in 0 to be in bucket 1, all user_id's that end in a 1 to be in bucket 2, etc. For other datatypes, it's a little tricky. In particular, the hash of a BIGINT is not the same as the BIGINT. And the hash of a string or a complex datatype will be some number that's derived from the value, but not anything humanly-recognizable. For example, if user_id were a STRING, then the user_id's in bucket 1 would probably not end in 0. In general, distributing rows based on the hash will give you a even distribution in the buckets. -</p> - -</section> - -<section name="What can go wrong?" href="bucketing_gone_wrong?"> -<p> -So, what can go wrong? As long as you -<strong>set hive.enforce.bucketing = true</strong>, and use the syntax above, -the tables should be populated properly. Things can go wrong if the bucketing -column type is different during the insert and on read, or if you manually -cluster by a value that's different from the table definition. -</p> -</section> -</body> -</document> http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/docs/xdocs/udf/reflect.xml ---------------------------------------------------------------------- diff --git a/docs/xdocs/udf/reflect.xml b/docs/xdocs/udf/reflect.xml deleted file mode 100644 index 435f025..0000000 --- a/docs/xdocs/udf/reflect.xml +++ /dev/null @@ -1,51 +0,0 @@ -<?xml version="1.0" encoding="UTF-8"?> -<!-- - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, - software distributed under the License is distributed on an - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - KIND, either express or implied. See the License for the - specific language governing permissions and limitations - under the License. ---> - -<document> - - <properties> - <title>Hadoop Hive- Reflect User Defined Function</title> - <author email="[email protected]">Hadoop Hive Documentation Team</author> - </properties> - - <body> -<section name="Reflect (Generic) UDF" href="reflect"> - -<p>A java class and method often exists to handle the exact function a user would like to use in hive. Rather -then having to write a wrapper UDF to call this method, the majority of these methods can be called using reflect udf. Reflect uses -java reflection to instantiate and call methods of objects, it can also call static functions. The method must return a primative type -or a type that hive knows how to serialize. -</p> - -<source><![CDATA[SELECT reflect("java.lang.String", "valueOf", 1), - reflect("java.lang.String", "isEmpty"), - reflect("java.lang.Math", "max", 2, 3), - reflect("java.lang.Math", "min", 2, 3), - reflect("java.lang.Math", "round", 2.5), - reflect("java.lang.Math", "exp", 1.0), - reflect("java.lang.Math", "floor", 1.9) -FROM src LIMIT 1; - - -1 true 3 2 3 2.7182818284590455 1.0]]></source> - -</section> -</body> -</document> http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/druid-handler/pom.xml ---------------------------------------------------------------------- diff --git a/druid-handler/pom.xml b/druid-handler/pom.xml index fc70185..3f8a74f 100644 --- a/druid-handler/pom.xml +++ b/druid-handler/pom.xml @@ -19,7 +19,7 @@ <parent> <groupId>org.apache.hive</groupId> <artifactId>hive</artifactId> - <version>2.2.0-SNAPSHOT</version> + <version>3.0.0-SNAPSHOT</version> <relativePath>../pom.xml</relativePath> </parent> @@ -107,6 +107,10 @@ <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> </exclusion> + <exclusion> + <groupId>io.druid</groupId> + <artifactId>druid-aws-common</artifactId> + </exclusion> </exclusions> </dependency> <dependency> @@ -218,6 +222,17 @@ <groupId>org.apache.calcite</groupId> <artifactId>calcite-druid</artifactId> <version>${calcite.version}</version> + <exclusions> + <exclusion> + <groupId>org.apache.calcite.avatica</groupId> + <artifactId>avatica-core</artifactId> + </exclusion> + </exclusions> + </dependency> + <dependency> + <groupId>org.apache.calcite.avatica</groupId> + <artifactId>avatica</artifactId> + <version>${avatica.version}</version> </dependency> <!-- test inter-project --> <dependency> @@ -325,6 +340,7 @@ <include>org.jdbi:*</include> <include>net.jpountz.lz4:*</include> <include>org.apache.commons:*</include> + <include>org.roaringbitmap:*</include> </includes> </artifactSet> <filters> http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java ---------------------------------------------------------------------- diff --git a/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java b/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java index d4f6865..daee2fe 100644 --- a/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java +++ b/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandler.java @@ -33,7 +33,6 @@ import com.metamx.common.lifecycle.Lifecycle; import com.metamx.http.client.HttpClient; import com.metamx.http.client.HttpClientConfig; import com.metamx.http.client.HttpClientInit; -import io.druid.indexer.SQLMetadataStorageUpdaterJobHandler; import io.druid.metadata.MetadataStorageConnectorConfig; import io.druid.metadata.MetadataStorageTablesConfig; import io.druid.metadata.SQLMetadataConnector; @@ -56,7 +55,6 @@ import org.apache.hadoop.hive.metastore.HiveMetaHook; import org.apache.hadoop.hive.metastore.MetaStoreUtils; import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.metastore.api.Table; -import org.apache.hadoop.hive.ql.metadata.DefaultStorageHandler; import org.apache.hadoop.hive.ql.metadata.HiveException; import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler; import org.apache.hadoop.hive.ql.plan.TableDesc; @@ -96,8 +94,6 @@ public class DruidStorageHandler extends DefaultHiveMetaHook implements HiveStor private final SQLMetadataConnector connector; - private final SQLMetadataStorageUpdaterJobHandler druidSqlMetadataStorageUpdaterJobHandler; - private final MetadataStorageTablesConfig druidMetadataStorageTablesConfig; private HttpClient httpClient; @@ -151,17 +147,14 @@ public class DruidStorageHandler extends DefaultHiveMetaHook implements HiveStor } else { throw new IllegalStateException(String.format("Unknown metadata storage type [%s]", dbType)); } - druidSqlMetadataStorageUpdaterJobHandler = new SQLMetadataStorageUpdaterJobHandler(connector); } @VisibleForTesting public DruidStorageHandler(SQLMetadataConnector connector, - SQLMetadataStorageUpdaterJobHandler druidSqlMetadataStorageUpdaterJobHandler, MetadataStorageTablesConfig druidMetadataStorageTablesConfig, HttpClient httpClient ) { this.connector = connector; - this.druidSqlMetadataStorageUpdaterJobHandler = druidSqlMetadataStorageUpdaterJobHandler; this.druidMetadataStorageTablesConfig = druidMetadataStorageTablesConfig; this.httpClient = httpClient; } @@ -256,6 +249,12 @@ public class DruidStorageHandler extends DefaultHiveMetaHook implements HiveStor @Override public void commitCreateTable(Table table) throws MetaException { + LOG.debug(String.format("commit create table [%s]", table.getTableName())); + publishSegments(table, true); + } + + + public void publishSegments(Table table, boolean overwrite) throws MetaException { if (MetaStoreUtils.isExternalTable(table)) { return; } @@ -266,15 +265,19 @@ public class DruidStorageHandler extends DefaultHiveMetaHook implements HiveStor List<DataSegment> segmentList = DruidStorageHandlerUtils .getPublishedSegments(tableDir, getConf()); LOG.info(String.format("Found [%d] segments under path [%s]", segmentList.size(), tableDir)); - druidSqlMetadataStorageUpdaterJobHandler.publishSegments( - druidMetadataStorageTablesConfig.getSegmentsTable(), + final String dataSourceName = table.getParameters().get(Constants.DRUID_DATA_SOURCE); + + DruidStorageHandlerUtils.publishSegments( + connector, + druidMetadataStorageTablesConfig, + dataSourceName, segmentList, - DruidStorageHandlerUtils.JSON_MAPPER + DruidStorageHandlerUtils.JSON_MAPPER, + overwrite ); final String coordinatorAddress = HiveConf .getVar(getConf(), HiveConf.ConfVars.HIVE_DRUID_COORDINATOR_DEFAULT_ADDRESS); int maxTries = HiveConf.getIntVar(getConf(), HiveConf.ConfVars.HIVE_DRUID_MAX_TRIES); - final String dataSourceName = table.getParameters().get(Constants.DRUID_DATA_SOURCE); LOG.info(String.format("checking load status from coordinator [%s]", coordinatorAddress)); // check if the coordinator is up @@ -488,7 +491,7 @@ public class DruidStorageHandler extends DefaultHiveMetaHook implements HiveStor public void commitInsertTable(Table table, boolean overwrite) throws MetaException { if (overwrite) { LOG.debug(String.format("commit insert overwrite into table [%s]", table.getTableName())); - this.commitCreateTable(table); + this.publishSegments(table, overwrite); } else { throw new MetaException("Insert into is not supported yet"); } http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java ---------------------------------------------------------------------- diff --git a/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java b/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java index 8d48e14..adf013b 100644 --- a/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java +++ b/druid-handler/src/java/org/apache/hadoop/hive/druid/DruidStorageHandlerUtils.java @@ -22,6 +22,7 @@ import com.fasterxml.jackson.databind.jsontype.NamedType; import com.fasterxml.jackson.dataformat.smile.SmileFactory; import com.google.common.base.Throwables; import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; import com.google.common.collect.Interner; import com.google.common.collect.Interners; import com.google.common.collect.Lists; @@ -43,6 +44,8 @@ import io.druid.segment.IndexMergerV9; import io.druid.segment.column.ColumnConfig; import io.druid.timeline.DataSegment; import io.druid.timeline.partition.LinearShardSpec; +import io.druid.timeline.partition.NoneShardSpec; + import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; @@ -54,9 +57,11 @@ import org.apache.hadoop.io.retry.RetryProxy; import org.apache.hadoop.util.StringUtils; import org.jboss.netty.handler.codec.http.HttpHeaders; import org.jboss.netty.handler.codec.http.HttpMethod; +import org.joda.time.DateTime; import org.skife.jdbi.v2.FoldController; import org.skife.jdbi.v2.Folder3; import org.skife.jdbi.v2.Handle; +import org.skife.jdbi.v2.PreparedBatch; import org.skife.jdbi.v2.StatementContext; import org.skife.jdbi.v2.TransactionCallback; import org.skife.jdbi.v2.TransactionStatus; @@ -335,14 +340,7 @@ public final class DruidStorageHandlerUtils { new HandleCallback<Void>() { @Override public Void withHandle(Handle handle) throws Exception { - handle.createStatement( - String.format("UPDATE %s SET used=false WHERE dataSource = :dataSource", - metadataStorageTablesConfig.getSegmentsTable() - ) - ) - .bind("dataSource", dataSource) - .execute(); - + disableDataSourceWithHandle(handle, metadataStorageTablesConfig, dataSource); return null; } } @@ -355,6 +353,64 @@ public final class DruidStorageHandlerUtils { return true; } + public static void publishSegments(final SQLMetadataConnector connector, + final MetadataStorageTablesConfig metadataStorageTablesConfig, + final String dataSource, + final List<DataSegment> segments, final ObjectMapper mapper, boolean overwrite) + { + connector.getDBI().inTransaction( + new TransactionCallback<Void>() + { + @Override + public Void inTransaction(Handle handle, TransactionStatus transactionStatus) throws Exception + { + if(overwrite){ + disableDataSourceWithHandle(handle, metadataStorageTablesConfig, dataSource); + } + final PreparedBatch batch = handle.prepareBatch( + String.format( + "INSERT INTO %1$s (id, dataSource, created_date, start, \"end\", partitioned, version, used, payload) " + + "VALUES (:id, :dataSource, :created_date, :start, :end, :partitioned, :version, :used, :payload)", + metadataStorageTablesConfig.getSegmentsTable() + ) + ); + for (final DataSegment segment : segments) { + + batch.add( + new ImmutableMap.Builder<String, Object>() + .put("id", segment.getIdentifier()) + .put("dataSource", segment.getDataSource()) + .put("created_date", new DateTime().toString()) + .put("start", segment.getInterval().getStart().toString()) + .put("end", segment.getInterval().getEnd().toString()) + .put("partitioned", (segment.getShardSpec() instanceof NoneShardSpec) ? false : true) + .put("version", segment.getVersion()) + .put("used", true) + .put("payload", mapper.writeValueAsBytes(segment)) + .build() + ); + + LOG.info("Published %s", segment.getIdentifier()); + + } + batch.execute(); + + return null; + } + } + ); + } + + public static void disableDataSourceWithHandle(Handle handle, MetadataStorageTablesConfig metadataStorageTablesConfig, String dataSource){ + handle.createStatement( + String.format("UPDATE %s SET used=false WHERE dataSource = :dataSource", + metadataStorageTablesConfig.getSegmentsTable() + ) + ) + .bind("dataSource", dataSource) + .execute(); + } + /** * @param connector SQL connector to metadata * @param metadataStorageTablesConfig Tables configuration http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java ---------------------------------------------------------------------- diff --git a/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java b/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java index 40a2022..fbdd4c9 100644 --- a/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java +++ b/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidOutputFormat.java @@ -33,6 +33,9 @@ import io.druid.granularity.QueryGranularity; import io.druid.query.aggregation.AggregatorFactory; import io.druid.query.aggregation.DoubleSumAggregatorFactory; import io.druid.query.aggregation.LongSumAggregatorFactory; +import io.druid.segment.IndexSpec; +import io.druid.segment.data.ConciseBitmapSerdeFactory; +import io.druid.segment.data.RoaringBitmapSerdeFactory; import io.druid.segment.indexing.DataSchema; import io.druid.segment.indexing.RealtimeTuningConfig; import io.druid.segment.indexing.granularity.GranularitySpec; @@ -200,6 +203,12 @@ public class DruidOutputFormat<K, V> implements HiveOutputFormat<K, DruidWritabl } Integer maxRowInMemory = HiveConf.getIntVar(jc, HiveConf.ConfVars.HIVE_DRUID_MAX_ROW_IN_MEMORY); + IndexSpec indexSpec; + if ("concise".equals(HiveConf.getVar(jc, HiveConf.ConfVars.HIVE_DRUID_BITMAP_FACTORY_TYPE))) { + indexSpec = new IndexSpec(new ConciseBitmapSerdeFactory(), null, null, null); + } else { + indexSpec = new IndexSpec(new RoaringBitmapSerdeFactory(true), null, null, null); + } RealtimeTuningConfig realtimeTuningConfig = new RealtimeTuningConfig(maxRowInMemory, null, null, @@ -208,7 +217,7 @@ public class DruidOutputFormat<K, V> implements HiveOutputFormat<K, DruidWritabl null, null, null, - null, + indexSpec, true, 0, 0, @@ -232,6 +241,6 @@ public class DruidOutputFormat<K, V> implements HiveOutputFormat<K, DruidWritabl @Override public void checkOutputSpecs(FileSystem ignored, JobConf job) throws IOException { - throw new UnsupportedOperationException("not implemented yet"); + // NOOP } } http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java ---------------------------------------------------------------------- diff --git a/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java b/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java index 0b35428..53624e1 100644 --- a/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java +++ b/druid-handler/src/java/org/apache/hadoop/hive/druid/io/DruidQueryBasedInputFormat.java @@ -20,6 +20,7 @@ package org.apache.hadoop.hive.druid.io; import java.io.IOException; import java.io.InputStream; import java.net.URL; +import java.net.URLEncoder; import java.util.ArrayList; import java.util.Arrays; import java.util.HashMap; @@ -170,7 +171,8 @@ public class DruidQueryBasedInputFormat extends InputFormat<NullWritable, DruidW // Create Select query SelectQueryBuilder builder = new Druids.SelectQueryBuilder(); builder.dataSource(dataSource); - builder.intervals(Arrays.asList(DruidTable.DEFAULT_INTERVAL)); + final List<Interval> intervals = Arrays.asList(); + builder.intervals(intervals); builder.pagingSpec(PagingSpec.newSpec(1)); Map<String, Object> context = new HashMap<>(); context.put(Constants.DRUID_QUERY_FETCH, false); @@ -212,7 +214,7 @@ public class DruidQueryBasedInputFormat extends InputFormat<NullWritable, DruidW StringUtils.join(query.getIntervals(), ","); // Comma-separated intervals without brackets final String request = String.format( "http://%s/druid/v2/datasources/%s/candidates?intervals=%s";, - address, query.getDataSource().getNames().get(0), intervals); + address, query.getDataSource().getNames().get(0), URLEncoder.encode(intervals, "UTF-8")); final InputStream response; try { response = DruidStorageHandlerUtils.submitRequest(client, new Request(HttpMethod.GET, new URL(request))); @@ -413,11 +415,15 @@ public class DruidQueryBasedInputFormat extends InputFormat<NullWritable, DruidW private static List<List<Interval>> createSplitsIntervals(List<Interval> intervals, int numSplits ) { - final long totalTime = DruidDateTimeUtils.extractTotalTime(intervals); + long startTime = intervals.get(0).getStartMillis(); long endTime = startTime; long currTime = 0; List<List<Interval>> newIntervals = new ArrayList<>(); + long totalTime = 0; + for (Interval interval: intervals) { + totalTime += interval.getEndMillis() - interval.getStartMillis(); + } for (int i = 0, posIntervals = 0; i < numSplits; i++) { final long rangeSize = Math.round((double) (totalTime * (i + 1)) / numSplits) - Math.round((double) (totalTime * i) / numSplits); http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidGroupByQueryRecordReader.java ---------------------------------------------------------------------- diff --git a/druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidGroupByQueryRecordReader.java b/druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidGroupByQueryRecordReader.java index 9e8b439..f0bdb9e 100644 --- a/druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidGroupByQueryRecordReader.java +++ b/druid-handler/src/java/org/apache/hadoop/hive/druid/serde/DruidGroupByQueryRecordReader.java @@ -99,7 +99,7 @@ public class DruidGroupByQueryRecordReader indexes[i]--; for (int j = i + 1; j < indexes.length; j++) { indexes[j] = current.getDimension( - query.getDimensions().get(j).getDimension()).size() - 1; + query.getDimensions().get(j).getOutputName()).size() - 1; } return true; } @@ -110,7 +110,7 @@ public class DruidGroupByQueryRecordReader indexes = new int[query.getDimensions().size()]; for (int i = 0; i < query.getDimensions().size(); i++) { DimensionSpec ds = query.getDimensions().get(i); - indexes[i] = current.getDimension(ds.getDimension()).size() - 1; + indexes[i] = current.getDimension(ds.getOutputName()).size() - 1; } return true; } @@ -131,7 +131,7 @@ public class DruidGroupByQueryRecordReader // 2) The dimension columns for (int i = 0; i < query.getDimensions().size(); i++) { DimensionSpec ds = query.getDimensions().get(i); - List<String> dims = current.getDimension(ds.getDimension()); + List<String> dims = current.getDimension(ds.getOutputName()); if (dims.size() == 0) { // NULL value for dimension value.getValue().put(ds.getOutputName(), null); @@ -170,7 +170,7 @@ public class DruidGroupByQueryRecordReader // 2) The dimension columns for (int i = 0; i < query.getDimensions().size(); i++) { DimensionSpec ds = query.getDimensions().get(i); - List<String> dims = current.getDimension(ds.getDimension()); + List<String> dims = current.getDimension(ds.getOutputName()); if (dims.size() == 0) { // NULL value for dimension value.getValue().put(ds.getOutputName(), null); http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java ---------------------------------------------------------------------- diff --git a/druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java b/druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java index da6610a..05e3ec5 100644 --- a/druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java +++ b/druid-handler/src/test/org/apache/hadoop/hive/druid/TestDruidStorageHandler.java @@ -18,10 +18,13 @@ package org.apache.hadoop.hive.druid; +import com.google.common.base.Throwables; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Lists; import io.druid.indexer.JobHelper; import io.druid.indexer.SQLMetadataStorageUpdaterJobHandler; +import io.druid.metadata.MetadataStorageTablesConfig; +import io.druid.metadata.SQLMetadataSegmentManager; import io.druid.segment.loading.SegmentLoadingException; import io.druid.timeline.DataSegment; import io.druid.timeline.partition.NoneShardSpec; @@ -42,10 +45,16 @@ import org.junit.Test; import org.junit.rules.TemporaryFolder; import org.mockito.Mockito; import org.skife.jdbi.v2.Handle; +import org.skife.jdbi.v2.StatementContext; +import org.skife.jdbi.v2.tweak.HandleCallback; +import org.skife.jdbi.v2.tweak.ResultSetMapper; import java.io.IOException; import java.io.OutputStream; +import java.sql.ResultSet; +import java.sql.SQLException; import java.util.Arrays; +import java.util.List; import java.util.Map; import java.util.UUID; @@ -85,7 +94,6 @@ public class TestDruidStorageHandler { public void testPreCreateTableWillCreateSegmentsTable() throws MetaException { DruidStorageHandler druidStorageHandler = new DruidStorageHandler( derbyConnectorRule.getConnector(), - new SQLMetadataStorageUpdaterJobHandler(derbyConnectorRule.getConnector()), derbyConnectorRule.metadataTablesConfigSupplier().get(), null ); @@ -114,7 +122,6 @@ public class TestDruidStorageHandler { ); DruidStorageHandler druidStorageHandler = new DruidStorageHandler( derbyConnectorRule.getConnector(), - new SQLMetadataStorageUpdaterJobHandler(derbyConnectorRule.getConnector()), derbyConnectorRule.metadataTablesConfigSupplier().get(), null ); @@ -126,7 +133,6 @@ public class TestDruidStorageHandler { throws MetaException, IOException { DruidStorageHandler druidStorageHandler = new DruidStorageHandler( derbyConnectorRule.getConnector(), - new SQLMetadataStorageUpdaterJobHandler(derbyConnectorRule.getConnector()), derbyConnectorRule.metadataTablesConfigSupplier().get(), null ); @@ -158,7 +164,6 @@ public class TestDruidStorageHandler { public void testCommitInsertTable() throws MetaException, IOException { DruidStorageHandler druidStorageHandler = new DruidStorageHandler( derbyConnectorRule.getConnector(), - new SQLMetadataStorageUpdaterJobHandler(derbyConnectorRule.getConnector()), derbyConnectorRule.metadataTablesConfigSupplier().get(), null ); @@ -184,7 +189,6 @@ public class TestDruidStorageHandler { public void testDeleteSegment() throws IOException, SegmentLoadingException { DruidStorageHandler druidStorageHandler = new DruidStorageHandler( derbyConnectorRule.getConnector(), - new SQLMetadataStorageUpdaterJobHandler(derbyConnectorRule.getConnector()), derbyConnectorRule.metadataTablesConfigSupplier().get(), null ); @@ -221,4 +225,68 @@ public class TestDruidStorageHandler { localFileSystem.exists(segmentOutputPath.getParent().getParent().getParent()) ); } + + @Test + public void testCommitInsertOverwriteTable() throws MetaException, IOException { + DerbyConnectorTestUtility connector = derbyConnectorRule.getConnector(); + MetadataStorageTablesConfig metadataStorageTablesConfig = derbyConnectorRule + .metadataTablesConfigSupplier().get(); + + DruidStorageHandler druidStorageHandler = new DruidStorageHandler( + connector, + metadataStorageTablesConfig, + null + ); + druidStorageHandler.preCreateTable(tableMock); + Configuration config = new Configuration(); + config.set(String.valueOf(HiveConf.ConfVars.HIVEQUERYID), UUID.randomUUID().toString()); + config.set(String.valueOf(HiveConf.ConfVars.DRUID_WORKING_DIR), tableWorkingPath); + druidStorageHandler.setConf(config); + LocalFileSystem localFileSystem = FileSystem.getLocal(config); + Path taskDirPath = new Path(tableWorkingPath, druidStorageHandler.makeStagingName()); + Path descriptorPath = DruidStorageHandlerUtils.makeSegmentDescriptorOutputPath(dataSegment, + new Path(taskDirPath, DruidStorageHandler.SEGMENTS_DESCRIPTOR_DIR_NAME) + ); + List<DataSegment> existingSegments = Arrays.asList(DataSegment.builder().dataSource(DATA_SOURCE_NAME).version("v0") + .interval(new Interval(1, 10)).shardSpec(NoneShardSpec.instance()).build()); + DruidStorageHandlerUtils.publishSegments(connector, metadataStorageTablesConfig, DATA_SOURCE_NAME, + existingSegments, + DruidStorageHandlerUtils.JSON_MAPPER, + true + ); + DruidStorageHandlerUtils.writeSegmentDescriptor(localFileSystem, dataSegment, descriptorPath); + druidStorageHandler.commitInsertTable(tableMock, true); + Assert.assertArrayEquals(Lists.newArrayList(DATA_SOURCE_NAME).toArray(), Lists.newArrayList( + DruidStorageHandlerUtils.getAllDataSourceNames(connector, + metadataStorageTablesConfig + )).toArray()); + + final List<DataSegment> dataSegmentList = connector.getDBI() + .withHandle(new HandleCallback<List<DataSegment>>() { + @Override + public List<DataSegment> withHandle(Handle handle) throws Exception { + return handle + .createQuery(String.format("SELECT payload FROM %s WHERE used=true", + metadataStorageTablesConfig.getSegmentsTable())) + .map(new ResultSetMapper<DataSegment>() { + + @Override + public DataSegment map(int i, ResultSet resultSet, + StatementContext statementContext) + throws SQLException { + try { + return DruidStorageHandlerUtils.JSON_MAPPER.readValue( + resultSet.getBytes("payload"), + DataSegment.class + ); + } catch (IOException e) { + throw Throwables.propagate(e); + } + } + }).list(); + } + }); + Assert.assertEquals(1, dataSegmentList.size()); + + } } http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/druid-handler/src/test/org/apache/hadoop/hive/ql/io/TestDruidRecordWriter.java ---------------------------------------------------------------------- diff --git a/druid-handler/src/test/org/apache/hadoop/hive/ql/io/TestDruidRecordWriter.java b/druid-handler/src/test/org/apache/hadoop/hive/ql/io/TestDruidRecordWriter.java index 9ec82c0..d9e01fe 100644 --- a/druid-handler/src/test/org/apache/hadoop/hive/ql/io/TestDruidRecordWriter.java +++ b/druid-handler/src/test/org/apache/hadoop/hive/ql/io/TestDruidRecordWriter.java @@ -37,8 +37,10 @@ import io.druid.granularity.QueryGranularities; import io.druid.query.aggregation.AggregatorFactory; import io.druid.query.aggregation.LongSumAggregatorFactory; import io.druid.query.aggregation.hyperloglog.HyperUniquesAggregatorFactory; +import io.druid.segment.IndexSpec; import io.druid.segment.QueryableIndex; import io.druid.segment.QueryableIndexStorageAdapter; +import io.druid.segment.data.RoaringBitmapSerdeFactory; import io.druid.segment.indexing.DataSchema; import io.druid.segment.indexing.RealtimeTuningConfig; import io.druid.segment.indexing.granularity.UniformGranularitySpec; @@ -139,8 +141,10 @@ public class TestDruidRecordWriter { objectMapper ); - RealtimeTuningConfig tuningConfig = RealtimeTuningConfig - .makeDefaultTuningConfig(temporaryFolder.newFolder()); + IndexSpec indexSpec = new IndexSpec(new RoaringBitmapSerdeFactory(true), null, null, null); + RealtimeTuningConfig tuningConfig = new RealtimeTuningConfig(null, null, null, + temporaryFolder.newFolder(), null, null, null, null, indexSpec, null, 0, 0, null, null + ); LocalFileSystem localFileSystem = FileSystem.getLocal(config); DataSegmentPusher dataSegmentPusher = new LocalDataSegmentPusher( new LocalDataSegmentPusherConfig() { http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/errata.txt ---------------------------------------------------------------------- diff --git a/errata.txt b/errata.txt index 6f464cf..949ed9a 100644 --- a/errata.txt +++ b/errata.txt @@ -1,6 +1,9 @@ Commits with the wrong or no JIRA referenced: git commit branch jira url +f1aae85f197de09d4b86143f7f13d5aa21d2eb85 master HIVE-16431 https://issues.apache.org/jira/browse/HIVE-16431 +cbab5b29f26ceb3d4633ade9647ce8bcb2f020a0 master HIVE-16422 https://issues.apache.org/jira/browse/HIVE-16422 +e6143de2b0c3f53d32db8a743119e3a8080d4f85 master HIVE-16425 https://issues.apache.org/jira/browse/HIVE-16425 3f90794d872e90c29a068f16cdf3f45b1cf52c74 master HIVE-15579 https://issues.apache.org/jira/browse/HIVE-15579 5a576b6fbf1680ab4dd8f275cad484a2614ef2c1 master HIVE-10391 https://issues.apache.org/jira/browse/HIVE-10391 582f4e1bc39b9605d11f762480b29561a44688ae llap HIVE-10217 https://issues.apache.org/jira/browse/HIVE-10217 @@ -86,3 +89,4 @@ d8298e1c85a515150562b0df68af89c18c468638 llap HIVE-9418 https://issues.ap d16d4f1bcc43d6ebcab0eaf5bc635fb88b60be5f master HIVE-9423 https://issues.apache.org/jira/browse/HIVE-9423 130617443bb05d79c18420c0c4e903a76da3651c master HIVE-14909 https://issues.apache.org/jira/browse/HIVE-14909 6dace60af4b6ab4d5200310a0ad94c4530c2bec3 master HIVE-13335 https://issues.apache.org/jira/browse/HIVE-13335 +5facfbb863366d7a661c21c57011b8dbe43f52e0 master HIVE-16307 https://issues.apache.org/jira/browse/HIVE-16307 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hbase-handler/pom.xml ---------------------------------------------------------------------- diff --git a/hbase-handler/pom.xml b/hbase-handler/pom.xml index 8dc47e9..7f57b77 100644 --- a/hbase-handler/pom.xml +++ b/hbase-handler/pom.xml @@ -19,7 +19,7 @@ <parent> <groupId>org.apache.hive</groupId> <artifactId>hive</artifactId> - <version>2.2.0-SNAPSHOT</version> + <version>3.0.0-SNAPSHOT</version> <relativePath>../pom.xml</relativePath> </parent> http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hbase-handler/src/test/queries/negative/hbase_ddl.q ---------------------------------------------------------------------- diff --git a/hbase-handler/src/test/queries/negative/hbase_ddl.q b/hbase-handler/src/test/queries/negative/hbase_ddl.q new file mode 100644 index 0000000..2913bcd --- /dev/null +++ b/hbase-handler/src/test/queries/negative/hbase_ddl.q @@ -0,0 +1,9 @@ +DROP TABLE hbase_table_1; +CREATE TABLE hbase_table_1(key int comment 'It is a column key', value string comment 'It is the column string value') +STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' +WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf:string") +TBLPROPERTIES ("hbase.table.name" = "hbase_table_0"); + +DESCRIBE EXTENDED hbase_table_1; + +alter table hbase_table_1 change column key newkey string; http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hbase-handler/src/test/queries/positive/hbase_ddl.q ---------------------------------------------------------------------- diff --git a/hbase-handler/src/test/queries/positive/hbase_ddl.q b/hbase-handler/src/test/queries/positive/hbase_ddl.q new file mode 100644 index 0000000..a8bae75 --- /dev/null +++ b/hbase-handler/src/test/queries/positive/hbase_ddl.q @@ -0,0 +1,20 @@ +DROP TABLE hbase_table_1; +CREATE TABLE hbase_table_1(key int comment 'It is a column key', value string comment 'It is the column string value') +STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' +WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf:string") +TBLPROPERTIES ("hbase.table.name" = "hbase_table_0"); + +DESCRIBE EXTENDED hbase_table_1; + +select * from hbase_table_1; + +EXPLAIN FROM src INSERT OVERWRITE TABLE hbase_table_1 SELECT * WHERE (key%2)=0; +FROM src INSERT OVERWRITE TABLE hbase_table_1 SELECT * WHERE (key%2)=0; + +ALTER TABLE hbase_table_1 SET TBLPROPERTIES('hbase.mapred.output.outputtable'='kkk'); + +desc formatted hbase_table_1; + +ALTER TABLE hbase_table_1 unset TBLPROPERTIES('hbase.mapred.output.outputtable'); + +desc formatted hbase_table_1; http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hbase-handler/src/test/queries/positive/hbase_queries.q ---------------------------------------------------------------------- diff --git a/hbase-handler/src/test/queries/positive/hbase_queries.q b/hbase-handler/src/test/queries/positive/hbase_queries.q index 49fa829..43efd6c 100644 --- a/hbase-handler/src/test/queries/positive/hbase_queries.q +++ b/hbase-handler/src/test/queries/positive/hbase_queries.q @@ -180,6 +180,7 @@ DROP TABLE IF EXISTS hbase_table_10; CREATE TABLE hbase_table_10 (id bigint, data map<int, int>, str string) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ("hbase.columns.mapping" = ":key,cf:map_col2,cf:str2_col"); +set hive.cbo.enable=false; insert overwrite table hbase_table_10 select 1 as id, map(10, cast(null as int)) as data , null as str from src limit 1; insert into table hbase_table_10 select 2 as id, map(20, cast(null as int)) as data , '1234' as str from src limit 1; insert into table hbase_table_10 select 3 as id, map(30, 31) as data , '1234' as str from src limit 1; http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hbase-handler/src/test/results/negative/hbase_ddl.q.out ---------------------------------------------------------------------- diff --git a/hbase-handler/src/test/results/negative/hbase_ddl.q.out b/hbase-handler/src/test/results/negative/hbase_ddl.q.out new file mode 100644 index 0000000..b5aad70 --- /dev/null +++ b/hbase-handler/src/test/results/negative/hbase_ddl.q.out @@ -0,0 +1,29 @@ +PREHOOK: query: DROP TABLE hbase_table_1 +PREHOOK: type: DROPTABLE +POSTHOOK: query: DROP TABLE hbase_table_1 +POSTHOOK: type: DROPTABLE +PREHOOK: query: CREATE TABLE hbase_table_1(key int comment 'It is a column key', value string comment 'It is the column string value') +STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' +WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf:string") +TBLPROPERTIES ("hbase.table.name" = "hbase_table_0") +PREHOOK: type: CREATETABLE +PREHOOK: Output: database:default +PREHOOK: Output: default@hbase_table_1 +POSTHOOK: query: CREATE TABLE hbase_table_1(key int comment 'It is a column key', value string comment 'It is the column string value') +STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' +WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf:string") +TBLPROPERTIES ("hbase.table.name" = "hbase_table_0") +POSTHOOK: type: CREATETABLE +POSTHOOK: Output: database:default +POSTHOOK: Output: default@hbase_table_1 +PREHOOK: query: DESCRIBE EXTENDED hbase_table_1 +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@hbase_table_1 +POSTHOOK: query: DESCRIBE EXTENDED hbase_table_1 +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@hbase_table_1 +key int It is a column key +value string It is the column string value + +#### A masked pattern was here #### +FAILED: SemanticException [Error 10134]: ALTER TABLE can only be used for [ADDPROPS, DROPPROPS] to a non-native table hbase_table_1 http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hbase-handler/src/test/results/positive/hbase_ddl.q.out ---------------------------------------------------------------------- diff --git a/hbase-handler/src/test/results/positive/hbase_ddl.q.out b/hbase-handler/src/test/results/positive/hbase_ddl.q.out new file mode 100644 index 0000000..8cb88ed --- /dev/null +++ b/hbase-handler/src/test/results/positive/hbase_ddl.q.out @@ -0,0 +1,186 @@ +PREHOOK: query: DROP TABLE hbase_table_1 +PREHOOK: type: DROPTABLE +POSTHOOK: query: DROP TABLE hbase_table_1 +POSTHOOK: type: DROPTABLE +PREHOOK: query: CREATE TABLE hbase_table_1(key int comment 'It is a column key', value string comment 'It is the column string value') +STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' +WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf:string") +TBLPROPERTIES ("hbase.table.name" = "hbase_table_0") +PREHOOK: type: CREATETABLE +PREHOOK: Output: database:default +PREHOOK: Output: default@hbase_table_1 +POSTHOOK: query: CREATE TABLE hbase_table_1(key int comment 'It is a column key', value string comment 'It is the column string value') +STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' +WITH SERDEPROPERTIES ("hbase.columns.mapping" = "cf:string") +TBLPROPERTIES ("hbase.table.name" = "hbase_table_0") +POSTHOOK: type: CREATETABLE +POSTHOOK: Output: database:default +POSTHOOK: Output: default@hbase_table_1 +PREHOOK: query: DESCRIBE EXTENDED hbase_table_1 +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@hbase_table_1 +POSTHOOK: query: DESCRIBE EXTENDED hbase_table_1 +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@hbase_table_1 +key int It is a column key +value string It is the column string value + +#### A masked pattern was here #### +PREHOOK: query: select * from hbase_table_1 +PREHOOK: type: QUERY +PREHOOK: Input: default@hbase_table_1 +#### A masked pattern was here #### +POSTHOOK: query: select * from hbase_table_1 +POSTHOOK: type: QUERY +POSTHOOK: Input: default@hbase_table_1 +#### A masked pattern was here #### +PREHOOK: query: EXPLAIN FROM src INSERT OVERWRITE TABLE hbase_table_1 SELECT * WHERE (key%2)=0 +PREHOOK: type: QUERY +POSTHOOK: query: EXPLAIN FROM src INSERT OVERWRITE TABLE hbase_table_1 SELECT * WHERE (key%2)=0 +POSTHOOK: type: QUERY +STAGE DEPENDENCIES: + Stage-0 is a root stage + Stage-2 + Stage-1 is a root stage + Stage-3 is a root stage + +STAGE PLANS: + Stage: Stage-0 + Alter Table Operator: + Alter Table + type: drop props + old name: default.hbase_table_1 + properties: + COLUMN_STATS_ACCURATE + + Stage: Stage-2 + Insert operator: + Insert + + Stage: Stage-1 + Pre Insert operator: + Pre-Insert task + + Stage: Stage-3 + Map Reduce + Map Operator Tree: + TableScan + alias: src + Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE + Filter Operator + predicate: ((UDFToDouble(key) % 2.0) = 0.0) (type: boolean) + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: UDFToInteger(key) (type: int), value (type: string) + outputColumnNames: _col0, _col1 + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE + File Output Operator + compressed: false + Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE + table: + input format: org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat + output format: org.apache.hadoop.hive.hbase.HiveHBaseTableOutputFormat + serde: org.apache.hadoop.hive.hbase.HBaseSerDe + name: default.hbase_table_1 + +PREHOOK: query: FROM src INSERT OVERWRITE TABLE hbase_table_1 SELECT * WHERE (key%2)=0 +PREHOOK: type: QUERY +PREHOOK: Input: default@src +PREHOOK: Output: default@hbase_table_1 +POSTHOOK: query: FROM src INSERT OVERWRITE TABLE hbase_table_1 SELECT * WHERE (key%2)=0 +POSTHOOK: type: QUERY +POSTHOOK: Input: default@src +POSTHOOK: Output: default@hbase_table_1 +PREHOOK: query: ALTER TABLE hbase_table_1 SET TBLPROPERTIES('hbase.mapred.output.outputtable'='kkk') +PREHOOK: type: ALTERTABLE_PROPERTIES +PREHOOK: Input: default@hbase_table_1 +PREHOOK: Output: default@hbase_table_1 +POSTHOOK: query: ALTER TABLE hbase_table_1 SET TBLPROPERTIES('hbase.mapred.output.outputtable'='kkk') +POSTHOOK: type: ALTERTABLE_PROPERTIES +POSTHOOK: Input: default@hbase_table_1 +POSTHOOK: Output: default@hbase_table_1 +PREHOOK: query: desc formatted hbase_table_1 +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@hbase_table_1 +POSTHOOK: query: desc formatted hbase_table_1 +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@hbase_table_1 +# col_name data_type comment + +key int It is a column key +value string It is the column string value + +# Detailed Table Information +Database: default +#### A masked pattern was here #### +Retention: 0 +#### A masked pattern was here #### +Table Type: MANAGED_TABLE +Table Parameters: + hbase.mapred.output.outputtable kkk + hbase.table.name hbase_table_0 +#### A masked pattern was here #### + numFiles 0 + numRows 0 + rawDataSize 0 + storage_handler org.apache.hadoop.hive.hbase.HBaseStorageHandler + totalSize 0 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.hbase.HBaseSerDe +InputFormat: null +OutputFormat: null +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + hbase.columns.mapping cf:string + serialization.format 1 +PREHOOK: query: ALTER TABLE hbase_table_1 unset TBLPROPERTIES('hbase.mapred.output.outputtable') +PREHOOK: type: ALTERTABLE_PROPERTIES +PREHOOK: Input: default@hbase_table_1 +PREHOOK: Output: default@hbase_table_1 +POSTHOOK: query: ALTER TABLE hbase_table_1 unset TBLPROPERTIES('hbase.mapred.output.outputtable') +POSTHOOK: type: ALTERTABLE_PROPERTIES +POSTHOOK: Input: default@hbase_table_1 +POSTHOOK: Output: default@hbase_table_1 +PREHOOK: query: desc formatted hbase_table_1 +PREHOOK: type: DESCTABLE +PREHOOK: Input: default@hbase_table_1 +POSTHOOK: query: desc formatted hbase_table_1 +POSTHOOK: type: DESCTABLE +POSTHOOK: Input: default@hbase_table_1 +# col_name data_type comment + +key int It is a column key +value string It is the column string value + +# Detailed Table Information +Database: default +#### A masked pattern was here #### +Retention: 0 +#### A masked pattern was here #### +Table Type: MANAGED_TABLE +Table Parameters: + hbase.table.name hbase_table_0 +#### A masked pattern was here #### + numFiles 0 + numRows 0 + rawDataSize 0 + storage_handler org.apache.hadoop.hive.hbase.HBaseStorageHandler + totalSize 0 +#### A masked pattern was here #### + +# Storage Information +SerDe Library: org.apache.hadoop.hive.hbase.HBaseSerDe +InputFormat: null +OutputFormat: null +Compressed: No +Num Buckets: -1 +Bucket Columns: [] +Sort Columns: [] +Storage Desc Params: + hbase.columns.mapping cf:string + serialization.format 1 http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hcatalog/build.properties ---------------------------------------------------------------------- diff --git a/hcatalog/build.properties b/hcatalog/build.properties index dea1a44..3767cf3 100644 --- a/hcatalog/build.properties +++ b/hcatalog/build.properties @@ -49,7 +49,7 @@ clover.report.dir=${build.dir}/test/clover/reports clover.pdf.report.dir=${build.dir}/test/clover/pdf/reports # junit jvm args -junit.jvm.args=-XX:-UseSplitVerifier -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=128M +junit.jvm.args=-XX:-UseSplitVerifier -XX:+CMSClassUnloadingEnabled apache-rat.version=0.8 http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hcatalog/core/pom.xml ---------------------------------------------------------------------- diff --git a/hcatalog/core/pom.xml b/hcatalog/core/pom.xml index 506bf22..d151374 100644 --- a/hcatalog/core/pom.xml +++ b/hcatalog/core/pom.xml @@ -25,7 +25,7 @@ <parent> <groupId>org.apache.hive.hcatalog</groupId> <artifactId>hive-hcatalog</artifactId> - <version>2.2.0-SNAPSHOT</version> + <version>3.0.0-SNAPSHOT</version> <relativePath>../pom.xml</relativePath> </parent> @@ -191,7 +191,7 @@ <artifactId>commons-logging</artifactId> </exclusion> </exclusions> - </dependency> + </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-mapreduce-client-jobclient</artifactId> @@ -208,13 +208,19 @@ <artifactId>commons-logging</artifactId> </exclusion> </exclusions> - </dependency> + </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-yarn-server-tests</artifactId> <version>${hadoop.version}</version> <classifier>tests</classifier> <scope>test</scope> + <exclusions> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + </exclusions> </dependency> <dependency> <groupId>org.apache.pig</groupId> http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/TestPermsGrp.java ---------------------------------------------------------------------- diff --git a/hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/TestPermsGrp.java b/hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/TestPermsGrp.java index 8aa510f..66a5dd4 100644 --- a/hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/TestPermsGrp.java +++ b/hcatalog/core/src/test/java/org/apache/hive/hcatalog/cli/TestPermsGrp.java @@ -116,7 +116,7 @@ public class TestPermsGrp extends TestCase { Table tbl = getTable(dbName, tblName, typeName); msc.createTable(tbl); Database db = Hive.get(hcatConf).getDatabase(dbName); - Path dfsPath = clientWH.getTablePath(db, tblName); + Path dfsPath = clientWH.getDefaultTablePath(db, tblName); cleanupTbl(dbName, tblName, typeName); // Next user did specify perms. @@ -126,7 +126,7 @@ public class TestPermsGrp extends TestCase { assertTrue(e instanceof ExitException); assertEquals(((ExitException) e).getStatus(), 0); } - dfsPath = clientWH.getTablePath(db, tblName); + dfsPath = clientWH.getDefaultTablePath(db, tblName); assertTrue(dfsPath.getFileSystem(hcatConf).getFileStatus(dfsPath).getPermission().equals(FsPermission.valueOf("drwx-wx---"))); cleanupTbl(dbName, tblName, typeName); @@ -141,7 +141,7 @@ public class TestPermsGrp extends TestCase { assertTrue(me instanceof ExitException); } // No physical dir gets created. - dfsPath = clientWH.getTablePath(db, tblName); + dfsPath = clientWH.getDefaultTablePath(db, tblName); try { dfsPath.getFileSystem(hcatConf).getFileStatus(dfsPath); assert false; http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hcatalog/hcatalog-pig-adapter/pom.xml ---------------------------------------------------------------------- diff --git a/hcatalog/hcatalog-pig-adapter/pom.xml b/hcatalog/hcatalog-pig-adapter/pom.xml index 3a72260..c50a4d5 100644 --- a/hcatalog/hcatalog-pig-adapter/pom.xml +++ b/hcatalog/hcatalog-pig-adapter/pom.xml @@ -25,7 +25,7 @@ <parent> <groupId>org.apache.hive.hcatalog</groupId> <artifactId>hive-hcatalog</artifactId> - <version>2.2.0-SNAPSHOT</version> + <version>3.0.0-SNAPSHOT</version> <relativePath>../pom.xml</relativePath> </parent> http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hcatalog/pom.xml ---------------------------------------------------------------------- diff --git a/hcatalog/pom.xml b/hcatalog/pom.xml index 34de177..9a73c84 100644 --- a/hcatalog/pom.xml +++ b/hcatalog/pom.xml @@ -24,7 +24,7 @@ <parent> <groupId>org.apache.hive</groupId> <artifactId>hive</artifactId> - <version>2.2.0-SNAPSHOT</version> + <version>3.0.0-SNAPSHOT</version> <relativePath>../pom.xml</relativePath> </parent> @@ -71,6 +71,28 @@ <version>${pig.version}</version> <classifier>h2</classifier> <scope>test</scope> + <exclusions> + <exclusion> + <groupId>org.mortbay.jetty</groupId> + <artifactId>jetty-util</artifactId> + </exclusion> + <exclusion> + <groupId>org.mortbay.jetty</groupId> + <artifactId>jetty</artifactId> + </exclusion> + <exclusion> + <groupId>org.mortbay.jetty</groupId> + <artifactId>jsp-api-2.1</artifactId> + </exclusion> + <exclusion> + <groupId>org.mortbay.jetty</groupId> + <artifactId>jsp-2.1</artifactId> + </exclusion> + <exclusion> + <groupId>org.mortbay.jetty</groupId> + <artifactId>servlet-api-2.5</artifactId> + </exclusion> + </exclusions> </dependency> </dependencies> </project> http://git-wip-us.apache.org/repos/asf/hive/blob/187eb760/hcatalog/server-extensions/pom.xml ---------------------------------------------------------------------- diff --git a/hcatalog/server-extensions/pom.xml b/hcatalog/server-extensions/pom.xml index 7d34543..797341c 100644 --- a/hcatalog/server-extensions/pom.xml +++ b/hcatalog/server-extensions/pom.xml @@ -25,7 +25,7 @@ <parent> <groupId>org.apache.hive.hcatalog</groupId> <artifactId>hive-hcatalog</artifactId> - <version>2.2.0-SNAPSHOT</version> + <version>3.0.0-SNAPSHOT</version> <relativePath>../pom.xml</relativePath> </parent>
