[jira] [Created] (HIVE-21583) KillTriggerActionHandler should use "hive" credential
Daniel Dai created HIVE-21583: - Summary: KillTriggerActionHandler should use "hive" credential Key: HIVE-21583 URL: https://issues.apache.org/jira/browse/HIVE-21583 Project: Hive Issue Type: Bug Reporter: Daniel Dai Assignee: Daniel Dai Currently SessionState.username is set to null, which is invalid as KillQueryImplementation will valid the user privilege. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 70372: HIVE-21427: Syslog storage handler
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70372/ --- (Updated April 4, 2019, 9:28 p.m.) Review request for hive, Ashutosh Chauhan and Jason Dere. Changes --- fixes a bug where input row can have data from previous row. Bugs: HIVE-21427 https://issues.apache.org/jira/browse/HIVE-21427 Repository: hive-git Description --- HIVE-21427: Syslog storage handler Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174 data/files/syslog-hs2-2.log PRE-CREATION data/files/syslog-hs2.log PRE-CREATION itests/src/test/resources/testconfiguration.properties 96dfbc4b56 llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java 9ef7af4eb0 ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 27fe828b75 ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java PRE-CREATION ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out PRE-CREATION standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java eafe0c6d46 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java 6282078411 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java PRE-CREATION Diff: https://reviews.apache.org/r/70372/diff/3/ Changes: https://reviews.apache.org/r/70372/diff/2-3/ Testing --- Thanks, Prasanth_J
[jira] [Created] (HIVE-21582) Prefix msck configs with metastore
Prasanth Jayachandran created HIVE-21582: Summary: Prefix msck configs with metastore Key: HIVE-21582 URL: https://issues.apache.org/jira/browse/HIVE-21582 Project: Hive Issue Type: Bug Affects Versions: 4.0.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran HIVE-20707 moved msck configs to metastore but the configs are not prefixed with "metastore". Will be good to prefix it with "metastore" for consistency with other configs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21581) Remove Lock in GetInputSummary
David Mollitor created HIVE-21581:
-
Summary: Remove Lock in GetInputSummary
Key: HIVE-21581
URL: https://issues.apache.org/jira/browse/HIVE-21581
Project: Hive
Issue Type: Improvement
Components: HiveServer2
Affects Versions: 4.0.0, 3.2.0
Reporter: David Mollitor
Assignee: David Mollitor
Fix For: 4.0.0
Now that Hive compile lock has been relaxed in [HIVE-20535], remove the
{{getInputSummary}} lock:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2459]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Review Request 70372: HIVE-21427: Syslog storage handler
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70372/ --- (Updated April 4, 2019, 6:18 p.m.) Review request for hive, Ashutosh Chauhan and Jason Dere. Changes --- review comment update Bugs: HIVE-21427 https://issues.apache.org/jira/browse/HIVE-21427 Repository: hive-git Description --- HIVE-21427: Syslog storage handler Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java aa27a01174 data/files/syslog-hs2-2.log PRE-CREATION data/files/syslog-hs2.log PRE-CREATION itests/src/test/resources/testconfiguration.properties 96dfbc4b56 llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java 9ef7af4eb0 ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 27fe828b75 ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogInputFormat.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogParser.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogSerDe.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/log/syslog/SyslogStorageHandler.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/log/TestSyslogInputFormat.java PRE-CREATION ql/src/test/queries/clientpositive/syslog_parser.q PRE-CREATION ql/src/test/queries/clientpositive/syslog_parser_file_pruning.q PRE-CREATION ql/src/test/results/clientpositive/llap/syslog_parser.q.out PRE-CREATION ql/src/test/results/clientpositive/llap/syslog_parser_file_pruning.q.out PRE-CREATION standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/MetastoreSchemaTool.java eafe0c6d46 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolCommandLine.java 6282078411 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/tools/schematool/SchemaToolTaskCreateLogsTable.java PRE-CREATION Diff: https://reviews.apache.org/r/70372/diff/2/ Changes: https://reviews.apache.org/r/70372/diff/1-2/ Testing --- Thanks, Prasanth_J
[jira] [Created] (HIVE-21580) Introduce ISO 8601 week numbering SQL:2016 formats
Karen Coppage created HIVE-21580: Summary: Introduce ISO 8601 week numbering SQL:2016 formats Key: HIVE-21580 URL: https://issues.apache.org/jira/browse/HIVE-21580 Project: Hive Issue Type: Improvement Reporter: Karen Coppage Enable Hive to parse the following datetime formats when any combination/subset of these or previously implemented patterns is provided in one string. Also catch combinations that conflict. * IYYY * IYY * IY * I * IW [https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21579) Introduce more complex SQL:2016 datetime formats
Karen Coppage created HIVE-21579: Summary: Introduce more complex SQL:2016 datetime formats Key: HIVE-21579 URL: https://issues.apache.org/jira/browse/HIVE-21579 Project: Hive Issue Type: Improvement Reporter: Karen Coppage Enable Hive to parse the following datetime formats when any combination/subset of these or previously implemented patterns is provided in one string. Also catch combinations that conflict. * MONTH * MON * D * DAY * DY * Q * WW * W [https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21578) Introduce SQL:2016 formats FM, FX, and nested strings
Karen Coppage created HIVE-21578: Summary: Introduce SQL:2016 formats FM, FX, and nested strings Key: HIVE-21578 URL: https://issues.apache.org/jira/browse/HIVE-21578 Project: Hive Issue Type: Improvement Reporter: Karen Coppage Enable Hive to parse the following datetime formats when any combination or subset of these or previously implemented formats is provided in one string. * "text" (nested strings) * FM * FX [Definitions here|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21577) Introduce remaining basic SQL:2016 formats
Karen Coppage created HIVE-21577: Summary: Introduce remaining basic SQL:2016 formats Key: HIVE-21577 URL: https://issues.apache.org/jira/browse/HIVE-21577 Project: Hive Issue Type: Improvement Reporter: Karen Coppage Introduce the following SQL:2016 formats to Hive: * YYY * YY * Y * * RR * DDD * HH12 * HH24 * S * FF[1..9] * AM/A.M. * PM/P.M. * TZH * TZM -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21576) Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats
Karen Coppage created HIVE-21576: Summary: Introduce CAST...FORMAT and limited list of SQL:2016 datetime formats Key: HIVE-21576 URL: https://issues.apache.org/jira/browse/HIVE-21576 Project: Hive Issue Type: Improvement Reporter: Karen Coppage Introduce FORMAT clause to CAST statements as well as the below limited list of SQL:2016 datetime formats to Hive in general. These can be used if a session-level feature flag is turned on. * * MM * DD * HH * MI * SS Definitions of these formats here: [https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/|https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21575) Add support for SQL:2016 datetime templates/patterns/masks and CAST(... AS ... FORMAT )
Karen Coppage created HIVE-21575:
Summary: Add support for SQL:2016 datetime
templates/patterns/masks and CAST(... AS ... FORMAT )
Key: HIVE-21575
URL: https://issues.apache.org/jira/browse/HIVE-21575
Project: Hive
Issue Type: Improvement
Reporter: Karen Coppage
Assignee: Karen Coppage
*Summary*
Timestamp and date handling and formatting is currently implemented in Hive
using (sometimes very specific) [Java SimpleDateFormat
patterns|http://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html]
, however, it is not what most standard SQL systems use. For example see
[Vertica|https://my.vertica.com/docs/7.2.x/HTML/Content/Authoring/SQLReferenceManual/Functions/Formatting/TemplatePatternsForDateTimeFormatting.htm],
[Netezza|http://www.ibm.com/support/knowledgecenter/SSULQD_7.2.1/com.ibm.nz.dbu.doc/r_dbuser_ntz_sql_extns_templ_patterns_date_time_conv.html],
[Oracle|https://docs.oracle.com/database/121/SQLRF/sql_elements004.htm#SQLRF00212],
and
[PostgreSQL|https://www.postgresql.org/docs/9.5/static/functions-formatting.html#FUNCTIONS-FORMATTING-DATETIME-TABLE].
*Cast...Format*
SQL:2016 also introduced the FORMAT clause for CAST which is the standard way
to do string <-> datetime conversions
For example:
{code:java}
CAST( AS [FORMAT ])
CAST( AS [FORMAT ])
cast(dt as string format 'DD-MM-')
cast('01-05-2017' as date format 'DD-MM-')
{code}
[Stuff like this|http://bigdataprogrammers.com/string-date-conversion-hive/]
wouldn't need to happen.
*New SQL:2016 Patterns*
Examples:
{code:java}
--old SimpleDateFormat
select date_format('2015-05-15 12:00:00', 'MMM dd, HH:mm:ss');
--new SQL:2016 format
select date_format('2015-05-15 12:00:00', 'mon dd, hh:mi:ss');
{code}
Some other conflicting examples:
SimpleDateTime: 'MMM dd, HH:mm:ss'
SQL:2016: 'mon dd, hh:mi:ss'
SimpleDateTime: '-MM-dd HH:mm:ss'
SQL:2016: '-mm-dd hh24:mi:ss'
We will have a session-level feature flag to revert to the legacy Java
SimpleDateFormat patterns. This would allow users to chose the behavior they
desire and scope it to a session if need be.
For the full list of patterns, see subsection "Proposal for Impala’s datetime
patterns" in this doc:
[https://docs.google.com/document/d/1V7k6-lrPGW7_uhqM-FhKl3QsxwCRy69v2KIxPsGjc1k/edit]
*Existing Hive functions affected*
Other functions use SimpleDateFormat internally; these are the ones afaik where
SimpleDateFormat or some similar format is part of the input:
* from_unixtime(bigint unixtime[, string format])
* unix_timestamp(string date, string pattern)
* to_unix_timestamp(date[, pattern])
* add_months(string start_date, int num_months, output_date_format)
* trunc(string date, string format) - currently only supports
'MONTH'/'MON'/'MM', 'QUARTER'/'Q' and 'YEAR'/''/'YY' as format.
* date_format(date/timestamp/string ts, string fmt)
This description is a heavily edited description of IMPALA-4018.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-21574) return wrong result when excuting left join sql
Panda Song created HIVE-21574:
-
Summary: return wrong result when excuting left join sql
Key: HIVE-21574
URL: https://issues.apache.org/jira/browse/HIVE-21574
Project: Hive
Issue Type: Bug
Affects Versions: 3.1.0
Environment: hive 3.1.0 hdfs 3.1.1
Reporter: Panda Song
when I use a table instead of the sub select,I get the right result,much more
rows are joined together(metrics old_uv is bigger!!!)
Is there some bugs here?
Please help me ,thanks a lot!!
{quote}{{select }}
{{a.event_date,}}
{{count(distinct a.device_id) as uv,}}
{{count(distinct case when b.device_id is not null then b.device_id end) as
old_uv,}}
{{count(distinct a.device_id) - count(distinct case when b.device_id is not
null then b.device_id end) as new_uv}}
{{from}}
{{(}}
{{select}}
{{event_date,}}
{{device_id,}}
{{qingting_id}}
{{from datacenter.bl_page_chain_day}}
{{where event_date = '2019-03-31'}}
{{and (current_content like 'https://a.qingting.fm/membership5%'}}
{{or current_content like 'https://m.qingting.fm/vips/members%'}}
{{or current_content like 'https://sss.qingting.fm/vips/members/v2/%')}}
{{)a}}
{{left join}}
{{(select}}
{{ b.device_id}}
{{from}}
{{lzq_test.first_buy_vip a}}
{{inner join datacenter.device_qingting b on a.qingting_id = b.qingting_id}}
{{where a.first_buy < '2019-03-31'}}
{{group by b.device_id}}
{{)b}}
{{on a.device_id = b.device_id}}
{{group by a.event_date;}}{quote}
plan:
{quote}Plan optimized by CBO. |
Vertex dependency in root stage
Map 1 <- Map 3 (BROADCAST_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 5 <- Map 4 (CUSTOM_SIMPLE_EDGE), Reducer 2 (ONE_TO_ONE_EDGE)
Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 6
File Output Operator [FS_26]
Select Operator [SEL_25] (rows=35527639 width=349)
Output:["_col0","_col1","_col2","_col3"]
Group By Operator [GBY_24] (rows=35527639 width=349)
Output:["_col0","_col1","_col2"],aggregations:["count(DISTINCT
KEY._col1:0._col0)","count(DISTINCT KEY._col1:1._col0)"],keys:KEY._col0
<-Reducer 5 [SIMPLE_EDGE]
SHUFFLE [RS_23]
PartitionCols:_col0
Group By Operator [GBY_22] (rows=71055278 width=349)
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["count(DISTINCT
_col1)","count(DISTINCT _col2)"],keys:true, _col1, _col2
Select Operator [SEL_20] (rows=71055278 width=349)
Output:["_col1","_col2"]
Map Join Operator [MAPJOIN_45] (rows=71055278 width=349)
Conds:RS_17.KEY.reducesinkkey0=RS_18.KEY.reducesinkkey0(Right
Outer),Output:["_col0","_col1"]
<-Reducer 2 [ONE_TO_ONE_EDGE]
FORWARD [RS_17]
PartitionCols:_col0
Group By Operator [GBY_12] (rows=21738609 width=235)
Output:["_col0"],keys:KEY._col0
<-Map 1 [SIMPLE_EDGE]
SHUFFLE [RS_11]
PartitionCols:_col0
Group By Operator [GBY_10] (rows=43477219 width=235)
Output:["_col0"],keys:_col0
Map Join Operator [MAPJOIN_44] (rows=43477219 width=235)
Conds:SEL_2._col1=RS_7._col0(Inner),Output:["_col0"]
<-Map 3 [BROADCAST_EDGE]
BROADCAST [RS_7]
PartitionCols:_col0
Select Operator [SEL_5] (rows=301013 width=228)
Output:["_col0"]
Filter Operator [FIL_32] (rows=301013 width=228)
predicate:((first_buy < DATE'2019-03-31') and qingting_id is not null)
TableScan [TS_3] (rows=1062401 width=228)
lzq_test@first_buy_vip,a, transactional
table,Tbl:COMPLETE,Col:NONE,Output:["qingting_id","first_buy"]
<-Select Operator [SEL_2] (rows=39524744 width=235)
Output:["_col0","_col1"]
Filter Operator [FIL_31] (rows=39524744 width=235)
predicate:qingting_id is not null
TableScan [TS_0] (rows=39524744 width=235)
datacenter@device_qingting,b, ACID
table,Tbl:COMPLETE,Col:COMPLETE,Output:["device_id","qingting_id"]
<-Map 4 [CUSTOM_SIMPLE_EDGE]
PARTITION_ONLY_SHUFFLE [RS_18]
PartitionCols:_col0
Select Operator [SEL_16] (rows=64595706 width=349)
Output:["_col0"]
Filter Operator [FIL_33] (rows=64595706 width=349)
predicate:((current_content like 'https://a.qingting.fm/membership5%') or
(current_content like 'https://m.qingting.fm/vips/members%') or
(current_content like 'https://sss.qingting.fm/vips/members/v2/%'))
TableScan [TS_14] (rows=64595706 width=349)
datacenter@bl_page_chain_day,bl_page_chain_day,Tbl:COMPLETE,Col:NONE,Output:["device_id","current_content"]
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
