[jira] [Reopened] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping reopened HIVE-23748: - > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Assignee: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8
[jira] [Resolved] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping resolved HIVE-23748. - Assignee: wanguangping Resolution: Auto Closed > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Assignee: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) > INFO : Completed compiling > c
[jira] [Reopened] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping reopened HIVE-23748: - > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f); > Time taken: 78.5
[jira] [Issue Comment Deleted] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping updated HIVE-23748: Comment: was deleted (was: not fix ，problem still exist. I just want delete sensitive information in this issue，help me) > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) >
[jira] [Commented] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148425#comment-17148425 ] wanguangping commented on HIVE-23748: - not fix ，problem still exist. I just want delete sensitive information in this issue，help me > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:nu
[jira] [Commented] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148422#comment-17148422 ] wanguangping commented on HIVE-23748: - 1.not fix，problem still exists. 2.i just want to delete sensitive information in this issue.Although I edit description section to delete sensitive information，but in history section still retain sensitive information. 3.if you can directly delete sensitive information in history section ,i will reopen this issue. or if you delete sensitive information only by delete this issue ,i will new a issue for problem please help me delete sensitive information! aheartfulguy 邮箱：aheartful...@126.com 签名由 网易邮箱大师 定制 On 06/30/2020 15:42, Karen Coppage (Jira) wrote: [ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148396#comment-17148396 ] Karen Coppage commented on HIVE-23748: -- Hi [~wanguangping], this was resolved as Fixed, was there a fix? -- This message was sent by Atlassian Jira (v8.3.4#803005) > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, typ
[jira] [Resolved] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping resolved HIVE-23748. - Resolution: Fixed > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f
[jira] [Reopened] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping reopened HIVE-23748: - > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f); > Time taken: 78.5
[jira] [Reopened] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping reopened HIVE-23748: - > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f); > Time taken: 78.5
[jira] [Resolved] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping resolved HIVE-23748. - Resolution: Fixed > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f
[jira] [Resolved] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping resolved HIVE-23748. - Resolution: Fixed > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Connecting to jdbc:hive2://xxx:1/prod > Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) > Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) > Transaction isolation: TRANSACTION_REPEATABLE_READ > INFO : Compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.887 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): > use prod > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); > Time taken: 0.197 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (1.096 seconds) > No rows affected (0.004 seconds) > INFO : Compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 1.324 seconds > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : Executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): > drop table if exists temp.shawnlee_newbase_devicebase > INFO : Starting task [Stage-0:DDL] in serial mode > INFO : Completed executing > command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); > Time taken: 12.895 seconds > INFO : OK > INFO : Concurrency mode is disabled, not creating a lock manager > No rows affected (14.229 seconds) > INFO : Compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): > x > INFO : Concurrency mode is disabled, not creating a lock manager > INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, > event > INFO : Semantic Analysis Completed (retrial = false) > INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, > type:string, comment:null), FieldSchema(name:device_id, type:string, > comment:null), FieldSchema(name:is_new, type:int, comment:null), > FieldSchema(name:first_attribute, type:map, comment:null), > FieldSchema(name:first_app_version, type:string, comment:null), > FieldSchema(name:first_platform_type, type:string, comment:null), > FieldSchema(name:first_manufacturer, type:string, comment:null), > FieldSchema(name:first_model, type:string, comment:null), > FieldSchema(name:first_ipprovince, type:string, comment:null), > FieldSchema(name:first_ipcity, type:string, comment:null), > FieldSchema(name:last_attribute, type:map, comment:null), > FieldSchema(name:last_app_version, type:string, comment:null), > FieldSchema(name:last_platform_type, type:string, comment:null), > FieldSchema(name:last_manufacturer, type:string, comment:null), > FieldSchema(name:last_model, type:string, comment:null), > FieldSchema(name:last_ipprovince, type:string, comment:null), > FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) > INFO : Completed compiling > command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f
[jira] [Updated] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping updated HIVE-23748: Description: h1. background * SQL on TEZ * it's a Occasional problem h1. hiveserver2 log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://xxx:1/prod Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) Transaction isolation: TRANSACTION_REPEATABLE_READ No rows affected (0.04 seconds) No rows affected (0.004 seconds) No rows affected (0.003 seconds) No rows affected (0.004 seconds) No rows affected (0.003 seconds) No rows affected (0.004 seconds) No rows affected (0.003 seconds) No rows affected (0.003 seconds) No rows affected (0.004 seconds) INFO : Compiling command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): use prod INFO : Concurrency mode is disabled, not creating a lock manager INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); Time taken: 0.887 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): use prod INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); Time taken: 0.197 seconds INFO : OK INFO : Concurrency mode is disabled, not creating a lock manager No rows affected (1.096 seconds) No rows affected (0.004 seconds) INFO : Compiling command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): drop table if exists temp.shawnlee_newbase_devicebase INFO : Concurrency mode is disabled, not creating a lock manager INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); Time taken: 1.324 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): drop table if exists temp.shawnlee_newbase_devicebase INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); Time taken: 12.895 seconds INFO : OK INFO : Concurrency mode is disabled, not creating a lock manager No rows affected (14.229 seconds) INFO : Compiling command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): x INFO : Concurrency mode is disabled, not creating a lock manager INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, event INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, type:string, comment:null), FieldSchema(name:device_id, type:string, comment:null), FieldSchema(name:is_new, type:int, comment:null), FieldSchema(name:first_attribute, type:map, comment:null), FieldSchema(name:first_app_version, type:string, comment:null), FieldSchema(name:first_platform_type, type:string, comment:null), FieldSchema(name:first_manufacturer, type:string, comment:null), FieldSchema(name:first_model, type:string, comment:null), FieldSchema(name:first_ipprovince, type:string, comment:null), FieldSchema(name:first_ipcity, type:string, comment:null), FieldSchema(name:last_attribute, type:map, comment:null), FieldSchema(name:last_app_version, type:string, comment:null), FieldSchema(name:last_platform_type, type:string, comment:null), FieldSchema(name:last_manufacturer, type:string, comment:null), FieldSchema(name:last_model, type:string, comment:null), FieldSchema(name:last_ipprovince, type:string, comment:null), FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f); Time taken: 78.517 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): INFO : Query ID = hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f INFO : Total jobs = 3 INFO
[jira] [Updated] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping updated HIVE-23748: Description: h1. background * SQL on TEZ * it's a Occasional problem h1. hiveserver2 log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://xxx:1/prod Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) Transaction isolation: TRANSACTION_REPEATABLE_READ INFO : Compiling command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): use prod INFO : Concurrency mode is disabled, not creating a lock manager INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); Time taken: 0.887 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): use prod INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); Time taken: 0.197 seconds INFO : OK INFO : Concurrency mode is disabled, not creating a lock manager No rows affected (1.096 seconds) No rows affected (0.004 seconds) INFO : Compiling command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): drop table if exists temp.shawnlee_newbase_devicebase INFO : Concurrency mode is disabled, not creating a lock manager INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); Time taken: 1.324 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): drop table if exists temp.shawnlee_newbase_devicebase INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); Time taken: 12.895 seconds INFO : OK INFO : Concurrency mode is disabled, not creating a lock manager No rows affected (14.229 seconds) INFO : Compiling command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): x INFO : Concurrency mode is disabled, not creating a lock manager INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, event INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, type:string, comment:null), FieldSchema(name:device_id, type:string, comment:null), FieldSchema(name:is_new, type:int, comment:null), FieldSchema(name:first_attribute, type:map, comment:null), FieldSchema(name:first_app_version, type:string, comment:null), FieldSchema(name:first_platform_type, type:string, comment:null), FieldSchema(name:first_manufacturer, type:string, comment:null), FieldSchema(name:first_model, type:string, comment:null), FieldSchema(name:first_ipprovince, type:string, comment:null), FieldSchema(name:first_ipcity, type:string, comment:null), FieldSchema(name:last_attribute, type:map, comment:null), FieldSchema(name:last_app_version, type:string, comment:null), FieldSchema(name:last_platform_type, type:string, comment:null), FieldSchema(name:last_manufacturer, type:string, comment:null), FieldSchema(name:last_model, type:string, comment:null), FieldSchema(name:last_ipprovince, type:string, comment:null), FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f); Time taken: 78.517 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): INFO : Query ID = hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f INFO : Total jobs = 3 INFO : Launching Job 1 out of 3 INFO : Starting task [Stage-1:MAPRED] in serial mode INFO : Subscribed to counters: [] for queryId: hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f INFO : Tez session hasn't been created yet. Opening session INFO : Dag name: INFO : Status: Running (Executing
[jira] [Updated] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping updated HIVE-23748: Description: h1. background * SQL on TEZ * it's a Occasional problem h1. hiveserver2 log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://xxx:1/prod Connected to: Apache Hive (version 3.1.0.3.1.4.0-315) Driver: Hive JDBC (version 3.1.0.3.1.4.0-315) Transaction isolation: TRANSACTION_REPEATABLE_READ No rows affected (0.04 seconds) No rows affected (0.004 seconds) No rows affected (0.003 seconds) No rows affected (0.004 seconds) No rows affected (0.003 seconds) No rows affected (0.004 seconds) No rows affected (0.003 seconds) No rows affected (0.003 seconds) No rows affected (0.004 seconds) INFO : Compiling command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): use prod INFO : Concurrency mode is disabled, not creating a lock manager INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); Time taken: 0.887 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2): use prod INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20200609033313_ed882b48-7ab4-42a2-84e4-c9ef764271e2); Time taken: 0.197 seconds INFO : OK INFO : Concurrency mode is disabled, not creating a lock manager No rows affected (1.096 seconds) No rows affected (0.004 seconds) INFO : Compiling command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): drop table if exists temp.shawnlee_newbase_devicebase INFO : Concurrency mode is disabled, not creating a lock manager INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); Time taken: 1.324 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23): drop table if exists temp.shawnlee_newbase_devicebase INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20200609033314_cba66b08-ad42-4b94-ad61-d15fe48efe23); Time taken: 12.895 seconds INFO : OK INFO : Concurrency mode is disabled, not creating a lock manager No rows affected (14.229 seconds) INFO : Compiling command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): x INFO : Concurrency mode is disabled, not creating a lock manager INFO : No Stats for user_profile@dw_uba_event_daily, Columns: attribute, event INFO : Semantic Analysis Completed (retrial = false) INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:day, type:string, comment:null), FieldSchema(name:device_id, type:string, comment:null), FieldSchema(name:is_new, type:int, comment:null), FieldSchema(name:first_attribute, type:map, comment:null), FieldSchema(name:first_app_version, type:string, comment:null), FieldSchema(name:first_platform_type, type:string, comment:null), FieldSchema(name:first_manufacturer, type:string, comment:null), FieldSchema(name:first_model, type:string, comment:null), FieldSchema(name:first_ipprovince, type:string, comment:null), FieldSchema(name:first_ipcity, type:string, comment:null), FieldSchema(name:last_attribute, type:map, comment:null), FieldSchema(name:last_app_version, type:string, comment:null), FieldSchema(name:last_platform_type, type:string, comment:null), FieldSchema(name:last_manufacturer, type:string, comment:null), FieldSchema(name:last_model, type:string, comment:null), FieldSchema(name:last_ipprovince, type:string, comment:null), FieldSchema(name:last_ipcity, type:string, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f); Time taken: 78.517 seconds INFO : Concurrency mode is disabled, not creating a lock manager INFO : Executing command(queryId=hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f): INFO : Query ID = hive_20200609033329_3fbf0a38-e5b0-4e3a-ae8b-ef95f400b50f INFO : Total jobs = 3 INFO
[jira] [Updated] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[
https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wanguangping updated HIVE-23748:
Description:
h1. background
* SQL on TEZ
* it's a Occasional problem
h1. hiveserver2 log
flume failed to collect logs ,collect local log files now.
***
***
TIME 1 RUN*
***
***
2020-06-09 03:33:11 INFO log dir:/home/data/wwwuser/falcon-runner/logs
2020-06-09 03:33:11 INFO sql usage1: comments
2020-06-09 03:33:11 INFO set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.auto.convert.join=true;
set mapred.job.queue.name=product;
set tez.queue.name=product;
set tez.queue.name=product;
set mapred.job.name=hivetask_smile.wang_994328;
set hive.task.name=hivetask_smile.wang_994328;
set mapred.job.priority=NORMAL;
use prod;
--SQL is written below--
set tez.queue.name=superman;
drop table if exists temp.shawnlee_newbase_devicebase;
create table if not exists temp.shawnlee_newbase_devicebase as
with is_first_day as (
select
day,
attribute['\$device_id'] device_id
from user_profile.dw_uba_event_daily
where day >= date_sub(current_date(),20) and day <=date_sub(current_date(),1)
and event !='\$AppStartPassively'
and attribute['\$is_first_day'] = 1
and attribute['platform_type'] in ('Android','iOS')
and project = 'default'
and attribute['\$device_id'] is not null
group by day,attribute['\$device_id']
),
last_msg as (
select
a.day,
a.device_id,
attribute as last_attribute
from
(select
day,
attribute['time'] ts,
attribute['\$device_id'] device_id,
attribute,
row_number() over(partition by day,attribute['\$device_id'] order by
attribute['time'] desc) row_number
from user_profile.dw_uba_event_daily
where day >= date_sub(current_date(),20) and day <=date_sub(current_date(),1)
and event !='\$AppStartPassively'
and attribute['platform_type'] in ('Android','iOS')
and attribute['\$device_id'] is not null
and project = 'default') a
where a.row_number=1
),
first_msg as (
select
a.day,
a.device_id,
attribute as first_attribute
from
(select
day,
attribute['time'] ts,
attribute['\$device_id'] device_id,
attribute,
row_number() over(partition by day,attribute['\$device_id'] order by
attribute['time'] ) row_number
from user_profile.dw_uba_event_daily
where day >= date_sub(current_date(),20) and day <=date_sub(current_date(),1)
and event !='\$AppStartPassively'
and attribute['platform_type'] in ('Android','iOS')
and attribute['\$device_id'] is not null
and project = 'default') a
where a.row_number=1
),
rihuo as (
select
day,
attribute['\$device_id'] device_id
from user_profile.dw_uba_event_daily
where day >= date_sub(current_date(),20) and day <=date_sub(current_date(),1)
and event !='\$AppStartPassively'
and attribute['platform_type'] in ('Android','iOS')
and project = 'default'
and attribute['\$device_id'] is not null
group by day,attribute['\$device_id']
)
select
a.day,
a.device_id,
(case when b.device_id is not null then 1 else 0 end) is_new,
c.first_attribute,
c.first_attribute['\$app_version'] as
first_app_version,c.first_attribute['platform_type'] as first_platform_type,
c.first_attribute['\$manufacturer'] as
first_manufacturer,c.first_attribute['\$model'] as first_model,
c.first_attribute['\$province'] as
first_ipprovince,c.first_attribute['\$city'] as first_ipcity,
d.last_attribute,
d.last_attribute['\$app_version'] as
last_app_version,d.last_attribute['platform_type'] as last_platform_type,
d.last_attribute['\$manufacturer'] as
last_manufacturer,d.last_attribute['\$model'] as last_model,
d.last_attribute['\$province'] as last_ipprovince,d.last_attribute['\$city']
as last_ipcity
from rihuo a
left join is_first_day b on a.day=b.day and a.device_id=b.device_id
left join first_msg c on a.day=c.day and a.device_id=c.device_id
left join last_msg d on a.day=d.day and a.device_id=d.device_id
;
insert overwrite table user_devicebase_shawnlee_app PARTITION (day)
select
device_id,
is_new,
first_attribute,
first_app_version,
first_platform_type,
first_manufacturer,
first_model,
first_ipprovince,
first_ipcity,
last_attribute,
last_app_version,
last_platform_type,
last_manufacturer,
last_model,
last_ipprovince,
last_ipcity,
day
from temp.shawnlee_newbase_devicebase
;
--SQL is written above--
"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/hdp/3.1.4.0-315/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/hdp/3.1.4.0-315/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/co
[jira] [Updated] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping updated HIVE-23748: Description: h1. background * SQL on TEZ * it's a Occasional problem h1. hiveserver2 log was: h1. [^hiveserver2 log.txt]background * SQL on TEZ * it's a Occasional problem h1. hiveserver2 log [^hiveserver2 log.txt] > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23748) tez task with File Merge operator generate tmp file with wrong suffix
[ https://issues.apache.org/jira/browse/HIVE-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanguangping updated HIVE-23748: Description: h1. [^hiveserver2 log.txt]background * SQL on TEZ * it's a Occasional problem h1. hiveserver2 log [^hiveserver2 log.txt] was: h1. background * SQL on TEZ * it's a Occasional problem h1. hiveserver2 log [^hiveserver2 log.txt] > tez task with File Merge operator generate tmp file with wrong suffix > - > > Key: HIVE-23748 > URL: https://issues.apache.org/jira/browse/HIVE-23748 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: wanguangping >Priority: Major > > h1. [^hiveserver2 log.txt]background > * SQL on TEZ > * it's a Occasional problem > h1. hiveserver2 log > [^hiveserver2 log.txt] > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
