[jira] [Updated] (MAPREDUCE-6736) import hive table from parquet files, there is no 'job.splitmetainfo' file message
[ https://issues.apache.org/jira/browse/MAPREDUCE-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ha, Hun Cheol updated MAPREDUCE-6736: - Description: same issue : https://issues.apache.org/jira/browse/MAPREDUCE-3056 which is created on 2011.09.21 and fixed 2011.10.04 there is user(Sergey) who has same issue on 2015.05.13 too! (look last comment of above link) on beeline prompt, I've tried import hive table from parquet files that are exported from another hive table but there was no "job.splitmetainfo" file on staging directory. So FileNotFoundException occurs I tried with yarn user, hdfs user with different permissions and directory ownership, but nothing changed. full log messages below == Log Type: syslog Log Upload Time: Wed Jul 20 17:57:36 +0900 2016 Log Length: 21439 2016-07-20 17:57:26,139 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1468834620182_0036_01 2016-07-20 17:57:26,417 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2016-07-20 17:57:26,463 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2016-07-20 17:57:26,463 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@52af26ee) 2016-07-20 17:57:26,510 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. 2016-07-20 17:57:26,991 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2016-07-20 17:57:27,091 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2016-07-20 17:57:27,154 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2016-07-20 17:57:27,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.tools.mapred.CopyCommitter 2016-07-20 17:57:27,231 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2016-07-20 17:57:27,232 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2016-07-20 17:57:27,233 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2016-07-20 17:57:27,234 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 2016-07-20 17:57:27,235 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2016-07-20 17:57:27,241 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2016-07-20 17:57:27,242 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2016-07-20 17:57:27,243 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2016-07-20 17:57:27,292 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020] 2016-07-20 17:57:27,322 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020] 2016-07-20 17:57:27,352 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020] 2016-07-20 17:57:27,382 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Perms after creating 448, Expected: 448 2016-07-20 17:57:27,387 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled 2016-07-20 17:57:27,428 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher:
[jira] [Updated] (MAPREDUCE-6736) import hive table from parquet files, there is no 'job.splitmetainfo' file message
[ https://issues.apache.org/jira/browse/MAPREDUCE-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ha, Hun Cheol updated MAPREDUCE-6736: - Description: same issue : https://issues.apache.org/jira/browse/MAPREDUCE-3056 which is created on 2011.09.21 and fixed 2011.10.04 there is user(Sergey) who has same issue on 2015.05.13 too! (look last comment of above link) on beeline prompt, I've tried import hive table from parquet files that are exported from another hive table but there was no "job.splitmetainfo" file on staging directory. So FileNotFoundException occurs full log messages below == Log Type: syslog Log Upload Time: Wed Jul 20 17:57:36 +0900 2016 Log Length: 21439 2016-07-20 17:57:26,139 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1468834620182_0036_01 2016-07-20 17:57:26,417 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2016-07-20 17:57:26,463 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2016-07-20 17:57:26,463 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@52af26ee) 2016-07-20 17:57:26,510 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. 2016-07-20 17:57:26,991 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2016-07-20 17:57:27,091 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2016-07-20 17:57:27,154 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2016-07-20 17:57:27,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.tools.mapred.CopyCommitter 2016-07-20 17:57:27,231 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2016-07-20 17:57:27,232 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2016-07-20 17:57:27,233 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2016-07-20 17:57:27,234 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 2016-07-20 17:57:27,235 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2016-07-20 17:57:27,241 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2016-07-20 17:57:27,242 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2016-07-20 17:57:27,243 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2016-07-20 17:57:27,292 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020] 2016-07-20 17:57:27,322 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020] 2016-07-20 17:57:27,352 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020] 2016-07-20 17:57:27,382 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Perms after creating 448, Expected: 448 2016-07-20 17:57:27,387 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled 2016-07-20 17:57:27,428 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
[jira] [Updated] (MAPREDUCE-6736) import hive table from parquet files, there is no 'job.splitmetainfo' file message
[ https://issues.apache.org/jira/browse/MAPREDUCE-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ha, Hun Cheol updated MAPREDUCE-6736: - Environment: Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) Hadoop 2.6.0-cdh5.7.0 was: Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) Hadoop 2.6.0-cdh5.7.0 Subversion http://github.com/cloudera/hadoop -r c00978c67b0d3fe9f3b896b5030741bd40bf541a Compiled by jenkins on 2016-03-23T18:36Z Compiled with protoc 2.5.0 >From source with checksum b2eabfa328e763c88cb14168f9b372 This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.7.0.jar > import hive table from parquet files, there is no 'job.splitmetainfo' file > message > -- > > Key: MAPREDUCE-6736 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6736 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 2.6.0 > Environment: Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) > Hadoop 2.6.0-cdh5.7.0 >Reporter: Ha, Hun Cheol >Priority: Blocker > > same issue : https://issues.apache.org/jira/browse/MAPREDUCE-3056 > which is created on 2011.09.21 and fixed 2011.10.04 > there is user(Sergey) who has same issue on 2015.05.13 too! (look last > comment of above link) > on beeline prompt, try import hive table from parquet files that are exported > from another hive table, there is no 'job.splitmetainfo' file on staging > directory : FileNotFoundException occurs > full log messages below > == > Log Type: syslog > Log Upload Time: Wed Jul 20 17:57:36 +0900 2016 > Log Length: 21439 > 2016-07-20 17:57:26,139 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for > application appattempt_1468834620182_0036_01 > 2016-07-20 17:57:26,417 WARN [main] org.apache.hadoop.util.NativeCodeLoader: > Unable to load native-hadoop library for your platform... using builtin-java > classes where applicable > 2016-07-20 17:57:26,463 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: > 2016-07-20 17:57:26,463 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, > Service: , Ident: > (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@52af26ee) > 2016-07-20 17:57:26,510 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. > 2016-07-20 17:57:26,991 WARN [main] > org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit > local reads feature cannot be used because libhadoop cannot be loaded. > 2016-07-20 17:57:27,091 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config > null > 2016-07-20 17:57:27,154 INFO [main] > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output > Committer Algorithm version is 1 > 2016-07-20 17:57:27,159 INFO [main] > org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is > org.apache.hadoop.tools.mapred.CopyCommitter > 2016-07-20 17:57:27,231 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.jobhistory.EventType for class > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler > 2016-07-20 17:57:27,232 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher > 2016-07-20 17:57:27,233 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher > 2016-07-20 17:57:27,234 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher > 2016-07-20 17:57:27,235 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler > 2016-07-20 17:57:27,241 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher > 2016-07-20 17:57:27,242 INFO [main] > org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class > org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class >
[jira] [Created] (MAPREDUCE-6736) import hive table from parquet files, there is no 'job.splitmetainfo' file message
Ha, Hun Cheol created MAPREDUCE-6736: Summary: import hive table from parquet files, there is no 'job.splitmetainfo' file message Key: MAPREDUCE-6736 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6736 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 2.6.0 Environment: Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64) Hadoop 2.6.0-cdh5.7.0 Subversion http://github.com/cloudera/hadoop -r c00978c67b0d3fe9f3b896b5030741bd40bf541a Compiled by jenkins on 2016-03-23T18:36Z Compiled with protoc 2.5.0 >From source with checksum b2eabfa328e763c88cb14168f9b372 This command was run using /usr/lib/hadoop/hadoop-common-2.6.0-cdh5.7.0.jar Reporter: Ha, Hun Cheol Priority: Blocker same issue : https://issues.apache.org/jira/browse/MAPREDUCE-3056 which is created on 2011.09.21 and fixed 2011.10.04 there is user(Sergey) who has same issue on 2015.05.13 too! (look last comment of above link) on beeline prompt, try import hive table from parquet files that are exported from another hive table, there is no 'job.splitmetainfo' file on staging directory : FileNotFoundException occurs full log messages below == Log Type: syslog Log Upload Time: Wed Jul 20 17:57:36 +0900 2016 Log Length: 21439 2016-07-20 17:57:26,139 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1468834620182_0036_01 2016-07-20 17:57:26,417 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2016-07-20 17:57:26,463 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: 2016-07-20 17:57:26,463 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@52af26ee) 2016-07-20 17:57:26,510 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. 2016-07-20 17:57:26,991 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 2016-07-20 17:57:27,091 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 2016-07-20 17:57:27,154 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1 2016-07-20 17:57:27,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.tools.mapred.CopyCommitter 2016-07-20 17:57:27,231 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 2016-07-20 17:57:27,232 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 2016-07-20 17:57:27,233 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 2016-07-20 17:57:27,234 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 2016-07-20 17:57:27,235 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 2016-07-20 17:57:27,241 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 2016-07-20 17:57:27,242 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 2016-07-20 17:57:27,243 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 2016-07-20 17:57:27,292 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://da74:8020] 2016-07-20 17:57:27,322 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system
