[jira] [Commented] (HIVE-14369) submit a task with hive on spark to other yarn cluster failed
[
https://issues.apache.org/jira/browse/HIVE-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399006#comment-15399006
]
Xiaomeng Huang commented on HIVE-14369:
---
Hi [~ruili]
It can't work, too. I think I should set both hivecluster and sparkcluster in
hdfs-site.xml of sparkcluster.
> submit a task with hive on spark to other yarn cluster failed
> -
>
> Key: HIVE-14369
> URL: https://issues.apache.org/jira/browse/HIVE-14369
> Project: Hive
> Issue Type: Bug
>Reporter: Xiaomeng Huang
>
> In our environment, we have two hadoop clusters with HA, named hivecluster
> and sparkcluster.
> hivecluster is a HA hadoop cluster for hive, which has large hard disk.
> sparkcluster is a HA hadoop cluster for spark, which has large memory.
> e.g. below is a hdfs-site.xml of hivecluster:
> {code}
>
> dfs.ha.namenodes.hivecluster
> nn1,nn2
>
>
> dfs.namenode.rpc-address.hivecluster.nn1
> 10.17.21.32:9000
>
>
> dfs.namenode.rpc-address.hivecluster.nn2
> 10.17.21.77:9000
>
>
> dfs.namenode.http-address.hivecluster.nn1
> 10.17.21.32:50070
>
>
> dfs.namenode.http-address.hivecluster.nn2
> 10.17.21.77:50070
>
> {code}
> Firstly, I created a hive table located as
> hdfs://hivecluster/hive/warehouse/xxx
> If I use hive on mr, it will run successfully.
> But if I use hive on spark to submit a task to yarn cluster of sparkcluster,
> it says:
> {code}
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> yarn日志显示：
> Diagnostics: java.lang.IllegalArgumentException:
> java.net.UnknownHostException: hivecluster
> Failing this attempt. Failing the application.
> {code}
> I didn't set host of hivecluster into hdfs-site.xml of sparkcluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14369) submit a task with hive on spark to other yarn cluster failed
[
https://issues.apache.org/jira/browse/HIVE-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398610#comment-15398610
]
Rui Li commented on HIVE-14369:
---
[~Huang Xiaomeng] discussed this with me offline and we think the possible
reason is sparkcluster cannot resolve the logical URI of the hive table.
I'm not sure if this is specific to HoS. Xiaomeng, how did you submit the spark
job to sparkcluster? Can you try the same thing with HoMR?
> submit a task with hive on spark to other yarn cluster failed
> -
>
> Key: HIVE-14369
> URL: https://issues.apache.org/jira/browse/HIVE-14369
> Project: Hive
> Issue Type: Bug
>Reporter: Xiaomeng Huang
>
> In our environment, we have two hadoop clusters with HA, named hivecluster
> and sparkcluster.
> hivecluster is a HA hadoop cluster for hive, which has large hard disk.
> sparkcluster is a HA hadoop cluster for spark, which has large memory.
> e.g. below is a hdfs-site.xml of hivecluster:
> {code}
>
> dfs.ha.namenodes.hivecluster
> nn1,nn2
>
>
> dfs.namenode.rpc-address.hivecluster.nn1
> 10.17.21.32:9000
>
>
> dfs.namenode.rpc-address.hivecluster.nn2
> 10.17.21.77:9000
>
>
> dfs.namenode.http-address.hivecluster.nn1
> 10.17.21.32:50070
>
>
> dfs.namenode.http-address.hivecluster.nn2
> 10.17.21.77:50070
>
> {code}
> Firstly, I created a hive table located as
> hdfs://hivecluster/hive/warehouse/xxx
> If I use hive on mr, it will run successfully.
> But if I use hive on spark to submit a task to yarn cluster of sparkcluster,
> it says:
> {code}
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.spark.SparkTask
> yarn日志显示：
> Diagnostics: java.lang.IllegalArgumentException:
> java.net.UnknownHostException: hivecluster
> Failing this attempt. Failing the application.
> {code}
> I didn't set host of hivecluster into hdfs-site.xml of sparkcluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
