Hello Uday,
I don’t believe anyone has tried running 2 dags in parallel in local mode
within the same TezClient ( and definitely not for HiveServer2 ). If this is
with 2 instances of Tez client, this could likely be a bug in terms of either
how Hive is setting up the TezClient for local mode with the same directories
or a bug somewhere in Tez where clashing directories for intermediate data
might be causing an issue. FWIW, the Tez AM does not support running 2 dags in
parallel and quite a bit of this code path is used with local mode.
It would be great if you could file a JIRA for this with more detailed logs and
then take help of the dev community to come up with a patch that addresses the
issue in your environment.
thanks
— Hitesh
> On Aug 25, 2016, at 8:34 AM, Chitragar, Uday (KMLWG)
> <uday.chitra...@kantarmedia.com> wrote:
>
> Hello,
>
> When running parallel queries (simultaneous connections by two beeline
> clients to HS2), I get the following exception (full debug attached),
> interestingly running the queries one after the other completes without any
> problem.
>
> The setup is Hive (1.2.1) and Tez (0.8.4) running in local mode.
> Apologies in advance if this forum is not the right place for this question,
> thank you.
>
> 2016-08-25 15:45:41,333 DEBUG
> [TezTaskEventRouter{attempt_1472136335089_0001_1_01_000000_0}]:
> impl.ShuffleInputEventHandlerImpl
> (ShuffleInputEventHandlerImpl.java:processDataMovementEvent(127)) - DME
> srcIdx: 0, targetIndex: 9, attemptNum
> : 0, payload: [hasEmptyPartitions: true, host: , port: 0, pathComponent: ,
> runDuration: 0]
> 2016-08-25 15:45:41,557 ERROR [TezChild]: tez.MapRecordSource
> (MapRecordSource.java:processRow(90)) - java.lang.IllegalStateException:
> Invalid input path
> file:/acorn/QC/OraExtract/20160131/Devices/Devices_extract_20160229T080613_3
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.getNominalPath(MapOperator.java:415)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:457)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1069)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:501)
>
>
>
> 2016-08-25 15:45:41,817 INFO [TezChild]: io.HiveContextAwareRecordReader
> (HiveContextAwareRecordReader.java:doNext(326)) –
> Cannot get partition description from
> file:/acorn/QC/reportlib/VM_ValEdit.24656because cannot find dir = file:/ac
> orn/QC/reportlib/VM_ValEdit.24656 in pathToPartitionInfo:
> [file:/acorn/QC/OraExtract/20160131/Devices]
>
>
>
> Regards,
> Uday
>
>
>
>
> Kantar Disclaimer <hive.log.submit.gz>
