[
https://issues.apache.org/jira/browse/KYLIN-4427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076779#comment-17076779
]
Preeti V commented on KYLIN-4427:
---------------------------------
[~hit_lacus], thank you for the pointer about it being case-sensitive. I
updated it and now I don't see the wrong FS error anymore.
I do have a new error now "hive.HiveReservoirReporter:119 :
java.io.FileNotFoundException: /hdp/apps/2.6.5.1175-1/tez/tez.tar.gz: No such
file or directory." (detailed log at the end) and the metrics are still not
showing in the dashboard.
I checked on the Kylin machines and the file is under
/usr/hdp/2.6.5.1175-1/tez/lib. There is no "apps' folder.
!image-2020-04-06-15-46-37-356.png!
I looked at the Tez URI in Ambari and it is pointed to the folder name with
apps in it
!image-2020-04-06-17-14-52-443.png!
If I update it to /hdp/${hdp.version}/tez/tez.tar.gz or
/hdp/apps/${hdp.version}/tez/lib/tez.tar.gz, the BlockingReservoir still gives
the file not found error. In addition to that all Kylin Build jobs started
failing as soon as I updated the tez.lib.uris, with a tez error about file not
found. So I assume the original uri was needed for Kylin builds.
Is there any property I can set in the kylinMetrics.xml for the tez endpoint?
Detailed error log
2020-04-06 21:42:36,961 INFO [metrics-blocking-reservoir-scheduler-0]
client.AHSProxy:42 : Connecting to Application History server at
bn01eap00002f79.ap.gbl/10.175.241.176:10200
2020-04-06 21:42:36,962 INFO [metrics-blocking-reservoir-scheduler-0]
client.TezClient:351 : Session mode. Starting session.
2020-04-06 21:42:36,962 INFO [metrics-blocking-reservoir-scheduler-0]
client.TezClientUtils:173 : Using tez.lib.uris value from configuration:
/hdp/apps/2.6.5.1175-1/tez/tez.tar.gz
2020-04-06 21:42:36,968 ERROR [metrics-blocking-reservoir-scheduler-0]
hive.HiveReservoirReporter:119 : java.io.FileNotFoundException:
/hdp/apps/2.6.5.1175-1/tez/tez.tar.gz: No such file or directory.
java.lang.RuntimeException: java.io.FileNotFoundException:
/hdp/apps/2.6.5.1175-1/tez/tez.tar.gz: No such file or directory.
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582)
at
org.apache.kylin.metrics.lib.impl.hive.HiveProducer.write(HiveProducer.java:154)
at
org.apache.kylin.metrics.lib.impl.hive.HiveProducer.send(HiveProducer.java:122)
at
org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter$HiveReservoirListener.onRecordUpdate(HiveReservoirReporter.java:117)
at
org.apache.kylin.metrics.lib.impl.BlockingReservoir.notifyListenerOfUpdatedRecord(BlockingReservoir.java:105)
at
org.apache.kylin.metrics.lib.impl.BlockingReservoir.onRecordUpdate(BlockingReservoir.java:93)
at
org.apache.kylin.metrics.lib.impl.BlockingReservoir.access$300(BlockingReservoir.java:33)
at
org.apache.kylin.metrics.lib.impl.BlockingReservoir$ReporterRunnable.run(BlockingReservoir.java:152)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException:
/hdp/apps/2.6.5.1175-1/tez/tez.tar.gz: No such file or directory.
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatusInternal(NativeAzureFileSystem.java:2732)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(NativeAzureFileSystem.java:2644)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:771)
at
org.apache.tez.client.TezClientUtils.getLRFileStatus(TezClientUtils.java:132)
at
org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:183)
at org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:831)
at org.apache.tez.client.TezClient.start(TezClient.java:355)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:197)
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:579)
... 8 more
> Wrong FileSystem error when trying to enable system cubes and Dashboard in
> Kylin 2.6.4
> --------------------------------------------------------------------------------------
>
> Key: KYLIN-4427
> URL: https://issues.apache.org/jira/browse/KYLIN-4427
> Project: Kylin
> Issue Type: Bug
> Components: Metrics
> Affects Versions: v2.6.4
> Reporter: Preeti V
> Assignee: Xiaoxiang Yu
> Priority: Major
> Attachments: KylinMetrics.JPG, image-2020-04-03-10-45-15-290.png,
> image-2020-04-03-10-45-20-859.png, image-2020-04-03-10-45-45-894.png,
> image-2020-04-06-15-46-37-356.png, image-2020-04-06-17-14-52-443.png
>
>
> I am trying to enable system cubes for the Dashboard using Kylin version
> 2.6.4 The tables are created correctly and the cube builds successfully but
> there is no query or job data on the dashboard, it shows 0.
>
> We use Azure storage for Hive(wasb:// file system). I can see that there is
> no data being updated on the Hive_Metrics tables in Azure. In Kylin logs I
> see the below error
>
>
> 2020-03-12 20:02:41,790 ERROR [metrics-blocking-reservoir-scheduler-0]
> hive.HiveReservoirReporter:119 : Wrong FS:
> wasb://*****.blob.core.windows.net/hive/warehouse/kylin.db/hive_metrics_query_cube_qa/kday_date=2020-03-12,
> expected: hdfs://*****-prod-bn01
> java.lang.IllegalArgumentException: Wrong FS:
> wasb://*****.blob.core.windows.net/hive/warehouse/kylin.db/hive_metrics_query_cube_qa/kday_date=2020-03-12,
> expected: hdfs://*****-prod-bn01
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:666)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:214)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1442)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$26.doCall(DistributedFileSystem.java:1438)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1454)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1448)
> at
> org.apache.kylin.metrics.lib.impl.hive.HiveProducer.write(HiveProducer.java:137)
> at
> org.apache.kylin.metrics.lib.impl.hive.HiveProducer.send(HiveProducer.java:122)
> at
> org.apache.kylin.metrics.lib.impl.hive.HiveReservoirReporter$HiveReservoirListener.onRecordUpdate(HiveReservoirReporter.java:117)
> at
> org.apache.kylin.metrics.lib.impl.BlockingReservoir.notifyListenerOfUpdatedRecord(BlockingReservoir.java:105)
>
>
> I checked the hive configs and it has the warehouse metastore dir correctly
> pointing to azure. I found another thread with similar problem where they are
> trying to use S3 instead of hdfs.
> [http://apache-kylin.74782.x6.nabble.com/jira-Created-KYLIN-4385-KYLIN-system-cube-failing-to-update-table-when-run-on-EMR-with-S3-as-storageS-td14234.html]
>
>
> I also followed the recommendations here
> [https://www.mail-archive.com/[email protected]/msg04347.html] and
> enabled all the necessary config values.
> Is this a bug in Kylin or a configuration issue on my cluster? Any help or
> guidance is appreciated.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
