Hi, We have just solved this problem. I would like to talk about our solution to contribute to the community. Maybe other friends may face the same problem. We noticed the reason for that problem is trying to connect CDH's HDFS without client deployment. So, firstly we deployed our Hadoop client files into /opt/apache-drill/conf folder. Then we were able to query data without block error and connect hdfs namenode with name service name (as high available).
F.y.i. Have a nice day. Apache <[email protected]>, 4 Mar 2021 Per, 18:34 tarihinde şunu yazdı: > Hi, > I think the solution to this problem depends on your initiative. Because I > planned to download CDH 6.3.3, but it seems that Cloudera have requested a > license from February 1. Orz... > So, Is it possible that you are using mini cluster (installed HDFS roles > service only) to confirm that drill can running based on 6.3.3. thanks > > > 在 2021年3月3日，15:30，Mehmet - <[email protected]> 写道： > > > > Hi, > > > > I don't have any plan yet but I was trying it on our Dev cluster, I'm > > thinking of trying it on the prod cluster as well. > > Btw, you're right, as far as I see in drillbit.log and > > "jars/3rdparty/hadoop-client-3.2.1.jar" folder, I have been using > > hadoop-client-3.2.1.jar (My Hadoop version is 3.0-cdh6.3.3). Actually > > Drill's client is higher than mine. I think there shouldn't be a problem. > > What do you think? > > > > Thank you, > > BR. > > > > luoc <[email protected]>, 1 Mar 2021 Pzt, 13:18 tarihinde şunu yazdı: > > > >> Hi, > >> I am so sorry to hear that. Do you have any plans? And, have you used > >> hadoop-client-3.2.1.jar to connect the cluster directly? because of the > >> drill include the hadoop version is 3.2.1. > >>> 2021年2月28日 上午4:28，Mehmet - <[email protected]> 写道： > >>> Hi, > >>> Yes, you're right and I agree with you. Regarding your last comment; I > >> can get and read any file from HDFS via terminal. I have attached the > >> ScreenShot. > >>> Thank you. > >>> 27 Şub 2021 Cmt, saat 05:58 tarihinde luoc <[email protected] <mailto: > >> [email protected]>> şunu yazdı: > >>> Hi, > >>> Drill 1.18 works well on CDH 5.13. the difference is that I did not > >> enable krb, then drill is supported for hadoop 3.x. (CDH 5.x based on > >> hadoop 2.6, 6.x based on hadoop 3.x) > >>> May the `list` commands only need to connect NameNode for get metadata > >> (without DataNode), So recommend that you using DFS client to connect > the > >> cluster and test whether the file can be read. Because of the log point > out > >> the dfs client have a problem on read file. > >>> Don’t paste the image for apache email, this is not supported. as an > >> attachment file is a simple way. > >>>> 2021年2月27日 上午12:10，Mehmet - <[email protected] <mailto: > >> [email protected]>> 写道： > >>>> Hi, > >>>> Stack Trace: https://paste.ubuntu.com/p/nFvygSpcjy/ < > >> https://paste.ubuntu.com/p/nFvygSpcjy/> < > >> https://paste.ubuntu.com/p/nFvygSpcjy/ < > >> https://paste.ubuntu.com/p/nFvygSpcjy/>> > >>>> Yes DataNode port is opening and accessible and any firewall problem > >> is impossible because the Drill was established on the same nodes with > >> Cloudera cluster. > >>>> I consider there is no problem with authentication. Because I can list > >> the hdfs folder clearly as below: > >>>> Via Terminal (Note: I have used the same kerberos user that I set > >> within Drill's jaas.conf folder. ); > >>>> Thank you. > >>>> BR. > >>>> luoc <[email protected] <mailto:[email protected]> <mailto: > [email protected] > >> <mailto:[email protected]>>>, 26 Şub 2021 Cum, 17:58 tarihinde şunu > yazdı: > >>>> Hi, > >>>> The storage config is correct. then enabled the Kerberos security. > >> So, please check the java stack trace to ensure not the authentication > >> problem. > >>>> Is it possible to use dfs client connect the HDFS for reading the > >> csv file? > >>>> The DataNode port is opening and accessible? > >>>>> 2021年2月26日 下午9:52，Mehmet - <[email protected] <mailto: > >> [email protected]> <mailto:[email protected] <mailto: > >> [email protected]>>> 写道： > >>>>> Hi, > >>>>> 1. Drill version: 1.18.0 > >>>>> 2. HDFS Version: Hadoop 3.0-cdh6.3.3 > >>>>> 3. Storage config: https://paste.ubuntu.com/p/5Dk9jVCxYr/ < > >> https://paste.ubuntu.com/p/5Dk9jVCxYr/> < > >> https://paste.ubuntu.com/p/5Dk9jVCxYr/ < > >> https://paste.ubuntu.com/p/5Dk9jVCxYr/>> > >>>>> 4. drill-env.sh file: https://paste.ubuntu.com/p/MGNG4zhbrk/ < > >> https://paste.ubuntu.com/p/MGNG4zhbrk/> < > >> https://paste.ubuntu.com/p/MGNG4zhbrk/ < > >> https://paste.ubuntu.com/p/MGNG4zhbrk/>> > >>>>> Thank you. > >>>>> BR. > >>>>> luoc <[email protected] <mailto:[email protected]> <mailto: > >> [email protected] <mailto:[email protected]>>>, 26 Şub 2021 Cum, 16:14 > >> tarihinde şunu yazdı: > >>>>>> Hi, > >>>>>> That does not seem like an issues with Drill. > >>>>>> Would you please provides more helpful information : > >>>>>> 1. Drill version > >>>>>> 2. HDFS version > >>>>>> 3. Storage config > >>>>>>> 2021年2月26日 下午3:32，Mehmet - <[email protected] <mailto: > >> [email protected]> <mailto:[email protected] <mailto: > >> [email protected]>>> 写道： > >>>>>>> Hi Team, > >>>>>>> I have a problem with Hdfs query on Drill. When I run a "SHOW > >> FILES in > >>>>>>> root.`tmp/` ", I can list the files correctly. > >>>>>>> Bu when I run a select query like "Select * from root.`tmp/` it > >> throws > >>>>>>> below error. > >>>>>>> Notes: > >>>>>>> - I have already checked the state of hdfs health(via dfsadmin and > >> hdfs > >>>>>> ui) > >>>>>>> and there is no any corruption or block error. > >>>>>>> - Drillbits are on the same cluster with Hadoop. So I think any > >> network > >>>>>>> problem is impossible. > >>>>>>> - I have also set dfs.client.use.datanode.hostname as true ( > >>>>>>> https://stackoverflow.com/a/55290406/7894534 < > >> https://stackoverflow.com/a/55290406/7894534> < > >> https://stackoverflow.com/a/55290406/7894534 < > >> https://stackoverflow.com/a/55290406/7894534>> ) > >>>>>>> org.apache.drill.common.exceptions.UserRemoteException: DATA_READ > >> ERROR: > >>>>>>> Could not obtain block: BP-2026912985-<namenode_ip>- > >>>>>>> 1569935018133:blk_1073842201_101390 file=/tmp/2015-summary.csv > >>>>>>> File Path: hdfs://<drillbit_ip>:8020/tmp/2015-summary.csv > >>>>>>> Fragment: 0:0 [Error Id: 466835bd-6512-4854-b231-eaa439eba6f2 on > >>>>>>> <drillbit_ip>:31010] > >>>>>>> Thank you. > >>>>>>> -- > >>>>>>> Mehmet ERSOY > >>>>> -- > >>>>> Mehmet ERSOY > >>>> -- > >>>> Mehmet ERSOY > > > > -- > > Mehmet ERSOY > > -- Mehmet ERSOY
