I saw that your hive table is with property "transactional" = "true",
And Doris only support ACID table with Hive 3, not support with Hive 2.x.
So you may need to create non-transactional hive table for Doris to visit.
--
Best Regards
Mingyu Chen
Email:
morning...@apache.org
在 2023-10-05 13:30:25，"Joice Jacob" <joicejacob1...@gmail.com> 写道：
Dear Doris Community & Jiafeng.Zhang
,
Thank you for your prompt response and your willingness to assist with the
issue I've been encountering with the integration of Doris and Cloudera's Hive.
To provide you with the necessary information, here are the details you
requested:
1. Hive Catalog Creation Statement:
My Hive catalog creation statement is as follows:
CREATE CATALOG hive PROPERTIES (
'type'='hms',
'hive.metastore.uris' = 'thrift://10.128.0.4:9083',
'hive.version' = '3.1.3',
'hive.metastore.sasl.enabled' = 'true',
'hive.metastore.kerberos.principal' = 'hive/instanc...@hadoop.com',
'hadoop.security.authentication' = 'kerberos',
'hadoop.kerberos.keytab' = '/home/techuser/doris/hive.keytab',
'hadoop.kerberos.principal' = 'hive/instanc...@hadoop.com',
'yarn.resourcemanager.principal' = 'yarn/instanc...@hadoop.com'
);
2. CDP Hive Version:
The CDP Hive version I am using is: Hive 3.1.3000.7.1.7.0-551
3. Configuration Attempt:
I have tried specifying the Hive version in my catalog configuration with the
following statement:
"hive.version=2.1.0"
Despite attempting to set the Hive version to 2.1.0, I continue to experience
the same issue, which is detailed in my previous emails.
4. Logs:
I will attach both the fe.log and fe.warn logs to this email for your
reference. These logs should provide additional context regarding the issue I'm
facing.
Once again, I want to express my gratitude for your assistance and support in
resolving this matter. I look forward to your insights and recommendations
based on the provided logs and catalog configuration.
Please feel free to let me know if you require any further information or if
there are additional steps I should take to assist in diagnosing and resolving
the issue.
fe.log log at hive catlog creation time
Hive managed table script
Best regards,
Joice
On Thu, Oct 5, 2023 at 8:10 AM Jiafeng.Zhang <zhang...@gmail.com> wrote:
Can you provide your hive catalog creation statement, your fe.log log at
that time, and your cdp hive version? This will help us locate the problem,
thank you.
You can also try specifying your hive version in your catalog statement:
"hive.version=2.1.0"
Joice Jacob <joicejacob1...@gmail.com> 于2023年10月5日周四 00:45写道：
> I am reaching out once again to seek assistance and share a specific issue
> I've encountered while integrating Doris with Cloudera-flavored Hive. The
> error message I'm facing is as follows:
> detailMessage = get file split failed for table: baby1, err:
> java.lang.Exception: Hive 2.x versioned full-acid tables need to run major
> compaction.
>
> This error message appears when I attempt to query a Hive table from
> Doris, and it seems to be related to Hive 2.x versioned full-ACID tables
> requiring a major compaction.
> I would like to ask the Doris community for guidance on how to handle this
> issue effectively. Specifically, I am interested in understanding the best
> practices and steps to follow when dealing with Cloudera-flavored Hive
> tables that require major compaction for Doris integration.
>
> If anyone in the community has successfully addressed this issue or can
> provide insights into how to configure and manage Cloudera-flavored Hive
> tables for integration with Doris, your expertise would be highly
> appreciated.
>
> Thank you for your time and support, and I look forward to receiving your
> valuable input.
> Joice
>
> On Wed, Oct 4, 2023 at 9:23 PM Joice Jacob <joicejacob1...@gmail.com>
> wrote:
>
>> Dear Doris Community,
>> I have an update on the issue I previously mentioned regarding the
>> integration of Hive on a CDP distribution with Doris 2.0.1.1.
>>
>> After further investigation, I have identified that the issue is related
>> to Hive managed tables being transactional by default. This appears to be
>> causing the error I encountered earlier.
>>
>> To address this issue, I am seeking guidance from the community on any
>> specific configurations or settings that need to be adjusted for Hive
>> transactional tables when using Hive as the catalog in Doris. Are there any
>> recommended configurations or best practices that I should follow to ensure
>> smooth integration and query execution?
>>
>> Any insights or recommendations from the Doris community would be greatly
>> appreciated. Your expertise and guidance will be instrumental in helping me
>> resolve this challenge.
>>
>> Thank you for your continued support, and I look forward to your valuable
>> input.
>>
>> Best regards,
>> Joice
>>
>> On Wed, Oct 4, 2023 at 6:54 PM Joice Jacob <joicejacob1...@gmail.com>
>> wrote:
>>
>>> Dear Doris Community,
>>>
>>> I am reaching out to the community to seek assistance with an
>>> integration issue I've encountered while trying to use Hive on a CDP
>>> distribution with Doris 2.0.1.1.
>>>
>>> Here are the details of my setup:
>>>
>>> Doris Version: 2.0.1.1
>>> Hive Version: 3.1.3
>>> Cluster Security: Kerberized
>>>
>>> I have successfully created a Hive catalog in Doris and have been able
>>> to set up the integration between Hive and Doris. However, when I attempt
>>> to query a Hive table using Doris, I encounter the following error:
>>>
>>> ERROR 1105 (HY000): errCode = 2, detailMessage = get file split failed
>>> for table: baby1, err: java.lang.Exception: Hive 2.x versioned full-acid
>>> tables need to run major compaction.
>>>
>>> I have already performed a major compaction as recommended, but I am
>>> still encountering the same error.
>>>
>>> I would greatly appreciate any insights, guidance, or solutions that the
>>> Doris community can offer to help me resolve this issue. If anyone has
>>> encountered a similar problem or has expertise in integrating Hive with
>>> Doris, your assistance would be invaluable.
>>>
>>> Thank you in advance for your time and support. I look forward to
>>> hearing from the community and working together to find a solution to this
>>> challenge.
>>>
>>> [image: baby_table.JPG]
>>> [image: doris_hive_catlog_result.JPG]
>>>
>>> [image: doris_fe_log.JPG]
>>>
>>> [image: hive_metastore_error.JPG]
>>>
>>> Thanks
>>> Joice
>>>
>>>
>>>
>>>
>>>
--
张家峰
