Re: Hive External Table with Zero Bytes files

[Mahender Sarangam]

Thanks Nishanth. We are also cleaning up files of size Zero byes




On 5/2/2018 8:53 AM, Nishanth S wrote:
I have run into similar issue with avro files . The solution was to fix 
upstream jobs  that were writing data to those directories . In our case the  
writers were not flushed/closed correctly during certain  events which caused 
the   issue . Fixing those prevented these 0 sized files.

-NS

On Wed, May 2, 2018 at 1:52 AM, Mahender Sarangam 
> wrote:

ping..

On 5/1/2018 3:57 AM, Mahender Sarangam wrote:

Thanks Thai. I have mentioned wrongly Folder Name, it 's same DAY=20180325 
(Folder) and same has Filename. actually in our upstream, our source table is 
partitioned by Date. Whenever a table is partitioned, we see Zero Byte. Now 
when we create external table with partitioned by columns and fire select query 
no data is returned. . If I delete manually those files (Zero Bytes), we were 
able to read.


/Mahender

On 4/28/2018 6:36 AM, Thai Bui wrote:
Your external table is referencing the .../day=201803250 location which is 
empty. Point your table to the capital .../DAY=201803250 and you should be able 
to read the data there.

Also, it looks like you want external partitioned table. You’ll need to create 
an external table with a partition clause, then alter the table and add 
partition for each of the ../DAY=someday path that you have.

On Sat, Apr 28, 2018 at 4:05 AM Mahender Sarangam 
> wrote:

Gentle Ping. Please help me on below issue. Has any one faced same issue

On 4/27/2018 1:28 AM, Mahender Sarangam wrote:

Hi,

Can any one faced issue while fetching data from external table. We are copying 
data from upstream system into our storage S3. As part of copy, directories 
along with Zero bytes files are been copied. Source File Format is in JSON 
format.  Below is Folder Hierarchy Structure


 DATE  -->  

   

Re: Hive External Table with Zero Bytes files

[Nishanth S]

I have run into similar issue with avro files . The solution was to fix
upstream jobs  that were writing data to those directories . In our case
the  writers were not flushed/closed correctly during certain  events which
caused the   issue . Fixing those prevented these 0 sized files.

-NS

On Wed, May 2, 2018 at 1:52 AM, Mahender Sarangam <
mahender.bigd...@outlook.com> wrote:

> ping..
>
> On 5/1/2018 3:57 AM, Mahender Sarangam wrote:
>
> Thanks Thai. I have mentioned wrongly Folder Name, it 's same DAY=20180325
> (Folder) and same has Filename. actually in our upstream, our source table
> is partitioned by Date. Whenever a table is partitioned, we see Zero Byte.
> Now when we create external table with partitioned by columns and fire
> select query no data is returned. . If I delete manually those files (Zero
> Bytes), we were able to read.
>
>
> /Mahender
>
> On 4/28/2018 6:36 AM, Thai Bui wrote:
>
> Your external table is referencing the .../day=201803250 location which is
> empty. Point your table to the capital .../DAY=201803250 and you should be
> able to read the data there.
>
> Also, it looks like you want external partitioned table. You’ll need to
> create an external table with a partition clause, then alter the table and
> add partition for each of the ../DAY=someday path that you have.
>
> On Sat, Apr 28, 2018 at 4:05 AM Mahender Sarangam <
> mahender.bigd...@outlook.com> wrote:
>
>> Gentle Ping. Please help me on below issue. Has any one faced same issue
>>
>> On 4/27/2018 1:28 AM, Mahender Sarangam wrote:
>>
>> Hi,
>>
>> Can any one faced issue while fetching data from external table. We are
>> copying data from upstream system into our storage S3. As part of copy,
>> directories along with Zero bytes files are been copied. Source File Format
>> is in JSON format.  Below is Folder Hierarchy Structure
>>
>>
>>  DATE  -->  
>>
>>

Re: Hive External Table with Zero Bytes files

[Mahender Sarangam]

ping..

On 5/1/2018 3:57 AM, Mahender Sarangam wrote:

Thanks Thai. I have mentioned wrongly Folder Name, it 's same DAY=20180325 
(Folder) and same has Filename. actually in our upstream, our source table is 
partitioned by Date. Whenever a table is partitioned, we see Zero Byte. Now 
when we create external table with partitioned by columns and fire select query 
no data is returned. . If I delete manually those files (Zero Bytes), we were 
able to read.


/Mahender

On 4/28/2018 6:36 AM, Thai Bui wrote:
Your external table is referencing the .../day=201803250 location which is 
empty. Point your table to the capital .../DAY=201803250 and you should be able 
to read the data there.

Also, it looks like you want external partitioned table. You’ll need to create 
an external table with a partition clause, then alter the table and add 
partition for each of the ../DAY=someday path that you have.

On Sat, Apr 28, 2018 at 4:05 AM Mahender Sarangam 
> wrote:

Gentle Ping. Please help me on below issue. Has any one faced same issue

On 4/27/2018 1:28 AM, Mahender Sarangam wrote:

Hi,

Can any one faced issue while fetching data from external table. We are copying 
data from upstream system into our storage S3. As part of copy, directories 
along with Zero bytes files are been copied. Source File Format is in JSON 
format.  Below is Folder Hierarchy Structure


 DATE  -->  

   

Re: Hive External Table with Zero Bytes files

[Mahender Sarangam]

Thanks Thai. I have mentioned wrongly Folder Name, it 's same DAY=20180325 
(Folder) and same has Filename. actually in our upstream, our source table is 
partitioned by Date. Whenever a table is partitioned, we see Zero Byte. Now 
when we create external table with partitioned by columns and fire select query 
no data is returned. . If I delete manually those files (Zero Bytes), we were 
able to read.


/Mahender

On 4/28/2018 6:36 AM, Thai Bui wrote:
Your external table is referencing the .../day=201803250 location which is 
empty. Point your table to the capital .../DAY=201803250 and you should be able 
to read the data there.

Also, it looks like you want external partitioned table. You’ll need to create 
an external table with a partition clause, then alter the table and add 
partition for each of the ../DAY=someday path that you have.

On Sat, Apr 28, 2018 at 4:05 AM Mahender Sarangam 
> wrote:

Gentle Ping. Please help me on below issue. Has any one faced same issue

On 4/27/2018 1:28 AM, Mahender Sarangam wrote:

Hi,

Can any one faced issue while fetching data from external table. We are copying 
data from upstream system into our storage S3. As part of copy, directories 
along with Zero bytes files are been copied. Source File Format is in JSON 
format.  Below is Folder Hierarchy Structure


 DATE  -->  

   

Re: Hive External Table with Zero Bytes files

[Gopal Vijayaraghavan]

> We are copying data from upstream system into our storage S3. As part of 
> copy, directories along with Zero bytes files are been copied. 

Is this exactly the same issue as the previous thread or a different one?



Cheers,
Gopal

Re: Hive External Table with Zero Bytes files

[Thai Bui]

Your external table is referencing the .../day=201803250 location which is
empty. Point your table to the capital .../DAY=201803250 and you should be
able to read the data there.

Also, it looks like you want external partitioned table. You’ll need to
create an external table with a partition clause, then alter the table and
add partition for each of the ../DAY=someday path that you have.

On Sat, Apr 28, 2018 at 4:05 AM Mahender Sarangam <
mahender.bigd...@outlook.com> wrote:

> Gentle Ping. Please help me on below issue. Has any one faced same issue
>
> On 4/27/2018 1:28 AM, Mahender Sarangam wrote:
>
> Hi,
>
> Can any one faced issue while fetching data from external table. We are
> copying data from upstream system into our storage S3. As part of copy,
> directories along with Zero bytes files are been copied. Source File Format
> is in JSON format.  Below is Folder Hierarchy Structure
>
>
>  DATE  -->  
>
>

Re: Hive External Table with Zero Bytes files

[Mahender Sarangam]

Gentle Ping. Please help me on below issue. Has any one faced same issue

On 4/27/2018 1:28 AM, Mahender Sarangam wrote:

Hi,

Can any one faced issue while fetching data from external table. We are copying 
data from upstream system into our storage S3. As part of copy, directories 
along with Zero bytes files are been copied. Source File Format is in JSON 
format.  Below is Folder Hierarchy Structure


 DATE  -->