[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
The code is committed to master branch. Thanks [~xuefuz] for the review!
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch, HIVE-17257.3.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Attachment: (was: HIVE-17257.3.patch)
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch, HIVE-17257.3.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Attachment: HIVE-17257.3.patch
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch, HIVE-17257.3.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Attachment: (was: HIVE-17257.3.patch)
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch, HIVE-17257.3.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Attachment: HIVE-17257.3.patch
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch, HIVE-17257.3.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Attachment: HIVE-17257.3.patch
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Major
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch, HIVE-17257.3.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Attachment: HIVE-17257.2.patch
Re-attach patch v0 to trigger tests.
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch,
> HIVE-17257.2.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Attachment: HIVE-17257.1.patch
Submitting patch v1 for testing (it is not ready yet).
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-17257.0.patch, HIVE-17257.1.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Status: Patch Available (was: Open)
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-17257.0.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17257) Hive should merge empty files
[
https://issues.apache.org/jira/browse/HIVE-17257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-17257:
Attachment: HIVE-17257.0.patch
Submitting patch v0 for testing.
> Hive should merge empty files
> -
>
> Key: HIVE-17257
> URL: https://issues.apache.org/jira/browse/HIVE-17257
> Project: Hive
> Issue Type: Bug
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-17257.0.patch
>
>
> Currently if merging file option is turned on and the dest dir contains large
> number of empty files, Hive will not trigger merge task:
> {code}
> private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
> AverageSize averageSize = getAverageSize(inpFs, dirPath);
> if (averageSize.getTotalSize() <= 0) {
> return -1;
> }
> if (averageSize.getNumFiles() <= 1) {
> return -1;
> }
> if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
> return averageSize.getTotalSize();
> }
> return -1;
> }
> {code}
> This logic doesn't seem right as the it seems better to combine these empty
> files into one.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
