[jira] [Updated] (HIVE-6979) Hadoop-2 test failures related to quick stats not being populated correctly
[ https://issues.apache.org/jira/browse/HIVE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6979: --- Resolution: Fixed Fix Version/s: 0.14.0 Status: Resolved (was: Patch Available) Committed this to trunk. Thanks, Prasanth! Lets track remaining failures in follow-up. > Hadoop-2 test failures related to quick stats not being populated correctly > --- > > Key: HIVE-6979 > URL: https://issues.apache.org/jira/browse/HIVE-6979 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Prasanth J >Assignee: Prasanth J > Fix For: 0.14.0 > > Attachments: HIVE-6979.1.patch, HIVE-6979.2.patch, HIVE-6979.3.patch > > > The test failures that are currently reported by Hive QA running on hadoop-2 > (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) > are related to difference in the way hadoop FileSystem.globStatus() api > behaves. For a directory structure like below > {code} > dir1/file1 > dir1/file2 > {code} > Two level of path pattern like dir1/*/* will return both files in hadoop 1.x > but will return empty result in hadoop 2.x (in fact it will say no such file > or directory and return empty file status array). Hadoop 2.x seems to be > compliant to linux behaviour (ls dir1/*/*) but hadoop 1.x is not. > As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are > populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6979) Hadoop-2 test failures related to quick stats not being populated correctly
[ https://issues.apache.org/jira/browse/HIVE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6979: --- Status: Open (was: Patch Available) > Hadoop-2 test failures related to quick stats not being populated correctly > --- > > Key: HIVE-6979 > URL: https://issues.apache.org/jira/browse/HIVE-6979 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Prasanth J >Assignee: Prasanth J > Attachments: HIVE-6979.1.patch, HIVE-6979.2.patch, HIVE-6979.3.patch > > > The test failures that are currently reported by Hive QA running on hadoop-2 > (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) > are related to difference in the way hadoop FileSystem.globStatus() api > behaves. For a directory structure like below > {code} > dir1/file1 > dir1/file2 > {code} > Two level of path pattern like dir1/*/* will return both files in hadoop 1.x > but will return empty result in hadoop 2.x (in fact it will say no such file > or directory and return empty file status array). Hadoop 2.x seems to be > compliant to linux behaviour (ls dir1/*/*) but hadoop 1.x is not. > As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are > populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6979) Hadoop-2 test failures related to quick stats not being populated correctly
[ https://issues.apache.org/jira/browse/HIVE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6979: --- Status: Patch Available (was: Open) > Hadoop-2 test failures related to quick stats not being populated correctly > --- > > Key: HIVE-6979 > URL: https://issues.apache.org/jira/browse/HIVE-6979 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Prasanth J >Assignee: Prasanth J > Attachments: HIVE-6979.1.patch, HIVE-6979.2.patch, HIVE-6979.3.patch > > > The test failures that are currently reported by Hive QA running on hadoop-2 > (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) > are related to difference in the way hadoop FileSystem.globStatus() api > behaves. For a directory structure like below > {code} > dir1/file1 > dir1/file2 > {code} > Two level of path pattern like dir1/*/* will return both files in hadoop 1.x > but will return empty result in hadoop 2.x (in fact it will say no such file > or directory and return empty file status array). Hadoop 2.x seems to be > compliant to linux behaviour (ls dir1/*/*) but hadoop 1.x is not. > As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are > populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6979) Hadoop-2 test failures related to quick stats not being populated correctly
[ https://issues.apache.org/jira/browse/HIVE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6979: - Attachment: HIVE-6979.3.patch union_remove_25 was failing as it wasn't able to find partition. So increased the limit count so that the specific partition value is always present. Updated stats_partscan_1_23 test as well. Other tests seems to pass in my local setup (Mac OS X). > Hadoop-2 test failures related to quick stats not being populated correctly > --- > > Key: HIVE-6979 > URL: https://issues.apache.org/jira/browse/HIVE-6979 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Prasanth J >Assignee: Prasanth J > Attachments: HIVE-6979.1.patch, HIVE-6979.2.patch, HIVE-6979.3.patch > > > The test failures that are currently reported by Hive QA running on hadoop-2 > (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) > are related to difference in the way hadoop FileSystem.globStatus() api > behaves. For a directory structure like below > {code} > dir1/file1 > dir1/file2 > {code} > Two level of path pattern like dir1/*/* will return both files in hadoop 1.x > but will return empty result in hadoop 2.x (in fact it will say no such file > or directory and return empty file status array). Hadoop 2.x seems to be > compliant to linux behaviour (ls dir1/*/*) but hadoop 1.x is not. > As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are > populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6979) Hadoop-2 test failures related to quick stats not being populated correctly
[ https://issues.apache.org/jira/browse/HIVE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6979: - Status: Patch Available (was: Open) > Hadoop-2 test failures related to quick stats not being populated correctly > --- > > Key: HIVE-6979 > URL: https://issues.apache.org/jira/browse/HIVE-6979 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Prasanth J >Assignee: Prasanth J > Attachments: HIVE-6979.1.patch, HIVE-6979.2.patch > > > The test failures that are currently reported by Hive QA running on hadoop-2 > (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) > are related to difference in the way hadoop FileSystem.globStatus() api > behaves. For a directory structure like below > {code} > dir1/file1 > dir1/file2 > {code} > Two level of path pattern like dir1/*/* will return both files in hadoop 1.x > but will return empty result in hadoop 2.x (in fact it will say no such file > or directory and return empty file status array). Hadoop 2.x seems to be > compliant to linux behaviour (ls dir1/*/*) but hadoop 1.x is not. > As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are > populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6979) Hadoop-2 test failures related to quick stats not being populated correctly
[ https://issues.apache.org/jira/browse/HIVE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6979: - Attachment: HIVE-6979.2.patch Addressed [~ashutoshc]'s review comments. [~ashutoshc] I fixed the recent test failures. Can you please take a look at the changes in RB? > Hadoop-2 test failures related to quick stats not being populated correctly > --- > > Key: HIVE-6979 > URL: https://issues.apache.org/jira/browse/HIVE-6979 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Prasanth J >Assignee: Prasanth J > Attachments: HIVE-6979.1.patch, HIVE-6979.2.patch > > > The test failures that are currently reported by Hive QA running on hadoop-2 > (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) > are related to difference in the way hadoop FileSystem.globStatus() api > behaves. For a directory structure like below > {code} > dir1/file1 > dir1/file2 > {code} > Two level of path pattern like dir1/*/* will return both files in hadoop 1.x > but will return empty result in hadoop 2.x (in fact it will say no such file > or directory and return empty file status array). Hadoop 2.x seems to be > compliant to linux behaviour (ls dir1/*/*) but hadoop 1.x is not. > As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are > populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6979) Hadoop-2 test failures related to quick stats not being populated correctly
[ https://issues.apache.org/jira/browse/HIVE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-6979: --- Status: Open (was: Patch Available) Seems like a progress all union_remove tests are now passing. But seems like some new failures got introduced. I cannot repro some, but able to repro few others, like create_like.q, database_drop.q etc. > Hadoop-2 test failures related to quick stats not being populated correctly > --- > > Key: HIVE-6979 > URL: https://issues.apache.org/jira/browse/HIVE-6979 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Prasanth J >Assignee: Prasanth J > Attachments: HIVE-6979.1.patch > > > The test failures that are currently reported by Hive QA running on hadoop-2 > (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) > are related to difference in the way hadoop FileSystem.globStatus() api > behaves. For a directory structure like below > {code} > dir1/file1 > dir1/file2 > {code} > Two level of path pattern like dir1/*/* will return both files in hadoop 1.x > but will return empty result in hadoop 2.x (in fact it will say no such file > or directory and return empty file status array). Hadoop 2.x seems to be > compliant to linux behaviour (ls dir1/*/*) but hadoop 1.x is not. > As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are > populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6979) Hadoop-2 test failures related to quick stats not being populated correctly
[ https://issues.apache.org/jira/browse/HIVE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6979: - Status: Patch Available (was: Open) > Hadoop-2 test failures related to quick stats not being populated correctly > --- > > Key: HIVE-6979 > URL: https://issues.apache.org/jira/browse/HIVE-6979 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Prasanth J >Assignee: Prasanth J > Attachments: HIVE-6979.1.patch > > > The test failures that are currently reported by Hive QA running on hadoop-2 > (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) > are related to difference in the way hadoop FileSystem.globStatus() api > behaves. For a directory structure like below > {code} > dir1/file1 > dir1/file2 > {code} > Two level of path pattern like dir1/*/* will return both files in hadoop 1.x > but will return empty result in hadoop 2.x (in fact it will say no such file > or directory and return empty file status array). Hadoop 2.x seems to be > compliant to linux behaviour (ls dir1/*/*) but hadoop 1.x is not. > As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are > populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HIVE-6979) Hadoop-2 test failures related to quick stats not being populated correctly
[ https://issues.apache.org/jira/browse/HIVE-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth J updated HIVE-6979: - Attachment: HIVE-6979.1.patch > Hadoop-2 test failures related to quick stats not being populated correctly > --- > > Key: HIVE-6979 > URL: https://issues.apache.org/jira/browse/HIVE-6979 > Project: Hive > Issue Type: Bug >Affects Versions: 0.14.0 >Reporter: Prasanth J >Assignee: Prasanth J > Attachments: HIVE-6979.1.patch > > > The test failures that are currently reported by Hive QA running on hadoop-2 > (https://issues.apache.org/jira/browse/HIVE-6968?focusedCommentId=13980570&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13980570) > are related to difference in the way hadoop FileSystem.globStatus() api > behaves. For a directory structure like below > {code} > dir1/file1 > dir1/file2 > {code} > Two level of path pattern like dir1/*/* will return both files in hadoop 1.x > but will return empty result in hadoop 2.x (in fact it will say no such file > or directory and return empty file status array). Hadoop 2.x seems to be > compliant to linux behaviour (ls dir1/*/*) but hadoop 1.x is not. > As a result of this, the fast statistics (NUM_FILES and TOTAL_SIZE) are > populated wrongly causing diffs in qfile tests for hadoop-1 and hadoop-2. -- This message was sent by Atlassian JIRA (v6.2#6252)