[jira] [Comment Edited] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725861#comment-16725861
]
Adam Szita edited comment on PIG-5371 at 12/20/18 1:56 PM:
---
Hi [~abstractdog],
Yeah sorry that's a typo, indeed -Dtestcase should be used.
The test doesn't hang on my side, it finishes successfully in 9 minutes.
Do you see which test method completes on your side and which one doesn't?
In the past when I faced with the hanging issue was due to my Mac's HDD had
over 90% utilisation which some HDFS code in MiniCluster did not like
was (Author: szita):
Hi [~abstractdog],
Yeah sorry that's a typo, indeed -Dtestcase should be used.
The doesn't hang on my side, it finishes successfully in 9 minutes.
Do you see which test method completes on your side and which one doesn't?
In the past when I faced with the hanging issue was due to my Mac's HDD had
over 90% utilisation which some HDFS code in MiniCluster did not like
> Hdfs bytes written assertions fail in TestPigRunner
> ---
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the size of the jar files as well. The problem is this could change
> very easily, so in my opinion the best would be to remove these assertions
> from TestPigRunner as this is just causing intermittent and/or persistent
> failures.
> The test class is for basic testing of PigRunner, and this is achieved well
> enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16722171#comment-16722171
]
Laszlo Bodor edited comment on PIG-5371 at 12/15/18 2:05 PM:
-
Thanks [~szita] ! I've two notes on this:
1. -Dtest doesn't filter test cases (all tests will run), -Dtestcase should be
used instead
2. 'ant clean jar' is ok, built successfully (I'm on apache trunk), but then
'ant test -Dtestcase=TestPigRunner' hangs, 10 minutes, and counting (i haven't
manage to debug it yet
{code:java}
...
test-core:
[delete] Deleting directory /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /tmp/pig_junit_tmp667875539
[echo] Tests in /home/abstractdog/apache/pig/test/excluded-tests-mr will be
excluded
[junit] Running org.apache.pig.test.TestPigRunner
{code}
could you please confirm that you can successfully run (and finish)
TestPigRunner?
was (Author: abstractdog):
Thanks [~szita] ! I've wo notes on this:
1. -Dtest doesn't filter test cases (all tests will run), -Dtestcase should be
used instead
2. 'ant clean jar' is ok, built successfully (I'm on apache trunk), but then
'ant test -Dtestcase=TestPigRunner' hangs, 10 minutes, and counting (i haven't
manage to debug it yet
{code}
...
test-core:
[delete] Deleting directory /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /tmp/pig_junit_tmp667875539
[echo] Tests in /home/abstractdog/apache/pig/test/excluded-tests-mr will be
excluded
[junit] Running org.apache.pig.test.TestPigRunner
{code}
could you please confirm that you can successfully run (and finish)
TestPigRunner?
> Hdfs bytes written assertions fail in TestPigRunner
> ---
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the size of the jar files as well. The problem is this could change
> very easily, so in my opinion the best would be to remove these assertions
> from TestPigRunner as this is just causing intermittent and/or persistent
> failures.
> The test class is for basic testing of PigRunner, and this is achieved well
> enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16697158#comment-16697158
]
Laszlo Bodor edited comment on PIG-5371 at 11/23/18 2:15 PM:
-
[~daijy], [~szita]: could you please review this small patch?
was (Author: abstractdog):
[~szita]: could you please review this small patch?
> Hdfs bytes written assertions fail in TestPigRunner
> ---
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the size of the jar files as well. The problem is this could change
> very easily, so in my opinion the best would be to remove these assertions
> from TestPigRunner as this is just causing intermittent and/or persistent
> failures.
> The test class is for basic testing of PigRunner, and this is achieved well
> enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
