[jira] [Commented] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725861#comment-16725861
]
Adam Szita commented on PIG-5371:
-
Hi [~abstractdog],
Yeah sorry that's a typo, indeed -Dtestcase should be used.
The doesn't hang on my side, it finishes successfully in 9 minutes.
Do you see which test method completes on your side and which one doesn't?
In the past when I faced with the hanging issue was due to my Mac's HDD had
over 90% utilisation which some HDFS code in MiniCluster did not like
> Hdfs bytes written assertions fail in TestPigRunner
> ---
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the size of the jar files as well. The problem is this could change
> very easily, so in my opinion the best would be to remove these assertions
> from TestPigRunner as this is just causing intermittent and/or persistent
> failures.
> The test class is for basic testing of PigRunner, and this is achieved well
> enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722171#comment-16722171
]
Laszlo Bodor commented on PIG-5371:
---
Thanks [~szita] ! I've wo notes on this:
1. -Dtest doesn't filter test cases (all tests will run), -Dtestcase should be
used instead
2. 'ant clean jar' is ok, built successfully (I'm on apache trunk), but then
'ant test -Dtestcase=TestPigRunner' hangs, 10 minutes, and counting (i haven't
manage to debug it yet
{code}
...
test-core:
[delete] Deleting directory /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /home/abstractdog/apache/pig/build/test/logs
[mkdir] Created dir: /tmp/pig_junit_tmp667875539
[echo] Tests in /home/abstractdog/apache/pig/test/excluded-tests-mr will be
excluded
[junit] Running org.apache.pig.test.TestPigRunner
{code}
could you please confirm that you can successfully run (and finish)
TestPigRunner?
> Hdfs bytes written assertions fail in TestPigRunner
> ---
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the size of the jar files as well. The problem is this could change
> very easily, so in my opinion the best would be to remove these assertions
> from TestPigRunner as this is just causing intermittent and/or persistent
> failures.
> The test class is for basic testing of PigRunner, and this is achieved well
> enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721419#comment-16721419
]
Adam Szita commented on PIG-5371:
-
Hi [~abstractdog], can you please elaborate on
{quote}TestPigRunner - work, only on an internal maintenance line
{quote}
I am able to run TestPigRunner checked out from trunk as per:
{code:java}
ant clean jar
ant test -Dtest=TestPigRunner{code}
..and it succeeds:
{code:java}
BUILD SUCCESSFUL
Total time: 8 minutes 47 seconds{code}
> Hdfs bytes written assertions fail in TestPigRunner
> ---
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the size of the jar files as well. The problem is this could change
> very easily, so in my opinion the best would be to remove these assertions
> from TestPigRunner as this is just causing intermittent and/or persistent
> failures.
> The test class is for basic testing of PigRunner, and this is achieved well
> enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712834#comment-16712834
]
Laszlo Bodor commented on PIG-5371:
---
[~rohini]: thanks for the review, I agree, will have to take a look at it again.
I would appreciate some guideline about running pig unit tests on trunk (I
didn't manage to make them - e.g. TestPigRunner - work, only on an internal
maintenance line)
> Hdfs bytes written assertions fail in TestPigRunner
> ---
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the size of the jar files as well. The problem is this could change
> very easily, so in my opinion the best would be to remove these assertions
> from TestPigRunner as this is just causing intermittent and/or persistent
> failures.
> The test class is for basic testing of PigRunner, and this is achieved well
> enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701160#comment-16701160
]
Rohini Palaniswamy commented on PIG-5371:
-
bq. It seems like HDFS counter 'HDFS_BYTES_WRITTEN' returns the byte count not
only for the result of pig store operator, but it includes the size of the jar
files as well.
Are you sure that jar files are included in it? It is a job counter. Don't
understand how that can happen.
> Hdfs bytes written assertions fail in TestPigRunner
> ---
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the size of the jar files as well. The problem is this could change
> very easily, so in my opinion the best would be to remove these assertions
> from TestPigRunner as this is just causing intermittent and/or persistent
> failures.
> The test class is for basic testing of PigRunner, and this is achieved well
> enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PIG-5371) Hdfs bytes written assertions fail in TestPigRunner
[
https://issues.apache.org/jira/browse/PIG-5371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16697158#comment-16697158
]
Laszlo Bodor commented on PIG-5371:
---
[~szita]: could you please review this small patch?
> Hdfs bytes written assertions fail in TestPigRunner
> ---
>
> Key: PIG-5371
> URL: https://issues.apache.org/jira/browse/PIG-5371
> Project: Pig
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: PIG-5371.01.patch, simpleTest.out
>
>
> Attached [^simpleTest.out]. It seems like HDFS counter 'HDFS_BYTES_WRITTEN'
> returns the byte count not only for the result of pig store operator, but it
> includes the size of the jar files as well. The problem is this could change
> very easily, so in my opinion the best would be to remove these assertions
> from TestPigRunner as this is just causing intermittent and/or persistent
> failures.
> The test class is for basic testing of PigRunner, and this is achieved well
> enough without the asserts.
> {code}
> 2018-11-23 10:14:52,661 [IPC Server handler 5 on 54929] INFO
> org.apache.hadoop.hdfs.StateChange - BLOCK* allocate blk_1073741827_1003,
> replicas=127.0.0.1:54934, 127.0.0.1:54930, 127.0.0.1:54943 for
> /tmp/temp-157262781/tmp-1057655772/automaton-1.11-8.jar
> ...
> 2018-11-23 10:14:52,735 [PacketResponder:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003,
> type=HAS_DOWNSTREAM_IN_PIPELINE, downstreams=2:[127.0.0.1:54930,
> 127.0.0.1:54943]] INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace - src:
> /127.0.0.1:54978, dest: /127.0.0.1:54934, bytes: 176285, op: HDFS_WRITE,
> cliID: DFSClient_NONMAPREDUCE_-1959727442_1, offset: 0, srvID:
> 108c4000-1ae0-402e-82cf-bf403629c0f7, blockid:
> BP-26001448-10.200.50.195-1542964474138:blk_1073741827_1003, duration(ns):
> 57162859
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
