Re: Re: spark master - run-tests error

[wei....@kaiyuandao.com]

it works like a charm for me. thanks for the quick workaround



 
From: Ted Yu
Date: 2015-12-04 10:45
To: wei@kaiyuandao.com
CC: user
Subject: Re: spark master - run-tests error
From dev/run-tests.py :

def identify_changed_files_from_git_commits(patch_sha, target_branch=None, 
target_ref=None):
"""
Given a git commit and target ref, use the set of files changed in the diff 
in order to
determine which modules' tests should be run.

Looks like the script needs git commit to determine which modules test suite 
should be run.

I use the following command to run tests:

mvn -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 -Dhadoop.version=2.7.0 
package

FYI

On Thu, Dec 3, 2015 at 6:09 PM, wei@kaiyuandao.com <wei@kaiyuandao.com> 
wrote:

hi, is there anyone knowing why I came to the following error when running 
tests after a successful full build? thanks

[root@sandbox spark_git]# dev/run-tests
**
File "./dev/run-tests.py", line 68, in 
__main__.identify_changed_files_from_git_commits
Failed example:
[x.name for x in determine_modules_for_files( 
identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
Exception raised:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
compileflags, 1) in test.globs
  File "", 
line 1, in 
[x.name for x in determine_modules_for_files( 
identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
  File "./dev/run-tests.py", line 84, in 
identify_changed_files_from_git_commits
raw_output = subprocess.check_output(['git', 'diff', '--name-only', 
patch_sha, diff_target],
AttributeError: 'module' object has no attribute 'check_output'
**
File "./dev/run-tests.py", line 70, in 
__main__.identify_changed_files_from_git_commits
Failed example:
'root' in [x.name for x in determine_modules_for_files(  
identify_changed_files_from_git_commits("50a0496a43", target_ref="6765ef9"))]
Exception raised:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
compileflags, 1) in test.globs
  File "", 
line 1, in 
'root' in [x.name for x in determine_modules_for_files(  
identify_changed_files_from_git_commits("50a0496a43", target_ref="6765ef9"))]

Re: spark master - run-tests error

[Ted Yu]

The commit on last line led to:

commit 50a0496a43f09d70593419efc38587c8441843bf
Author: Brennon York 
Date:   Wed Jun 17 12:00:34 2015 -0700

When did you last update your workspace ?

Cheers

On Thu, Dec 3, 2015 at 6:09 PM, wei@kaiyuandao.com <
wei@kaiyuandao.com> wrote:

>
> hi, is there anyone knowing why I came to the following error when running
> tests after a successful full build? thanks
>
> [root@sandbox spark_git]# dev/run-tests
> **
>
> File "./dev/run-tests.py", line 68, in 
> __main__.identify_changed_files_from_git_commits
> Failed example:
> [x.name
>  for x in determine_modules_for_files( 
> identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
> Exception raised:
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
> compileflags, 1) in test.globs
>
>   File "", 
> line 1, in 
> [x.name
>  for x in determine_modules_for_files( 
> identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
>
>   File "./dev/run-tests.py", line 84, in 
> identify_changed_files_from_git_commits
>
> raw_output = subprocess.check_output(['git', 'diff', '--name-only', 
> patch_sha, diff_target],
> AttributeError: 'module' object has no attribute 'check_output'
> **
>
> File "./dev/run-tests.py", line 70, in 
> __main__.identify_changed_files_from_git_commits
> Failed example:
> 'root' in [x.name
>  for x in determine_modules_for_files(  
> identify_changed_files_from_git_commits("50a0496a43", target_ref="6765ef9"))]
> Exception raised:
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
> compileflags, 1) in test.globs
>
>   File "", 
> line 1, in 
> 'root' in [x.name
>  for x in determine_modules_for_files(  
> identify_changed_files_from_git_commits("50a0496a43", target_ref="6765ef9"))]
>
>

Re: Re: spark master - run-tests error

[wei....@kaiyuandao.com]

I was using the latest master branch. 



 
From: Ted Yu
Date: 2015-12-04 10:14
To: wei@kaiyuandao.com
CC: user
Subject: Re: spark master - run-tests error
The commit on last line led to:

commit 50a0496a43f09d70593419efc38587c8441843bf
Author: Brennon York <brennon.y...@capitalone.com>
Date:   Wed Jun 17 12:00:34 2015 -0700

When did you last update your workspace ?

Cheers

On Thu, Dec 3, 2015 at 6:09 PM, wei@kaiyuandao.com <wei@kaiyuandao.com> 
wrote:

hi, is there anyone knowing why I came to the following error when running 
tests after a successful full build? thanks

[root@sandbox spark_git]# dev/run-tests
**
File "./dev/run-tests.py", line 68, in 
__main__.identify_changed_files_from_git_commits
Failed example:
[x.name for x in determine_modules_for_files( 
identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
Exception raised:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
compileflags, 1) in test.globs
  File "", 
line 1, in 
[x.name for x in determine_modules_for_files( 
identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
  File "./dev/run-tests.py", line 84, in 
identify_changed_files_from_git_commits
raw_output = subprocess.check_output(['git', 'diff', '--name-only', 
patch_sha, diff_target],
AttributeError: 'module' object has no attribute 'check_output'
**
File "./dev/run-tests.py", line 70, in 
__main__.identify_changed_files_from_git_commits
Failed example:
'root' in [x.name for x in determine_modules_for_files(  
identify_changed_files_from_git_commits("50a0496a43", target_ref="6765ef9"))]
Exception raised:
Traceback (most recent call last):
  File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
compileflags, 1) in test.globs
  File "", 
line 1, in 
'root' in [x.name for x in determine_modules_for_files(  
identify_changed_files_from_git_commits("50a0496a43", target_ref="6765ef9"))]

Re: spark master - run-tests error

[Ted Yu]

>From dev/run-tests.py :

def identify_changed_files_from_git_commits(patch_sha, target_branch=None,
target_ref=None):
"""
Given a git commit and target ref, use the set of files changed in the
diff in order to
determine which modules' tests should be run.

Looks like the script needs git commit to determine which modules test
suite should be run.

I use the following command to run tests:

mvn -Phive -Phive-thriftserver -Pyarn -Phadoop-2.4 -Dhadoop.version=2.7.0
package

FYI

On Thu, Dec 3, 2015 at 6:09 PM, wei@kaiyuandao.com <
wei@kaiyuandao.com> wrote:

>
> hi, is there anyone knowing why I came to the following error when running
> tests after a successful full build? thanks
>
> [root@sandbox spark_git]# dev/run-tests
> **
>
> File "./dev/run-tests.py", line 68, in 
> __main__.identify_changed_files_from_git_commits
> Failed example:
> [x.name
>  for x in determine_modules_for_files( 
> identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
> Exception raised:
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
> compileflags, 1) in test.globs
>
>   File "", 
> line 1, in 
> [x.name
>  for x in determine_modules_for_files( 
> identify_changed_files_from_git_commits("fc0a1475ef", target_ref="5da21f07"))]
>
>   File "./dev/run-tests.py", line 84, in 
> identify_changed_files_from_git_commits
>
> raw_output = subprocess.check_output(['git', 'diff', '--name-only', 
> patch_sha, diff_target],
> AttributeError: 'module' object has no attribute 'check_output'
> **
>
> File "./dev/run-tests.py", line 70, in 
> __main__.identify_changed_files_from_git_commits
> Failed example:
> 'root' in [x.name
>  for x in determine_modules_for_files(  
> identify_changed_files_from_git_commits("50a0496a43", target_ref="6765ef9"))]
> Exception raised:
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
> compileflags, 1) in test.globs
>
>   File "", 
> line 1, in 
> 'root' in [x.name
>  for x in determine_modules_for_files(  
> identify_changed_files_from_git_commits("50a0496a43", target_ref="6765ef9"))]
>
>