Hey
i have been writing unit tests for beam pipelines using TestPipeline - in
python though, and i dont have such usecases
wondering if this satisfy your usecase?
If not, please give me some pointers to try to reproduce your usecase. To
me it seems, one of your transformations raises an exception, i am
expecting the pipeline to die after that?
class TestNewsPipelineWithExc(unittest.TestCase):
def prepare_for_big_query_tst(self, items):
raise Exception('Raising an Exception')
def test_prepare_for_bigquery(self):
with self.assertRaises(Exception):
with TestPipeline() as p:
lst = ['AMZN', 'TestHeadline', 0.8]
lst2 = ['ABBV', 'TESTABBV', 0.5]
# Calling DataFrame constructor on list
df = pd.DataFrame([lst, lst2], columns=['ticker',
'headline', 0])
input = p | beam.Create([df])
res = self.prepare_for_big_query_tst(input)
On Thu, Nov 19, 2020 at 5:02 PM Pablo Estrada <[email protected]> wrote:
> Hi Dave!
> I don't have a lot of experience with coders, but I would include the Beam
> user@ list (added just now) to see if someone else has done this.
> Best
> -P.
>
> On Wed, Nov 18, 2020 at 7:22 AM Dave Anderson <[email protected]>
> wrote:
>
>> Pablo,
>>
>> Also, for now I've created tests that exercise the encode() and decode()
>> methods directly on the Coder. I was just curious if there was a way to
>> also test it in the context of a pipeline.
>>
>> thanks,
>>
>> Dave
>>
>> *David Anderson*
>>
>> Solution Engineer - Cloud Healthcare & Life Sciences
>>
>> [email protected]
>>
>>
>>
>> On Wed, Nov 18, 2020 at 9:17 AM Dave Anderson <[email protected]>
>> wrote:
>>
>>> Hi Pablo,
>>>
>>> I was wondering if you could point me to any resources that might help
>>> me address a challenge I've run into when unit testing a Custom Coder I
>>> have created as part of the Whistle ELP project.
>>>
>>> The regular tests where I'm using PAssert to check the value returning
>>> from a call to
>>> p.apply( "Create from test objects",
>>> Create.of(runtimeContext).withCoder(RuntimeContextCoder.of()));
>>>
>>> are working fine. The issue arises when I throw an exception from the
>>> coder, for example when a null object is passed in to the encode() or
>>> decode() methods.
>>>
>>> I've tried to use Assert.assertThrows() for the test case, and it is
>>> actually succeeding in that check, but the pipeline continues to run and I
>>> end up with a NPE.
>>>
>>> Outputs for non-root node Create from test objects are null
>>> java.lang.NullPointerException: Outputs for non-root node Create from
>>> test objects are null
>>>
>>> Have you ever written unit tests using the TestPipeline that are testing
>>> for exceptions being thrown? Am I missing something really obvious?
>>>
>>> Also, feel free to direct me to a mailing list if there's an appropriate
>>> one for this. You have just been incredibly helpful with the FhirIO,
>>> HL7v2IO, and the DicomIO efforts, so I figured I'd ping you to get some
>>> direction.
>>>
>>> thanks,
>>>
>>> Dave
>>>
>>> [image: image.png]
>>>
>>> *David Anderson*
>>>
>>> Solution Engineer - Cloud Healthcare & Life Sciences
>>>
>>> [email protected]
>>>
>>>
>>>
