Hi Mikhail, I’m surprised using asLong works. What you need to do is
int maxRowSize =
ctx.getProperty(MAX_ROW_SIZE)
.asDataSize(DataUnit.B)
.intValue();
You may specify other units. If the property is not set the asDataSize call
returns null, if this possible likely check first. This method returns a Double.
> On 15 Jan 2018, at 15:04, Mikhail Sosonkin <[email protected]> wrote:
>
> Joe,
>
> I think there might a difference in behavior for how processor attributes
> are evaluated in test suite vs real execution. I'm not entirely sure if it
> is an expected/acceptable behavior. It has to with the DATA_SIZE_VALIDATOR.
> Mine was defined as follows:
>
> public static final PropertyDescriptor MAX_ROW_SIZE = new
> PropertyDescriptor
> .Builder().name(BigQueryAttributes.MAX_ROW_SIZE_ATTR)
> .displayName("Max row size")
> .description(BigQueryAttributes.MAX_ROW_SIZE_DESC)
> .required(true)
> .defaultValue("1 MB")
> .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
> .build();
>
> Then I would use it like this in the onTrigger method:
>
> final long max_row_size = context.getProperty(MAX_ROW_SIZE).asLong();
>
> I used it this way based on the examples I've seen in the other processors.
> In the runtime it seems to be working great and gives me the byte values.
> However, in testing it would through an exception because it would try to
> parse "1 MB" (the default value) as an actual Long. Of course, if I just
> place a long value (like 1024) it would not pass the validation function.
> So, I did a more explicit "cast" using the asDataSize method:
>
> final long max_row_size =
> context.getProperty(MAX_ROW_SIZE).asDataSize(DataUnit.B).longValue();
>
> This solved my problems and makes the code a bit more explicit which is
> nice. The annoying part is that the StandardProcessorTestRunner does not
> keep the stacktrace from the exceptions generated in onTrigger of the
> processor (StandardProcessorTestRunner:199)
>
> final Throwable thrown = future.get();
>
> That is what caused confusion for me. I hadn't realized that the exception
> was being thrown via my code. So, if anything the test suite needs to do a
> better job at delivering stack traces for the exceptions to help with
> debugging.
>
> I'll post this up on a ticket.
>
> Thanks,
> Mike.
>
>
>
>> On Sun, Jan 14, 2018 at 6:12 AM, Joe Witt <[email protected]> wrote:
>>
>> mike
>>
>> can you explain what was happening and the exception that came back?
>> Filing a JIRA for this might help the next person avoid the confusion.
>>
>> Thanks
>>
>> On Sun, Jan 14, 2018 at 12:44 AM, Mikhail Sosonkin
>> <[email protected]> wrote:
>>> Sorry for the alarm, after some debugging, I've realized that the issue
>> is
>>> on my side. Just wasn't clear from the exceptions I was receiving.
>>>
>>> Thanks,
>>> Mike.
>>>
>>> On Fri, Jan 12, 2018 at 7:27 PM, Mikhail Sosonkin <[email protected]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I'm getting some strange behavior on unit tests.
>>>>
>>>> (https://github.com/nologic/nifi/tree/NIFI-4731, https://
>>>> github.com/nologic/nifi/blob/3a4749a7b467fff0188174bc256e53
>>>> 818d43b7ba/nifi-nar-bundles/nifi-gcp-bundle/nifi-gcp-
>>>> processors/src/test/java/org/apache/nifi/processors/gcp/bigquery/
>>>> PutBigQueryStreamTest.java#L65)
>>>>
>>>> The property looks like this:
>>>> public static final PropertyDescriptor MAX_ROW_SIZE = new
>>>> PropertyDescriptor
>>>> .Builder().name(BigQueryAttributes.MAX_ROW_SIZE_ATTR)
>>>> .displayName("Max row size")
>>>> .description(BigQueryAttributes.MAX_ROW_SIZE_DESC)
>>>> .required(true)
>>>> .defaultValue("1 MB")
>>>> .addValidator(StandardValidators.DATA_SIZE_VALIDATOR)
>>>> .build();
>>>>
>>>> This works great in NiFi, but when I run it via a unit test. I get an
>>>> exception (even with a default value):
>>>> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.543
>> sec
>>>> <<< FAILURE! - in org.apache.nifi.processors.gcp.bigquery.
>>>> PutBigQueryStreamTest
>>>> testSuccessfulInsert(org.apache.nifi.processors.gcp.bigquery.
>> PutBigQueryStreamTest)
>>>> Time elapsed: 1.326 sec <<< FAILURE!
>>>> java.lang.AssertionError: java.lang.NumberFormatException: For input
>>>> string: "2 MB"
>>>> at org.apache.nifi.processors.gcp.bigquery.PutBigQueryStreamTest.
>>>> testSuccessfulInsert(PutBigQueryStreamTest.java:79)
>>>> Caused by: java.lang.NumberFormatException: For input string: "2 MB"
>>>>
>>>>
>>>> So, I tried changing the value to a number, which fails the test as
>>>> expected:
>>>> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.665
>> sec
>>>> <<< FAILURE! - in org.apache.nifi.processors.gcp.bigquery.
>>>> PutBigQueryStreamTest
>>>> testSuccessfulInsert(org.apache.nifi.processors.gcp.bigquery.
>> PutBigQueryStreamTest)
>>>> Time elapsed: 1.437 sec <<< FAILURE!
>>>> java.lang.AssertionError:
>>>> Processor has 1 validation failures:
>>>> 'bq.row.size' validated against '1024' is invalid because Must be of
>>>> format <Data Size> <Data Unit> where <Data Size> is a non-negative
>> integer
>>>> and <Data Unit> is a supported Data Unit, such as: B, KB, MB, GB, TB
>>>>
>>>>
>>>> What am I missing? Why is the default throwing an exception? Maybe the
>>>> validator is broken?
>>>>
>>>> Thanks,
>>>> Mike.
>>>>
>>>
>>> --
>>> This email may contain material that is confidential for the sole use of
>>> the intended recipient(s). Any review, reliance or distribution or
>>> disclosure by others without express permission is strictly prohibited.
>> If
>>> you are not the intended recipient, please contact the sender and delete
>>> all copies of this message.
>>
>
> --
> This email may contain material that is confidential for the sole use of
> the intended recipient(s). Any review, reliance or distribution or
> disclosure by others without express permission is strictly prohibited. If
> you are not the intended recipient, please contact the sender and delete
> all copies of this message.
