Re: Jenkins

[Driesprong, Fokko]

Thanks for the reply Niels.

The only thing that makes Jenkins more appropriate right now is that Apache
offers Jenkins hosts For now we only build against Java8. But in the future
it might also make sense to have a build matrix to build also against Java
10 and 11. That would be easier to configure in, for example, Travis since
there is native support. Personally, as long it has proper Github
integration (support for PR's and easy links to the build logs), I don't
really care which CI service we're using. Since we rely on Docker and
Yetus, we're not really locked in to a specific service.

Cheers, Fokko




Op do 11 okt. 2018 om 21:53 schreef Niels Basjes :

> I fully agree with automated builds.
>
> I'm just curious what makes Jenkins more appropriate?
>
> Niels
> On Thu, Oct 11, 2018 at 6:09 PM Driesprong, Fokko 
> wrote:
> >
> > Hi all,
> >
> > I'd like to revive the Jenkins build on Avro. The last runs are a bit
> over
> > a year old: https://builds.apache.org/view/A/view/Avro/
> >
> > I've opened a PR based on Apache Yetus
> >  to configure the precommit
> task
> > to run CI jobs, as Sean Busbey initially suggested. Initially I would
> like
> > to go for a Travis CI, as we have with Apache Airflow, but I think the
> Jenkins
> > of Apache itself  might be more appropriate.
> >
> > The current Yetus scripts that I've written are still a bit crude, but I
> > think we should have the automated process from end to end up as soon as
> > possible, and then we can start polishing. Would it be possible to get
> > access to the Apache Jenkins
> > <
> https://cwiki.apache.org/confluence/display/INFRA/Jenkins#Jenkins-HowdoIgetanaccount
> >,
> > or is someone able to set this up?
> >
> > I would like to configure the Github pull request builder, based on
> > the CloudBees
> > plugin as suggested on the Apache wiki
> > <
> https://cwiki.apache.org/confluence/display/INFRA/GitHub+Pull+Request+Builder
> >
> > .
> >
> > Hope to get your feedback on the plan, and hope to move forward :-)
> >
> > Cheers, Fokko
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>

[jira] [Commented] (AVRO-1256) C++ API compileJsonSchema ignores "doc" and custom attributes on a field/record

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647277#comment-16647277
 ] 

ASF GitHub Bot commented on AVRO-1256:
--

aniket486 commented on issue #345:  AVRO-1256: C++ API compileJsonSchema 
ignores "doc" and custom attributes on a field/record
URL: https://github.com/apache/avro/pull/345#issuecomment-429174954
 
 
   @thiru-apache can you please help us review this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> C++ API compileJsonSchema ignores "doc" and custom attributes on a 
> field/record
> ---
>
> Key: AVRO-1256
> URL: https://issues.apache.org/jira/browse/AVRO-1256
> Project: Avro
>  Issue Type: Improvement
>  Components: c++
>Affects Versions: 1.7.2
> Environment: Running on all platforms (Windows, OSX, Linux)
>Reporter: Tim Menninger
>Priority: Minor
> Attachments: AVRO-1256.patch
>
>
> It appears that when my JSON is compiled into a valid schema object it is 
> ignoring all types of "documentation" that I am trying to adorn with each 
> field in my record. Reading through the Java issues it seems that this was a 
> bug and fixed (AVRO-601, AVRO-612, AVRO-779) but it seems the C++ 
> implementation has yet to adopt this feature? This is my sample schema, I 
> have attempted to insert both "doc" and "mycustom" in multiple places to see 
> if it is supported at any level. Please excuse if there appears to be a 
> syntax error in the JSON I hand tweaked some of this. The schema is valid and 
> successfully parses.
> {
>   "type": "record",
>   "name": "myschema",
>   "doc": "Doc Meta",
>   "mycustom": "My Custom",
>   "fields": [
>   { "name":"field_a","type":["string","null"], "doc":"Doc Meta", 
> "mycustom":"My Custom A"},
>   { "name":"field_b","type":["string","null"], "doc":"Doc Meta", 
> "mycustom":"My Custom B"},
>   { "name":"field_c","type":["string","null"], "doc":"Doc Meta", 
> "mycustom":"My Custom C"}
>   ]
> }
> I looked through the SchemaTests.cc code for 1.7.3 and there was not a test 
> case for this there so i didn't think this was addressed in that version. I 
> am running 1.7.2. When this schema is used to load with compileJsonSchema and 
> then a file is serialized the file schema looks like this.
> {
>   "type":"record",
>   "name":"myschema",
>   "fields": [
>   { "name":"field_a","type":["string","null"]},
>   { "name":"field_b","type":["string","null"]},
>   { "name":"field_c","type":["string","null"]}
>   ]
> }



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-1256) C++ API compileJsonSchema ignores "doc" and custom attributes on a field/record

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647275#comment-16647275
 ] 

ASF GitHub Bot commented on AVRO-1256:
--

aniket486 commented on a change in pull request #345:  AVRO-1256: C++ API 
compileJsonSchema ignores "doc" and custom attributes on a field/record
URL: https://github.com/apache/avro/pull/345#discussion_r224649978
 
 

 ##
 File path: lang/c++/impl/NodeImpl.cc
 ##
 @@ -125,54 +125,60 @@ NodeUnion::resolve(const Node ) const
 return match;
 }
 
-SchemaResolution 
+SchemaResolution
 NodeFixed::resolve(const Node ) const
 {
 if(reader.type() == AVRO_FIXED) {
 return (
 (reader.fixedSize() == fixedSize()) &&
-(reader.name() == name()) 
-) ? 
+(reader.name() == name())
+) ?
 RESOLVE_MATCH : RESOLVE_NO_MATCH;
 }
 return furtherResolution(reader);
 }
 
-SchemaResolution 
+SchemaResolution
 NodeSymbolic::resolve(const Node ) const
 {
 const NodePtr  = leafAt(0);
 return node->resolve(reader);
 }
 
-// Wrap an indentation in a struct for ostream operator<< 
-struct indent { 
+// Wrap an indentation in a struct for ostream operator<<
+struct indent {
 indent(int depth) :
 d(depth)
 { }
-int d; 
+int d;
 };
 
 /// ostream operator for indent
 std::ostream& operator <<(std::ostream , indent x)
 {
 static const std::string spaces("");
 while(x.d--) {
-os << spaces; 
+os << spaces;
 }
 return os;
 }
 
-void 
+void
 NodePrimitive::printJson(std::ostream , int depth) const
 {
 os << '\"' << type() << '\"';
+if (getDoc().size()) {
+os << ",\n" << indent(depth) << "\"doc\": \"" << getDoc() << "\"";
 
 Review comment:
   ack


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> C++ API compileJsonSchema ignores "doc" and custom attributes on a 
> field/record
> ---
>
> Key: AVRO-1256
> URL: https://issues.apache.org/jira/browse/AVRO-1256
> Project: Avro
>  Issue Type: Improvement
>  Components: c++
>Affects Versions: 1.7.2
> Environment: Running on all platforms (Windows, OSX, Linux)
>Reporter: Tim Menninger
>Priority: Minor
> Attachments: AVRO-1256.patch
>
>
> It appears that when my JSON is compiled into a valid schema object it is 
> ignoring all types of "documentation" that I am trying to adorn with each 
> field in my record. Reading through the Java issues it seems that this was a 
> bug and fixed (AVRO-601, AVRO-612, AVRO-779) but it seems the C++ 
> implementation has yet to adopt this feature? This is my sample schema, I 
> have attempted to insert both "doc" and "mycustom" in multiple places to see 
> if it is supported at any level. Please excuse if there appears to be a 
> syntax error in the JSON I hand tweaked some of this. The schema is valid and 
> successfully parses.
> {
>   "type": "record",
>   "name": "myschema",
>   "doc": "Doc Meta",
>   "mycustom": "My Custom",
>   "fields": [
>   { "name":"field_a","type":["string","null"], "doc":"Doc Meta", 
> "mycustom":"My Custom A"},
>   { "name":"field_b","type":["string","null"], "doc":"Doc Meta", 
> "mycustom":"My Custom B"},
>   { "name":"field_c","type":["string","null"], "doc":"Doc Meta", 
> "mycustom":"My Custom C"}
>   ]
> }
> I looked through the SchemaTests.cc code for 1.7.3 and there was not a test 
> case for this there so i didn't think this was addressed in that version. I 
> am running 1.7.2. When this schema is used to load with compileJsonSchema and 
> then a file is serialized the file schema looks like this.
> {
>   "type":"record",
>   "name":"myschema",
>   "fields": [
>   { "name":"field_a","type":["string","null"]},
>   { "name":"field_b","type":["string","null"]},
>   { "name":"field_c","type":["string","null"]}
>   ]
> }



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Jenkins

[Niels Basjes]

I fully agree with automated builds.

I'm just curious what makes Jenkins more appropriate?

Niels
On Thu, Oct 11, 2018 at 6:09 PM Driesprong, Fokko  wrote:
>
> Hi all,
>
> I'd like to revive the Jenkins build on Avro. The last runs are a bit over
> a year old: https://builds.apache.org/view/A/view/Avro/
>
> I've opened a PR based on Apache Yetus
>  to configure the precommit task
> to run CI jobs, as Sean Busbey initially suggested. Initially I would like
> to go for a Travis CI, as we have with Apache Airflow, but I think the Jenkins
> of Apache itself  might be more appropriate.
>
> The current Yetus scripts that I've written are still a bit crude, but I
> think we should have the automated process from end to end up as soon as
> possible, and then we can start polishing. Would it be possible to get
> access to the Apache Jenkins
> ,
> or is someone able to set this up?
>
> I would like to configure the Github pull request builder, based on
> the CloudBees
> plugin as suggested on the Apache wiki
> 
> .
>
> Hope to get your feedback on the plan, and hope to move forward :-)
>
> Cheers, Fokko



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

[jira] [Issue Comment Deleted] (AVRO-2199) Validate that field defaults have the correct type

[Daniel Orner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Orner updated AVRO-2199:
---
Comment: was deleted

(was: [~theturtle32] I definitely feel your pain. :( But the fact is that the 
current implementation is incorrect - the fact that you have bad data out there 
is a result of the incorrect implementation. In my case we had other Avro 
consumers not written in Ruby whose parsers were stricter that were crashing 
because we were sending bad data, so IMO it's more important to fix the 
behavior in Ruby which doesn't match the Avro specification.

We should definitely make this a breaking change (1.9.0 instead of 1.8.3) so 
people can be made aware of this.)

>  Validate that field defaults have the correct type
> ---
>
> Key: AVRO-2199
> URL: https://issues.apache.org/jira/browse/AVRO-2199
> Project: Avro
>  Issue Type: Improvement
>  Components: ruby
>Affects Versions: 1.8.2
>Reporter: Daniel Orner
>Priority: Minor
>
> Currently, on the master branch, when a schema is parsed, it is possible to 
> define a field with a type and a default of a totally different type. E.g. if 
> the field has type "string", the default can be set to "null".
> I'd like to open a PR which will fix this by running the default through the 
> SchemaValidator whenever a new Field is created. See 
> [https://github.com/salsify/avro-patches/pull/16]
> cc: [~tjwp]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646752#comment-16646752
 ] 

ASF GitHub Bot commented on AVRO-2199:
--

dorner commented on issue #320: AVRO-2199:  Validate that field defaults have 
the correct type
URL: https://github.com/apache/avro/pull/320#issuecomment-429034100
 
 
   @theturtle32 I definitely feel your pain. But the fact is that the current 
implementation is incorrect - the fact that you have bad data out there is a 
result of the incorrect implementation. In my case we had other Avro consumers 
not written in Ruby whose parsers were stricter that were crashing because we 
were sending bad data, so IMO it's more important to fix the behavior in Ruby 
which doesn't match the Avro specification.
   
   We should definitely make this a breaking change (1.9.0 instead of 1.8.3) so 
people can be made aware of this.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


>  Validate that field defaults have the correct type
> ---
>
> Key: AVRO-2199
> URL: https://issues.apache.org/jira/browse/AVRO-2199
> Project: Avro
>  Issue Type: Improvement
>  Components: ruby
>Affects Versions: 1.8.2
>Reporter: Daniel Orner
>Priority: Minor
>
> Currently, on the master branch, when a schema is parsed, it is possible to 
> define a field with a type and a default of a totally different type. E.g. if 
> the field has type "string", the default can be set to "null".
> I'd like to open a PR which will fix this by running the default through the 
> SchemaValidator whenever a new Field is created. See 
> [https://github.com/salsify/avro-patches/pull/16]
> cc: [~tjwp]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2241) Fix Jsr310/Joda tests when comparing datetimes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646712#comment-16646712
 ] 

ASF GitHub Bot commented on AVRO-2241:
--

Fokko opened a new pull request #346: AVRO-2241. Fix Jsr310/Joda datetime tests
URL: https://github.com/apache/avro/pull/346
 
 
   Whe comparing Instant, DateTime, LocalTime with its Joda equivalents the 
tests fail since the java.time.* adds support for microseconds, and Joda only 
miliseconds. For example:
   
   ```
   Failed tests:
 
TestSpecificLogicalTypes.testAbilityToReadJodaRecordWrittenAsJsr310Record:183
   Expected: is "11:45:07.170331"
but: was "11:45:07.170"
 TestSpecificLogicalTypes.testRecordWithJsr310LogicalTypes:115 Should match 
written record expected:
   
   <{"b": true, "i32": 34, "i64": 35, "f32": 3.14, "f64": 3019.34, "s": null, 
"d": 2018-10-11, "t": 11:45:06.513252, "ts": 2018-10-11T11:45:06.513358Z, 
"dec": 123.45}>
   
   but was:
   
   <{"b": true, "i32": 34, "i64": 35, "f32": 3.14, "f64": 3019.34, "s": null, 
"d": 2018-10-11, "t": 11:45:06.513, "ts": 2018-10-11T11:45:06.513Z, "dec": 
123.45}>
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Fix Jsr310/Joda tests when comparing datetimes
> --
>
> Key: AVRO-2241
> URL: https://issues.apache.org/jira/browse/AVRO-2241
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.9.0
>
>
> Whe comparing Instant, DateTime, LocalTime with its Joda equivalents the 
> tests fail since the java.time.* adds support for microseconds, and Joda only 
> miliseconds. For example:
> Failed tests: 
>   
> TestSpecificLogicalTypes.testAbilityToReadJodaRecordWrittenAsJsr310Record:183 
> Expected: is "11:45:07.170331"
>  but: was "11:45:07.170"
>   TestSpecificLogicalTypes.testRecordWithJsr310LogicalTypes:115 Should match 
> written record expected:<{"b": true, "i32": 34, "i64": 35, "f32": 3.14, 
> "f64": 3019.34, "s": null, "d": 2018-10-11, "t": 11:45:06.513252, "ts": 
> 2018-10-11T11:45:06.513358Z, "dec": 123.45}> but was:<{"b": true, "i32": 34, 
> "i64": 35, "f32": 3.14, "f64": 3019.34, "s": null, "d": 2018-10-11, "t": 
> 11:45:06.513, "ts": 2018-10-11T11:45:06.513Z, "dec": 123.45}>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2241) Fix Jsr310/Joda tests when comparing datetimes

[Fokko Driesprong (JIRA)]

Fokko Driesprong created AVRO-2241:
--

 Summary: Fix Jsr310/Joda tests when comparing datetimes
 Key: AVRO-2241
 URL: https://issues.apache.org/jira/browse/AVRO-2241
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.8.2
Reporter: Fokko Driesprong
Assignee: Fokko Driesprong
 Fix For: 1.9.0


Whe comparing Instant, DateTime, LocalTime with its Joda equivalents the tests 
fail since the java.time.* adds support for microseconds, and Joda only 
miliseconds. For example:

Failed tests: 
  TestSpecificLogicalTypes.testAbilityToReadJodaRecordWrittenAsJsr310Record:183 
Expected: is "11:45:07.170331"
 but: was "11:45:07.170"
  TestSpecificLogicalTypes.testRecordWithJsr310LogicalTypes:115 Should match 
written record expected:<{"b": true, "i32": 34, "i64": 35, "f32": 3.14, "f64": 
3019.34, "s": null, "d": 2018-10-11, "t": 11:45:06.513252, "ts": 
2018-10-11T11:45:06.513358Z, "dec": 123.45}> but was:<{"b": true, "i32": 34, 
"i64": 35, "f32": 3.14, "f64": 3019.34, "s": null, "d": 2018-10-11, "t": 
11:45:06.513, "ts": 2018-10-11T11:45:06.513Z, "dec": 123.45}>





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2229) Ability to test using a Docker image

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646696#comment-16646696
 ] 

ASF GitHub Bot commented on AVRO-2229:
--

Fokko commented on issue #343: [AVRO-2229][WIP] Test using a Docker image
URL: https://github.com/apache/avro/pull/343#issuecomment-429016706
 
 
   Lets wait with this one until we get #344 in


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Ability to test using a Docker image
> 
>
> Key: AVRO-2229
> URL: https://issues.apache.org/jira/browse/AVRO-2229
> Project: Avro
>  Issue Type: Improvement
>Affects Versions: 1.8.2
>Reporter: Fokko Driesprong
>Assignee: Fokko Driesprong
>Priority: Major
> Fix For: 1.9.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins

[Driesprong, Fokko]

Hi all,

I'd like to revive the Jenkins build on Avro. The last runs are a bit over
a year old: https://builds.apache.org/view/A/view/Avro/

I've opened a PR based on Apache Yetus
 to configure the precommit task
to run CI jobs, as Sean Busbey initially suggested. Initially I would like
to go for a Travis CI, as we have with Apache Airflow, but I think the Jenkins
of Apache itself  might be more appropriate.

The current Yetus scripts that I've written are still a bit crude, but I
think we should have the automated process from end to end up as soon as
possible, and then we can start polishing. Would it be possible to get
access to the Apache Jenkins
,
or is someone able to set this up?

I would like to configure the Github pull request builder, based on
the CloudBees
plugin as suggested on the Apache wiki

.

Hope to get your feedback on the plan, and hope to move forward :-)

Cheers, Fokko

[jira] [Commented] (AVRO-1887) set up precommit checks via Apache Yetus

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/AVRO-1887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646676#comment-16646676
 ] 

ASF GitHub Bot commented on AVRO-1887:
--

Fokko commented on issue #344: [AVRO-1887][WIP] Setup precommit hooks using 
Apache Yetus
URL: https://github.com/apache/avro/pull/344#issuecomment-429012884
 
 
   @busbey I've got a first version of the Apache Yetus script working. It took 
me a while since it is new to me, so be gentle on the review ;)
   I've written a small plugin that will locate all the `build.sh` scripts, and 
subsequently invoke the `test` step. I know this is still a bit crude, but this 
allows us to reuse the existing build scripts.
   
   Please note that the tests are still failing, but I'm working on patches in 
a new PR.
   
   ```
_ _ __ 
   |  ___|_ _(_) |_   _ _ __ ___| |
   | |_ / _` | | | | | | '__/ _ \ |
   |  _| (_| | | | |_| | | |  __/_|
   |_|  \__,_|_|_|\__,_|_|  \___(_)
   
   
   
   | Vote | Subsystem |  Runtime   | Comment
   
   |   0  |   reexec  |   0m  9s   | Docker mode activated. 
   |  |   || Prechecks 
   |  |   || Compile Tests 
   |  -1  |buildtest  |  14m 46s   | The testsuite failed, please check the 
   |  |   || output
   |  |   || Other Tests 
   |  |   |  14m 55s   | 
   ```
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> set up precommit checks via Apache Yetus
> 
>
> Key: AVRO-1887
> URL: https://issues.apache.org/jira/browse/AVRO-1887
> Project: Avro
>  Issue Type: Task
>  Components: build, community
>Reporter: Sean Busbey
>Assignee: Suraj Acharya
>Priority: Blocker
> Attachments: AVRO-1887-0.patch, AVRO-1887-1.patch, AVRO-1887-2.patch
>
>
> The lack of automated checks for things like doing a full build is negatively 
> impacting my ability to do timely reviews. We should leverage the work done 
> by other communities and get precommit checks going with Apache Yetus ([ref 
> Apache Yetus 
> Precommit|http://yetus.apache.org/documentation/0.3.0/precommit-basic/])
> Yetus has built in support for using Docker as a containerized build 
> environment ([ref 
> docs|https://yetus.apache.org/documentation/0.3.0/precommit-advanced/#Docker_Support]),
>  but I don't know if we can use the one we have directly or if we'll need to 
> do some refactoring.
> Implementing this will ultimately require a committer (so that they can make 
> changes to builds.apache.org), but a non-cmmitter can get the bulk of the 
> work done by managing to do local yetus checks with the docker feature active.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)